Compare commits

..
65 Commits
Author SHA1 Message Date
ChuckandClaude Opus 5.5 4ddc3a3620 chore: prepare the 3.8.0 release (#709)
* chore(deprecation): remove the 35 APIs deprecated for 3.8.0

The usage scan (docs/DEPRECATIONS_3.8.md, regenerated 2026-10-01 and
committed here) finds no call or override of any of them in the 46
monorepo plugins or the 8 third-party plugins plugins.json lists; the
only core callers were other deprecated methods removed alongside.

- CacheManager: 13 methods, plus the private helpers only
  has_data_changed used (_has_*_changed, _is_market_open).
- DisplayManager: 7 methods, plus WEATHER_COLORS and the private
  _draw_sun/_cloud/_rain/_snow/_storm helpers only the icon methods used.
- FontManager: 14 methods, plus size_tokens, _save_overrides and
  _clear_plugin_font_cache. font_overrides and _load_overrides stay:
  resolve_font() still applies config/font_overrides.json.
  performance_stats stays: get_font() keeps it and tests read it.
- PluginManager.get_enabled_plugins.

test_deprecation.py pins only the two 3.9.0 markers now; the scanner
tests run against a stand-in core instead of the real markers. The
memory-tier tests read stats through log_memory_cache_stats() and the
component, and the test of the removed _clear_plugin_font_cache goes.
Docs drop the removed methods' reference entries; the Deprecated APIs
table becomes "Removed in 3.8.0". CHANGELOG gains a Removed section.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(deprecation): drop the test harness's copies of the removed icon methods

VisualTestDisplayManager still drew weather icons that DisplayManager no
longer has, so a plugin's visual tests could pass on calls that raise
AttributeError on the real display. Its draw_sun/draw_cloud/draw_rain/
draw_snow/draw_weather_icon/draw_text_with_icons, WEATHER_COLORS and the
private helpers go, with the tests that exercised them. The CHANGELOG's
Deprecations entries no longer say nothing is removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore: prepare the 3.8.0 release

Bumps src.__version__ to 3.8.0 and turns Unreleased into ## 3.8.0, with a
summary and a New modules list (vegas_elements, testing.vegas, sports_vegas;
display_watchdog, plugin_catalog, plugin_runtime) for plugins flooring on
3.8.0. Adds the CHANGELOG line #701's second commit lacked (blocks laid out
off the render thread).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore: 3.8.0 also ships what landed on main since the prep

#703, #705, #706 and #693 merged after this branch was cut; their CHANGELOG
entries now sit under 3.8.0. The summary and New modules list name them
(sports consolidation stage 4's four modules, src/ipc, field_model), stage
4's section says to floor on 3.8.0, and src/common/README.md marks its four
modules 3.8.0 instead of Unreleased.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 11:07:02 -04:00
ChuckandClaude Opus 5.5 2601cb4cbb chore(deprecation): remove the 35 APIs deprecated for 3.8.0 (#708)
* chore(deprecation): remove the 35 APIs deprecated for 3.8.0

The usage scan (docs/DEPRECATIONS_3.8.md, regenerated 2026-10-01 and
committed here) finds no call or override of any of them in the 46
monorepo plugins or the 8 third-party plugins plugins.json lists; the
only core callers were other deprecated methods removed alongside.

- CacheManager: 13 methods, plus the private helpers only
  has_data_changed used (_has_*_changed, _is_market_open).
- DisplayManager: 7 methods, plus WEATHER_COLORS and the private
  _draw_sun/_cloud/_rain/_snow/_storm helpers only the icon methods used.
- FontManager: 14 methods, plus size_tokens, _save_overrides and
  _clear_plugin_font_cache. font_overrides and _load_overrides stay:
  resolve_font() still applies config/font_overrides.json.
  performance_stats stays: get_font() keeps it and tests read it.
- PluginManager.get_enabled_plugins.

test_deprecation.py pins only the two 3.9.0 markers now; the scanner
tests run against a stand-in core instead of the real markers. The
memory-tier tests read stats through log_memory_cache_stats() and the
component, and the test of the removed _clear_plugin_font_cache goes.
Docs drop the removed methods' reference entries; the Deprecated APIs
table becomes "Removed in 3.8.0". CHANGELOG gains a Removed section.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(deprecation): drop the test harness's copies of the removed icon methods

VisualTestDisplayManager still drew weather icons that DisplayManager no
longer has, so a plugin's visual tests could pass on calls that raise
AttributeError on the real display. Its draw_sun/draw_cloud/draw_rain/
draw_snow/draw_weather_icon/draw_text_with_icons, WEATHER_COLORS and the
private helpers go, with the tests that exercised them. The CHANGELOG's
Deprecations entries no longer say nothing is removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 10:45:28 -04:00
ChuckandClaude Opus 5.5 695ff92009 feat(ipc): display control socket, stage 1 - on-demand with acks (#706)
The display serves a control socket (/run/ledmatrix/control.sock) carrying versioned JSON commands, one per line, each answered. Stage 1 covers on-demand start, stop and status; commands are queued on the socket thread and applied on the render thread through the mailbox's own handler, and the web interface falls back to the file mailbox when the socket is unavailable. Protocol and security model: docs/IPC_CONTROL_SOCKET.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 10:33:02 -04:00
ChuckandClaude Opus 5.5 74696d2108 fix(display): routine rotation log lines are DEBUG (30% fewer journal lines) (#693)
"Processing mode", "display() returned False" and "No content to display" repeated what "Switching to mode" already logs on every rotation; they are now DEBUG. On ledpi this cut the display's journal lines by about 30%; the measured SD-write saving is small (within noise).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 10:21:22 -04:00
ChuckandClaude Opus 5.5 3ad0438e75 feat(common): sports consolidation stage 4 -- the identical sweep (plugin host, live scroll, display rules, font path) (#705)
Moves the code every scoreboard plugin carries identically into core: src.common.sports_plugin_host, sports_live_scroll, sports_display_rules and sports_font_path, with unit tests and a parity test against the ledmatrix-plugins copies (LEDMATRIX_PLUGINS).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 10:10:22 -04:00
ChuckandClaude Opus 5.5 c6701ac00d feat(web): ES-module page lifecycle and one schema field model (stage 1) (#703)
Adds a native ES-module layer to the web UI (core/boot, registry, api, facade; window.LEDMatrix as the one global), a page lifecycle that the Cache tab is converted to as the reference, text/javascript serving and revalidation for unversioned module requests, and src/plugin_system/field_model.py with a parity test against the render_field macro. Also: the cache page toggles its grey 'Not configured' style instead of only adding it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 10:00:26 -04:00
ChuckandClaude Opus 5.5 795834811f perf(scroll): extend and trim the Vegas strip in place (#701)
* perf(timing): say which render-thread work a late frame followed

The soak already says how often a moving frame reached the panel late, but
not what the render thread was doing just before it. Vegas does two kinds of
work there between frames -- building its strip (compose, extend) and, with
live elements, patching changed pixels into it -- and deciding whether either
is affordable needs their own numbers.

- FrameTimingRecorder.note_op(kind, nbytes) tags the next presented frame.
  Totals gain op_frames, late_op_frames, op_freezes and op_bytes per kind;
  aggregate() still takes frames without ops. The file schema is unchanged.
- Vegas tags compose and every strip extension (with the bytes it copied).
- frame_soak prints an "after work" table: frames, late %, freezes and MB
  moved per kind, only when something tagged its work.
- render_bench gains --strip-screens (Vegas-sized strips), --patch-bytes /
  --patch-every / --patch-where (in-place column writes, as a live element
  update does) and --extend-every-screens / --extend-width (append + trim on
  a fixed cadence that holds the strip's width).

No runtime behaviour changes: this is the measurement gate for live Vegas
elements.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(changelog): note the frame-op attribution and bench modes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(scroll): build the strip's PIL image only when something reads it

Every Vegas strip extension rebuilt ScrollHelper.cached_image from
cached_array in full, twice (append, then trim), on the render thread:
Image.fromarray is 1.7ms for an 8,000px strip and 3.8ms for 20,000px on a
Pi 4 (measured on ledpi), about two thirds of an extension's render-thread
cost. Nothing on the frame path reads the image's pixels; every frame is cut
from the array.

cached_image is now a property. append_content and drop_scrolled_prefix
defer it; the first read builds it from the array it started with and keeps
it only if the strip has not changed meanwhile, so a sync push racing an
extension cannot leave a stale image cached. Assigning cached_image stores
exactly what was assigned, as before. has_strip() says whether there is a
strip without building its image; the helper's frame path, Vegas and the
adapter's scroll-cache invalidation use it. The strip is also no longer held
in memory twice.

In Vegas the image is now built only by a multi-display sync push.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(vegas): live elements -- a plugin API for content that changes while it scrolls

Vegas bakes each plugin's pictures into one strip, so a card already on its
way across the panel keeps what it showed when it was drawn. This adds the
API and bookkeeping for content that can be updated in place; the worker
that redraws and swaps it follows separately. No shipped plugin implements
the hook yet, so nothing changes for users.

Plugin API (core 3.8.0), all no-ops by default:
- BasePlugin.get_vegas_elements() -> [VegasElement(key, image, version,
  live, refresh_hz)]: named, fixed-width pieces of Vegas content.
- BasePlugin.redraw_vegas_element(key, width, height, at): a lock-free
  redraw for content that changes with time.
- BasePlugin.notify_vegas_data_changed(): data that lands outside update().
- src/plugin_system/vegas_elements.py (VegasElement, re-exported from
  base_plugin).

Core:
- PluginAdapter asks a plugin that implements the hook for elements on the
  background fetch only (under its lock, on its own canvas); every other
  path keeps get_vegas_content(). Live elements are pinned (padded with
  content_padding, never trimmed), tagged with their key, digest and data
  epoch in Image.info so the existing cache and group plumbing carry them
  unchanged, and untagged if a width budget crops them.
- RenderPipeline records where each live element lands (ElementRecord), in
  absolute strip columns a trim does not move; the block-start arithmetic
  is shared with the STATIC markers.
- PluginManager update listeners (add/remove_update_listener,
  notify_data_changed): told the moment update() completes, not at the
  next ~4s Vegas poll. The coordinator uses one to move each plugin's data
  epoch on.
- vegas_scroll.live_refresh (kill switch), live_max_hz, live_min_interval,
  live_lead_screens; per-plugin core-owned vegas_live. Live elements are
  off under multi-display sync, in swap mode and with offscreen_prefetch off.
- scripts/check_plugin.py checks the element contract
  (src/plugin_system/testing/vegas.py); test/fixtures/plugins/vegas-live-stub
  is a working example.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(vegas): live elements update in place while they scroll

One background worker (src/vegas_mode/live_worker.py) redraws a plugin's
live elements when its data epoch moves on (update listener) or on their
refresh_hz, nearest the screen first, and hands changed pixels lock-free to
the render thread, which copies them into the strip between frames
(RenderPipeline.apply_live_patches, ScrollHelper.patch_columns): at most
four patches or two screens of bytes a frame, no drawing or locks there.
The worker takes over group prefetch once a live element is placed, runs
inside the render gate, and is supervised. Update tick 1s while live
elements exist. Web UI switch for live_refresh. OFFSCREEN_RENDERING.md
describes what was built and why SegmentStrip was not needed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(sports): live Vegas cards for the scoreboards (shared layer)

One live element per game, drawn only when what the card shows changes, so
a score changes on a card already crossing the panel. The shared part, so
each scoreboard adopts it in a few lines:

- src/common/sports_vegas.py: game_key, game_fingerprint (the whole game
  dict, frozen: no drawn field can be missed), dedupe_games, VegasCardCache,
  StickyOdds (odds a live poll left out stay drawn), finished_games /
  with_finished_games (a game that just went final keeps its card, after its
  league's live games; one a heuristic only judged over keeps its live
  state, so a tied end of regulation never shows FINAL early).
- SportsScrollDisplay.make_vegas_renderer() is the override point;
  build_vegas_elements() and SportsScrollDisplayManager
  .get_vegas_elements_for() do the rest. A card's version includes its
  teams' ranks, which the renderer draws from the rankings cache.
- SportsLiveSharedMixin._record_finished_game() / finished_games_snapshot():
  held for FINISHED_GAME_TTL after it leaves the live list.

A sport that does not implement make_vegas_renderer keeps its ordinary Vegas
content, so no scoreboard changes until it opts in.

scripts/render_plugin.py --vegas renders a plugin's Vegas block as the
ticker lays it out, and --timeline stacks it at successive moments as
the ticker would update it in place; the join is now
render_pipeline.join_plugin_rows().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(vegas): keep live games in the ticker by default

display.vegas_scroll.live_in_ticker now defaults to true: through a live
game the marquee keeps running and the live scoreboard takes extra turns in
it -- its cards updating in place while they scroll -- instead of the ticker
giving way to the full-screen scoreboard.

The new default would reach nobody on its own: every existing config holds
an explicit false copied from the template (there was no control for it),
and the template merge only adds missing keys. ConfigManager therefore turns
a stored false on once, with a backup, and records live_in_ticker_migrated
so a false chosen afterwards stays. The marker is never in the template.

A "Keep live games in the ticker" checkbox under Vegas mode sets it. Tests
that pin the full-screen takeover now say live_in_ticker=false.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(dev): preview a plugin's Vegas strip in the dev server

The dev server's View selector gains "Vegas strip (live elements)" and
"Vegas strip (plain Vegas content)": the plugin's block of the Vegas ticker,
laid out by the ticker's own code (render_vegas_strip, as render_plugin.py
--vegas uses), with its live elements listed. /api/render takes
"vegas": "live" | "plain"; the display view is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(scroll): extend and trim the Vegas strip in place

Every strip extension rebuilt the whole strip (np.concatenate: 2-2.6 ms for
a 10-14k px strip at 512x64 on a Pi 4) and every trim copied what was left
(1.2-1.8 ms), on the render thread. With ~4 ms of slack per refresh, every
extension frame on hdpi missed its refresh (5/5 in each soak run).

The strip now lives in a buffer with spare room; cached_array is a view of
its live columns. An append writes only the new columns (~0.2 ms), a trim
only moves the view's start, and the one full copy happens when the buffer
is reallocated (STRIP_SPARE_FACTOR 3: about once every two strip-lengths
scrolled). A cached_array set from outside -- the multi-display follower's
read-only one, create_scrolling_image's -- is never written through, and a
new strip lets the old buffer go. last_copy_bytes says what was copied, and
the Vegas frame-timing attribution reports that instead of the whole strip.

test_scroll_helper_in_place.py: the buffer is reused and only new columns
copied, trims copy nothing, reallocation when the room runs out, outside
arrays untouched, and random appends/trims/patches/scrolling checked frame
by frame against the old copying strip (mutation-checked).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(sports): a default _determine_game_type on SportsScrollDisplay

render_vegas_card looked the method up with getattr and a None default, which
static analysis (Codacy) reports as calling something that may not be
callable. The base class now has the default -- the card type from the game's
state -- and the plugins that define their own override it as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(scroll): no assert in _extended_strip

An assert vanishes under python -O (Codacy); a real check says the same.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: review follow-ups on the shared live-card layer

- The reused Vegas renderer always gets the current rankings, empty
  included, so ranks cleared since are not kept drawn.
- render_plugin.py: --timeline refuses --no-live (a timeline shows live
  elements changing), --timeline/--no-live need --vegas, and the Vegas
  paths create the output's directory like the display path does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(vegas): lay a group's blocks out before the extension, off the render thread

#701 cut the strip copy, but the frame after an extension was still late:
the render thread also joined each plugin's rows (separation_gap measures
every pair) and pasted the blocks into an addition image, ~37 ms on hdpi
(Pi 4, 512x64) against ~3.75 ms of slack.

The thread that fetched the group now does that as each member arrives:
RenderPipeline.prepare_group_member joins the rows and turns the block into
pixels (the prefetch thread and the live worker, under the render gate).
extend_scroll_content takes those blocks, and ScrollHelper.append_content
writes items -- images or RGB arrays -- straight into the strip's spare room,
blanking only the gaps. A member that was not prepared (an inline fetch) is
joined at the extension as before; the strip is identical either way.

On hdpi the extension's render-thread work goes from 37.5 ms to 3.2 ms p50
in place (6.9 ms when the buffer is reallocated). The helper's per-append
INFO line, a duplicate of the pipeline's, is now DEBUG.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 09:48:47 -04:00
ChuckandClaude Opus 5.5 dfd67c7c8b feat(dev): preview a plugin's Vegas strip in the dev server (#700)
* perf(timing): say which render-thread work a late frame followed

The soak already says how often a moving frame reached the panel late, but
not what the render thread was doing just before it. Vegas does two kinds of
work there between frames -- building its strip (compose, extend) and, with
live elements, patching changed pixels into it -- and deciding whether either
is affordable needs their own numbers.

- FrameTimingRecorder.note_op(kind, nbytes) tags the next presented frame.
  Totals gain op_frames, late_op_frames, op_freezes and op_bytes per kind;
  aggregate() still takes frames without ops. The file schema is unchanged.
- Vegas tags compose and every strip extension (with the bytes it copied).
- frame_soak prints an "after work" table: frames, late %, freezes and MB
  moved per kind, only when something tagged its work.
- render_bench gains --strip-screens (Vegas-sized strips), --patch-bytes /
  --patch-every / --patch-where (in-place column writes, as a live element
  update does) and --extend-every-screens / --extend-width (append + trim on
  a fixed cadence that holds the strip's width).

No runtime behaviour changes: this is the measurement gate for live Vegas
elements.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(changelog): note the frame-op attribution and bench modes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(scroll): build the strip's PIL image only when something reads it

Every Vegas strip extension rebuilt ScrollHelper.cached_image from
cached_array in full, twice (append, then trim), on the render thread:
Image.fromarray is 1.7ms for an 8,000px strip and 3.8ms for 20,000px on a
Pi 4 (measured on ledpi), about two thirds of an extension's render-thread
cost. Nothing on the frame path reads the image's pixels; every frame is cut
from the array.

cached_image is now a property. append_content and drop_scrolled_prefix
defer it; the first read builds it from the array it started with and keeps
it only if the strip has not changed meanwhile, so a sync push racing an
extension cannot leave a stale image cached. Assigning cached_image stores
exactly what was assigned, as before. has_strip() says whether there is a
strip without building its image; the helper's frame path, Vegas and the
adapter's scroll-cache invalidation use it. The strip is also no longer held
in memory twice.

In Vegas the image is now built only by a multi-display sync push.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(vegas): live elements -- a plugin API for content that changes while it scrolls

Vegas bakes each plugin's pictures into one strip, so a card already on its
way across the panel keeps what it showed when it was drawn. This adds the
API and bookkeeping for content that can be updated in place; the worker
that redraws and swaps it follows separately. No shipped plugin implements
the hook yet, so nothing changes for users.

Plugin API (core 3.8.0), all no-ops by default:
- BasePlugin.get_vegas_elements() -> [VegasElement(key, image, version,
  live, refresh_hz)]: named, fixed-width pieces of Vegas content.
- BasePlugin.redraw_vegas_element(key, width, height, at): a lock-free
  redraw for content that changes with time.
- BasePlugin.notify_vegas_data_changed(): data that lands outside update().
- src/plugin_system/vegas_elements.py (VegasElement, re-exported from
  base_plugin).

Core:
- PluginAdapter asks a plugin that implements the hook for elements on the
  background fetch only (under its lock, on its own canvas); every other
  path keeps get_vegas_content(). Live elements are pinned (padded with
  content_padding, never trimmed), tagged with their key, digest and data
  epoch in Image.info so the existing cache and group plumbing carry them
  unchanged, and untagged if a width budget crops them.
- RenderPipeline records where each live element lands (ElementRecord), in
  absolute strip columns a trim does not move; the block-start arithmetic
  is shared with the STATIC markers.
- PluginManager update listeners (add/remove_update_listener,
  notify_data_changed): told the moment update() completes, not at the
  next ~4s Vegas poll. The coordinator uses one to move each plugin's data
  epoch on.
- vegas_scroll.live_refresh (kill switch), live_max_hz, live_min_interval,
  live_lead_screens; per-plugin core-owned vegas_live. Live elements are
  off under multi-display sync, in swap mode and with offscreen_prefetch off.
- scripts/check_plugin.py checks the element contract
  (src/plugin_system/testing/vegas.py); test/fixtures/plugins/vegas-live-stub
  is a working example.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(vegas): live elements update in place while they scroll

One background worker (src/vegas_mode/live_worker.py) redraws a plugin's
live elements when its data epoch moves on (update listener) or on their
refresh_hz, nearest the screen first, and hands changed pixels lock-free to
the render thread, which copies them into the strip between frames
(RenderPipeline.apply_live_patches, ScrollHelper.patch_columns): at most
four patches or two screens of bytes a frame, no drawing or locks there.
The worker takes over group prefetch once a live element is placed, runs
inside the render gate, and is supervised. Update tick 1s while live
elements exist. Web UI switch for live_refresh. OFFSCREEN_RENDERING.md
describes what was built and why SegmentStrip was not needed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(sports): live Vegas cards for the scoreboards (shared layer)

One live element per game, drawn only when what the card shows changes, so
a score changes on a card already crossing the panel. The shared part, so
each scoreboard adopts it in a few lines:

- src/common/sports_vegas.py: game_key, game_fingerprint (the whole game
  dict, frozen: no drawn field can be missed), dedupe_games, VegasCardCache,
  StickyOdds (odds a live poll left out stay drawn), finished_games /
  with_finished_games (a game that just went final keeps its card, after its
  league's live games; one a heuristic only judged over keeps its live
  state, so a tied end of regulation never shows FINAL early).
- SportsScrollDisplay.make_vegas_renderer() is the override point;
  build_vegas_elements() and SportsScrollDisplayManager
  .get_vegas_elements_for() do the rest. A card's version includes its
  teams' ranks, which the renderer draws from the rankings cache.
- SportsLiveSharedMixin._record_finished_game() / finished_games_snapshot():
  held for FINISHED_GAME_TTL after it leaves the live list.

A sport that does not implement make_vegas_renderer keeps its ordinary Vegas
content, so no scoreboard changes until it opts in.

scripts/render_plugin.py --vegas renders a plugin's Vegas block as the
ticker lays it out, and --timeline stacks it at successive moments as
the ticker would update it in place; the join is now
render_pipeline.join_plugin_rows().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(vegas): keep live games in the ticker by default

display.vegas_scroll.live_in_ticker now defaults to true: through a live
game the marquee keeps running and the live scoreboard takes extra turns in
it -- its cards updating in place while they scroll -- instead of the ticker
giving way to the full-screen scoreboard.

The new default would reach nobody on its own: every existing config holds
an explicit false copied from the template (there was no control for it),
and the template merge only adds missing keys. ConfigManager therefore turns
a stored false on once, with a backup, and records live_in_ticker_migrated
so a false chosen afterwards stays. The marker is never in the template.

A "Keep live games in the ticker" checkbox under Vegas mode sets it. Tests
that pin the full-screen takeover now say live_in_ticker=false.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(dev): preview a plugin's Vegas strip in the dev server

The dev server's View selector gains "Vegas strip (live elements)" and
"Vegas strip (plain Vegas content)": the plugin's block of the Vegas ticker,
laid out by the ticker's own code (render_vegas_strip, as render_plugin.py
--vegas uses), with its live elements listed. /api/render takes
"vegas": "live" | "plain"; the display view is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(sports): a default _determine_game_type on SportsScrollDisplay

render_vegas_card looked the method up with getattr and a None default, which
static analysis (Codacy) reports as calling something that may not be
callable. The base class now has the default -- the card type from the game's
state -- and the plugins that define their own override it as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: review follow-ups on the shared live-card layer

- The reused Vegas renderer always gets the current rankings, empty
  included, so ranks cleared since are not kept drawn.
- render_plugin.py: --timeline refuses --no-live (a timeline shows live
  elements changing), --timeline/--no-live need --vegas, and the Vegas
  paths create the output's directory like the display path does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 08:37:15 -04:00
ChuckandClaude Opus 5.5 7f06cc9c3b feat(vegas): keep live games in the ticker by default (#699)
* perf(timing): say which render-thread work a late frame followed

The soak already says how often a moving frame reached the panel late, but
not what the render thread was doing just before it. Vegas does two kinds of
work there between frames -- building its strip (compose, extend) and, with
live elements, patching changed pixels into it -- and deciding whether either
is affordable needs their own numbers.

- FrameTimingRecorder.note_op(kind, nbytes) tags the next presented frame.
  Totals gain op_frames, late_op_frames, op_freezes and op_bytes per kind;
  aggregate() still takes frames without ops. The file schema is unchanged.
- Vegas tags compose and every strip extension (with the bytes it copied).
- frame_soak prints an "after work" table: frames, late %, freezes and MB
  moved per kind, only when something tagged its work.
- render_bench gains --strip-screens (Vegas-sized strips), --patch-bytes /
  --patch-every / --patch-where (in-place column writes, as a live element
  update does) and --extend-every-screens / --extend-width (append + trim on
  a fixed cadence that holds the strip's width).

No runtime behaviour changes: this is the measurement gate for live Vegas
elements.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(changelog): note the frame-op attribution and bench modes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(scroll): build the strip's PIL image only when something reads it

Every Vegas strip extension rebuilt ScrollHelper.cached_image from
cached_array in full, twice (append, then trim), on the render thread:
Image.fromarray is 1.7ms for an 8,000px strip and 3.8ms for 20,000px on a
Pi 4 (measured on ledpi), about two thirds of an extension's render-thread
cost. Nothing on the frame path reads the image's pixels; every frame is cut
from the array.

cached_image is now a property. append_content and drop_scrolled_prefix
defer it; the first read builds it from the array it started with and keeps
it only if the strip has not changed meanwhile, so a sync push racing an
extension cannot leave a stale image cached. Assigning cached_image stores
exactly what was assigned, as before. has_strip() says whether there is a
strip without building its image; the helper's frame path, Vegas and the
adapter's scroll-cache invalidation use it. The strip is also no longer held
in memory twice.

In Vegas the image is now built only by a multi-display sync push.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(vegas): live elements -- a plugin API for content that changes while it scrolls

Vegas bakes each plugin's pictures into one strip, so a card already on its
way across the panel keeps what it showed when it was drawn. This adds the
API and bookkeeping for content that can be updated in place; the worker
that redraws and swaps it follows separately. No shipped plugin implements
the hook yet, so nothing changes for users.

Plugin API (core 3.8.0), all no-ops by default:
- BasePlugin.get_vegas_elements() -> [VegasElement(key, image, version,
  live, refresh_hz)]: named, fixed-width pieces of Vegas content.
- BasePlugin.redraw_vegas_element(key, width, height, at): a lock-free
  redraw for content that changes with time.
- BasePlugin.notify_vegas_data_changed(): data that lands outside update().
- src/plugin_system/vegas_elements.py (VegasElement, re-exported from
  base_plugin).

Core:
- PluginAdapter asks a plugin that implements the hook for elements on the
  background fetch only (under its lock, on its own canvas); every other
  path keeps get_vegas_content(). Live elements are pinned (padded with
  content_padding, never trimmed), tagged with their key, digest and data
  epoch in Image.info so the existing cache and group plumbing carry them
  unchanged, and untagged if a width budget crops them.
- RenderPipeline records where each live element lands (ElementRecord), in
  absolute strip columns a trim does not move; the block-start arithmetic
  is shared with the STATIC markers.
- PluginManager update listeners (add/remove_update_listener,
  notify_data_changed): told the moment update() completes, not at the
  next ~4s Vegas poll. The coordinator uses one to move each plugin's data
  epoch on.
- vegas_scroll.live_refresh (kill switch), live_max_hz, live_min_interval,
  live_lead_screens; per-plugin core-owned vegas_live. Live elements are
  off under multi-display sync, in swap mode and with offscreen_prefetch off.
- scripts/check_plugin.py checks the element contract
  (src/plugin_system/testing/vegas.py); test/fixtures/plugins/vegas-live-stub
  is a working example.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(vegas): live elements update in place while they scroll

One background worker (src/vegas_mode/live_worker.py) redraws a plugin's
live elements when its data epoch moves on (update listener) or on their
refresh_hz, nearest the screen first, and hands changed pixels lock-free to
the render thread, which copies them into the strip between frames
(RenderPipeline.apply_live_patches, ScrollHelper.patch_columns): at most
four patches or two screens of bytes a frame, no drawing or locks there.
The worker takes over group prefetch once a live element is placed, runs
inside the render gate, and is supervised. Update tick 1s while live
elements exist. Web UI switch for live_refresh. OFFSCREEN_RENDERING.md
describes what was built and why SegmentStrip was not needed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(sports): live Vegas cards for the scoreboards (shared layer)

One live element per game, drawn only when what the card shows changes, so
a score changes on a card already crossing the panel. The shared part, so
each scoreboard adopts it in a few lines:

- src/common/sports_vegas.py: game_key, game_fingerprint (the whole game
  dict, frozen: no drawn field can be missed), dedupe_games, VegasCardCache,
  StickyOdds (odds a live poll left out stay drawn), finished_games /
  with_finished_games (a game that just went final keeps its card, after its
  league's live games; one a heuristic only judged over keeps its live
  state, so a tied end of regulation never shows FINAL early).
- SportsScrollDisplay.make_vegas_renderer() is the override point;
  build_vegas_elements() and SportsScrollDisplayManager
  .get_vegas_elements_for() do the rest. A card's version includes its
  teams' ranks, which the renderer draws from the rankings cache.
- SportsLiveSharedMixin._record_finished_game() / finished_games_snapshot():
  held for FINISHED_GAME_TTL after it leaves the live list.

A sport that does not implement make_vegas_renderer keeps its ordinary Vegas
content, so no scoreboard changes until it opts in.

scripts/render_plugin.py --vegas renders a plugin's Vegas block as the
ticker lays it out, and --timeline stacks it at successive moments as
the ticker would update it in place; the join is now
render_pipeline.join_plugin_rows().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(vegas): keep live games in the ticker by default

display.vegas_scroll.live_in_ticker now defaults to true: through a live
game the marquee keeps running and the live scoreboard takes extra turns in
it -- its cards updating in place while they scroll -- instead of the ticker
giving way to the full-screen scoreboard.

The new default would reach nobody on its own: every existing config holds
an explicit false copied from the template (there was no control for it),
and the template merge only adds missing keys. ConfigManager therefore turns
a stored false on once, with a backup, and records live_in_ticker_migrated
so a false chosen afterwards stays. The marker is never in the template.

A "Keep live games in the ticker" checkbox under Vegas mode sets it. Tests
that pin the full-screen takeover now say live_in_ticker=false.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(sports): a default _determine_game_type on SportsScrollDisplay

render_vegas_card looked the method up with getattr and a None default, which
static analysis (Codacy) reports as calling something that may not be
callable. The base class now has the default -- the card type from the game's
state -- and the plugins that define their own override it as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: review follow-ups on the shared live-card layer

- The reused Vegas renderer always gets the current rankings, empty
  included, so ranks cleared since are not kept drawn.
- render_plugin.py: --timeline refuses --no-live (a timeline shows live
  elements changing), --timeline/--no-live need --vegas, and the Vegas
  paths create the output's directory like the display path does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 08:27:16 -04:00
ChuckandClaude Opus 5.5 a12be7c3c5 feat(sports): live Vegas cards for the scoreboards (shared layer) (#698)
* perf(timing): say which render-thread work a late frame followed

The soak already says how often a moving frame reached the panel late, but
not what the render thread was doing just before it. Vegas does two kinds of
work there between frames -- building its strip (compose, extend) and, with
live elements, patching changed pixels into it -- and deciding whether either
is affordable needs their own numbers.

- FrameTimingRecorder.note_op(kind, nbytes) tags the next presented frame.
  Totals gain op_frames, late_op_frames, op_freezes and op_bytes per kind;
  aggregate() still takes frames without ops. The file schema is unchanged.
- Vegas tags compose and every strip extension (with the bytes it copied).
- frame_soak prints an "after work" table: frames, late %, freezes and MB
  moved per kind, only when something tagged its work.
- render_bench gains --strip-screens (Vegas-sized strips), --patch-bytes /
  --patch-every / --patch-where (in-place column writes, as a live element
  update does) and --extend-every-screens / --extend-width (append + trim on
  a fixed cadence that holds the strip's width).

No runtime behaviour changes: this is the measurement gate for live Vegas
elements.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(changelog): note the frame-op attribution and bench modes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(scroll): build the strip's PIL image only when something reads it

Every Vegas strip extension rebuilt ScrollHelper.cached_image from
cached_array in full, twice (append, then trim), on the render thread:
Image.fromarray is 1.7ms for an 8,000px strip and 3.8ms for 20,000px on a
Pi 4 (measured on ledpi), about two thirds of an extension's render-thread
cost. Nothing on the frame path reads the image's pixels; every frame is cut
from the array.

cached_image is now a property. append_content and drop_scrolled_prefix
defer it; the first read builds it from the array it started with and keeps
it only if the strip has not changed meanwhile, so a sync push racing an
extension cannot leave a stale image cached. Assigning cached_image stores
exactly what was assigned, as before. has_strip() says whether there is a
strip without building its image; the helper's frame path, Vegas and the
adapter's scroll-cache invalidation use it. The strip is also no longer held
in memory twice.

In Vegas the image is now built only by a multi-display sync push.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(vegas): live elements -- a plugin API for content that changes while it scrolls

Vegas bakes each plugin's pictures into one strip, so a card already on its
way across the panel keeps what it showed when it was drawn. This adds the
API and bookkeeping for content that can be updated in place; the worker
that redraws and swaps it follows separately. No shipped plugin implements
the hook yet, so nothing changes for users.

Plugin API (core 3.8.0), all no-ops by default:
- BasePlugin.get_vegas_elements() -> [VegasElement(key, image, version,
  live, refresh_hz)]: named, fixed-width pieces of Vegas content.
- BasePlugin.redraw_vegas_element(key, width, height, at): a lock-free
  redraw for content that changes with time.
- BasePlugin.notify_vegas_data_changed(): data that lands outside update().
- src/plugin_system/vegas_elements.py (VegasElement, re-exported from
  base_plugin).

Core:
- PluginAdapter asks a plugin that implements the hook for elements on the
  background fetch only (under its lock, on its own canvas); every other
  path keeps get_vegas_content(). Live elements are pinned (padded with
  content_padding, never trimmed), tagged with their key, digest and data
  epoch in Image.info so the existing cache and group plumbing carry them
  unchanged, and untagged if a width budget crops them.
- RenderPipeline records where each live element lands (ElementRecord), in
  absolute strip columns a trim does not move; the block-start arithmetic
  is shared with the STATIC markers.
- PluginManager update listeners (add/remove_update_listener,
  notify_data_changed): told the moment update() completes, not at the
  next ~4s Vegas poll. The coordinator uses one to move each plugin's data
  epoch on.
- vegas_scroll.live_refresh (kill switch), live_max_hz, live_min_interval,
  live_lead_screens; per-plugin core-owned vegas_live. Live elements are
  off under multi-display sync, in swap mode and with offscreen_prefetch off.
- scripts/check_plugin.py checks the element contract
  (src/plugin_system/testing/vegas.py); test/fixtures/plugins/vegas-live-stub
  is a working example.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(vegas): live elements update in place while they scroll

One background worker (src/vegas_mode/live_worker.py) redraws a plugin's
live elements when its data epoch moves on (update listener) or on their
refresh_hz, nearest the screen first, and hands changed pixels lock-free to
the render thread, which copies them into the strip between frames
(RenderPipeline.apply_live_patches, ScrollHelper.patch_columns): at most
four patches or two screens of bytes a frame, no drawing or locks there.
The worker takes over group prefetch once a live element is placed, runs
inside the render gate, and is supervised. Update tick 1s while live
elements exist. Web UI switch for live_refresh. OFFSCREEN_RENDERING.md
describes what was built and why SegmentStrip was not needed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(sports): live Vegas cards for the scoreboards (shared layer)

One live element per game, drawn only when what the card shows changes, so
a score changes on a card already crossing the panel. The shared part, so
each scoreboard adopts it in a few lines:

- src/common/sports_vegas.py: game_key, game_fingerprint (the whole game
  dict, frozen: no drawn field can be missed), dedupe_games, VegasCardCache,
  StickyOdds (odds a live poll left out stay drawn), finished_games /
  with_finished_games (a game that just went final keeps its card, after its
  league's live games; one a heuristic only judged over keeps its live
  state, so a tied end of regulation never shows FINAL early).
- SportsScrollDisplay.make_vegas_renderer() is the override point;
  build_vegas_elements() and SportsScrollDisplayManager
  .get_vegas_elements_for() do the rest. A card's version includes its
  teams' ranks, which the renderer draws from the rankings cache.
- SportsLiveSharedMixin._record_finished_game() / finished_games_snapshot():
  held for FINISHED_GAME_TTL after it leaves the live list.

A sport that does not implement make_vegas_renderer keeps its ordinary Vegas
content, so no scoreboard changes until it opts in.

scripts/render_plugin.py --vegas renders a plugin's Vegas block as the
ticker lays it out, and --timeline stacks it at successive moments as
the ticker would update it in place; the join is now
render_pipeline.join_plugin_rows().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(sports): a default _determine_game_type on SportsScrollDisplay

render_vegas_card looked the method up with getattr and a None default, which
static analysis (Codacy) reports as calling something that may not be
callable. The base class now has the default -- the card type from the game's
state -- and the plugins that define their own override it as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: review follow-ups on the shared live-card layer

- The reused Vegas renderer always gets the current rankings, empty
  included, so ranks cleared since are not kept drawn.
- render_plugin.py: --timeline refuses --no-live (a timeline shows live
  elements changing), --timeline/--no-live need --vegas, and the Vegas
  paths create the output's directory like the display path does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 21:32:07 -04:00
ChuckandClaude Opus 5.5 f4bda50710 feat(vegas): live elements update in place while they scroll (#697)
* perf(timing): say which render-thread work a late frame followed

The soak already says how often a moving frame reached the panel late, but
not what the render thread was doing just before it. Vegas does two kinds of
work there between frames -- building its strip (compose, extend) and, with
live elements, patching changed pixels into it -- and deciding whether either
is affordable needs their own numbers.

- FrameTimingRecorder.note_op(kind, nbytes) tags the next presented frame.
  Totals gain op_frames, late_op_frames, op_freezes and op_bytes per kind;
  aggregate() still takes frames without ops. The file schema is unchanged.
- Vegas tags compose and every strip extension (with the bytes it copied).
- frame_soak prints an "after work" table: frames, late %, freezes and MB
  moved per kind, only when something tagged its work.
- render_bench gains --strip-screens (Vegas-sized strips), --patch-bytes /
  --patch-every / --patch-where (in-place column writes, as a live element
  update does) and --extend-every-screens / --extend-width (append + trim on
  a fixed cadence that holds the strip's width).

No runtime behaviour changes: this is the measurement gate for live Vegas
elements.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(changelog): note the frame-op attribution and bench modes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(scroll): build the strip's PIL image only when something reads it

Every Vegas strip extension rebuilt ScrollHelper.cached_image from
cached_array in full, twice (append, then trim), on the render thread:
Image.fromarray is 1.7ms for an 8,000px strip and 3.8ms for 20,000px on a
Pi 4 (measured on ledpi), about two thirds of an extension's render-thread
cost. Nothing on the frame path reads the image's pixels; every frame is cut
from the array.

cached_image is now a property. append_content and drop_scrolled_prefix
defer it; the first read builds it from the array it started with and keeps
it only if the strip has not changed meanwhile, so a sync push racing an
extension cannot leave a stale image cached. Assigning cached_image stores
exactly what was assigned, as before. has_strip() says whether there is a
strip without building its image; the helper's frame path, Vegas and the
adapter's scroll-cache invalidation use it. The strip is also no longer held
in memory twice.

In Vegas the image is now built only by a multi-display sync push.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(vegas): live elements -- a plugin API for content that changes while it scrolls

Vegas bakes each plugin's pictures into one strip, so a card already on its
way across the panel keeps what it showed when it was drawn. This adds the
API and bookkeeping for content that can be updated in place; the worker
that redraws and swaps it follows separately. No shipped plugin implements
the hook yet, so nothing changes for users.

Plugin API (core 3.8.0), all no-ops by default:
- BasePlugin.get_vegas_elements() -> [VegasElement(key, image, version,
  live, refresh_hz)]: named, fixed-width pieces of Vegas content.
- BasePlugin.redraw_vegas_element(key, width, height, at): a lock-free
  redraw for content that changes with time.
- BasePlugin.notify_vegas_data_changed(): data that lands outside update().
- src/plugin_system/vegas_elements.py (VegasElement, re-exported from
  base_plugin).

Core:
- PluginAdapter asks a plugin that implements the hook for elements on the
  background fetch only (under its lock, on its own canvas); every other
  path keeps get_vegas_content(). Live elements are pinned (padded with
  content_padding, never trimmed), tagged with their key, digest and data
  epoch in Image.info so the existing cache and group plumbing carry them
  unchanged, and untagged if a width budget crops them.
- RenderPipeline records where each live element lands (ElementRecord), in
  absolute strip columns a trim does not move; the block-start arithmetic
  is shared with the STATIC markers.
- PluginManager update listeners (add/remove_update_listener,
  notify_data_changed): told the moment update() completes, not at the
  next ~4s Vegas poll. The coordinator uses one to move each plugin's data
  epoch on.
- vegas_scroll.live_refresh (kill switch), live_max_hz, live_min_interval,
  live_lead_screens; per-plugin core-owned vegas_live. Live elements are
  off under multi-display sync, in swap mode and with offscreen_prefetch off.
- scripts/check_plugin.py checks the element contract
  (src/plugin_system/testing/vegas.py); test/fixtures/plugins/vegas-live-stub
  is a working example.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(vegas): live elements update in place while they scroll

One background worker (src/vegas_mode/live_worker.py) redraws a plugin's
live elements when its data epoch moves on (update listener) or on their
refresh_hz, nearest the screen first, and hands changed pixels lock-free to
the render thread, which copies them into the strip between frames
(RenderPipeline.apply_live_patches, ScrollHelper.patch_columns): at most
four patches or two screens of bytes a frame, no drawing or locks there.
The worker takes over group prefetch once a live element is placed, runs
inside the render gate, and is supervised. Update tick 1s while live
elements exist. Web UI switch for live_refresh. OFFSCREEN_RENDERING.md
describes what was built and why SegmentStrip was not needed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 21:07:31 -04:00
ChuckandClaude Opus 5.5 56947298d6 a plugin API for content that changes while it scrolls (#696)
* perf(timing): say which render-thread work a late frame followed

The soak already says how often a moving frame reached the panel late, but
not what the render thread was doing just before it. Vegas does two kinds of
work there between frames -- building its strip (compose, extend) and, with
live elements, patching changed pixels into it -- and deciding whether either
is affordable needs their own numbers.

- FrameTimingRecorder.note_op(kind, nbytes) tags the next presented frame.
  Totals gain op_frames, late_op_frames, op_freezes and op_bytes per kind;
  aggregate() still takes frames without ops. The file schema is unchanged.
- Vegas tags compose and every strip extension (with the bytes it copied).
- frame_soak prints an "after work" table: frames, late %, freezes and MB
  moved per kind, only when something tagged its work.
- render_bench gains --strip-screens (Vegas-sized strips), --patch-bytes /
  --patch-every / --patch-where (in-place column writes, as a live element
  update does) and --extend-every-screens / --extend-width (append + trim on
  a fixed cadence that holds the strip's width).

No runtime behaviour changes: this is the measurement gate for live Vegas
elements.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(changelog): note the frame-op attribution and bench modes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(scroll): build the strip's PIL image only when something reads it

Every Vegas strip extension rebuilt ScrollHelper.cached_image from
cached_array in full, twice (append, then trim), on the render thread:
Image.fromarray is 1.7ms for an 8,000px strip and 3.8ms for 20,000px on a
Pi 4 (measured on ledpi), about two thirds of an extension's render-thread
cost. Nothing on the frame path reads the image's pixels; every frame is cut
from the array.

cached_image is now a property. append_content and drop_scrolled_prefix
defer it; the first read builds it from the array it started with and keeps
it only if the strip has not changed meanwhile, so a sync push racing an
extension cannot leave a stale image cached. Assigning cached_image stores
exactly what was assigned, as before. has_strip() says whether there is a
strip without building its image; the helper's frame path, Vegas and the
adapter's scroll-cache invalidation use it. The strip is also no longer held
in memory twice.

In Vegas the image is now built only by a multi-display sync push.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(vegas): live elements -- a plugin API for content that changes while it scrolls

Vegas bakes each plugin's pictures into one strip, so a card already on its
way across the panel keeps what it showed when it was drawn. This adds the
API and bookkeeping for content that can be updated in place; the worker
that redraws and swaps it follows separately. No shipped plugin implements
the hook yet, so nothing changes for users.

Plugin API (core 3.8.0), all no-ops by default:
- BasePlugin.get_vegas_elements() -> [VegasElement(key, image, version,
  live, refresh_hz)]: named, fixed-width pieces of Vegas content.
- BasePlugin.redraw_vegas_element(key, width, height, at): a lock-free
  redraw for content that changes with time.
- BasePlugin.notify_vegas_data_changed(): data that lands outside update().
- src/plugin_system/vegas_elements.py (VegasElement, re-exported from
  base_plugin).

Core:
- PluginAdapter asks a plugin that implements the hook for elements on the
  background fetch only (under its lock, on its own canvas); every other
  path keeps get_vegas_content(). Live elements are pinned (padded with
  content_padding, never trimmed), tagged with their key, digest and data
  epoch in Image.info so the existing cache and group plumbing carry them
  unchanged, and untagged if a width budget crops them.
- RenderPipeline records where each live element lands (ElementRecord), in
  absolute strip columns a trim does not move; the block-start arithmetic
  is shared with the STATIC markers.
- PluginManager update listeners (add/remove_update_listener,
  notify_data_changed): told the moment update() completes, not at the
  next ~4s Vegas poll. The coordinator uses one to move each plugin's data
  epoch on.
- vegas_scroll.live_refresh (kill switch), live_max_hz, live_min_interval,
  live_lead_screens; per-plugin core-owned vegas_live. Live elements are
  off under multi-display sync, in swap mode and with offscreen_prefetch off.
- scripts/check_plugin.py checks the element contract
  (src/plugin_system/testing/vegas.py); test/fixtures/plugins/vegas-live-stub
  is a working example.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 20:59:16 -04:00
ChuckandClaude Opus 5.5 596809acc3 perf(scroll): build the strip's PIL image only when something reads it (#695)
* perf(timing): say which render-thread work a late frame followed

The soak already says how often a moving frame reached the panel late, but
not what the render thread was doing just before it. Vegas does two kinds of
work there between frames -- building its strip (compose, extend) and, with
live elements, patching changed pixels into it -- and deciding whether either
is affordable needs their own numbers.

- FrameTimingRecorder.note_op(kind, nbytes) tags the next presented frame.
  Totals gain op_frames, late_op_frames, op_freezes and op_bytes per kind;
  aggregate() still takes frames without ops. The file schema is unchanged.
- Vegas tags compose and every strip extension (with the bytes it copied).
- frame_soak prints an "after work" table: frames, late %, freezes and MB
  moved per kind, only when something tagged its work.
- render_bench gains --strip-screens (Vegas-sized strips), --patch-bytes /
  --patch-every / --patch-where (in-place column writes, as a live element
  update does) and --extend-every-screens / --extend-width (append + trim on
  a fixed cadence that holds the strip's width).

No runtime behaviour changes: this is the measurement gate for live Vegas
elements.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(changelog): note the frame-op attribution and bench modes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(scroll): build the strip's PIL image only when something reads it

Every Vegas strip extension rebuilt ScrollHelper.cached_image from
cached_array in full, twice (append, then trim), on the render thread:
Image.fromarray is 1.7ms for an 8,000px strip and 3.8ms for 20,000px on a
Pi 4 (measured on ledpi), about two thirds of an extension's render-thread
cost. Nothing on the frame path reads the image's pixels; every frame is cut
from the array.

cached_image is now a property. append_content and drop_scrolled_prefix
defer it; the first read builds it from the array it started with and keeps
it only if the strip has not changed meanwhile, so a sync push racing an
extension cannot leave a stale image cached. Assigning cached_image stores
exactly what was assigned, as before. has_strip() says whether there is a
strip without building its image; the helper's frame path, Vegas and the
adapter's scroll-cache invalidation use it. The strip is also no longer held
in memory twice.

In Vegas the image is now built only by a multi-display sync push.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 20:50:34 -04:00
ChuckandClaude Opus 5.5 77862b631b perf(timing): say which render-thread work a late frame followed (#694)
* perf(timing): say which render-thread work a late frame followed

The soak already says how often a moving frame reached the panel late, but
not what the render thread was doing just before it. Vegas does two kinds of
work there between frames -- building its strip (compose, extend) and, with
live elements, patching changed pixels into it -- and deciding whether either
is affordable needs their own numbers.

- FrameTimingRecorder.note_op(kind, nbytes) tags the next presented frame.
  Totals gain op_frames, late_op_frames, op_freezes and op_bytes per kind;
  aggregate() still takes frames without ops. The file schema is unchanged.
- Vegas tags compose and every strip extension (with the bytes it copied).
- frame_soak prints an "after work" table: frames, late %, freezes and MB
  moved per kind, only when something tagged its work.
- render_bench gains --strip-screens (Vegas-sized strips), --patch-bytes /
  --patch-every / --patch-where (in-place column writes, as a live element
  update does) and --extend-every-screens / --extend-width (append + trim on
  a fixed cadence that holds the strip's width).

No runtime behaviour changes: this is the measurement gate for live Vegas
elements.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(changelog): note the frame-op attribution and bench modes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 20:40:39 -04:00
ChuckandClaude Opus 5.5 7804ea8f69 feat(update): stable/beta update channel; stable follows release tags (#684)
Adds auto_update.channel: stable follows the newest vX.Y.Z release tag
(detached HEAD; pre-releases and other tags ignored), beta follows main as
before. Nothing ever moves a device backwards: a checkout newer than the
newest release keeps following main (or stays put when detached) until a
release contains its commit. Legacy configs migrate to stable when they
reach a release. Update Code, the weekly updater's preflight, and the
verifier's rollback (back to old_ref: branch or detached release) all
honour the channel. General tab Update Channel select, GET/POST
/api/v3/system/update-channel, release-aware Overview banner and Tools git
panel. New installs default to stable.

Rig fix (ledpi): /system/check-update reports update_available: false when
the channel's action is none (a detached HEAD newer than the newest
release), matching Update Code; the Tools panel no longer calls every
detached HEAD "a release".

Merged with main through #687 (heartbeat verifier, #683 login, #688
plugin_catalog, #685 Tailwind build).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 15:35:26 -04:00
ChuckandClaude Opus 5.5 64c7289593 feat(display): systemd watchdog and heartbeat for a frozen render loop (#687)
If the render loop gets stuck inside a plugin's display(), ledmatrix.service
stays active and the panel stays frozen. This adds a way to detect that.

- src/display_watchdog.py (standard library only) sends sd_notify over
  $NOTIFY_SOCKET and writes /run/ledmatrix/display-heartbeat.json. Only the
  render thread counts: beats from other threads are ignored.
- ledmatrix.service: WatchdogSec=120, NotifyAccess=main,
  RuntimeDirectory=ledmatrix (0755), RestartSteps=4 and
  RestartMaxDelaySec=2min. It stays Type=simple. run.py widens the watchdog
  to 15 min for start-up, and load_plugin() does the same on the render
  thread. The loop arms after its first frame.
- /api/v3/health adds checks.display_loop: running, stalled (no heartbeat
  for over 60s, which makes the status degraded) or not_reported. With web
  login on, a caller who is not logged in still gets only healthy/degraded,
  and a stall degrades that answer.
- The update verifier requires a fresh heartbeat from the restarted display
  when the display it replaced was writing one. A frozen panel is rolled
  back.
- Existing installs get the systemd watchdog only after install_service.sh
  is re-run. The heartbeat works right away.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 11:15:31 -04:00
ChuckandClaude Opus 5.5 b09434a418 refactor(plugins): the display publishes plugin runtime state; retire plugin_state.json (#690)
Stage 2 of the web plugin catalog, after #688.

- The display publishes a plugin runtime snapshot (plugin_runtime.py) to
  the shared cache: per plugin loaded, lifecycle state, a short redacted
  error summary, the version it loaded and when, plus published_at /
  stale_after / running. Written on change (throttled to 10 s; the
  RUNNING/ENABLED flip of an ordinary update is not a change) and once a
  minute otherwise; cleanup() publishes running: false.
- The web reads it back and restores loaded / state / error_info in
  /api/v3/plugins/installed (plus loaded_version, loaded_at and
  data.runtime). Only a live snapshot counts; stale, stopped or missing
  answers null and says which.
- data/plugin_state.json is retired: every reader and writer moved to
  config + disk (desired) or the snapshot (observed). Nothing in it was
  non-derivable, so nothing is migrated and an existing file is left
  unread. The web-side PluginStateManager (state_manager.py) is removed;
  the display's plugin_state.PluginStateManager is the only state machine.
- StateReconciliation compares config + disk with the snapshot, reporting
  enabled-but-not-loaded and older-version-loaded as no_action findings.
- Backups list installed manifests with enabled from config.json.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 10:48:14 -04:00
ChuckandClaude Opus 5.5 7ab6fb1aff refactor(web): read plugins through a PluginCatalog; only the display runs them (#688)
The web process built its own PluginManager and loaded plugins into itself:
store installs and updates loaded or reloaded a web-side copy, and config
saves and enable/disable called on_config_change, on_enable and on_disable
on it. None of that reached the panel, and /plugins/installed reported
runtime state from those copies.

- Add PluginCatalog (src/plugin_system/plugin_catalog.py): manifests,
  directories, display modes, installed version, schema and config reads,
  with no way to run a plugin. app.py and both blueprints use it; the
  plugin_manager blueprint attribute is gone.
- Remove every lifecycle call from the web routes. Config changes already
  reach the display through ConfigService (on_config_change) and the
  enabled-set reconcile.
- Health and metrics readers move to api_v3.health_tracker /
  resource_monitor. /plugins/installed reports loaded/state/error_info as
  null (the display does not publish them) and enabled by the display's
  rule.
- Store install, update and uninstall answer restart_required when the
  running display will not pick the change up by itself
  (display_restart_required). The restart banner follows the flag via
  window.noteRestartRequired instead of the /config/main URL heuristic;
  /config/main now sends restart_required: true.
- The one remaining in-process import of plugin code (Starlark helper
  modules, oauth_flow action scripts) goes through
  _import_plugin_code_in_web_process() until a web-entry contract.
- /plugins/installed reports vegas_participation (from #682) from the
  user's setting or the manifest, with vegas_participation_source; when
  only the plugin's code decides it, null with source 'runtime', since the
  web process no longer has plugin instances to ask.
- Check & Update All keeps its restart flags when the final list refresh
  fails, and asks for a restart when an enabled plugin's first request got
  no answer and the re-sent one found it up to date.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 10:39:44 -04:00
ChuckandClaude Opus 5.5 ba6eccb489 build(web): generate the UI's Tailwind CSS with the pinned standalone CLI (#685)
Replaces the hand-written Tailwind subset in app.css with a real, purged
Tailwind build: scripts/build_css.py runs the pinned, SHA-256-checked
standalone Tailwind CLI (no Node), the generated tailwind.css and
plugin-frame.css are committed, and CI fails when they are stale. The Pi
never builds anything. The login page (#683) now links tailwind.css too,
and the load-order test covers every template that links app.css.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 09:38:16 -04:00
ChuckandClaude Opus 5.5 5ea0d511dc feat(store): read ledmatrix_min_version, aliases and commit from the registry (#686)
The store reads three optional registry fields: ledmatrix_min_version
(an incompatible install/update is refused before any download, with a
"Needs LEDMatrix X+" card badge), aliases (update/uninstall/reinstall by
registry id find a plugin installed under its manifest id, with registry
proof only), and commit (shown and linked on the store card). An older
plugins.json behaves as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 09:31:16 -04:00
ChuckandClaude Opus 5.5 1c928b2033 feat(vegas): one declared participation per plugin (scroll | pause | exclude) (#682)
A plugin takes part in Vegas mode in one declared way: 'scroll', 'pause'
or 'exclude', resolved from the user's vegas_participation setting, the
manifest field, then the legacy hooks, so no plugin changes behaviour.
The stream manager decides inclusion and pauses through it; the installed
plugins API and the Vegas plugin-order list report it. Deprecates
get_supported_vegas_modes, get_vegas_segment_width and vegas_panel_count
for removal in 3.9.0, and regenerates docs/DEPRECATIONS_3.8.md to include
them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 09:30:37 -04:00
ChuckandClaude Opus 5.5 b2df0fda1b docs(sports): record the ufc round-break verify outcome (refuted) (#692)
ESPN sends the break between rounds as STATUS_END_OF_ROUND with
displayClock "-", not "0:00", so the shared game-over rule never
drops a five-round fight at the round 4 break. Verified against
recorded payloads in ChuckBuilds/ledmatrix-plugins#580, which pins it.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 09:20:30 -04:00
ChuckandClaude Opus 5.5 15c61def67 fix(web): make the SSE streams' 200/min rate limit actually apply (#691)
app.py called limiter.limit("200 per minute")(stream_x) after the routes
were registered and discarded the result. flask-limiter 3.x enforces a
decorated limit in the wrapper limit() returns, and marks the original
function so the before_request middleware skips it, so the streams had
no limit at all -- not even the 1000/min default. Register the wrapper
as the view instead.

The new test (skipped without flask-limiter) reconnects to each stream
201 times and expects the last to get a 429; it fails on the old code.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 09:12:23 -04:00
ChuckandClaude Opus 5.5 c8a0ddcf7b chore(deprecation): retarget the 35 deprecations to 3.8.0 and add a usage scan (#681)
Moves the 35 @deprecated markers from 3.7.0 (already shipped with them in
place) to 3.8.0, and adds scripts/plugin_api_usage.py plus the generated
docs/DEPRECATIONS_3.8.md: who still calls or overrides each deprecated
method across core, the monorepo and third-party plugins. Removes nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 09:11:36 -04:00
ChuckandClaude Opus 5.5 9fe23af432 docs(sports): reconcile-then-promote roadmap, drift report and report-only CI job (#680)
Rewrites the roadmap in docs/SPORTS_UNIFICATION.md for the
reconcile-then-promote decision (stages 0-3 recorded as done), and adds the
sports drift report script with a report-only CI job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 09:11:02 -04:00
ChuckandClaude Opus 5.5 c0d97e4867 fix(plugins): one hung plugin no longer stops every plugin from updating (#677)
The update worker no longer blocks forever on a plugin whose display()
never returns. It waits at most PLUGIN_LOCK_TIMEOUT (5s) for a plugin's
lock, then skips that plugin's update (a report-only "busy skip" in
health) and keeps updating every other plugin. display() frames are timed
(slow calls logged and counted; calls past the executor timeout recorded as
hangs), a hung update() is recorded, and on_config_change() now runs under
the plugin lock or is deferred to the worker. The plugin-facing API is
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 09:10:41 -04:00
ChuckandClaude Opus 5.5 e3c85cece6 feat(web): optional web login and API tokens, off by default (stacked on #674) (#683)
Optional web login, off by default: a device that sets no password behaves
exactly as before. Set under General > Security; then every page and API
route needs a session login or an API token (Authorization: Bearer).
Loopback, the Wi-Fi setup flow in AP mode, static files, captive-portal
probes and a reduced /api/v3/health stay open. Secrets live in the web_auth
section of config_secrets.json and no API returns them.
scripts/reset_web_password.py turns login off. Stacked on #674.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 09:09:40 -04:00
ChuckandClaude Opus 5.5 ba38a83c2c fix(web): refuse cross-site state-changing requests (Origin/Referer check) (#674)
The web interface refuses state-changing requests (POST/PUT/PATCH/DELETE)
whose Origin (or, without one, Referer) is not the host they were sent to,
or is null: 403 CROSS_SITE_REQUEST (web_interface/origin_guard.py). Any
website a LAN user visited could otherwise make their browser POST a plain
form to the Pi. /api/v3/system/action also refuses form-encoded and
text/plain bodies (415) unless sent by HTMX. Clients that send no Origin or
Referer (curl, requests, Home Assistant, the MQTT bridge) are unaffected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 09:02:01 -04:00
ChuckandClaude Opus 5.5 c3a7a110c4 fix(display): on-demand loads a disabled plugin live instead of failing (#678)
* fix(web): on-demand no longer restarts a running display service

POST /display/on-demand/start treated start_service (default true, sent by
"Preview on display", the on-demand dialog and the MQTT bridge) as
"restart": with the service running it ran systemctl stop, slept 1.5s and
started it again. Every request cold-started the display process -- every
plugin reloaded, panel blank -- to deliver a request the running process
already reads from the cache mailbox every ON_DEMAND_POLL_INTERVAL (0.25s),
including mid-dwell, mid-screen and mid-Vegas. The restart bought nothing:
startup only restores a session the display saved itself
(display_on_demand_config), so the new request arrived through the same
mailbox either way.

start_service now means "start it if it is not running". The stop route
coerces stop_service to a boolean so "false" no longer stops the service.
test_api_v3_on_demand_restart.py pinned the old restart path; it now pins
the replacement. Docs updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(display): on-demand loads a disabled plugin live instead of failing

The display process only loads enabled plugins, so an on-demand request for
a disabled one -- "Preview on display" offers it on every config page, with a
note that the plugin will be enabled for the preview -- failed with
invalid-mode. Nothing enabled it short of a restart, and the on-demand route
no longer restarts the service.

_activate_on_demand now loads an installed-but-not-running plugin through
the live-enable path (load_plugin + _register_loaded_plugin), with a new
load_plugin(force_enabled=True) so the instance runs enabled while
config.json keeps saying disabled. The plugin is tracked in
_on_demand_loaded_plugins, and the main loop unloads it through
_unregister_plugin once on-demand moves off it (stop, expiry, another
request, or a failed request that ends the session) -- right after its own
poll, where no display() is on the stack. A failed load publishes status
error with load-failed. A plugin enabled during the session stays loaded.

A session restored after a restart uses the same tracking instead of
setting enabled in the config dict config_manager caches, so its plugin is
unloaded when the session ends rather than staying loaded until the next
restart. Ending a session no longer resumes the rotation onto a plugin that
is about to be unloaded, which a restored session did.

Also: a stop sent while on-demand is inactive clears a failed request's
error, instead of /display/on-demand/status reporting status: error until
the state aged out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 17:59:12 -04:00
ChuckandClaude Opus 5.5 6047eb5e4e test: stop the suite reinstalling plugins into the real plugin-repos/ (#679)
Any test that imported web_interface.app and sent a request fired the app's
startup reconciliation, which runs against the checkout's real config.json
and plugin-repos/ and reinstalls every configured-but-missing plugin from the
live store. A full Windows run left basketball-scoreboard, calendar,
football-scoreboard, leaderboard and ledmatrix-stocks untracked in
plugin-repos/ (not gitignored) from that daemon thread.

test/conftest.py now installs an import hook that sets the app's run-once
_reconciliation_started latch as the module finishes executing, so lazy
imports, module-level imports and reloads all start disarmed.
StateReconciliation's own tests are unaffected. A regression test pins that
a request to the imported app launches no reconciliation thread.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 17:47:32 -04:00
ChuckandClaude Opus 5.5 c4c46d3ba7 fix(web): on-demand no longer restarts a running display service (#676)
POST /display/on-demand/start treated start_service (default true, sent by
"Preview on display", the on-demand dialog and the MQTT bridge) as
"restart": with the service running it ran systemctl stop, slept 1.5s and
started it again. Every request cold-started the display process -- every
plugin reloaded, panel blank -- to deliver a request the running process
already reads from the cache mailbox every ON_DEMAND_POLL_INTERVAL (0.25s),
including mid-dwell, mid-screen and mid-Vegas. The restart bought nothing:
startup only restores a session the display saved itself
(display_on_demand_config), so the new request arrived through the same
mailbox either way.

start_service now means "start it if it is not running". The stop route
coerces stop_service to a boolean so "false" no longer stops the service.
test_api_v3_on_demand_restart.py pinned the old restart path; it now pins
the replacement. Docs updated.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 14:42:58 -04:00
ChuckandClaude Opus 5.5 da9a999102 chore: prepare the 3.7.0 release (#673)
Bumps src.__version__ to 3.7.0 and turns Unreleased (#672: sports_celebration,
sports_fetch and sports_card_wrappers) into ## 3.7.0; src/common/README.md and
docs/SPORTS_UNIFICATION.md say 3.7.0 for the three modules.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 12:51:24 -04:00
ChuckandClaude Opus 5.5 1e4c890d59 feat(common): sports_celebration, sports_fetch and sports_card_wrappers, promoted from the scoreboards (sports consolidation stage 3) (#672)
Three new hardware-free modules holding code the scoreboard plugins carry
as identical copies (executable AST, docstrings stripped, checked across
every carrying plugin at ledmatrix-plugins 30455671). The bodies are the
plugins'; the changes are type annotations for the mypy ratchet, the
colour helpers losing their leading underscore as public free functions,
and two comments that described the plugins' files.

- src/common/sports_celebration.py: SportsCelebrationMixin, the score/win
  takeover drawn by afl, football, hockey, nrl and soccer
  (_draw_celebration_layout and the palette, backdrop, scenery, confetti,
  crest and _fit_font steps, with their class constants), plus the colour
  helpers (logo_palette, lift_color, cap_luminance, mix_color, ...). Only
  the drawing: _start_celebration, _check_for_goal/_check_for_score,
  _check_for_win and display() differ between the plugins and stay there.
- src/common/sports_fetch.py: SportsFetchMixin, the four SportsCore methods
  identical in all nine scoreboards: _fetch_season_directly,
  _background_fetches_espn_ranges, _needs_previous_day and
  _wants_live_odds, with _LOOKBACK_CUTOFF_HOUR and _LIVE_ODDS_LOOKAHEAD.
  _get_timezone, _extract_game_details and _fetch_data are as identical
  and stay behind, for the reasons sports_shared gives (a per-plugin
  import; the abstract contract); so does SportsUpcoming.__init__, since
  no src/common mixin has a constructor.
- src/common/sports_card_wrappers.py: SportsCardWrappersMixin, the
  seventeen sports_card delegations the eight game renderers carry (15 in
  all eight, 2 in all but football, whose own versions override them).
  _schema_font_size/_resolve_font_size look identical but read each
  plugin's own _SCHEMA_PATH, so they stay.

Each mixin has no __init__ and creates no attributes (the host contract is
declared as annotations only), defines no name the mixins beside it
define, and documents the attributes it reads; a host-contract test
parses each and fails on an undocumented read. A method kept on a
plugin's class wins over the mixin's.

Tests: behaviour ported from the plugins' celebration, odds, lookback and
date-range tests against stub hosts carrying exactly the contract, with
crests drawn by the test (test_sports_celebration.py, test_sports_fetch.py,
test_sports_card_wrappers.py), and test_sports_stage3_parity.py, which with
LEDMATRIX_PLUGINS set compares every body with every plugin copy that is
left (58 pass against the plugins today; a copy that is gone counts as
adopted). All three modules are on the mypy ratchet, in
src/common/README.md, the CHANGELOG's Unreleased section and
SPORTS_UNIFICATION's module table. Nothing in core uses them yet.

Full suite: the same 67 failing test ids as main (Windows-only), 77 more
passing.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 12:38:40 -04:00
ChuckandClaude Opus 5.5 7f96075076 chore: prepare the 3.6.2 release (#671)
Bumps src.__version__ to 3.6.2 and turns Unreleased (#670, the favourite
check's false "season has finished" for list-calendar competitions between
rounds) into ## 3.6.2.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 11:55:48 -04:00
ChuckandClaude Opus 5.5 439013b18c fix(common): favourite check no longer calls the Europa League finished between matchdays (#670)
On 2026-09-29 ESPN's uefa.europa scoreboard still showed the 17 September
matchday, so every event was past. Its calendar is a "list" of rounds
(League Phase to 30 Jan 2027, then the knockout rounds to the final), not
a match-day whitelist, and the league's season type is a soccer id rather
than 2/3, so neither 3.6.1 rule applied and the check said the season had
finished.

When every event is past, a round in a list calendar that has not started
yet now draws no conclusion. Only a round's start date counts: end dates
are padded past the last game (AFL's Grand Final round still had a day to
run three days after the Grand Final), and rounds in an offseason phase
(college football's All-Star week) are skipped. Season end dates are still
ignored, so PLL (season to 2027-01-01) stays "finished", as do the World
Cup and AFL. Of 28 live ESPN scoreboards only uefa.europa's message changes.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 11:49:36 -04:00
ChuckandClaude Opus 5.5 e5bbfa2ae3 chore: prepare the 3.6.1 release (#669)
Bumps src.__version__ to 3.6.1 and records #667 (the favourite check's false
"season has finished") under ## 3.6.1; #667 had no CHANGELOG entry. Plugins
that drop their bundled favourite-check copy floor on 3.6.1.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 10:59:20 -04:00
ChuckandClaude Opus 5.5 db49275075 fix(common): favourite check no longer calls a started postseason a finished season (#667)
* fix(common): favourite check no longer calls a started postseason a finished season

The day after a regular season ends, ESPN's default scoreboard still
returns that last regular-season day, while leagues[0].season has moved
to Postseason. All events were in the past, so the check logged "the
season has finished" for MLB on 2026-09-29 while the upcoming manager in
the same process was showing TB's wild-card games.

When every event is past and the league is in a later in-season phase
(regular season or postseason) than all of the returned events, draw no
conclusion. The offseason is excluded, so a genuinely finished season is
still reported as finished.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(common): favourite check reports the next matchday between soccer rounds

Between matchdays ESPN's soccer scoreboard keeps showing the last one, so
every event is in the past and in the league's current phase, which the
postseason rule does not cover; the check said the Premier League season
had finished on 2026-09-29 (last games 20 September, next 10 October).
When the league calendar is a "day" whitelist, its entries are days with
games, so a future one is used as the next fixture. MLB's day calendar is
a blacklist and is not read that way; PLL's whitelist has no future days
and is still reported as finished.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 10:52:38 -04:00
ChuckandClaude Opus 5.5 a11412dabb chore: prepare the 3.6.0 release (#666)
Turns the CHANGELOG's Unreleased section into ## 3.6.0 and bumps
src.__version__, the value plugin ledmatrix_min_version floors compare
against. 3.6.0 ships the two modules from #665 (favorite_team_check,
sports_timezone); nothing else has changed since 3.5.0. src/common/README.md
and docs/SPORTS_UNIFICATION.md say 3.6.0 for them instead of Unreleased.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 09:28:21 -04:00
ChuckandClaude Opus 5.5 fe5bed2886 feat(common): favorite_team_check and sports_timezone, promoted from the scoreboards (sports consolidation stage 2) (#665)
* feat(common): favorite_team_check and sports_timezone, promoted from the scoreboards (sports consolidation stage 2)

Two new hardware-free modules, taken from files the scoreboard plugins carry
as copies:

- src/common/favorite_team_check.py: FavoriteTeamCheck(logger, leagues), the
  seven byte-identical <sport>_favorite_check.py copies. Same code; the only
  additions are two type annotations (for the mypy ratchet).
- src/common/sports_timezone.py: resolve_timezone_name(), resolve_timezone(),
  system_timezone_name(), from the ten <sport>_timezone.py copies. They
  differed only in the plugin label named in the nothing-resolved warning and
  the write-back-bug values, which become keyword-only arguments
  (plugin_label, writeback_fixed_in). Same resolution order and log text.

Tests are ported from the plugins' own (test_favorite_check.py,
test_schedule_note_uses_game_dates.py, test_timezone_resolution.py; the
timezone ones run once per plugin's values and pin the exact warning text).
Both modules are on the mypy ratchet, in src/common/README.md, the CHANGELOG's
Unreleased section and SPORTS_UNIFICATION's module table. Nothing in core uses
them yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(common): bdf_font and json_body shipped in 3.5.0

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(common): annotate the favourite check's deliberate except/pass for Bandit

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 09:05:10 -04:00
ChuckandClaude Opus 5.5 5b30052b59 docs(changelog): fold Unreleased into 3.5.0 for the release (#664)
* docs(changelog): fold Unreleased into 3.5.0 for the release

Every Unreleased entry (#605-#663) moves into the 3.5.0 section, grouped
with the existing 3.5.0 areas; new groups for Display and Vegas, Plugin
error reporting, Wi-Fi, Fonts and Removed. "## Unreleased" stays as an
empty heading.

Module list: add src/common/json_body.py (espn_dates imports it with a
fallback) and src/common/bdf_font.py; list the other modules new since
v3.4.0 as core-internal; add the new names in existing modules
(handles_espn_date_ranges, register_plugin_fonts(plugin_dir),
forget_manager_fonts). Record the src.common and plugin_system modules
#608 deleted.

Add entries for merged PRs that had none: #604, #605, #606, #607, #608,
#609, #613, #616, #618, #622, #625, #628, #630, #633. Note that three
scripts named in older 3.5.0 entries were later deleted by #607.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(changelog): list the hardware-free test under developer tools, not plugin modules

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 16:05:22 -04:00
ChuckandClaude Opus 5.5 8557eff88a fix(plugins): put a (re)loading plugin's directory first on sys.path (#663)
Plugins import their own files by bare name (`from sports import ...`),
which resolves to the first directory on sys.path that has the file. The
loader added a plugin's directory only if it was missing, so on a reload --
a live re-enable from the web UI -- the plugin's directory stayed behind
every plugin loaded since, and its bare imports found their files first.

Seen on ledpi: re-enabling UFC with hockey running failed with "cannot
import name '_status_is_final' from 'sports'" (it got hockey's sports.py).
A loading plugin's directory is now always moved to the front. Every
scoreboard ships its own sports.py, so any of them was exposed on reload.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 15:02:01 -04:00
ChuckandClaude Opus 5.5 b8c01c69fb ci: mypy ratchet -- keep type-clean modules clean (71 modules, 536 -> 442 errors) (#661)
* ci: mypy ratchet -- keep type-clean modules clean

mypy-clean.txt lists the 71 modules under src/ that type-check clean;
scripts/check_types.py runs mypy (--follow-imports=silent) on exactly
those files and fails on any error or a missing/unsorted/duplicate entry.
A new "Type check (mypy ratchet)" CI job runs it with mypy 1.20.2 and
pinned stubs; the manual pre-commit mypy hook now runs the same script
(a local hook, so mypy sees the installed requirements like CI does).

35 modules were made clean with annotation-only fixes: hints, typing.cast,
TYPE_CHECKING imports, implicit-Optional defaults made explicit, and
annotations widened (never guards removed) where mypy called a defensive
isinstance check unreachable. No runtime behaviour change.

mypy.ini: numpy and orjson are treated as Any (follow_imports=skip, also
for stubs). numpy 2.3+ stubs use 3.12 `type` statements that mypy won't
parse at python_version 3.10, and orjson is optional, so seeing its stubs
made the result depend on whether it was installed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore: annotate check_types.py's list-form mypy subprocess

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 15:01:35 -04:00
ChuckandClaude Opus 5.5 e6e0a16140 ci: run the web UI DOM test suites; fix two stale suites (#660)
A new "Web UI JS tests" job installs jsdom, starts the web interface in
emulator mode and runs test/js/run_all.js with REQUIRE_DOM=1, which makes a
DOM suite that can't run a failure rather than a silent skip. (The unit
suites were already covered through pytest.)

Two suites failed against main when run for real:
- test_tools_sections rendered the Tools partial without LEDEscape, which
  base.html's app-early.js defines; it now installs it in beforeParse, and
  supplies two sample Starlark apps (one id with a quote) when the server
  has none, instead of assuming a device with apps and Pixlet.
- test_store_dom assumed the live registry had at most 48 plugins; it now
  checks pagination whichever side of 48 it is.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 15:01:08 -04:00
ChuckandClaude Opus 5.5 989eae9405 refactor(plugins): split PluginStoreManager into mixins (#659)
* refactor(plugins): split PluginStoreManager into mixins

src/plugin_system/store_manager.py (2,977 lines) keeps the class, its
shared state, locks, the uninstall registry, directory lookup and
uninstall; its methods are split by area into:
- store_registry.py (_RegistryMixin): registry, GitHub metadata, search,
  manifest validation
- store_install.py (_InstallMixin): install paths and dependencies
- store_update.py (_UpdateMixin): updates, rollback, local git state

Pure move: all 56 members are byte-identical (checked with ast) and the
assembled class has exactly the same attributes as before (checked at
runtime). PluginStoreManager is imported from store_manager.py as before.
Tests that patched shared modules (subprocess, requests, tempfile, shutil)
through store_manager now reach them through the module whose code they
exercise; a source-text contract test reads all store_*.py modules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore: annotate findings the split moved into new store modules

subprocess imports and a list-form git clone (no shell), and the config
template's placeholder token string -- existing code that Codacy reported
as new because it moved. Annotated with the repo's nosec/nosemgrep style.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore: annotate the default-branch git clone the split moved

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 15:00:45 -04:00
ChuckandClaude Opus 5.5 d469fe39d2 fix(odds): don't return the cached no-odds marker as odds (#662)
A game ESPN had no odds for is cached as {"no_odds": True}, so it isn't
re-requested on every update. On the next update get_odds() returned that
marker from the cache as if it were odds: a truthy dict that callers took
to mean the game had some. It's still a cache hit (its ttl decides when to
ask again), but get_odds() now returns None for it -- on the cache hit and
in the stale-cache fallback after a failed fetch -- as the plugins' bundled
copies already did.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 14:59:40 -04:00
ChuckandClaude Opus 5.5 09103a8a7d refactor(web): split api_v3/plugins.py by area (#658)
* refactor(web): split api_v3/plugins.py by area

web_interface/blueprints/api_v3/plugins.py (3,285 lines) becomes:
- plugins.py: installed list, enable/disable, plugin actions
- plugin_store.py: install, update, uninstall, store, saved repositories
- plugin_config.py: config get/save, schema, reset
- plugin_assets.py: asset uploads and plugin static files
- plugin_health.py: health, metrics, limits
- plugin_operations.py: operation history, state reconciliation
- plugin_calendar.py: calendar credentials and auth

Pure move: all 44 functions and 38 route decorators are byte-identical
(checked with ast), URLs and endpoint names are unchanged (url-map test).
Each module imports only what it uses. Tests and config.py that reached
into plugins.py for moved names now import from the new module.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(web): keep exception text out of calendar responses; annotate moved code

The split made scanners report existing findings in the moved code as new:
- CodeQL: the calendar auth and calendar-list routes returned exception
  text (redacted, but still derived from the exception). Both now log the
  exception and return a fixed message pointing at the log.
- MD5 in the asset upload only makes a filename unique: usedforsecurity=False.
- pickle reads/writes the calendar plugin's own OAuth token (as before):
  annotated. Token-status labels and a log line naming the secrets path are
  false positives: annotated with the repo's nosec/nosemgrep convention.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(web): name uploaded assets with SHA-256 instead of MD5

The hash only makes an uploaded image's filename unique. Codacy flags MD5
even with usedforsecurity=False, and SHA-256 does the job as well; existing
files keep their names.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(web): keep the redacted exception detail in calendar errors

Reverts the calendar part of 5e695b7c. The project's policy
(test_no_api_v3_handler_discards_its_exception) is that an API error
carries the redacted exception detail -- describe_exception runs it
through the credential redactor -- so a failure is diagnosable from the web
UI. Dropping it for CodeQL broke that; CodeQL can't see the redaction, so
its two alerts here are false positives, like the existing ones on main.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 14:58:53 -04:00
ChuckandClaude Opus 5.5 724673ba0b fix: one retry layer for background fetches; CI installs the web requirements; one Discord invite (#657)
- BackgroundDataService: the session adapter retried connection errors 3x
  inside each attempt of the service's own retry loop (up to 16 connection
  attempts per request on a dead network). The adapter no longer retries;
  ESPN date chunks, which bypass the loop and skip a failed chunk, get a
  small connection retry of their own (_ConnectionRetryingSession).
- CI installs web_interface/requirements.txt. The brotli header test now
  checks its intent (core never hand-sets br; requests may advertise it when
  a decoder is installed) instead of failing whenever brotli is present.
- Every Discord link uses the LEDMatrix server's invite (RdrC37rEag).

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 11:08:43 -04:00
ChuckandClaude Opus 5.5 6cfcf2e384 fix(web): plugin dir resolver in routes, nmcli AP detection, daemon config reload, upload safety, BDF preview (#655)
* fix(web): plugin dir resolver in routes, nmcli AP detection, daemon config reload, upload safety

- Route plugin lookups (installed list, update, recorded version, config
  form, web UI pages) through the plugin manager's resolver so plugins in
  ledmatrix-<id> directories work.
- Captive-portal detection also sees the nmcli fallback AP (cached).
- WiFi monitor daemon re-reads wifi_config.json when its mtime changes.
- Drop the AP check in disconnect_from_network that could never fire.
- LED status file per WiFiManager; config path falls back to this checkout.
- BDF font preview via src.common.bdf_font.
- Asset uploads validate every file before saving; metadata and calendar
  credentials written atomically; no absolute path in the response;
  asset delete answers 400 for a missing body.
- Coerce string booleans in plugin toggle, on-demand start and AP force.
- SSE broadcaster clears its thread handle before exiting.
- start.py log filter handles every exc_info form.
- Cleanups: unused plugins/fonts partial work, duplicate backup catch-alls,
  raw-config error helper, update-route tidy, redundant imports.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(web): request BDF font previews now that the server renders them

The Fonts tab skipped the preview request for .bdf files because the server
used to refuse them; /fonts/preview now draws BDF with the shared loader.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(web): take the update route's plugin directory from a directory listing

CodeQL flagged the path built from the request's plugin_id (the id was
already validated with safe_path_component, which CodeQL doesn't model; the
same flow on main is alerts 738/739). The directory is now the entry of
plugins_dir matched by name, so nothing built from user input reaches the
filesystem; an id with nothing installed goes to the store manager, which
reports it not found as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(web): read the blueprint's plugin_manager defensively in _plugin_directory

_get_plugin_version now goes through _plugin_directory, which read
api_v3.plugin_manager directly; the attribute exists only once the app sets
it, so test_path_traversal_guards::test_a_real_manifest_is_read failed
when run on its own (order-dependent in the full suite).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 10:42:07 -04:00
ChuckandClaude Opus 5.5 c00bf5e8e6 fix(plugin-system): unload/update race, failed-load cleanup, limits validation, schema lookup, install rollback (#653)
* fix(plugin-system): unload/update race, failed-load module cleanup, limits validation, schema lookup, install rollback, op-queue dedupe

- unload_plugin takes the per-plugin lock (5s bounded) before cleanup(),
  and an update() that finishes after its plugin was unloaded no longer
  sets the state back to ENABLED.
- A load that fails after import drops plugin_<id> and its submodules
  and forgets its manager fonts, so a fixed plugin reloads new code.
- Resource limits are validated as non-negative numbers: 400 at
  POST /plugins/limits, bad cached records ignored with one warning.
  Route docstrings note health/metrics reset and limits only change the
  web process's view.
- SchemaManager.get_schema_path resolves each search dir via
  resolve_plugin_dir (manifest id, ledmatrix-<id>) before the literal
  paths; plugins/ still before plugin-repos/. Misses cached 30s and
  logged once at DEBUG.
- install_from_url sets an existing copy aside and restores it if the
  move fails, under the per-plugin reinstall lock.
- Operation queue refuses a second pending op for a plugin and trims
  _operations with history.
- get_vegas_render_width reads display_manager.width first.
- get_logger in store/schema/health/resource/saved_repositories;
  UTF-8 reads in store_manager and state_manager.
- Docs: update_interval precedence (manifest over config) stated where
  users are told to set it in config.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(web): build the limits 400 message from the field name, not an exception

CodeQL flagged str(e) flowing into the response. invalid_limit_field()
returns the offending field without raising, and limits_from_dict uses it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 10:41:40 -04:00
ChuckandClaude Opus 5.5 0e9e2cabba fix(web): widget cache-busting, dead frontend code, and dependency pins (#656)
- Plugin-supplied widgets load as /static/plugin-widgets/...js?v=<plugin
  version>, so an update isn't hidden behind the year-long immutable cache.
- Fire-and-forget loadInstalledPlugins() calls catch the rejection it has
  already reported, so the global handler no longer adds a second toast.
- Timezone picker renders again when the General partial is re-injected.
- Remove dead code: executePluginAction's six plugin-id fallbacks and
  [DEBUG] logging, window.currentPluginConfig and every read of it, the
  file-upload JSON delete branch, unused PluginAPI / PluginInstallManager /
  PluginStateManager helpers, loadPluginWidgetsFromManifest, the stale
  install_manager.js and LEDVisibility fallbacks, error_handler.js's global
  escapeHtml, 13 unused CSS rules, and stale comments/no-op returns.
- pytz < 2027, psutil < 7 in requirements-test.txt, pytest-cov < 8.
- Pin anthropics/claude-code-action to the commit v1 resolves to.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 10:40:52 -04:00
ChuckandClaude Opus 5.5 65d82580bc test(starlark): cover the review fixes #535 shipped without tests (#650)
Six Starlark fixes are on main via #535 and #537, but a follow-up commit
carrying tests for half of them was pushed to fix/starlark-pixlet-install
six minutes after #535 merged, so those tests never landed. This ports
them onto the api_v3 package split:

- a failed toggle write answers 500, and a loaded app is not flipped in
  memory when the manifest write fails
- each manifest writer gets its own temp file; concurrent writes leave
  readable JSON; no temp files are left behind
- a failed dynamic import of tronbyte_repository / pixlet_renderer does
  not stay cached in sys.modules
- a failed save_config() leaves config and timing untouched and does not
  re-render; a successful save still applies

It also logs when the timing update to the manifest is not persisted.
_update_manifest_safe answers False rather than raising, so the existing
except branch never saw that failure.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 10:40:32 -04:00
ChuckandClaude Opus 5.5 f6c0fe55d9 fix(core): font zip cache, monotonic timers, resolver back-off, and other core/common fixes (#654)
* fix(core): font zip cache, monotonic timers, resolver back-off, and other core/common fixes

- font_manager: a .zip font URL is served as its extracted font after a
  restart (the cached-file check returned the archive first); downloads
  use requests with a 30s timeout into a temp file + os.replace.
- api_helper / sync_manager: rate-limit and heartbeat/leader timeouts use
  time.monotonic(); last_request_time and the status file's ts stay
  wall-clock. set_on_new_cycle docstring no longer claims core uses it.
- logo_helper: the placeholder uses the same scaled box as a real logo.
- permission_utils: one _sudo_bash_candidates() helper (with the sudoers
  exact-argv rationale) shared by sudo_remove_directory, which now retries
  the next bash path on a sudo refusal, and install_requirements_file.
- dynamic_team_resolver: failed/empty fetch backs off 5 min; duplicate
  INFO log and contradictory docstring example fixed.
- element_style: scale default looked up through element aliases.
- background_data_service: cache-hit callback runs outside the lock.
- config_arrays: union-aware type check (["array","null"]); stale
  dotToNested() reference removed.
- auto_update_setup: non-dict auto_update reads as off; temp result file
  unlinked when the write fails.
- exceptions: constructors copy the caller's context dict.
- logging_config: StructuredFormatter json.dumps(default=str).
- error_aggregator: removed unused export_path/export_to_file/_auto_export.
- Docstrings: validate_file_upload max_size_mb, raise_on_errors.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(sync): retry the status-file rename like the other atomic writers

On Windows os.replace can fail with "Access is denied" while a scanner
briefly holds the target open; config_manager_atomic._replace already
retries that (and re-raises at once on other platforms). The sync status
writer called os.replace directly, which made
test_concurrent_writers_each_use_their_own_temp_file flaky on Windows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 10:40:16 -04:00
ChuckandClaude Opus 5.5 6f45ff5e63 fix(display): thread-safety for deferred updates, BDF faces and follower image; one refresh default (#652)
- DisplayManager.defer_update()/process_deferred_updates(): one lock around
  every queue mutation (appends from the update thread were lost to the
  render thread's filter/slice reassignments); callables run outside it.
- FontManager and element_style no longer cache BDF freetype.Face objects
  process-wide (load_bdf_face caches them per thread); element_style's LRU
  is locked against get/move_to_end vs eviction races.
- limit_refresh_rate_hz default is one constant, DEFAULT_REFRESH_LIMIT_HZ =
  100 (the template's), for the library options, refresh_hz, the matrix
  guard, Vegas and scroll_config. Previously a missing key capped the panel
  at 90 while pacing assumed 100.
- Sync follower: the TCP thread queues the leader's scroll image; the render
  thread swaps image/array/width in between frames.
- update_display() error log rate-limited (traceback first, then once a
  minute with a count); swallowed DisplayController exceptions log at DEBUG.
- Root display_controller.py runs run.py via runpy.
- stream_manager: correct the RLock release comments; merge duplicate if.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 10:39:40 -04:00
ChuckandClaude Opus 5.5 1e62677257 fix(vegas): pause for STATIC plugins where their turn falls in the strip (#651)
The static trigger peeked at the front of StreamManager's segment buffer,
which continuous scrolling (the default) never advances -- it extends the
strip with take_next_group() -- so the same first segment was examined on
every frame. A STATIC plugin paused the scroll only if it was first, once,
at startup; otherwise it scrolled past as ordinary content. Swap mode had
the same problem for any STATIC plugin not first in its cycle.

The render pipeline now records a marker (strip column, plugin id) for
each STATIC plugin where the strip is built -- composition and every
extension -- shifts the markers when the scrolled prefix is trimmed, and
clears them on reset. The coordinator pauses when the scroll reaches the
next marker: a tuple comparison per frame instead of a lock, a plugin
lookup and a get_vegas_display_mode() call. take_next_group() no longer
renders STATIC plugins' content. The pause calls display() under the
plugin lock and is timed with the monotonic clock.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 10:39:14 -04:00
ChuckandClaude Opus 5.5 224847cebc fix(display): stop the run loop spinning when no mode has anything to show (#649)
* fix(display): stop the run loop spinning when no mode has anything to show

A mode whose display() reports nothing rotates to the next at once, with no
dwell. With every enabled mode empty (only a sports plugin in its
off-season, say) the loop went round with no sleep: on ledpi, 169% CPU and
~1,800 "No content" log lines every 10 seconds. After one full rotation of
empty passes it now pauses EMPTY_ROTATION_PAUSE (1s) per pass, servicing
plugin updates and returning early on on-demand or schedule changes; live
priority is still checked at the top of every pass, and the streak resets
as soon as any mode shows something.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(display): restart the loop if the empty-rotation pause starts on-demand; per-rotation streak

- an on-demand request serviced during the pause returned early into the
  on-demand branch, which advanced past the mode just requested; the loop
  now restarts when the pause changed the mode, on-demand state or schedule
- the streak is reset when the rotation changes (on-demand start/stop, a
  plugin enabled or disabled), so a streak from one rotation can't make
  another pause before its own modes are tried
- docstring: live content is picked up within the pause, not "at once"

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 09:07:08 -04:00
ChuckandClaude Opus 5.5 aeaeaa4e94 chore: make contributor tooling work; fix drifted docs (#648)
- mypy.ini parses again (multi-line exclude and inline value comments made
  mypy reject the file); the mypy pre-commit hook is manual-only until the
  ~500 existing errors in src/ are paid down, and CONTRIBUTING says so
- .gitignore: ignore all of config/ except the templates (ytm_auth.json and
  others weren't ignored)
- .gitattributes: LF for .sh and .service
- claude-code-review: skip fork PRs, which have no secrets
- check_system_compatibility.sh: 3.13 supported, <3.10 an error
- docs/scripts drift: emulator guide, README API Metrics, route count,
  docs index, scripts README; pyflakes nits in dev scripts

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 08:26:55 -04:00
ChuckandClaude Opus 5.5 76f5d8a336 fix(web): seven web UI bugs, and remove dead plugins_manager.js helpers (#647)
- Operation History: the plugin filter lists the installed plugin ids
  instead of one option, "plugins" (Object.keys of {plugins: [...]}).
- Ctrl/Cmd+S submits the active tab's first visible form with
  requestSubmit() (validation and onsubmit guards run) instead of a bare
  Event on the first form in the document; skipped inside a modal dialog
  and on tabs without a form.
- Overview "Check Updates" confirms like "Update Code", takes its button
  explicitly (no implicit global event) and shows the server's message.
  Both, and the Tools tab git pull, raise the restart-pending banner on
  restart_required.
- Tools: toolsAction and diagnostics show the server's error message;
  only a non-JSON body falls back to HTTP <status>.
- Installed list after uninstall: PluginAPI writes clear the throttler's
  GET cache, a forced loadInstalledPlugins clears it too, and the
  post-uninstall reload goes through refreshInstalledPlugins().
- Plugin widgets load from /static/plugin-widgets/ only (the other two
  paths have no route).
- Raw JSON editor escapes the parse error; slider escapes value/min/max/step.
- Removed the unreferenced array-of-objects and key-value helpers from
  plugins_manager.js, the textarea auto-resize and Ctrl+R handlers in
  app.js, and a redundant ?v= on the plugins_manager.js script tag.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 08:26:27 -04:00
ChuckandClaude Opus 5.5 7eb7a58d0c fix: web UI and src.common bugs (wifi wrong-password, plugin icon, starlark toggle, API caching, scroll/logo/font helpers) (#646)
- wifi: keep the "wrong_password:" prefix through the restore/AP fallback so
  the UI's incorrect-password prompt fires again.
- /plugins/installed returns the manifest's icon (string only).
- /starlark/apps/<id>/toggle coerces `enabled` and delegates to
  _toggle_starlark_app (disk before memory, no KeyError, "false" is false).
- /api/v3/ JSON GETs are sent Cache-Control: no-store; non-JSON keeps 5s.
- ScrollHelper.set_scrolling_image converts non-RGB input (alpha onto black);
  create/set_scrolling_image reset last_update_time like reset_scroll.
- LogoHelper backs off a failed download per path for
  MISSING_LOGO_RECHECK_SECONDS; cleared on invalidate/clear_cache.
- refresh_placeholder_timestamp saves atomically.
- FontManager.clear_cache / _clear_plugin_font_cache bump cache_generation.
- Odds manager: per-game logs to DEBUG; JSON decode error caught before
  RequestException (same cooldown).
- element_style mangled continuations; startup validator skips null plugin
  blocks and reuses the controller's discovery.
- src/common/README lists frame_timing, json_body, render_gate.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 08:26:05 -04:00
ChuckandClaude Opus 5.5 b518c51679 fix(plugin-system): load/enable failures, atomic state files, pip lock, test-double parity (#645)
- load_plugin: an on_enable() that raises unregisters the instance, so the
  next load retries instead of returning True "already loaded".
- get_plugin_info: guard plugin.get_info(); one plugin raising no longer
  breaks /api/v3/plugins/installed.
- plugin_state.json and the operation history are written with
  atomic_write_text under their lock.
- plugin_loader: module-level lock serialises pip installs across the
  parallel startup loaders.
- store_manager._install_via_download: extract dir cleanup moved to finally.
- Test doubles: draw_image() warns (DeprecationWarning; the real
  DisplayManager has none), MockDisplayManager.draw_text accepts the real
  signature's optional params, VisualTestDisplayManager logs draw errors at
  WARNING.
- Docs/comments: compatibility.py method name, PluginState.LOADED meaning,
  brittle schema count, why _report_skip_once uses setdefault.
- Remove unused PluginOperationQueue.get_active_operations().

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 08:25:45 -04:00
ChuckandClaude Opus 5.5 6bc13a8934 fix(display): Vegas resumes after live priority, and six smaller runtime fixes (#644)
- Vegas: a live-priority pause was only lifted from inside run_frame(),
  which returns before that check while paused, so the ticker never came
  back until a restart. run_iteration() now resumes it (the controller
  only calls it when nothing preempts Vegas); start()/stop() clear the
  pause state. Iteration length is timed with the monotonic clock.
- Dim schedule: a per-day disabled day now updates the minute-gate cache,
  so brightness no longer flips back to dim within each minute.
- On-demand: a second request no longer overwrites the rotation resume
  index with the first request's mode.
- Render pipeline: reset() drops the prepared group and deferred queue,
  and a prefetch in flight across a reset discards its result.
- Sync: stop() removes the status file (and the controller's cleanup now
  calls it), standalone removes a stale one at startup, and writes use a
  unique mkstemp temp file.
- render_gate.swap_releases_gil() delegates to frame_timing.
- Stale docstrings/comments corrected.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 08:25:26 -04:00
ChuckandClaude Opus 5.5 bcef1957a9 fix(security): refuse unsafe plugin ids, keep secrets private, validate request bodies (#643)
* fix(security): refuse unsafe plugin ids, keep secrets private, validate bodies

- install_from_url and the registry install's manifest rename refuse a
  plugin id that is not a single safe name (no ../ out of plugins_dir).
- Uninstall and config reset refuse core config sections and ids with
  path parts; uninstall of a plugin whose directory is gone still works.
- separate_secrets checks a field's own x-secret marker before recursing,
  so object/array secrets no longer land in config.json.
- Backup restore creates missing secrets/wifi/ytm files with mode 640;
  export skips non-object manifests and no longer collides on same-second
  exports.
- SYSTEM_FONTS includes every bundled font from BUNDLED_FONTS.
- Raw config/secrets saves and validate_request_json require a JSON object.
- A blank max_dynamic_duration_seconds keeps the stored value; other values
  are validated to 30-1800 instead of raising a 500.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(security): validate the id before install_plugin moves anything; claim backup names atomically

- install_plugin set aside plugins_dir / plugin_id before any id check, so
  "../x" moved a directory outside the plugins dir (the rollback moved it
  back, but only if the install path got that far)
- two exports finishing in the same second could both see a free name and
  the later os.replace destroyed the first archive; the name is now
  claimed with O_EXCL before the archive is swapped in

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 08:24:43 -04:00
ChuckandClaude Opus 5.5 da5937da3d fix: six bugs found testing main on a real Pi (#641)
* fix: six bugs found testing main on a real Pi (ledpi)

- Stopping the service now runs cleanup. systemd stops ledmatrix.service
  with SIGTERM, whose default action ended Python before run()'s finally
  block, so the update worker, Vegas and the panel were never torn down.
  main() now turns SIGTERM into KeyboardInterrupt, the Ctrl-C path.
- "Now showing" no longer turns into "unknown". display_current_state was
  only written on a mode change and the web UI reads it with max_age=120,
  so a live game or a single plugin on screen for longer read as unknown.
  It is republished every 30 s while unchanged.
- Switching Vegas on in the web UI works when it was off at startup. The
  coordinator was only created at startup; the config watcher now flags it
  and the render thread creates it.
- configure_web_sudo.sh finds reboot and poweroff in /usr/sbin. Run as the
  web user it could not, silently dropped their rules and still said it
  granted them, so the web UI's Reboot/Shutdown stopped working.
- check_system_compatibility.sh reports installed packages as installed.
  `dpkg -l | grep -q` under pipefail failed when grep exited early.
- A network failure fetching GitHub repo info logs a WARNING, not ERROR.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(changelog): fixes found testing on a Pi

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: run the Linux-only script tests correctly

The sbin-lookup test set PATH=/nonexistent and then could not find bash
itself; call it by absolute path. The dpkg-query stub read $4, but the
package name is the third (last) argument. Both now pass on a Pi.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(display): cover the follower, long-render and startup cases

Review follow-ups on the ledpi fixes:
- The pending Vegas start is applied before the sync-follower branch too
  (_apply_pending_vegas_init), which skips _is_vegas_mode_active() while a
  follower is connected but needs the coordinator for the leader's image.
- _service_pending_changes(), which runs inside Vegas iterations and long
  screens, republishes a stale display_current_state as well; the main
  loop alone could be away for a 240 s Vegas iteration.
- The SIGTERM handler is installed after DisplayController() is built, so a
  stop during parallel plugin loading keeps the default immediate exit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-27 18:13:49 -04:00
ChuckandClaude Opus 5 c4927e82a3 fix(config): normalize nullable arrays and objects instead of refusing them (#642)
* fix(config): normalize nullable arrays and objects instead of refusing them

`element_style._nullable` widens every `customization.modes.<mode>` override
with 'null' so a blank means "inherit the base", which turns a colour declared
"array" into ["array", "null"]. `normalize_config_values` only knew how to
convert null/integer/number/boolean out of a union, so a valid [0, 249, 0]
matched nothing and logged

    Could not normalize field customization.modes.upcoming.odds_text.text_color:
    value=[0, 249, 0], type=<class 'list'>, schema_type=['array', 'null']

The warning was the harmless half. It `continue`d past the single-type handling
below, where `prop_type == 'array'` coerces items, so a nullable array never had
its items normalized while a plain one did. A form posts numbers as strings, so
["0", "249", "0"] survived to the validator and was rejected with "Expected type
integer, got str" -- setting a per-mode colour in the web UI failed outright.
Every per-mode override of a structural or string type was exposed, across all
eight scoreboard plugins, not only colours.

Re-enter the single-type handling with the matched member rather than bailing,
accept a string that matches, and warn only on a genuine mismatch so the
diagnostic still reaches the validator.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ET8e5weDrb5Ju5QTKLU7zh

* fix(config): convert only integral numbers for integer array items

Review catch on the previous commit. Routing nullable arrays into the shared
item handling made its integer coercion reachable for them, and that coercion
called int(v) on any number: a client sending [2.5, 249, 0] for an RGB array
got 2 stored and a 200 back, so a wrong value was silently corrected into a
valid-looking one rather than refused.

Convert only genuinely integral values, at both the union-item and the plain
'array' item branch so the two cannot drift. A whole float -- 2.0, which is all
JSON can express for an integer -- still converts. This also settles an
inconsistency that predates the change: int('2.5') raises, so the string form
was always preserved and rejected while the numeric form was truncated.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ET8e5weDrb5Ju5QTKLU7zh

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-26 16:33:23 -04:00
ChuckandClaude Opus 5.5 9964dd2183 feat(vegas): render plugin content off the render thread, and keep it off the GIL when the panel needs it (#630)
DisplayManager.offscreen() gives a thread its own canvas, so Vegas renders every plugin's ticker content on its prefetch thread instead of pausing the scroll for canvas-bound plugins on the render thread. A render gate (src/common/render_gate.py, vegas_scroll.prefetch_gate, on by default with the GIL-releasing binding) lets the prefetch thread run Python only while the render thread waits in SwapOnVSync: on hdpi, frames 2+ refreshes late fell eightfold and late frames overall from 0.90% to 0.60%. See docs/OFFSCREEN_RENDERING.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-24 19:57:03 -04:00
ChuckandClaude Opus 5.5 865d62f67b feat(display): compensate for the panel's scan order while scrolling (#634)
A 1:N-scan HUB75 panel lights the two rows either side of its middle at opposite ends of each refresh, so a scroll at one pixel per refresh shows a 1px step across the middle of every panel. While something scrolls at one frame per refresh, DisplayManager now shows the half whose seam row lights first one refresh behind the other (src/scan_order.py), which lines the two up again. Only for layouts whose row order is known; display.scan_order_compensation "off" disables it. Confirmed on hdpi before and after.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-24 19:47:44 -04:00
428 changed files with 51644 additions and 11681 deletions
-1
View File
@@ -4,4 +4,3 @@ exclude_paths:
- "plugins/**" - "plugins/**"
- "assets/**" - "assets/**"
- "test/**" - "test/**"
- "scripts/debug/**"
+10
View File
@@ -1,2 +1,12 @@
# Auto detect text files and perform LF normalization # Auto detect text files and perform LF normalization
* text=auto * text=auto
# Files the Pi executes must stay LF even in a Windows checkout with
# core.autocrlf=true: a CRLF shebang line fails with "bad interpreter",
# and systemd rejects CRLF unit files.
*.sh text eol=lf
*.service text eol=lf
# Generated by scripts/build_css.py; collapsed in diffs, not hand-edited.
web_interface/static/v3/tailwind.css linguist-generated=true
web_interface/static/v3/plugin-frame.css linguist-generated=true
+5 -1
View File
@@ -6,6 +6,10 @@ on:
jobs: jobs:
claude-review: claude-review:
# Pull requests from forks get no repository secrets, so without this
# guard every outside contributor's PR showed this check red for a reason
# they can't fix. Skipped checks don't block merging.
if: github.event.pull_request.head.repo.full_name == github.repository
runs-on: ubuntu-latest runs-on: ubuntu-latest
permissions: permissions:
contents: read contents: read
@@ -21,7 +25,7 @@ jobs:
- name: Run Claude Code Review - name: Run Claude Code Review
id: claude-review id: claude-review
uses: anthropics/claude-code-action@v1 uses: anthropics/claude-code-action@756cc22e19660d20e8cc9496b4f242475a7f7790 # v1
with: with:
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }} claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
# Review PRs opened by the Claude GitHub App. Without this the action # Review PRs opened by the Claude GitHub App. Without this the action
+1 -1
View File
@@ -32,7 +32,7 @@ jobs:
- name: Run Claude Code - name: Run Claude Code
id: claude id: claude
uses: anthropics/claude-code-action@v1 uses: anthropics/claude-code-action@756cc22e19660d20e8cc9496b4f242475a7f7790 # v1
with: with:
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }} claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
+125 -3
View File
@@ -8,7 +8,7 @@ on:
# needs a re-run or didn't get created. # needs a re-run or didn't get created.
workflow_dispatch: workflow_dispatch:
# Both jobs only check out the repo and run pytest. # The jobs only check out the repo and run the tests.
permissions: permissions:
contents: read contents: read
@@ -35,7 +35,7 @@ jobs:
- name: Install dependencies - name: Install dependencies
run: | run: |
python -m pip install --upgrade pip python -m pip install --upgrade pip
pip install -r requirements.txt -r requirements-test.txt pip install -r requirements.txt -r web_interface/requirements.txt -r requirements-test.txt
pip install RGBMatrixEmulator pip install RGBMatrixEmulator
- name: Run plugin safety harness - name: Run plugin safety harness
@@ -58,7 +58,7 @@ jobs:
- name: Install dependencies - name: Install dependencies
run: | run: |
python -m pip install --upgrade pip python -m pip install --upgrade pip
pip install -r requirements.txt -r requirements-test.txt pip install -r requirements.txt -r web_interface/requirements.txt -r requirements-test.txt
pip install RGBMatrixEmulator pip install RGBMatrixEmulator
# Run the ENTIRE test tree (except test/plugins, which the # Run the ENTIRE test tree (except test/plugins, which the
@@ -73,3 +73,125 @@ jobs:
--cov=src --cov=web_interface \ --cov=src --cov=web_interface \
--cov-report=term \ --cov-report=term \
--cov-fail-under=52 --cov-fail-under=52
js-tests:
name: Web UI JS tests
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
persist-credentials: false
- uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5.3.0
with:
python-version: "3.12"
cache: pip
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
with:
node-version: "22"
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt -r web_interface/requirements.txt
npm install --no-audit --no-fund --prefix test/js
# The DOM suites test the real server-rendered pages and API, so they
# need the web interface running. REQUIRE_DOM turns "couldn't reach it"
# into a failure instead of a silent skip.
- name: Start the web interface
run: |
EMULATOR=true python -c "from web_interface.app import app; app.run(host='127.0.0.1', port=5000, threaded=True)" > web.log 2>&1 &
for i in $(seq 60); do curl -sf -o /dev/null http://127.0.0.1:5000/ && exit 0; sleep 1; done
cat web.log
exit 1
- name: Run JS suites
env:
BASE: http://127.0.0.1:5000
REQUIRE_DOM: "1"
run: node test/js/run_all.js
css-build:
name: Tailwind CSS is up to date
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
persist-credentials: false
- uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5.3.0
with:
python-version: "3.12"
# Downloads the pinned standalone Tailwind CLI (SHA-256 checked; no
# Node), rebuilds static/v3/tailwind.css and plugin-frame.css from the
# templates and JS, and fails if the committed files differ. Fix a
# failure by running `python3 scripts/build_css.py` and committing.
- name: Check the committed CSS matches a fresh build
run: python scripts/build_css.py --check
type-check:
name: Type check (mypy ratchet)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
persist-credentials: false
- uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5.3.0
with:
python-version: "3.12"
cache: pip
# The runtime requirements are installed so mypy sees the real types of
# PIL, requests, psutil and friends -- missing, they'd be Any and the
# result would differ from a developer's machine. mypy and the stubs are
# pinned so a new release can't turn this red without a code change.
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt -r web_interface/requirements.txt
pip install mypy==1.20.2 types-requests==2.33.0.20260906 types-pytz==2026.4.0.20260926
# mypy on exactly the modules in mypy-clean.txt; fails on any error in
# them, or if a listed file is missing. See CONTRIBUTING.md.
- name: Run mypy on the ratchet list
run: python scripts/check_types.py
sports-drift-report:
name: Sports drift report (report only)
runs-on: ubuntu-latest
# A progress measure for docs/SPORTS_UNIFICATION.md, never a gate: the
# monorepo's own check_sports_drift.py is the gate. The step summary shows
# how many bodies each scoreboard method family still has.
continue-on-error: true
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
persist-credentials: false
- name: Check out ledmatrix-plugins (main)
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
repository: ChuckBuilds/ledmatrix-plugins
path: ledmatrix-plugins
persist-credentials: false
- uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5.3.0
with:
python-version: "3.12"
# Stdlib only; exits 0 whatever it finds.
- name: Report method-family drift across the nine scoreboards
run: |
python scripts/sports_drift_report.py --plugins ledmatrix-plugins \
--markdown --json sports-drift.json >> "$GITHUB_STEP_SUMMARY"
python scripts/sports_drift_report.py --plugins ledmatrix-plugins
- name: Upload the full report
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: sports-drift-report
path: sports-drift.json
+7 -9
View File
@@ -3,15 +3,13 @@ __pycache__/
*.py[cod] *.py[cod]
*$py.class *$py.class
# Secrets # Secrets and per-device state. Everything the software writes into config/
config/config_secrets.json # is local to one device -- config.json, config_secrets.json, wifi_config.json,
# Atomic writes leave these behind when a save or a test is interrupted; # ytm_auth.json (a login session), saved_repositories.json, font_overrides.json,
# the suite drops several per run. # and the temp files atomic writes leave behind when interrupted -- so only
config/.config_secrets.json.tmp.* # the templates are tracked. Listing files one by one missed several.
config/config.json config/*
config/config.json.backup !config/*.template.json
config/wifi_config.json
config/uninstalled_plugins.json
credentials.json credentials.json
token.pickle token.pickle
+13 -5
View File
@@ -37,14 +37,22 @@ repos:
types: [python] types: [python]
pass_filenames: false pass_filenames: false
- repo: https://github.com/pre-commit/mirrors-mypy # The mypy ratchet -- the same check as CI's "Type check (mypy ratchet)"
rev: v1.8.0 # job: mypy on exactly the modules listed in mypy-clean.txt. Run it with
# pre-commit run mypy --hook-stage manual
# A local hook rather than mirrors-mypy so mypy sees the packages installed
# from requirements.txt, as CI does; an isolated hook env without them types
# PIL, requests and friends as Any and reports different errors. Needs
# mypy==1.20.2 (the version CI pins) in the environment you commit from.
- repo: local
hooks: hooks:
- id: mypy - id: mypy
additional_dependencies: [types-requests, types-pytz] name: mypy (ratchet, mypy-clean.txt)
args: [--ignore-missing-imports, --no-error-summary] entry: python scripts/check_types.py
language: system
pass_filenames: false pass_filenames: false
files: ^src/ always_run: true
stages: [manual]
- repo: https://github.com/PyCQA/bandit - repo: https://github.com/PyCQA/bandit
rev: 1.8.3 rev: 1.8.3
+1211 -222
View File
File diff suppressed because it is too large Load Diff
+1
View File
@@ -46,6 +46,7 @@
- Store manager (`PluginStoreManager` in `src/plugin_system/store_manager.py`) handles install/update/uninstall - Store manager (`PluginStoreManager` in `src/plugin_system/store_manager.py`) handles install/update/uninstall
- Monorepo plugins are installed without a `.git` directory: GitHub Trees API + raw downloads, falling back to ZIP extraction - Monorepo plugins are installed without a `.git` directory: GitHub Trees API + raw downloads, falling back to ZIP extraction
- Update detection for monorepo plugins uses version comparison (manifest version vs registry latest_version) - Update detection for monorepo plugins uses version comparison (manifest version vs registry latest_version)
- Optional registry entry fields (`store_registry.py`): `ledmatrix_min_version` refuses an incompatible install/update before the download (the post-download manifest gate stays as the fallback); `aliases` are the entry's other ids (manifest id `ledmatrix-weather` for `weather`), used with the `plugin_path` name by update/uninstall/reinstall to find the install — only this registry proof counts, never a bare `ledmatrix-<id>` folder (owner decision, #686; such a folder is only logged); `commit` is informational. An older plugins.json has none of them
- Plugin configs stored in `config/config.json`, NOT in plugin directories — safe across reinstalls - Plugin configs stored in `config/config.json`, NOT in plugin directories — safe across reinstalls
- Third-party plugins can use their own repo URL with empty `plugin_path` - Third-party plugins can use their own repo URL with empty `plugin_path`
+1 -1
View File
@@ -63,7 +63,7 @@ ChuckBuilds, and any other forums hosted by or affiliated with the project.
Instances of abusive, harassing, or otherwise unacceptable behavior may be Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported to the community leaders responsible for enforcement on the reported to the community leaders responsible for enforcement on the
[LEDMatrix Discord](https://discord.gg/uW36dVAtcT) (DM a moderator or [LEDMatrix Discord](https://discord.gg/RdrC37rEag) (DM a moderator or
ChuckBuilds directly) or by opening a private GitHub Security Advisory if ChuckBuilds directly) or by opening a private GitHub Security Advisory if
the issue involves account safety. All complaints will be reviewed and the issue involves account safety. All complaints will be reviewed and
investigated promptly and fairly. investigated promptly and fairly.
+17 -5
View File
@@ -9,7 +9,7 @@ improvements, and code changes.
- **Bugs / feature requests**: open an issue using one of the templates - **Bugs / feature requests**: open an issue using one of the templates
in [`.github/ISSUE_TEMPLATE/`](.github/ISSUE_TEMPLATE/). in [`.github/ISSUE_TEMPLATE/`](.github/ISSUE_TEMPLATE/).
- **Real-time discussion**: the - **Real-time discussion**: the
[LEDMatrix Discord](https://discord.gg/uW36dVAtcT). [LEDMatrix Discord](https://discord.gg/RdrC37rEag).
- **Plugin development**: - **Plugin development**:
[`docs/PLUGIN_DEVELOPMENT_GUIDE.md`](docs/PLUGIN_DEVELOPMENT_GUIDE.md) [`docs/PLUGIN_DEVELOPMENT_GUIDE.md`](docs/PLUGIN_DEVELOPMENT_GUIDE.md)
and the [`ledmatrix-plugins`](https://github.com/ChuckBuilds/ledmatrix-plugins) and the [`ledmatrix-plugins`](https://github.com/ChuckBuilds/ledmatrix-plugins)
@@ -58,12 +58,24 @@ integration tests.
3. **Keep PRs focused.** One conceptual change per PR. If you find 3. **Keep PRs focused.** One conceptual change per PR. If you find
adjacent bugs while working, fix them in a separate PR. adjacent bugs while working, fix them in a separate PR.
4. **Follow the existing code style.** The pre-commit hooks run 4. **Follow the existing code style.** The pre-commit hooks run
`flake8` (E9, F63, F7, F82 plus bugbear `B` checks), `mypy` on `flake8` (E9, F63, F7, F82 plus bugbear `B` checks), `bandit`,
`src/`, `bandit`, and `gitleaks` — install the CLI with and `gitleaks` — install the CLI with
`python -m pip install pre-commit`, then run `python -m pip install pre-commit`, then run
`pre-commit install` so they run on every commit; HTML/JS in `pre-commit install` so they run on every commit. Type checking
is a ratchet while the existing mypy errors in `src/` are paid
down: `mypy-clean.txt` lists the modules that type-check clean, and
CI runs `python scripts/check_types.py` (also the manual hook
`pre-commit run mypy --hook-stage manual`) to keep every listed
module clean. When you make another module clean, add it to the
list (sorted); don't take one off to get CI green. Keep type fixes
annotation-only where you can -- widen a hint rather than delete a
defensive runtime check mypy calls unreachable. HTML/JS in
`web_interface/` follows the patterns already in `templates/v3/` `web_interface/` follows the patterns already in `templates/v3/`
and `static/v3/`. and `static/v3/`. If you change a template or a static JS file,
run `python3 scripts/build_css.py` and commit the regenerated
`static/v3/tailwind.css` with it -- CI fails when the committed CSS
is out of date. It needs no Node; see
[`web_interface/README.md`](web_interface/README.md#styling-tailwind-css).
5. **Update documentation** alongside code changes. If you add a 5. **Update documentation** alongside code changes. If you add a
config key, document it in the relevant `*.md` file (or, for config key, document it in the relevant `*.md` file (or, for
plugins, in `config_schema.json` so the form is auto-generated). plugins, in `config_schema.json` so the form is auto-generated).
+2 -2
View File
@@ -33,7 +33,7 @@ I'm trying to be open to constructive criticism and support, as long as it's a r
- Show support on Youtube: https://www.youtube.com/@ChuckBuilds - Show support on Youtube: https://www.youtube.com/@ChuckBuilds
- Check out the write-up on my website: https://www.chuck-builds.com/led-matrix/ - Check out the write-up on my website: https://www.chuck-builds.com/led-matrix/
- Stay in touch on Instagram: https://www.instagram.com/ChuckBuilds/ - Stay in touch on Instagram: https://www.instagram.com/ChuckBuilds/
- Want to chat? Reach out on the LEDMatrix Discord: [https://discord.com/invite/uW36dVAtcT](https://discord.gg/dfFwsasa6W) - Want to chat? Reach out on the LEDMatrix Discord: [https://discord.gg/RdrC37rEag](https://discord.gg/RdrC37rEag)
- Feeling Generous? Consider sponsoring this project or sending a donation (these AI credits aren't cheap!) - Feeling Generous? Consider sponsoring this project or sending a donation (these AI credits aren't cheap!)
----------------------------------------------------------------------------------- -----------------------------------------------------------------------------------
@@ -948,7 +948,7 @@ sudo systemctl enable ledmatrix-web.service
- **On-Demand Controls**: Start specific displays (weather, stocks, sports) on demand - **On-Demand Controls**: Start specific displays (weather, stocks, sports) on demand
- **Service Management**: Start/stop the main display service - **Service Management**: Start/stop the main display service
- **System Controls**: Restart, update code, and manage the system - **System Controls**: Restart, update code, and manage the system
- **API Metrics**: Monitor API usage and system performance - **System Stats**: CPU, memory and temperature on the Overview tab
- **Logs**: View system logs in real-time - **Logs**: View system logs in real-time
### Troubleshooting Web Interface ### Troubleshooting Web Interface
+26 -3
View File
@@ -16,7 +16,7 @@ Use one of these channels, in order of preference:
maintainer. maintainer.
- Direct link: <https://github.com/ChuckBuilds/LEDMatrix/security/advisories/new> - Direct link: <https://github.com/ChuckBuilds/LEDMatrix/security/advisories/new>
2. **Discord DM**. Send a direct message to a moderator on the 2. **Discord DM**. Send a direct message to a moderator on the
[LEDMatrix Discord](https://discord.gg/uW36dVAtcT). Don't post in [LEDMatrix Discord](https://discord.gg/RdrC37rEag). Don't post in
public channels. public channels.
Please include: Please include:
@@ -61,8 +61,31 @@ Out of scope (please report upstream):
LEDMatrix is designed for trusted local networks. Several limitations LEDMatrix is designed for trusted local networks. Several limitations
are intentional rather than vulnerabilities: are intentional rather than vulnerabilities:
- **No web UI authentication.** The web interface assumes the network - **Web UI authentication is optional and off by default.** Out of the
it's running on is trusted. Don't expose port 5000 to the internet. box the web interface assumes the network it's running on is trusted.
Setting a password under **General > Security** makes every page and
API route require a login or an API token (`Authorization: Bearer`),
with wrong passwords rate-limited per address
(`web_interface/auth.py`). Deliberately left open even then: requests
from the Pi itself (loopback without proxy headers; a reverse proxy on
the Pi must add `X-Forwarded-For`, or every request it relays counts as
local), the Wi-Fi setup flow while the Pi is in access-point mode,
static files, and a status-only `/api/v3/health`. The password is a
werkzeug hash and tokens are stored as SHA-256, in
`config/config_secrets.json`, which no API returns. There is no TLS:
over plain HTTP the password and tokens cross the LAN in the clear, so
still don't expose port 5000 to the internet; put a TLS reverse proxy
or a VPN in front for remote access. Anyone with shell access to the Pi
can turn login off (`scripts/reset_web_password.py`), which is the
documented recovery path.
"Trusted network" does not mean "trusted websites", though: any page
a LAN user opens could make their browser POST to the Pi. So the
interface refuses a `POST`/`PUT`/`PATCH`/`DELETE` whose `Origin` (or
`Referer`) header names another site (`web_interface/origin_guard.py`),
and `/api/v3/system/action` only accepts JSON or HTMX requests. Tools
that send neither header (curl, Home Assistant, the MQTT bridge) are
unaffected. Not covered: DNS rebinding, and anyone who can reach the
port directly.
- **Plugins run unsandboxed.** Installed plugins execute in the same - **Plugins run unsandboxed.** Installed plugins execute in the same
Python process as the display loop with full file-system and Python process as the display loop with full file-system and
network access. Review plugin code (especially third-party plugins network access. Review plugin code (especially third-party plugins
+3 -2
View File
@@ -1,7 +1,8 @@
{ {
"web_display_autostart": true, "web_display_autostart": true,
"auto_update": { "auto_update": {
"enabled": false "enabled": false,
"channel": "stable"
}, },
"schedule": { "schedule": {
"enabled": false, "enabled": false,
@@ -133,7 +134,7 @@
"plugin_rotation_order": [], "plugin_rotation_order": [],
"use_short_date_format": true, "use_short_date_format": true,
"vegas_scroll": { "vegas_scroll": {
"live_in_ticker": false, "live_in_ticker": true,
"live_weight": 3, "live_weight": 3,
"favorite_live_weight": 5, "favorite_live_weight": 5,
"enabled": false, "enabled": false,
+15 -7
View File
@@ -1,12 +1,20 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""Legacy entry point: runs ``run.py``, which is the one to use.
``python3 run.py`` (``-e`` for the emulator, ``-d`` for debug logging) is how
the display service and the docs start LEDMatrix. This file used to import
``src.display_controller.main`` directly, which skipped what run.py sets up
first -- ``sys.dont_write_bytecode`` (root-owned ``__pycache__`` in plugin
directories blocks the web service from updating them), the ``-e``/``-d``
flags, and the logging configuration. It now runs run.py exactly as
``python3 run.py`` would, with the same arguments.
"""
import os import os
import sys import runpy
# Add the project root directory to Python path
sys.path.append(os.path.dirname(os.path.abspath(__file__)))
from src.display_controller import main
if __name__ == "__main__": if __name__ == "__main__":
main() runpy.run_path(
os.path.join(os.path.dirname(os.path.abspath(__file__)), "run.py"),
run_name="__main__",
)
+91 -75
View File
@@ -10,22 +10,27 @@ This guide covers advanced LEDMatrix features for users and developers, includin
Vegas scroll mode displays content from multiple plugins in a continuous horizontal scroll, similar to news tickers seen in Las Vegas casinos. Plugins contribute content segments that flow across the display in a seamless ticker-style presentation. Vegas scroll mode displays content from multiple plugins in a continuous horizontal scroll, similar to news tickers seen in Las Vegas casinos. Plugins contribute content segments that flow across the display in a seamless ticker-style presentation.
### Display Modes ### How a Plugin Takes Part
**SCROLL (Continuous Scrolling):** Each plugin has a *Vegas participation*:
- Content scrolls continuously left
- Smooth, fluid motion
- Best for news-ticker style displays
**FIXED_SEGMENT (Fixed-Width Block):** **`scroll` (the default):**
- Plugin gets fixed-width block on display - The plugin's content scrolls by with everyone else's
- Content doesn't scroll out of its segment - Best for news-ticker style content: scores, headlines, prices, the time
- Multiple plugins can share the display simultaneously
**STATIC (Scroll Pauses):** **`pause`:**
- Scrolling pauses when content is fully visible - The scroll stops when the plugin's turn comes round
- Displays for specified duration, then resumes scrolling - The plugin draws the whole panel for its display duration, then the
- Best for content that needs to be fully read scroll resumes
- Best for content that needs to be read in full, or alerts
**`exclude`:**
- The plugin is left out of Vegas mode
A plugin declares its default; set `vegas_participation` in a plugin's
config to override it (see [Per-Plugin Configuration](#per-plugin-configuration)).
Older documentation also describes a *fixed segment* mode; Vegas never
implemented one, and it has always behaved exactly like `scroll`.
### Configuration ### Configuration
@@ -70,17 +75,31 @@ total. See the full list in
### Live Content in the Ticker ### Live Content in the Ticker
By default, live content **preempts** Vegas mode: while any plugin reports By default (since 3.8.0) live content **stays in the ticker** and takes
live priority, the display controller refuses to run the ticker and shows **extra turns inside it**, and a scoreboard that supports live cards updates
that plugin's full-screen display instead. You get a big readable scoreboard, the score on a card already crossing the screen (`live_refresh`, "Update live
but the marquee stops entirely for the duration of the game. content while it scrolls").
Set `live_in_ticker` to keep the ticker running and let live content take To get the old behaviour back -- live content **preempts** Vegas mode: while
**extra turns inside it** instead: any plugin reports live priority the ticker stops and that plugin's
full-screen display is shown instead -- untick **Keep live games in the
ticker** under Vegas mode, or set `live_in_ticker` to `false`:
```json
"vegas_scroll": {
"live_in_ticker": false
}
```
Until 3.8.0 `false` was the default and every config held it, copied from
the template. The first start on 3.8.0 turns it on once (a backup of the
config is kept as `config.json.backup`, and `live_in_ticker_migrated` records
that it ran); a `false` set after that is left alone.
The weights below apply while live content is in the ticker:
```json ```json
"vegas_scroll": { "vegas_scroll": {
"live_in_ticker": true,
"live_weight": 3, "live_weight": 3,
"favorite_live_weight": 5 "favorite_live_weight": 5
} }
@@ -164,8 +183,7 @@ Override Vegas behavior for specific plugins:
{ {
"my_plugin": { "my_plugin": {
"enabled": true, "enabled": true,
"vegas_mode": "scroll", "vegas_participation": "pause",
"vegas_panel_count": 2,
"display_duration": 10 "display_duration": 10
} }
} }
@@ -175,19 +193,30 @@ Override Vegas behavior for specific plugins:
| Setting | Values | Description | | Setting | Values | Description |
|---------|--------|-------------| |---------|--------|-------------|
| `vegas_mode` | `scroll`, `fixed`, `static` | Display mode for this plugin | | `vegas_participation` | `scroll`, `pause`, `exclude` | How this plugin takes part: its content scrolls by, the scroll pauses for its turn and shows it full screen, or it is left out. Unset uses the plugin's own default |
| `vegas_panel_count` | any positive integer | Width in panels (1 panel = display width) | | `display_duration` | seconds | How long a `pause` plugin holds the screen |
| `display_duration` | seconds | Pause duration for STATIC mode | | `vegas_width_pct` | 10–100 | Width of this plugin's card, as a percentage of the panel |
| `vegas_overflow` | `rotate`, `truncate` | What to do when its content is wider than its allowance |
| `vegas_max_width_screens` | number of screens | The widest its card may be |
Plugins may also set `vegas_overflow` and `vegas_max_width_screens` in These are core-owned settings (see
their config section to control how oversized content is handled (see [PLUGIN_CONFIG_CORE_PROPERTIES.md](PLUGIN_CONFIG_CORE_PROPERTIES.md)): every
`PluginManager` in `src/plugin_system/plugin_manager.py`). plugin accepts them whether or not its own schema lists them. Set them in
the plugin's section of config.json, in the web UI's **Config Editor**
tab.
Some plugins also offer a `vegas_mode` setting of their own (`scroll`,
`fixed` or `static`). It still works — `static` pauses, the other two scroll
— but `vegas_participation` takes precedence, and `fixed` has never done
anything different from `scroll`. The old `vegas_panel_count` setting never
had an effect and is deprecated (removed in 3.9.0).
### Plugin Integration (Developer Guide) ### Plugin Integration (Developer Guide)
All of these have defaults in All of these have defaults in
[`BasePlugin`](../src/plugin_system/base_plugin.py); override only what you [`BasePlugin`](../src/plugin_system/base_plugin.py); override only what you
need. need. The reference is
[PLUGIN_API_REFERENCE.md](PLUGIN_API_REFERENCE.md#vegas-scroll-hooks).
**1. Implement Content Method:** **1. Implement Content Method:**
@@ -203,43 +232,41 @@ If it returns `None` (the default), Vegas falls back to the plugin's
(`PluginAdapter.get_content()` in (`PluginAdapter.get_content()` in
[`src/vegas_mode/plugin_adapter.py`](../src/vegas_mode/plugin_adapter.py)). [`src/vegas_mode/plugin_adapter.py`](../src/vegas_mode/plugin_adapter.py)).
**2. Specify Content Type:** **2. Declare how the plugin takes part:**
```python Most plugins need nothing: the default is `scroll`. A plugin that should
def get_vegas_content_type(self): pause the scroll, or stay out of Vegas, says so in `manifest.json`:
# 'multi' | 'static' | 'none' -- default is 'static'
return 'multi' ```json
{
"vegas_participation": "pause"
}
``` ```
`'none'` excludes the plugin from Vegas mode. The user's own `vegas_participation` setting overrides the manifest. When
the answer depends on state, override the method instead:
**3. Optionally Specify Display Mode:**
These return `VegasDisplayMode` members, not strings:
```python ```python
from src.plugin_system.base_plugin import VegasDisplayMode def get_vegas_participation(self):
# 'scroll' | 'pause' | 'exclude'
def get_vegas_display_mode(self): return 'pause' if self._alert_is_live() else 'scroll'
return VegasDisplayMode.SCROLL
def get_supported_vegas_modes(self):
return [VegasDisplayMode.SCROLL, VegasDisplayMode.STATIC]
``` ```
`VegasDisplayMode` has `SCROLL` (`"scroll"`), `FIXED_SEGMENT` (`"fixed"`) and A plugin written for an older core that declares nothing keeps its
`STATIC` (`"static"`). The default `get_vegas_display_mode()` uses the behaviour: `get_vegas_display_mode()` returning `VegasDisplayMode.STATIC`
plugin's `vegas_mode` config value if set, otherwise maps the content type pauses, `get_vegas_content_type()` returning `'none'` excludes, and
(`multi` to `SCROLL`, anything else to `FIXED_SEGMENT`). everything else scrolls. `get_supported_vegas_modes()`,
`get_vegas_segment_width()` and the SCROLL / FIXED_SEGMENT distinction are
deprecated (removed in 3.9.0): Vegas never read them.
### Content Rendering Guidelines ### Content Rendering Guidelines
**Image Dimensions:** **Image Dimensions:**
- **Height:** Must match display height (typically 32 pixels) - **Height:** Must match display height (typically 32 pixels)
- **Width:** Varies by mode: - **Width:** Any width for `scroll` (recommended 64-512 pixels);
- SCROLL: Any width (recommended 64-512 pixels) `get_vegas_render_width()` is the width Vegas would like, and it narrows
- FIXED_SEGMENT: `panel_count * display_width` `display_manager` to match while it asks. A `pause` plugin draws the
- STATIC: Any width, optimized for readability whole panel in `display()`.
**Color Mode:** **Color Mode:**
- Use RGB color mode - Use RGB color mode
@@ -289,17 +316,10 @@ class WeatherPlugin(BasePlugin):
def get_vegas_content(self): def get_vegas_content(self):
"""Return cached Vegas image""" """Return cached Vegas image"""
return self.vegas_image return self.vegas_image
def get_vegas_content_type(self):
return 'multi'
def get_vegas_display_mode(self):
return 'scroll'
def get_supported_vegas_modes(self):
return ['scroll', 'static']
``` ```
It scrolls, the default participation, so it declares nothing else.
### System Architecture ### System Architecture
Vegas mode consists of four core components working together to provide smooth 125 FPS continuous scrolling: Vegas mode consists of four core components working together to provide smooth 125 FPS continuous scrolling:
@@ -382,7 +402,8 @@ Vegas mode consists of four core components working together to provide smooth 1
**Responsibilities:** **Responsibilities:**
- Convert plugin content to scrollable images - Convert plugin content to scrollable images
- Handle different Vegas display modes (SCROLL, FIXED, STATIC) - Fetch the content of `scroll` plugins (a `pause` plugin is drawn by
its own `display()` when the scroll pauses; see StreamManager)
- Manage fallback for plugins without Vegas support - Manage fallback for plugins without Vegas support
- Cache plugin content for performance - Cache plugin content for performance
@@ -391,21 +412,16 @@ Vegas mode consists of four core components working together to provide smooth 1
- Calls `get_vegas_content()` if available - Calls `get_vegas_content()` if available
- Falls back to `display()` method if not - Falls back to `display()` method if not
2. **Handle display mode:** 2. **Participation** is decided by the StreamManager, not here
- SCROLL: Returns image as-is for continuous scrolling (`resolve_vegas_participation()` in
- FIXED_SEGMENT: Creates fixed-width block (panel_count * display_width) [`base_plugin.py`](../src/plugin_system/base_plugin.py)): `exclude`
- STATIC: Marks content for pause-when-visible behavior plugins never reach the adapter, and `pause` plugins are not fetched.
3. **Content type handling:**
- `multi`: Multiple segments (list of images)
- `static`: Single static image
- `none`: Skip this plugin in current cycle
**Fallback Behavior:** **Fallback Behavior:**
- If plugin doesn't implement Vegas methods: - If plugin doesn't implement Vegas methods:
- Calls plugin's `display()` method - Calls plugin's `display()` method
- Captures rendered display as static image - Captures rendered display as static image
- Treats as fixed segment - Scrolls it by as one block
- Ensures all plugins work in Vegas mode without explicit support - Ensures all plugins work in Vegas mode without explicit support
#### 4. RenderPipeline #### 4. RenderPipeline
@@ -508,7 +524,7 @@ All components use thread-safe patterns:
If a plugin doesn't implement Vegas methods: If a plugin doesn't implement Vegas methods:
- System calls the plugin's `display()` method - System calls the plugin's `display()` method
- Captures the rendered display as a static image - Captures the rendered display as a static image
- Treats it as a fixed segment - Scrolls it by as one block
This ensures all plugins work in Vegas mode, even without explicit support. This ensures all plugins work in Vegas mode, even without explicit support.
+4 -4
View File
@@ -27,7 +27,7 @@ Advanced patterns, examples, and best practices for developing LEDMatrix plugins
The Display Manager's icon methods — `draw_weather_icon()`, `draw_sun()`, The Display Manager's icon methods — `draw_weather_icon()`, `draw_sun()`,
`draw_cloud()`, `draw_rain()`, `draw_snow()` and `draw_text_with_icons()` — `draw_cloud()`, `draw_rain()`, `draw_snow()` and `draw_text_with_icons()` —
are deprecated, removed in 3.7.0. Draw your own icons instead: render them were removed in 3.8.0. Draw your own icons instead: render them
onto a PIL image and paste it onto `self.display_manager.image`, or ship onto a PIL image and paste it onto `self.display_manager.image`, or ship
icon images with the plugin. The weather plugin's `WeatherIcons` class is an icon images with the plugin. The weather plugin's `WeatherIcons` class is an
example. See [Deprecated APIs](PLUGIN_API_REFERENCE.md#deprecated-apis). example. See [Deprecated APIs](PLUGIN_API_REFERENCE.md#deprecated-apis).
@@ -194,7 +194,7 @@ def update(self):
sport_key = "nhl" sport_key = "nhl"
cache_key = f"{self.plugin_id}_{sport_key}_games" cache_key = f"{self.plugin_id}_{sport_key}_games"
# get_background_cached_data() is deprecated, removed in 3.7.0 — use get() # get_background_cached_data() was removed in 3.8.0 — use get()
cached = self.cache_manager.get(cache_key, max_age=60) cached = self.cache_manager.get(cache_key, max_age=60)
if cached: if cached:
@@ -596,8 +596,8 @@ def update(self):
```python ```python
def update(self): def update(self):
# get_enabled_plugins() is deprecated, removed in 3.7.0 — check the # get_enabled_plugins() was removed in 3.8.0 — check the instance's
# instance's `enabled` flag instead # `enabled` flag instead
weather_plugin = self.plugin_manager.get_plugin("weather") weather_plugin = self.plugin_manager.get_plugin("weather")
if weather_plugin is not None and weather_plugin.enabled: if weather_plugin is not None and weather_plugin.enabled:
# Use weather data # Use weather data
+198 -15
View File
@@ -41,19 +41,136 @@ each other. They share three things:
| State | Where | Written by | Read by | | State | Where | Written by | Read by |
|---|---|---|---| |---|---|---|---|
| On-demand request | cache `display_on_demand_request` | web: `start_on_demand_display()` / `stop_on_demand_display()` in [`api_v3/display.py`](../web_interface/blueprints/api_v3/display.py) | display: `_poll_on_demand_requests()` | | On-demand command | control socket `/run/ledmatrix/control.sock` ([IPC_CONTROL_SOCKET.md](IPC_CONTROL_SOCKET.md)) | web: `start_on_demand_display()` / `stop_on_demand_display()` in [`api_v3/display.py`](../web_interface/blueprints/api_v3/display.py), via [`src/ipc/client.py`](../src/ipc/client.py) | display: [`src/ipc/server.py`](../src/ipc/server.py) acks; the render thread applies it in `_poll_on_demand_requests()` |
| On-demand request (fallback) | cache `display_on_demand_request` | web, when the socket fails; four plugins write it directly | display: `_poll_on_demand_requests()` |
| On-demand state | cache `display_on_demand_state` | display: `_publish_on_demand_state()` | web: `/api/v3/display/on-demand/status` | | On-demand state | cache `display_on_demand_state` | display: `_publish_on_demand_state()` | web: `/api/v3/display/on-demand/status` |
| Current screen | cache `display_current_state` | display | web: `/api/v3/display/current-status` | | Current screen | cache `display_current_state` | display | web: `/api/v3/display/current-status` |
| Plugin errors | cache `plugin_error_snapshot` | display: `ErrorSnapshotPublisher` ([`src/error_aggregator.py`](../src/error_aggregator.py)) | web: `read_error_report()` for `/api/v3/errors/*` | | Plugin errors | cache `plugin_error_snapshot` | display: `ErrorSnapshotPublisher` ([`src/error_aggregator.py`](../src/error_aggregator.py)) | web: `read_error_report()` for `/api/v3/errors/*` |
| Error clear | cache `plugin_error_clear_request` | web | display | | Error clear | cache `plugin_error_clear_request` | web | display |
| Font usage | cache `font_usage_snapshot` | display: `FontUsagePublisher` ([`src/font_usage.py`](../src/font_usage.py)) | web: Fonts tab | | Font usage | cache `font_usage_snapshot` | display: `FontUsagePublisher` ([`src/font_usage.py`](../src/font_usage.py)) | web: Fonts tab |
| Plugin health | cache `plugin_health:<id>` | display (web writes on reset) | web: `/api/v3/plugins/health` | | Plugin health | cache `plugin_health:<id>` | display (web writes on reset) | web: `/api/v3/plugins/health` |
| Plugin runtime (loaded, state, last error, version) | cache `plugin_runtime_snapshot` | display: `PluginRuntimePublisher` ([`src/plugin_system/plugin_runtime.py`](../src/plugin_system/plugin_runtime.py)) | web: `read_plugin_runtime()` for `/api/v3/plugins/installed`, `/plugins/state`, reconciliation |
| Preview frame | `/tmp/led_matrix_preview.png` | display: `DisplayManager`, gated by [`snapshot_policy`](../src/common/snapshot_policy.py) | web: display SSE stream, `/api/v3/health` (file age) | | Preview frame | `/tmp/led_matrix_preview.png` | display: `DisplayManager`, gated by [`snapshot_policy`](../src/common/snapshot_policy.py) | web: display SSE stream, `/api/v3/health` (file age) |
| Preview viewer marker | `/tmp/led_matrix_preview_viewer` | web, while a preview is open | display: writes full-rate snapshots only while it is fresh | | Preview viewer marker | `/tmp/led_matrix_preview_viewer` | web, while a preview is open | display: writes full-rate snapshots only while it is fresh |
| Hardware init status | `/tmp/led_matrix_hw_status.json` | display | web: `/api/v3/hardware/status` | | Hardware init status | `/tmp/led_matrix_hw_status.json` | display | web: `/api/v3/hardware/status` |
| Render-loop heartbeat | `/run/ledmatrix/display-heartbeat.json` (tmpfs) | display: the render thread, via [`display_watchdog`](../src/display_watchdog.py) | web: `/api/v3/health` (`checks.display_loop`); the update health check |
The on-demand start route also restarts `ledmatrix.service` by default so the The on-demand start route starts `ledmatrix.service` when it is not running
request takes effect straight away. (`start_service`, on by default) but never restarts a running one. The routes
send the command over the display's control socket and get an ack; when that
fails (a stopped display, one older than the socket) they write the mailbox
instead, which the display reads every `ON_DEMAND_POLL_INTERVAL` (0.25s), from
its dwell sleep, its render loops and Vegas's interrupt check as well as the
main loop. Both ways end in the same handler, `_handle_on_demand_request()`.
The socket's handlers only queue; see [IPC_CONTROL_SOCKET.md](IPC_CONTROL_SOCKET.md)
for the protocol, the permission model and the plan to retire the mailboxes.
### Web and display processes: who runs plugins
Only the display process imports plugin code, instantiates plugins and calls
their lifecycle hooks (`update`, `display`, `on_config_change`, `on_enable`,
`on_disable`). The web process is metadata-only: it reads plugins as files
through `PluginCatalog`
([`src/plugin_system/plugin_catalog.py`](../src/plugin_system/plugin_catalog.py))
-- manifests, config schemas (through `SchemaManager`), each plugin's
section of `config.json`, and installed versions. The catalog keeps the
read-only method names of `PluginManager` and has nothing that can run a
plugin (no `load_plugin`, `get_plugin` or `plugins`).
How a web-side change reaches the running plugins:
| Change | How the display picks it up |
|---|---|
| Plugin settings saved, config reset | `ConfigService` sees the new `config.json` and calls the plugin's `on_config_change` with the prepared section |
| Plugin enabled or disabled | `ConfigService` → `_controller_config_change` flags a reconcile; `_reconcile_enabled_plugins` loads it (fresh from disk) or unloads it on the render thread |
| Plugin uninstalled (config removed) | the removed section flips its `enabled` flag, and the reconcile unloads it |
| Plugin installed, not enabled | nothing to do until it is enabled, which loads it |
| Plugin installed while already enabled, updated while enabled, or uninstalled with its config kept | **not picked up**: the display keeps running what it loaded. The route answers `restart_required: true` and the UI shows its restart banner |
`display_restart_required()` in `plugin_catalog.py` holds that last rule;
routes return it as `restart_required` (with the banner's wording in
`restart_message`), and `window.noteRestartRequired()` in
`static/v3/app.js` raises the banner for any response that carries it,
`POST /api/v3/config/main` included.
Runtime state shown in the UI comes from what the display publishes to the
shared cache: health and metrics (`/api/v3/plugins/health`,
`/plugins/metrics`), errors (`/api/v3/errors/*`), the current mode, and the
plugin runtime snapshot described below. `enabled` is read from
`config.json` by the display's rule (a missing flag is disabled).
Plugin code still runs in the web process in one place,
`_import_plugin_code_in_web_process()` in
[`api_v3/__init__.py`](../web_interface/blueprints/api_v3/__init__.py): the
Starlark routes import the starlark-apps plugin's `tronbyte_repository` and
`pixlet_renderer` helper modules (never the plugin class), and a web-UI
action with `oauth_flow` imports its script for `get_auth_url()`. Every
other web-UI action runs its script as a subprocess. A later, explicit
**plugin web-entry contract** -- a declared entry point for plugin web code
-- replaces that function.
Next stages: a **control socket** from the web process to the display
(reload one plugin, ask for its state) in place of `restart_required` and
the cache-key mailboxes, and the plugin web-entry contract above.
### Plugin state: desired, observed, and who owns it
There is one plugin state machine, and the display owns it:
`PluginStateManager` in
[`plugin_state.py`](../src/plugin_system/plugin_state.py) (unloaded →
loaded → enabled ⇄ running, error, disabled), held by the display's
`PluginManager`. It also records, per loaded plugin, the manifest version it
loaded and when. Nothing else keeps plugin state:
| Question | Answered by |
|---|---|
| Is it installed, at which version? | the plugins directory (`manifest.json`) |
| Should it run? | `config.json` (`<id>.enabled`, missing = disabled) |
| Has the user uninstalled it for good? | the store's uninstalled-plugins record |
| Is the display running it, at which version, and why not? | the display's runtime snapshot |
**The runtime snapshot.** `PluginRuntimePublisher`
([`plugin_runtime.py`](../src/plugin_system/plugin_runtime.py)), started by
`DisplayController` right after it creates the `PluginManager`, writes the
cache key `plugin_runtime_snapshot`: per plugin `loaded`, `state`, `error`
(type, a redacted message of at most 200 characters, when, recoverable),
`version` and `loaded_at`, plus `published_at`, `stale_after` and `running`.
The cache is on disk, usually the SD card, so it writes when something a
reader sees changes -- throttled to once per 10 s -- and otherwise once a
minute as a heartbeat. RUNNING, which every `update()` passes through, is
published as ENABLED, so plugin updates alone never cause a write.
`cleanup()` publishes `running: false`.
**Reading it.** `read_plugin_runtime()` judges the snapshot before anyone
uses it: `live` (fresh, from a running display), `stale` (older than
`stale_after`, 3 minutes: a hung or crashed display), `stopped` or
`unknown` (none, unreadable, or another schema). Only a live view reports
per-plugin facts; every other status answers `null` for them, so stale
truth cannot leak into a response. `/api/v3/plugins/installed` returns
`loaded`, `state`, `error_info`, `loaded_version` and `loaded_at` per
plugin and `data.runtime` (`status`, `published_at`, `age_seconds`);
`/api/v3/plugins/state` returns the same beside the desired state.
**Reconciliation**
([`state_reconciliation.py`](../src/plugin_system/state_reconciliation.py))
compares desired state (config + disk) with observed state (the snapshot).
It fixes desired-state gaps -- a plugin on disk with no config section gets
`{"enabled": false}`, a configured plugin missing from disk is reinstalled
unless the user uninstalled it -- and only reports observed-state gaps
(enabled but not loaded, loaded at an older version): the display loads and
unloads by config on its own, and a version gap needs a restart.
**`data/plugin_state.json` is retired.** The web process used to keep a
second `PluginStateManager` (`state_manager.py`) persisted to that file:
per plugin an enabled flag copied from config, a version copied from the
manifest (when set at all), a status derived from those, and install/update
timestamps. Reconciliation mostly synced it back to config and backups
merged it into their plugin list. Every field is derivable (the timestamps
from the operation history), so nothing is migrated: no code reads or
writes the file, and a copy left on a device is inert and safe to delete.
The two classes shared a name but not a concern -- a persisted install
record versus the live lifecycle -- so they were not merged; the persisted
one had nothing left to hold and was removed.
## Display loop ## Display loop
@@ -82,7 +199,11 @@ then normal rotation.
- **On-demand.** A request from the web interface pins one plugin (or mode) - **On-demand.** A request from the web interface pins one plugin (or mode)
for a duration. `_activate_on_demand()` / `_clear_on_demand()`; the for a duration. `_activate_on_demand()` / `_clear_on_demand()`; the
session is saved under `display_on_demand_config` so it survives a session is saved under `display_on_demand_config` so it survives a
restart. It also keeps the display on during scheduled off hours. restart. It also keeps the display on during scheduled off hours. A
request for a plugin that is disabled in config loads it live
(`_load_plugin_for_on_demand()`, `load_plugin(force_enabled=True)`)
without writing `config.json`; the main loop unloads it once on-demand
moves off it (`_release_on_demand_plugins()`).
- **Live priority.** `_check_live_priority()` looks for a plugin whose - **Live priority.** `_check_live_priority()` looks for a plugin whose
`has_live_priority()` and `has_live_content()` are both true and switches `has_live_priority()` and `has_live_content()` are both true and switches
to it, rotating between several live games. to it, rotating between several live games.
@@ -99,7 +220,8 @@ then normal rotation.
changes. The controller refreshes its cached settings; enabling or changes. The controller refreshes its cached settings; enabling or
disabling a plugin queues `_reconcile_enabled_plugins()`, which loads or disabling a plugin queues `_reconcile_enabled_plugins()`, which loads or
unloads it on the display thread; each plugin gets `on_config_change()` unloads it on the display thread; each plugin gets `on_config_change()`
for its own section. Set `LEDMATRIX_HOT_RELOAD=false` to turn this off. for its own section, under its plugin lock
(`PluginManager.apply_config_change()`). Set `LEDMATRIX_HOT_RELOAD=false` to turn this off.
Matrix hardware settings are only read at start-up. Matrix hardware settings are only read at start-up.
- **Vegas mode.** [`src/vegas_mode/`](../src/vegas_mode/): the display loop - **Vegas mode.** [`src/vegas_mode/`](../src/vegas_mode/): the display loop
calls `VegasModeCoordinator.run_iteration()` calls `VegasModeCoordinator.run_iteration()`
@@ -113,6 +235,43 @@ then normal rotation.
`sync.role`: a leader sends a follower its share of each frame over UDP `sync.role`: a leader sends a follower its share of each frame over UDP
(port 5765). (port 5765).
### Liveness
A render thread stuck inside a plugin leaves the service "active" and the
panel frozen, so liveness is reported by the render thread itself
([`src/display_watchdog.py`](../src/display_watchdog.py), standard library
only). `beat()` from any other thread is ignored: the update worker, Vegas's
tick thread and the prefetcher keep running while the render thread is stuck,
and must not vouch for it.
- **Check-in points.** The top of `run()`'s loop (`loop_pass()`), every
dwell second (`_sleep_with_plugin_updates`), every frame of the per-screen
loops (`_display_once`), every frame of Vegas's own loop and static pause
(`coordinator.run_iteration`), each plugin fetched for a Vegas cycle
(`StreamManager._fetch_plugin_content`), each update on the
`synchronous_updates` path, and every frame pushed
(`DisplayManager.update_display` -> `note_frame()`). Beats are
rate-limited to one ping and one heartbeat write every 5 s.
- **systemd watchdog.** `ledmatrix.service` is `Type=simple` with
`WatchdogSec=120` and `NotifyAccess=main`. `run.py` sends
`WATCHDOG_USEC` = 15 minutes before importing anything heavy (start-up loads
plugins and runs the 20 s update budget, and the watchdog clock starts with
the process). After the first frame -- or the first full pass, when there is
nothing to draw -- the loop sends `READY=1`, restores the unit's 120 s and
pings. `PluginManager.load_plugin()` on the render thread (a plugin enabled
from the web UI, or loaded for on-demand) gets 15 minutes again, since it
can run pip. A missed deadline is a SIGABRT; faulthandler, enabled on
arming, dumps every thread's stack to the journal.
- **Heartbeat.** `/run/ledmatrix/display-heartbeat.json`
(`{"pid", "mono", "wall"}`; `RuntimeDirectory=ledmatrix`, 0755, file 0644 so
the web user can read it). Readers compare `mono` with their own
`time.monotonic()` -- CLOCK_MONOTONIC is shared by every process and does not
jump when NTP first sets an RTC-less Pi's clock. `/api/v3/health` calls it
`stalled` past 60 s; no file is `not_reported` and changes nothing. A clean
stop removes it. Without `RuntimeDirectory=` (an older unit) the display,
as root, creates the directory itself; off Linux, or without root, there
is no heartbeat.
## Plugin system ## Plugin system
[`src/plugin_system/`](../src/plugin_system/): [`src/plugin_system/`](../src/plugin_system/):
@@ -121,13 +280,14 @@ then normal rotation.
|---|---| |---|---|
| Base class plugins implement | [`base_plugin.py`](../src/plugin_system/base_plugin.py) (`BasePlugin`, `VegasDisplayMode`) | | Base class plugins implement | [`base_plugin.py`](../src/plugin_system/base_plugin.py) (`BasePlugin`, `VegasDisplayMode`) |
| Finding a plugin's directory | [`plugin_dirs.py`](../src/plugin_system/plugin_dirs.py): manifest `id` first, then directory `<id>` or `ledmatrix-<id>` | | Finding a plugin's directory | [`plugin_dirs.py`](../src/plugin_system/plugin_dirs.py): manifest `id` first, then directory `<id>` or `ledmatrix-<id>` |
| Discovery, load, unload, scheduled updates | [`plugin_manager.py`](../src/plugin_system/plugin_manager.py) (`PluginManager`) | | Discovery, load, unload, scheduled updates (display process) | [`plugin_manager.py`](../src/plugin_system/plugin_manager.py) (`PluginManager`) |
| Manifest, schema, config and version reads (web process) | [`plugin_catalog.py`](../src/plugin_system/plugin_catalog.py) (`PluginCatalog`; see [who runs plugins](#web-and-display-processes-who-runs-plugins)) |
| Import and instantiate | [`plugin_loader.py`](../src/plugin_system/plugin_loader.py) (`PluginLoader.load_plugin()`: dependencies, module, class) | | Import and instantiate | [`plugin_loader.py`](../src/plugin_system/plugin_loader.py) (`PluginLoader.load_plugin()`: dependencies, module, class) |
| Timeouts | [`plugin_executor.py`](../src/plugin_system/plugin_executor.py) (`PluginExecutor`, 30 s default; a timed-out thread is abandoned, not killed) | | Timeouts | [`plugin_executor.py`](../src/plugin_system/plugin_executor.py) (`PluginExecutor`, 30 s default; a timed-out thread is abandoned, not killed) |
| Circuit breaker | [`plugin_health.py`](../src/plugin_system/plugin_health.py) (`PluginHealthTracker`: 3 consecutive failures open the circuit for 300 s) | | Circuit breaker | [`plugin_health.py`](../src/plugin_system/plugin_health.py) (`PluginHealthTracker`: 3 consecutive failures open the circuit for 300 s) |
| Resource metrics | [`resource_monitor.py`](../src/plugin_system/resource_monitor.py) | | Resource metrics | [`resource_monitor.py`](../src/plugin_system/resource_monitor.py) |
| Config schemas and defaults | [`schema_manager.py`](../src/plugin_system/schema_manager.py) | | Config schemas and defaults | [`schema_manager.py`](../src/plugin_system/schema_manager.py) |
| Install, update, uninstall | [`store_manager.py`](../src/plugin_system/store_manager.py) (`PluginStoreManager`) | | Install, update, uninstall | [`store_manager.py`](../src/plugin_system/store_manager.py) (`PluginStoreManager`), with its methods split across [`store_registry.py`](../src/plugin_system/store_registry.py) (registry, GitHub), [`store_install.py`](../src/plugin_system/store_install.py) and [`store_update.py`](../src/plugin_system/store_update.py) |
| Core-version gate | [`compatibility.py`](../src/plugin_system/compatibility.py) | | Core-version gate | [`compatibility.py`](../src/plugin_system/compatibility.py) |
Discovery scans only `plugin_system.plugins_directory` (default Discovery scans only `plugin_system.plugins_directory` (default
@@ -148,8 +308,9 @@ everything else through `_reinstall_with_rollback()`.
## Web interface ## Web interface
- **App.** [`web_interface/app.py`](../web_interface/app.py) builds the - **App.** [`web_interface/app.py`](../web_interface/app.py) builds the
Flask `app` at import time, creates the managers, and registers two Flask `app` at import time, creates the managers -- a `PluginCatalog`,
blueprints. `web_interface/start.py` runs it on port 5000. never a `PluginManager` -- and registers two blueprints.
`web_interface/start.py` runs it on port 5000.
- **Pages.** [`blueprints/pages_v3.py`](../web_interface/blueprints/pages_v3.py) - **Pages.** [`blueprints/pages_v3.py`](../web_interface/blueprints/pages_v3.py)
serves the shell `templates/v3/base.html` at `/` and each tab as a serves the shell `templates/v3/base.html` at `/` and each tab as a
partial at `/partials/<name>` (templates in partial at `/partials/<name>` (templates in
@@ -158,8 +319,13 @@ everything else through `_reinstall_with_rollback()`.
- **API.** [`blueprints/api_v3/`](../web_interface/blueprints/api_v3/) is one - **API.** [`blueprints/api_v3/`](../web_interface/blueprints/api_v3/) is one
blueprint at `/api/v3`, split by area: `backup.py`, `config.py`, blueprint at `/api/v3`, split by area: `backup.py`, `config.py`,
`display.py`, `fonts.py`, `misc.py` (health, logs, errors, cache, sync), `display.py`, `fonts.py`, `misc.py` (health, logs, errors, cache, sync),
`plugins.py`, `starlark.py`, `system.py` (service actions, updates, git), `starlark.py`, `system.py` (service actions, updates, git), `wifi.py`, and
`wifi.py`. `__init__.py` defines the blueprint and shared helpers and the plugin routes: `plugins.py` (installed list, enable/disable, plugin
actions), `plugin_store.py` (install, update, uninstall, store),
`plugin_config.py` (config, schema, reset), `plugin_assets.py` (uploads,
plugin static files), `plugin_health.py` (health, metrics, limits),
`plugin_operations.py` (operation history, state reconciliation) and
`plugin_calendar.py`. `__init__.py` defines the blueprint and shared helpers and
imports the modules so their routes register. Endpoints are listed in imports the modules so their routes register. Endpoints are listed in
[REST_API_REFERENCE.md](REST_API_REFERENCE.md). [REST_API_REFERENCE.md](REST_API_REFERENCE.md).
- **Front end.** HTMX loads each tab's partial on first open - **Front end.** HTMX loads each tab's partial on first open
@@ -176,8 +342,19 @@ everything else through `_reinstall_with_rollback()`.
- **Update Code** on the Overview tab and the automatic updater both call - **Update Code** on the Overview tab and the automatic updater both call
`perform_core_update()` in `perform_core_update()` in
[`api_v3/system.py`](../web_interface/blueprints/api_v3/system.py): [`api_v3/system.py`](../web_interface/blueprints/api_v3/system.py):
`git pull --rebase`, reinstall changed requirement files, report whether a fetch branches and tags, move the checkout for the update channel, reinstall
restart is needed. changed requirement files, report whether a restart is needed.
- **Update channels** (`auto_update.channel`):
[`web_interface/update_channel.py`](../web_interface/update_channel.py)
decides the move. `stable` checks out the newest `vX.Y.Z` tag (detached
HEAD) when it contains the current commit; `beta` is
`git pull --rebase --autostash` on the current branch, and leaves a
detached release for `main` first. A stable device newer than the newest
release keeps pulling `main` until a release contains its commit, so no
update ever moves backwards; a config without the key is written as
`stable` once the device reaches a release. Checkouts carry uncommitted
edits across with `git stash create`/`apply`, and keep them in the stash
list if they no longer apply.
- **Automatic updates** (`auto_update.enabled`, off by default): - **Automatic updates** (`auto_update.enabled`, off by default):
`AutoUpdater` in [`web_interface/auto_update.py`](../web_interface/auto_update.py) `AutoUpdater` in [`web_interface/auto_update.py`](../web_interface/auto_update.py)
runs in the web process, checks every 30 minutes, and updates at most runs in the web process, checks every 30 minutes, and updates at most
@@ -188,7 +365,11 @@ everything else through `_reinstall_with_rollback()`.
`ledmatrix-update-verify.path`, which runs the verifier as a separate unit `ledmatrix-update-verify.path`, which runs the verifier as a separate unit
(so restarting the web service does not kill it). The verifier restarts (so restarting the web service does not kill it). The verifier restarts
both services, waits for the web API to answer and the display service to both services, waits for the web API to answer and the display service to
stay up, and on failure resets to the previous commit and restarts again. stay up -- and, when the display wrote a heartbeat before the update, to
keep one fresh from the restarted process (see Liveness) -- and on failure
returns to where HEAD was (the branch, or detached on the previous
release; `old_ref` in the pending file), resets to the previous commit
and restarts again.
Plugin updates run only after a verified core update. State is in Plugin updates run only after a verified core update. State is in
`data/auto_update_state.json` and `data/auto_update_pending.json`. `data/auto_update_state.json` and `data/auto_update_pending.json`.
- **Startup validator.** `StartupValidator` - **Startup validator.** `StartupValidator`
@@ -196,7 +377,9 @@ everything else through `_reinstall_with_rollback()`.
`DisplayController.__init__`: config and cache directory first, then `DisplayController.__init__`: config and cache directory first, then
enabled plugins once the plugin manager exists. It also warns when an enabled plugins once the plugin manager exists. It also warns when an
installed systemd unit differs from its template in `systemd/`. Results installed systemd unit differs from its template in `systemd/`. Results
are logged; startup continues either way. are logged; startup continues either way. Nothing rewrites installed units
on update: a unit change such as the watchdog reaches an existing install
only when `install_service.sh` is re-run.
## Where to start reading ## Where to start reading
+10 -2
View File
@@ -17,6 +17,7 @@ tooling against it.
|---|---|---|---| |---|---|---|---|
| `web_display_autostart` | bool, `true` | Whether the web interface service starts with the system | `scripts/utils/start_web_conditionally.py` | | `web_display_autostart` | bool, `true` | Whether the web interface service starts with the system | `scripts/utils/start_web_conditionally.py` |
| `auto_update.enabled` | bool, `false` | Weekly automatic updates: LEDMatrix code first (health-checked, rolled back on failure), then installed plugins. Toggle in the General tab or install with `first_time_install.sh --enable-auto-update` | `web_interface/auto_update.py`, `src/auto_update_setup.py` (`is_enabled()`) | | `auto_update.enabled` | bool, `false` | Weekly automatic updates: LEDMatrix code first (health-checked, rolled back on failure), then installed plugins. Toggle in the General tab or install with `first_time_install.sh --enable-auto-update` | `web_interface/auto_update.py`, `src/auto_update_setup.py` (`is_enabled()`) |
| `auto_update.channel` | `"stable"` or `"beta"`, `"stable"` (template) | What Update Code and the weekly update install. `stable`: the newest `vX.Y.Z` release tag (pre-releases ignored), checked out with a detached HEAD. `beta`: `main`. Never moves a device backwards: one newer than the newest release keeps following `main` until a release contains its commit. Missing (configs from before channels) behaves like `stable` and is saved as `stable` once the device is on a release. General tab, Update Channel | `web_interface/update_channel.py` (`resolve()`) |
| `timezone` | string, `"America/New_York"` | IANA timezone for schedules and displays | `ConfigManager.get_timezone()` | | `timezone` | string, `"America/New_York"` | IANA timezone for schedules and displays | `ConfigManager.get_timezone()` |
| `target_fps` | int, `100` | Legacy "Scroll Frame Rate". Core scrolling no longer reads it: scroll frames are presented at `display.hardware.limit_refresh_rate_hz` divided by each scroll's frame hold, and speed comes from each plugin's scroll settings. Still exposed to plugins via `BasePlugin.global_config` | `src/plugin_system/base_plugin.py` | | `target_fps` | int, `100` | Legacy "Scroll Frame Rate". Core scrolling no longer reads it: scroll frames are presented at `display.hardware.limit_refresh_rate_hz` divided by each scroll's frame hold, and speed comes from each plugin's scroll settings. Still exposed to plugins via `BasePlugin.global_config` | `src/plugin_system/base_plugin.py` |
| `location` | object | `city` / `state` / `country`. Supplies the **default** for a plugin's own `location_city` / `location_state` / `location_country` setting, so weather, radar and friends follow this device without being configured twice. A value saved on the plugin itself still overrides it. Starlark (Tidbyt) apps get the same treatment: a `Location` field left blank on the app renders at this city (geocoded once via Open-Meteo, coordinates cached permanently) instead of the app author's default, which is usually San Francisco. If the city can't be looked up (no match, or the geocoder is unreachable; retried after 30 minutes), the app keeps its own default. | `SchemaManager.apply_device_location()`, then plugins via merged config; `src/device_location.py` for Starlark apps | | `location` | object | `city` / `state` / `country`. Supplies the **default** for a plugin's own `location_city` / `location_state` / `location_country` setting, so weather, radar and friends follow this device without being configured twice. A value saved on the plugin itself still overrides it. Starlark (Tidbyt) apps get the same treatment: a `Location` field left blank on the app renders at this city (geocoded once via Open-Meteo, coordinates cached permanently) instead of the app author's default, which is usually San Francisco. If the city can't be looked up (no match, or the geocoder is unreachable; retried after 30 minutes), the app keeps its own default. | `SchemaManager.apply_device_location()`, then plugins via merged config; `src/device_location.py` for Starlark apps |
@@ -128,6 +129,13 @@ Read by `src/vegas_mode/config.py` (`VegasScrollConfig.from_config`). See
| `min_content_separation` | int, `24` | | `min_content_separation` | int, `24` |
| `min_cut_gap` | int, `6` | | `min_cut_gap` | int, `6` |
| `continuous_scroll` | bool, `true` | | `continuous_scroll` | bool, `true` |
| `offscreen_prefetch` | bool, `true` — render every plugin's ticker content on the background thread, each on its own canvas. `false` restores handing canvas-bound plugins to the render thread, one pause at a time. Temporary; see [OFFSCREEN_RENDERING.md](OFFSCREEN_RENDERING.md) |
| `prefetch_gate` | bool, `true` — let that background thread run Python only while the render thread is waiting for the panel, so the render thread never waits for the GIL when a refresh comes round. Only takes effect with the rebuilt rgbmatrix binding (`scripts/build_rgbmatrix_nogil.sh`). See [OFFSCREEN_RENDERING.md](OFFSCREEN_RENDERING.md) |
| `switch_interval_ms` | float, `0` — experimental: shorten Python's GIL switch interval to this many ms while Vegas runs. `0` leaves the default (5 ms) alone |
| `live_refresh` | bool, `true` — live elements: a plugin that supports them (scores, the flight map) has what is already scrolling updated when its data changes, instead of freezing each card as it was drawn. Always off under multi-display sync, in swap mode and with `offscreen_prefetch` off. `false` restores the frozen behaviour exactly. Per plugin: `vegas_live` in the plugin's section |
| `live_max_hz` | float, `5` (0–10) — ceiling on how often an animated live element (a moving aircraft) is redrawn; `0` keeps data updates and turns animation off. Capped at 1 Hz without the rebuilt rgbmatrix binding |
| `live_min_interval` | float, `2` (0.5–60) — shortest time between two data redraws of one plugin; a faster plugin is redrawn at this rate, never skipped |
| `live_lead_screens` | float, `1` (0–5) — how far ahead of the screen, in screen widths, an animated element starts being redrawn |
| `smooth_scroll` | bool, `true` — move a whole number of pixels per panel refresh, locked to vsync. `scroll_speed` is snapped to the nearest speed the panel can show that way (at 95Hz: 95, 47.5, 31.7 px/s…), measured against the panel's real refresh rate once scrolling starts | | `smooth_scroll` | bool, `true` — move a whole number of pixels per panel refresh, locked to vsync. `scroll_speed` is snapped to the nearest speed the panel can show that way (at 95Hz: 95, 47.5, 31.7 px/s…), measured against the panel's real refresh rate once scrolling starts |
| `sub_pixel_blend` | bool, `false` — the older smoothing: advance by elapsed time and blend neighbouring pixel columns. Looks anti-aliased in the web preview but shimmers on the panel and is not locked to the refresh. Overrides `smooth_scroll` when on | | `sub_pixel_blend` | bool, `false` — the older smoothing: advance by elapsed time and blend neighbouring pixel columns. Looks anti-aliased in the web preview but shimmers on the panel and is not locked to the refresh. Overrides `smooth_scroll` when on |
| `extend_threshold_screens` | float, `2.0` | | `extend_threshold_screens` | float, `2.0` |
@@ -144,7 +152,7 @@ Read by `src/vegas_mode/config.py` (`VegasScrollConfig.from_config`). See
| `max_cycle_duration` | int, `240` | | `max_cycle_duration` | int, `240` |
| `frame_based_scrolling` | bool, `true` — does not step or set a frame rate; motion is by elapsed time either way. When `true`, `scroll_speed` passes through a clamp of 0.1–5 px per `scroll_delay` (see next row) | | `frame_based_scrolling` | bool, `true` — does not step or set a frame rate; motion is by elapsed time either way. When `true`, `scroll_speed` passes through a clamp of 0.1–5 px per `scroll_delay` (see next row) |
| `scroll_delay` | float, `0.02` — not a frame period. Only used with `frame_based_scrolling`: the applied speed is `clamp(scroll_speed × scroll_delay, 0.1, 5) / scroll_delay` px/s, so at `0.02` speeds under 5 px/s run at 5, and at `0.001` nothing runs slower than 100 px/s | | `scroll_delay` | float, `0.02` — not a frame period. Only used with `frame_based_scrolling`: the applied speed is `clamp(scroll_speed × scroll_delay, 0.1, 5) / scroll_delay` px/s, so at `0.02` speeds under 5 px/s run at 5, and at `0.001` nothing runs slower than 100 px/s |
| `live_in_ticker` | bool, `false` — keep scrolling during live games instead of handing the display to a full-screen scoreboard | | `live_in_ticker` | bool, `true` — keep scrolling during live games instead of handing the display to a full-screen scoreboard. `false` was the default before 3.8.0; the first start on 3.8.0 turns a stored `false` on once and sets `live_in_ticker_migrated` |
| `live_weight` | int, `3` (1–10) — slots per cycle for a plugin with live content | | `live_weight` | int, `3` (1–10) — slots per cycle for a plugin with live content |
| `favorite_live_weight` | int, `5` (1–10) — slots per cycle when a plugin reports a favorite team is live | | `favorite_live_weight` | int, `5` (1–10) — slots per cycle when a plugin reports a favorite team is live |
@@ -177,5 +185,5 @@ See [PLUGIN_CONFIG_CORE_PROPERTIES.md](PLUGIN_CONFIG_CORE_PROPERTIES.md).
| Key | Meaning | | Key | Meaning |
|---|---| |---|---|
| `github.api_token` | Optional GitHub token the Plugin Store uses to avoid API rate limits (`src/plugin_system/store_manager.py`) | | `github.api_token` | Optional GitHub token the Plugin Store uses to avoid API rate limits (`src/plugin_system/store_registry.py`) |
| `<plugin-id>.*` | Secrets a plugin declares with `"x-secret": true` in its config schema; merged into that plugin's config at load time | | `<plugin-id>.*` | Secrets a plugin declares with `"x-secret": true` in its config schema; merged into that plugin's config at load time |
+175
View File
@@ -0,0 +1,175 @@
# Deprecated plugin APIs: usage scan
Generated by `scripts/plugin_api_usage.py` — do not edit by hand; re-run it (see [How to re-run](#how-to-re-run)).
- Scanned: 2026-10-01, core 3.7.0
- Monorepo: [ChuckBuilds/ledmatrix-plugins](https://github.com/ChuckBuilds/ledmatrix-plugins) (main @ 4de1d134), 46 plugins
- Third-party plugins: 8 with their own repo in `plugins.json` (f1-live, gif-player, pga-tour-leaderboard, plex-marquee, ledmatrix-dresden-departures, tidbyt-baseball-scoreboard, sleeper-fantasy, ledmatrix-nascar)
**37 deprecated methods: 36 unused, 1 still used, 0 need review.**
Counted per plugin: a *call* is `<receiver>.method` on an object named like the owner (`cache_manager`, `display_manager`, `font_manager`, `plugin_manager`), or on `self`/`super()` in a subclass; an *override* is `def method` in a subclass of the owner. *Review* hits are `.method` on a receiver whose type the scan cannot tell. *Internal* hits sit inside another deprecated core method and go with it. *Unrelated* hits are a different class's own method with the same name (a name collision), and never block removal; neither do hits in test files.
| Method | Removal | Core | Plugins (calls / overrides) | Name collisions & tests | Verdict |
|---|---|---|---|---|---|
| `CacheManager.has_data_changed` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 |
| `CacheManager.update_cache` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 |
| `CacheManager.setup_persistent_cache` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 |
| `CacheManager.get_sport_live_interval` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 |
| `CacheManager.get_sport_key_from_cache_key` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 |
| `CacheManager.get_background_cached_data` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 |
| `CacheManager.is_background_data_available` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 |
| `CacheManager.record_cache_hit` | 3.8.0 | core (1 internal) | — | — | unused — safe to remove in 3.8.0 |
| `CacheManager.record_cache_miss` | 3.8.0 | core (1 internal) | — | — | unused — safe to remove in 3.8.0 |
| `CacheManager.record_fetch_time` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 |
| `CacheManager.get_cache_metrics` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 |
| `CacheManager.log_cache_metrics` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 |
| `CacheManager.get_memory_cache_stats` | 3.8.0 | core tests (3 test calls) | — | — | unused — safe to remove in 3.8.0 |
| `DisplayManager.draw_sun` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 |
| `DisplayManager.draw_cloud` | 3.8.0 | core (2 internals) | — | ledmatrix-weather (1 unrelated) | unused — safe to remove in 3.8.0 |
| `DisplayManager.draw_rain` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 |
| `DisplayManager.draw_snow` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 |
| `DisplayManager.draw_weather_icon` | 3.8.0 | core (1 internal) | — | ledmatrix-weather (5 unrelateds) | unused — safe to remove in 3.8.0 |
| `DisplayManager.draw_text_with_icons` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 |
| `DisplayManager.get_scrolling_stats` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 |
| `FontManager.get_manager_fonts` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 |
| `FontManager.get_detected_fonts` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 |
| `FontManager.unregister_plugin_fonts` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 |
| `FontManager.get_plugin_fonts` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 |
| `FontManager.set_override` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 |
| `FontManager.remove_override` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 |
| `FontManager.get_overrides` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 |
| `FontManager.get_available_fonts` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 |
| `FontManager.get_size_tokens` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 |
| `FontManager.get_performance_stats` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 |
| `FontManager.get_font_catalog` | 3.8.0 | core tests (1 test call) | — | — | unused — safe to remove in 3.8.0 |
| `FontManager.add_font` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 |
| `FontManager.remove_font` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 |
| `FontManager.validate_font` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 |
| `BasePlugin.get_supported_vegas_modes` | 3.9.0 | core tests (2 test reviews) | blackjack (1 call, 1 override); calendar (1 override); olympics (1 override) | — | still used by blackjack, calendar, olympics — keep or migrate first |
| `BasePlugin.get_vegas_segment_width` | 3.9.0 | core tests (1 test review) | — | — | unused — safe to remove in 3.9.0 |
| `PluginManager.get_enabled_plugins` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 |
## Unused — safe to remove (36)
`CacheManager.has_data_changed`, `CacheManager.update_cache`, `CacheManager.setup_persistent_cache`, `CacheManager.get_sport_live_interval`, `CacheManager.get_sport_key_from_cache_key`, `CacheManager.get_background_cached_data`, `CacheManager.is_background_data_available`, `CacheManager.record_cache_hit`, `CacheManager.record_cache_miss`, `CacheManager.record_fetch_time`, `CacheManager.get_cache_metrics`, `CacheManager.log_cache_metrics`, `CacheManager.get_memory_cache_stats`, `DisplayManager.draw_sun`, `DisplayManager.draw_cloud`, `DisplayManager.draw_rain`, `DisplayManager.draw_snow`, `DisplayManager.draw_weather_icon`, `DisplayManager.draw_text_with_icons`, `DisplayManager.get_scrolling_stats`, `FontManager.get_manager_fonts`, `FontManager.get_detected_fonts`, `FontManager.unregister_plugin_fonts`, `FontManager.get_plugin_fonts`, `FontManager.set_override`, `FontManager.remove_override`, `FontManager.get_overrides`, `FontManager.get_available_fonts`, `FontManager.get_size_tokens`, `FontManager.get_performance_stats`, `FontManager.get_font_catalog`, `FontManager.add_font`, `FontManager.remove_font`, `FontManager.validate_font`, `BasePlugin.get_vegas_segment_width`, `PluginManager.get_enabled_plugins`
## Still used — keep or migrate first (1)
`BasePlugin.get_supported_vegas_modes`
## Every hit
File paths are relative to the plugin's directory (core: the repo root).
| Method | Where | File:line | Kind | Code |
|---|---|---|---|---|
| `CacheManager.get_sport_live_interval` | core | src/cache/cache_strategy.py:28 | unrelated | `def get_sport_live_interval(self, sport_key: str) -> int:` |
| `CacheManager.get_sport_live_interval` | core | src/cache/cache_strategy.py:60 | unrelated | `live_interval = self.get_sport_live_interval(sport_key)` |
| `CacheManager.get_sport_live_interval` | core | src/cache_manager.py:785 | unrelated | `return self._strategy_component.get_sport_live_interval(sport_key)` |
| `CacheManager.get_sport_key_from_cache_key` | core | src/cache/cache_strategy.py:214 | unrelated | `def get_sport_key_from_cache_key(self, key: str) -> Optional[str]:` |
| `CacheManager.get_sport_key_from_cache_key` | core | src/cache_manager.py:806 | unrelated | `return self._strategy_component.get_sport_key_from_cache_key(key)` |
| `CacheManager.get_sport_key_from_cache_key` | core | src/cache_manager.py:816 | unrelated | `sport_key = self._strategy_component.get_sport_key_from_cache_key(key)` |
| `CacheManager.record_cache_hit` | core | src/cache_manager.py:869 | internal (in `CacheManager.get_background_cached_data`) | `self.record_cache_hit('background')` |
| `CacheManager.record_cache_miss` | core | src/cache_manager.py:876 | internal (in `CacheManager.get_background_cached_data`) | `self.record_cache_miss('background')` |
| `CacheManager.record_fetch_time` | core | src/cache/cache_metrics.py:67 | unrelated | `def record_fetch_time(self, duration: float) -> None:` |
| `CacheManager.record_fetch_time` | core | src/cache_manager.py:922 | unrelated | `self._metrics_component.record_fetch_time(duration)` |
| `CacheManager.get_memory_cache_stats` | core tests | test/test_cache_manager_memory_tier.py:43 | test call | `stats = cm.get_memory_cache_stats()` |
| `CacheManager.get_memory_cache_stats` | core tests | test/test_cache_manager_memory_tier.py:63 | test call | `assert cm.get_memory_cache_stats()["last_cleanup"] >= before` |
| `CacheManager.get_memory_cache_stats` | core tests | test/test_cache_manager_memory_tier.py:68 | test call | `stats = cm.get_memory_cache_stats()` |
| `DisplayManager.draw_sun` | core | src/plugin_system/testing/visual_display_manager.py:417 | unrelated | `def draw_sun(self, x: int, y: int, size: int = 16):` |
| `DisplayManager.draw_cloud` | core | src/display_manager.py:1359 | internal (in `DisplayManager.draw_rain`) | `self.draw_cloud(x, y, size)` |
| `DisplayManager.draw_cloud` | core | src/display_manager.py:1374 | internal (in `DisplayManager.draw_snow`) | `self.draw_cloud(x, y, size)` |
| `DisplayManager.draw_cloud` | core | src/plugin_system/testing/visual_display_manager.py:421 | unrelated | `def draw_cloud(self, x: int, y: int, size: int = 16, color: Tuple[int, int, int] = (200, 200, 200)):` |
| `DisplayManager.draw_cloud` | ledmatrix-weather | weather_icons.py:184 | unrelated | `def draw_cloud(draw: ImageDraw, x: int, y: int, size: int = 16, color: tuple = (200, 200, 200)):` |
| `DisplayManager.draw_rain` | core | src/plugin_system/testing/visual_display_manager.py:425 | unrelated | `def draw_rain(self, x: int, y: int, size: int = 16):` |
| `DisplayManager.draw_snow` | core | src/plugin_system/testing/visual_display_manager.py:429 | unrelated | `def draw_snow(self, x: int, y: int, size: int = 16):` |
| `DisplayManager.draw_weather_icon` | core | src/display_manager.py:1518 | internal (in `DisplayManager.draw_text_with_icons`) | `self.draw_weather_icon(icon_type, icon_x, icon_y)` |
| `DisplayManager.draw_weather_icon` | core | src/plugin_system/testing/visual_display_manager.py:510 | unrelated | `def draw_weather_icon(self, condition: str, x: int, y: int, size: int = 16) -> None:` |
| `DisplayManager.draw_weather_icon` | core | src/plugin_system/testing/visual_display_manager.py:533 | unrelated | `self.draw_weather_icon(icon_type, icon_x, icon_y)` |
| `DisplayManager.draw_weather_icon` | ledmatrix-weather | manager.py:84 | unrelated | `def draw_weather_icon(image, icon_code, x, y, size):` |
| `DisplayManager.draw_weather_icon` | ledmatrix-weather | manager.py:1280 | unrelated | `WeatherIcons.draw_weather_icon(img, icon_code, icon_x, icon_y,` |
| `DisplayManager.draw_weather_icon` | ledmatrix-weather | manager.py:1559 | unrelated | `WeatherIcons.draw_weather_icon(img, forecast['icon'], icon_x, icon_y, icon_size)` |
| `DisplayManager.draw_weather_icon` | ledmatrix-weather | manager.py:1650 | unrelated | `WeatherIcons.draw_weather_icon(img, forecast['icon'], icon_x, icon_y, icon_size)` |
| `DisplayManager.draw_weather_icon` | ledmatrix-weather | weather_icons.py:168 | unrelated | `def draw_weather_icon(image: Image.Image, icon_code: str, x: int, y: int, size: int = DEFAULT_SIZE):` |
| `DisplayManager.draw_text_with_icons` | core | src/plugin_system/testing/visual_display_manager.py:526 | unrelated | `def draw_text_with_icons(self, text: str, icons: List[tuple] = None,` |
| `FontManager.get_font_catalog` | core tests | test/test_deprecation.py:229 | test call | `assert fm.get_font_catalog() == fm.font_catalog` |
| `BasePlugin.get_supported_vegas_modes` | core tests | test/test_vegas_participation.py:356 | test review | `assert plugin.get_supported_vegas_modes() == [` |
| `BasePlugin.get_supported_vegas_modes` | core tests | test/test_vegas_participation.py:358 | test review | `assert plugin.get_supported_vegas_modes()` |
| `BasePlugin.get_supported_vegas_modes` | blackjack | manager.py:732 | override | `def get_supported_vegas_modes(self):` |
| `BasePlugin.get_supported_vegas_modes` | blackjack | manager.py:695 | call | `if mode in self.get_supported_vegas_modes():` |
| `BasePlugin.get_supported_vegas_modes` | calendar | manager.py:875 | override | `def get_supported_vegas_modes(self) -> List[VegasDisplayMode]:` |
| `BasePlugin.get_supported_vegas_modes` | olympics | manager.py:624 | override | `def get_supported_vegas_modes(self) -> List[VegasDisplayMode]:` |
| `BasePlugin.get_vegas_segment_width` | core tests | test/test_vegas_participation.py:359 | test review | `assert plugin.get_vegas_segment_width() == 2` |
## Sources scanned
| Source | Group | Python files | Hits |
|---|---|---|---|
| core | core | 172 | 20 |
| core tests | core-tests | 347 | 17 |
| 7-segment-clock | monorepo | 3 | 0 |
| afl-scoreboard | monorepo | 35 | 0 |
| baseball-scoreboard | monorepo | 61 | 0 |
| basketball-scoreboard | monorepo | 49 | 0 |
| birdnet-go | monorepo | 2 | 0 |
| blackjack | monorepo | 7 | 2 |
| calendar | monorepo | 5 | 1 |
| christmas-countdown | monorepo | 3 | 0 |
| clock-simple | monorepo | 2 | 0 |
| countdown | monorepo | 5 | 0 |
| cricket-scoreboard | monorepo | 8 | 0 |
| f1-scoreboard | monorepo | 15 | 0 |
| fantasy-blitz | monorepo | 13 | 0 |
| football-scoreboard | monorepo | 74 | 0 |
| geochron | monorepo | 10 | 0 |
| hello-world | monorepo | 2 | 0 |
| hockey-scoreboard | monorepo | 52 | 0 |
| incoming-packages | monorepo | 8 | 0 |
| jellyfin-now-playing | monorepo | 4 | 0 |
| lacrosse-scoreboard | monorepo | 40 | 0 |
| ledmatrix-elections | monorepo | 12 | 0 |
| ledmatrix-flights | monorepo | 48 | 0 |
| ledmatrix-leaderboard | monorepo | 9 | 0 |
| ledmatrix-music | monorepo | 11 | 0 |
| ledmatrix-stocks | monorepo | 7 | 0 |
| ledmatrix-weather | monorepo | 15 | 6 |
| march-madness | monorepo | 4 | 0 |
| masters-tournament | monorepo | 10 | 0 |
| mqtt-notifications | monorepo | 4 | 0 |
| news | monorepo | 6 | 0 |
| nfl-draft | monorepo | 3 | 0 |
| nfl-stat-leaders | monorepo | 8 | 0 |
| nrl-scoreboard | monorepo | 30 | 0 |
| odds-ticker | monorepo | 9 | 0 |
| of-the-day | monorepo | 14 | 0 |
| olympics | monorepo | 16 | 1 |
| on-air | monorepo | 2 | 0 |
| pomodoro-timer | monorepo | 3 | 0 |
| soccer-scoreboard | monorepo | 47 | 0 |
| static-image | monorepo | 3 | 0 |
| stock-news | monorepo | 3 | 0 |
| text-display | monorepo | 4 | 0 |
| tide-display | monorepo | 3 | 0 |
| ufc-scoreboard | monorepo | 38 | 0 |
| web-ui-info | monorepo | 2 | 0 |
| youtube-stats | monorepo | 5 | 0 |
| f1-live | third-party | 10 | 0 |
| gif-player | third-party | 1 | 0 |
| pga-tour-leaderboard | third-party | 2 | 0 |
| plex-marquee | third-party | 1 | 0 |
| ledmatrix-dresden-departures | third-party | 1 | 0 |
| tidbyt-baseball-scoreboard | third-party | 2 | 0 |
| sleeper-fantasy | third-party | 1 | 0 |
| ledmatrix-nascar | third-party | 1 | 0 |
## How to re-run
```bash
# Clones the monorepo and each third-party plugin (depth 1) into a temp cache:
python3 scripts/plugin_api_usage.py --output docs/DEPRECATIONS_3.8.md
# Or scan a local monorepo checkout (read only) instead of cloning it:
python3 scripts/plugin_api_usage.py --monorepo ../ledmatrix-plugins
```
Before removing a method in its release, re-run the scan against the current monorepo and registry: a plugin added since this file was generated may have started calling it. Remove only methods the fresh scan reports unused; move the rest to a later release (the test in `test/test_deprecation.py` fails while a marker names a release at or below `src.__version__`).
+5 -5
View File
@@ -54,8 +54,8 @@ rows = self.layout.bounds.inset(1).split_v(3, 1, gap=1)
self.draw_fit("12:34", rows[0]) # largest crisp font that fits self.draw_fit("12:34", rows[0]) # largest crisp font that fits
self.draw_image(logo, rows[1], mode="fill_height", crop_to_ink=True) self.draw_image(logo, rows[1], mode="fill_height", crop_to_ink=True)
# Weather icons: draw_weather_icon() is deprecated, removed in 3.7.0 — # Weather icons: draw_weather_icon() was removed in 3.8.0 — draw your
# draw your own icons (the weather plugin ships WeatherIcons) # own icons (the weather plugin ships WeatherIcons)
# Scrolling state # Scrolling state
display_manager.set_scrolling_state(True) display_manager.set_scrolling_state(True)
@@ -78,7 +78,7 @@ strategy = cache_manager.get_cache_strategy("weather")
``` ```
`get_background_cached_data()` (use `get()`) and `get_sport_live_interval()` `get_background_cached_data()` (use `get()`) and `get_sport_live_interval()`
are deprecated, removed in 3.7.0. See were removed in 3.8.0. See
[Deprecated APIs](PLUGIN_API_REFERENCE.md#deprecated-apis). [Deprecated APIs](PLUGIN_API_REFERENCE.md#deprecated-apis).
## Plugin Manager Quick Methods ## Plugin Manager Quick Methods
@@ -87,8 +87,8 @@ are deprecated, removed in 3.7.0. See
# Get plugins # Get plugins
plugin = plugin_manager.get_plugin("plugin-id") plugin = plugin_manager.get_plugin("plugin-id")
all_plugins = plugin_manager.get_all_plugins() all_plugins = plugin_manager.get_all_plugins()
# get_enabled_plugins() is deprecated, removed in 3.7.0 — check `enabled` # get_enabled_plugins() was removed in 3.8.0 — check `enabled` on the
# on the entries in plugin_manager.plugins # entries in plugin_manager.plugins
# Get info # Get info
info = plugin_manager.get_plugin_info("plugin-id") info = plugin_manager.get_plugin_info("plugin-id")
+6 -6
View File
@@ -17,13 +17,13 @@ The LEDMatrix emulator allows you to run and test LEDMatrix displays on your com
## Prerequisites ## Prerequisites
### System Requirements ### System Requirements
- Python 3.7 or higher - Python 3.10 or higher
- Windows, macOS, or Linux - Windows, macOS, or Linux
- At least 2GB RAM (4GB recommended) - At least 2GB RAM (4GB recommended)
- Internet connection for plugin downloads - Internet connection for plugin downloads
### Required Software ### Required Software
- Python 3.7+ - Python 3.10+
- pip (Python package manager) - pip (Python package manager)
- Git (for plugin management) - Git (for plugin management)
@@ -50,8 +50,7 @@ pip install -r requirements-emulator.txt
``` ```
This installs: This installs:
- `RGBMatrixEmulator` - The core emulation library - `RGBMatrixEmulator` - the emulation library (and whatever it depends on)
- Additional dependencies for display adapters
### 3. Install Standard Dependencies ### 3. Install Standard Dependencies
@@ -63,8 +62,9 @@ pip install -r requirements.txt
### 1. Emulator Configuration File ### 1. Emulator Configuration File
The emulator uses `emulator_config.json` for configuration. Here's the The emulator uses `emulator_config.json` for configuration. It isn't in
default configuration as it ships in the repo: the repo (it's gitignored): RGBMatrixEmulator writes it on first run.
A typical file looks like this:
```json ```json
{ {
+8 -8
View File
@@ -13,10 +13,9 @@
BDF fonts in `assets/fonts/`, registers fonts that plugins ship, and records BDF fonts in `assets/fonts/`, registers fonts that plugins ship, and records
which plugin uses which font so the web UI can show it. which plugin uses which font so the web UI can show it.
Several methods are deprecated and will be removed in LEDMatrix 3.7.0; they Several methods were removed in LEDMatrix 3.8.0 after a release of
log a warning on first call. They are listed in deprecation warnings; [Removed methods](#removed-methods) below lists them
[Deprecated methods](#deprecated-methods) below, and the full set is pinned in with what to use instead.
[`test/test_deprecation.py`](../test/test_deprecation.py).
## Getting the FontManager ## Getting the FontManager
@@ -128,8 +127,8 @@ font = self.font_manager.resolve_font(
`resolve_font()` still honours `config/font_overrides.json` (a map of `resolve_font()` still honours `config/font_overrides.json` (a map of
element key to `family` and/or `size_px`), which is read once at start-up. element key to `family` and/or `size_px`), which is read once at start-up.
The methods that edit it — `set_override()`, `remove_override()`, The methods that edited it — `set_override()`, `remove_override()`,
`get_overrides()` — are deprecated, and there is no web UI or REST endpoint `get_overrides()` — were removed in 3.8.0, and there is no web UI or REST endpoint
for overrides (the override editor and `/api/v3/fonts/overrides` were for overrides (the override editor and `/api/v3/fonts/overrides` were
removed). To let users choose a font, add a field to your plugin's config removed). To let users choose a font, add a field to your plugin's config
schema. schema.
@@ -207,9 +206,10 @@ Current methods:
| `clear_cache()` | Drop cached fonts and metrics | | `clear_cache()` | Drop cached fonts and metrics |
| `font_catalog` (attribute) | Family name → file path | | `font_catalog` (attribute) | Family name → file path |
### Deprecated methods ### Removed methods
Removed in 3.7.0. Each logs a warning on first call. Removed in 3.8.0, after logging a deprecation warning on first call since
3.5.0.
| Method | Use instead | | Method | Use instead |
|---|---| |---|---|
+16
View File
@@ -240,6 +240,22 @@ The fastest way to verify a plugin works without waiting for the rotation:
- Install community plugins straight from a GitHub URL via - Install community plugins straight from a GitHub URL via
**Install from GitHub** on the same tab. **Install from GitHub** on the same tab.
### Keep LEDMatrix Up to Date
- **Update Code** on the **Overview** tab installs the newest version, and a
banner at the top of the page says when one is available.
- **General → Automatic Updates** does it once a week, overnight, with a
health check that undoes an update that breaks the device.
- **General → Update Channel** picks which version that is. **Stable** (the
default) installs releases, which have been tested and have release
notes. **Beta** installs the newest code as soon as it is written, before
it is released: fixes arrive sooner, and so do new problems.
- Switching to Stable never installs an older version than the one you
have. If your device is already newer than the latest release (which is
normal if it was set up or updated from the newest code), it keeps
getting the newest code until the next release includes it, then follows
releases from there. The General tab says when this is the case.
### Enable Advanced Features ### Enable Advanced Features
**Vegas Scroll Mode:** **Vegas Scroll Mode:**
+25
View File
@@ -170,6 +170,31 @@ pytest test/test_config_manager.py
pytest pytest
``` ```
### Web UI JavaScript Tests
The suites in `test/js` need node; the DOM ones also need jsdom and a running
web interface (details in [`test/js/README.md`](../test/js/README.md)):
```bash
npm install --no-audit --no-fund --prefix test/js # jsdom; node_modules is gitignored
EMULATOR=true python3 web_interface/app.py # in another shell
BASE=http://localhost:5000 REQUIRE_DOM=1 node test/js/run_all.js
```
`pytest test/test_js_unit_suites.py` runs just the unit suites.
### Plugin Config Form Parity
`test/test_field_model_parity.py` checks `build_field_model` against the
`render_field` macro for every plugin schema it finds
([WEB_FRONTEND_ARCHITECTURE.md](WEB_FRONTEND_ARCHITECTURE.md)). It always
covers `plugin-repos/` and the test fixtures; point it at a checkout of the
official plugins to cover those too:
```bash
LEDMATRIX_MONOREPO_PLUGINS=../ledmatrix-plugins/plugins pytest test/test_field_model_parity.py
```
### Debug a Failing Test ### Debug a Failing Test
```bash ```bash
+250
View File
@@ -0,0 +1,250 @@
# Control socket (web → display)
The display process serves a Unix socket that the web interface uses to send
it commands and get an answer back. It replaces the cache-file "mailboxes" on
the SD card one command at a time. Stage 1, described here, carries on-demand
start/stop/status. The file mailbox stays as a fallback for one release.
| | |
|---|---|
| Socket | `/run/ledmatrix/control.sock` (tmpfs) |
| Served by | the display process ([`src/ipc/server.py`](../src/ipc/server.py)), started by `DisplayController.run()` |
| Used by | the web interface ([`src/ipc/client.py`](../src/ipc/client.py)): `POST /api/v3/display/on-demand/start` and `/stop` |
| Contract | [`src/ipc/contract.py`](../src/ipc/contract.py): messages, versions, framing and the socket path; both sides import it |
| Override | `LEDMATRIX_CONTROL_SOCKET=/some/path.sock` for both processes, or `=off` to disable it |
## Why
Before the socket, the web interface sent commands by writing a cache key
(`display_on_demand_request`) that the display read every 0.25 s.
- **No acknowledgement.** The route answered "success" once the file was
written, whether or not a display was running to read it.
- **Lost requests.** The display had to read the request and then delete it.
A request written between those two steps could be thrown away (see
`_consume_on_demand_request`). The cache has no atomic claim to prevent it.
- **Fragile.** Each channel repeated its own permission, atomic-write,
staleness and in-memory-cache rules. Two of them caused bugs: a `memory_ttl`
bug ignored every on-demand request after the first for an hour, and a
stopped display was still reported as "active" for two minutes.
The socket answers every command, carries one request per message (so nothing
can overwrite it), and belongs to the display process. If the display is not
running, the socket does not exist, and the web interface knows right away.
## Protocol (version 1)
**Framing.** One JSON object per line (newline-delimited JSON), UTF-8, at
most 64 KiB per line (`MAX_MESSAGE_BYTES`). Senders encode with
`ensure_ascii`, so a newline never appears inside a message. A connection
can carry several requests. Each request gets exactly one response, in order.
**Request**
```json
{"v": 1, "id": "5f0c…", "cmd": "on_demand.start",
"args": {"plugin_id": "clock", "mode": null, "duration": 30, "pinned": false}}
```
- `v` is the protocol version.
- `id` is a printable string of 1-128 characters. It is echoed back in the
response, and for on-demand commands it is also the on-demand `request_id`.
- `cmd` is a command name.
- `args` is an object. It may be omitted when a command takes no arguments.
**Response**
```json
{"v": 1, "id": "5f0c…", "ok": true, "result": {"accepted": true, "request_id": "5f0c…", "queued": 1}}
{"v": 1, "id": "5f0c…", "ok": false, "error": {"code": "busy", "message": "…"}}
```
`id` is `null` only when the request could not be parsed far enough to have
one. Clients branch on `error.code`, never on the message text.
**Commands**
| `cmd` | `args` | `result` | Kind |
|---|---|---|---|
| `hello` | `{versions: [int], client?: str}` | `{version, versions, commands, max_message_bytes, server}` | answered directly |
| `ping` | — | `{pong: true}` | answered directly |
| `on_demand.start` | `{plugin_id?, mode?, duration?, pinned?}` (at least one of `plugin_id` and `mode`) | ack | queued |
| `on_demand.stop` | — | ack | queued |
| `on_demand.status` | — | `{on_demand: {...}, current_mode, display_active}` | answered directly |
`duration` is a number of seconds, or a numeric string. `0`, `null` or `""`
mean "until stopped". `pinned` must be a real boolean: the REST route has
already converted strings like `"false"` before it sends the command. The
`on_demand` object in `on_demand.status` is the same dict the display
publishes to `display_on_demand_state`.
**Acknowledgements.** A queued command is *accepted*, not *done*.
`{"accepted": true, "request_id": …}` means the command is waiting in the
render thread's queue, and the render thread will apply it at its next
on-demand check. That is within one frame on a scrolling screen, 0.25 s
during a dwell, and up to 1 s on a static screen, whose frame loop sleeps a
second between frames. Except on a scrolling screen, where the mailbox waits
up to 0.25 s, these are the mailbox's delays too: stage 1 adds
acknowledgements, not speed. Any outcome is published as before
(`display_on_demand_state`, and `status`/`error` for a bad plugin or mode),
and it can be read with `on_demand.status`.
**Versions.** Every request carries `v`. For any command except `hello`, a
`v` the display does not speak gets `unsupported_version`. `hello` is checked
by its `versions` list instead, and its result names the highest version both
sides share, so a client can find out what a display supports before it
relies on anything newer. Stage 1's client sends `v: 1` and falls back to the
mailbox when the display refuses it. It does not send `hello` first, which
saves a round trip.
**Error codes:** `bad_json`, `bad_request`, `message_too_large`,
`unsupported_version`, `unknown_command`, `invalid_args`, `busy` (queue full,
or too many connections), `forbidden` (peer credentials refused), `internal`.
Try it on a device:
```bash
python3 - <<'EOF'
from src.ipc import client # run from the project directory
print(client.on_demand_status())
EOF
```
## How the display applies a command
The server's threads never touch rendering. A connection thread parses the
request, validates it against the contract, and then does one of two things:
- For a command that changes the panel, it puts a `QueuedCommand` on a
bounded queue (16 entries) and answers with the ack.
- For a query, it answers from a status snapshot the display provides
(`DisplayController._control_status`). The snapshot only reads attributes.
The render thread drains the queue in `_poll_on_demand_requests()`, the same
place it reads the mailbox, and hands each command to
`_handle_on_demand_request()`, which is the mailbox's own handler. The two
paths share all of their code: activation, the processed-id guard, error
publishing, and resuming the rotation afterwards. The 0.25 s floor on the
mailbox read does not apply to the queue, because draining it costs no disk
read. A queued command also lets `_service_pending_changes()` skip its own
floor, so a long scrolling screen or a Vegas iteration takes the command at
its next frame.
**Exactly once.** A command and a mailbox write for the same request share
one `request_id`. If the client times out after the display queued the
command and then also writes the mailbox, the display processes the request
once. The existing `on_demand_request_id` and processed-id checks drop the
second copy.
## Robustness
All of this runs inside the display process, so nothing a client does may
block the render loop or crash it:
- **Bounded connections.** Each connection gets its own daemon thread, with
at most 8 at once. One more is answered `busy` and closed.
- **Timeouts.** Each read and write times out after 2 s. A message must
arrive whole within 5 s of its first byte. An idle connection is closed
after 10 s. A slow or stuck client costs one thread for a few seconds.
- **Malformed input.** A line that is not JSON gets `bad_json`, and the
connection carries on. A line longer than 64 KiB gets `message_too_large`,
and the connection is closed, because the next message boundary cannot be
found. A client that disconnects mid-message is dropped silently. No
exception from a handler leaves the connection thread.
- **Full queue.** When the queue is full, the client gets `busy` and falls
back to the mailbox. A full queue means the render thread is stuck, and the
systemd watchdog deals with that.
- **Startup.** The server binds under a temporary name, sets the mode and the
group, then renames the socket into place, so it never appears with the
umask's permissions. It removes a stale socket (a file that nothing is
listening on). It never removes a live socket or a file that is not a
socket. `close()` removes the socket only if it is still the one this
process created.
- **Never fatal.** If the server cannot start (Windows, no `AF_UNIX`, a bind
failure, `LEDMATRIX_CONTROL_SOCKET=off`), it logs that and the display runs
as before. The web interface then uses the mailbox.
## Security model
The display runs as root and the web interface as the installing user (see
[PERMISSIONS.md](PERMISSIONS.md)). The socket admits exactly those two, plus
anything else in the group they share:
1. **The directory.** `/run/ledmatrix` is created by `RuntimeDirectory=ledmatrix`
in `ledmatrix.service` (#687): root-owned, `0755`, on tmpfs, and removed
when the display stops. Under an older unit, the display creates the
directory itself as root, as it does for the heartbeat. No installer
change is needed.
2. **The socket file.** The file is `root:<shared group>` with mode `0660`,
and the kernel refuses `connect()` to anyone without write permission on
it. The shared group is the cache directory's group whenever that
directory is group-writable. That is `ledmatrix` on an installed device
(`/var/cache/ledmatrix` is `root:ledmatrix 2775`), and it is the same rule
DiskCache uses for every file the two services share. Otherwise the group
is the project directory's (`get_shared_group_gid()`, which config files
use). With neither, the mode is `0600` and only root can connect.
3. **Peer credentials.** Where the kernel reports them (`SO_PEERCRED`, on
Linux), the server checks every connection again. It accepts root, the
display's own user, or a member of the shared group: the peer's primary
gid, or a supplementary group read from `/proc/<pid>/status`. If `/proc`
is unreadable, it uses the group database. Any other peer gets `forbidden`
and is disconnected. This covers a socket mode that someone loosened by
hand.
The commands are deliberately narrow. Stage 1 can start or stop on-demand
display and read its state, which anyone who can reach the web UI can already
do. Nothing on the socket runs a shell, writes a file, or names a path.
**Development.** A display that is not root and cannot write to
`/run/ledmatrix`, such as `python3 run.py -e` from a checkout, serves the
socket at `$TMPDIR/ledmatrix-<uid>/control.sock`. That directory is private
(`0700`), and the server refuses it if another user owns it. The web
interface, run by the same user, looks there after `/run/ledmatrix`. The test
suite sets `LEDMATRIX_CONTROL_SOCKET=off` (`test/conftest.py`), so a run on a
device never touches the live display.
## Stage plan
1. **On-demand, with acks (this stage).** Contract, server, client.
`on_demand.start`/`stop`/`status`, `hello`, `ping`. The REST routes try the
socket first and report `transport: "socket" | "mailbox"` (plus
`socket_error` on fallback). The mailbox is unchanged, and the plugins that
write it directly (birdnet-go, mqtt-notifications, on-air, pomodoro-timer)
keep working.
2. **Commands that are restarts or polls today.**
- `brightness.set`, transient and with no `config.json` write.
- `plugin.reload`, which replaces the `restart_required` answer from #688
with a live reload of the updated plugin on the render thread.
- `config.reload`, which applies a saved config without waiting for the 2 s
mtime poll and acks which sections changed.
- The dwell sleep and the static screen's 1 s frame sleep wait on the
queue instead of sleeping, so a command lands within milliseconds on
every kind of screen. Under WSL, with a static plugin on screen, a stop
takes 1.0 s by either path today.
3. **A state stream.** A `subscribe` command that keeps the connection open
and pushes events: mode changes, on-demand state, plugin runtime state and
the heartbeat. It replaces the polled `display_current_state`,
`plugin_runtime_snapshot` (#690) and `display-heartbeat.json` (#687) for
readers that hold a connection. The web interface relays it to its
existing SSE stream. The files remain for one release for older readers.
4. **Retire the mailboxes.** After a release in which every device has had the
socket, the web interface stops writing `display_on_demand_request`, and
the display stops polling it, logging the plugins that still write it so
they can move to an in-process `request_display()`. The other cache keys
used as messages (`plugin_error_clear_request` and the remaining
`display_*` keys) move to the socket or to tmpfs.
## Checking it on a device
```bash
ls -l /run/ledmatrix/control.sock # srw-rw---- root ledmatrix
sudo journalctl -u ledmatrix | grep "Control socket"
curl -s -X POST localhost:5000/api/v3/display/on-demand/start \
-H 'Content-Type: application/json' -d '{"plugin_id":"clock","duration":20}'
# ... "transport": "socket"
```
If the response says `"transport": "mailbox"`, `socket_error` gives the
reason. `no_socket` means the display is stopped or predates the socket.
`refused` usually means the web user is not in the socket's group, which
takes effect when the web service restarts after the user is added.
+354
View File
@@ -0,0 +1,354 @@
# Offscreen Rendering
**Status (2026-09-30):** offscreen rendering is implemented
(`DisplayManager.offscreen()`, the adapter on the prefetch thread, the plugin
lock), and so are live elements, which grew out of steps 2 and 3 below: see
*Live elements*. The segment strip proposed as step 2 was not needed; *Why not
a SegmentStrip* says why.
First soak of step 1 on hdpi (50 px/s, `pwm_bits` 7, preview open, 8-minute
runs, A/B/B/A):
| build | late | by 1 | 2 | 3–5 | 6+ | freezes | render-thread fetches |
|---|---|---|---|---|---|---|---|
| #628 | 0.53% | 82 | 2 | 3 | 2 | 3 | 6 |
| step 1 | 0.63% | 78 | 63 | 17 | 2 | 1 | 0 |
| step 1 | 0.42% | 77 | 23 | 10 | 0 | 0 | 0 |
| #628 | 0.37% | 84 | 5 | 3 | 3 | 2 | 14 |
It does what it was built to: no plugin is fetched on the render thread, and
freezes fell from 5 to 1. But frames 2–5 refreshes late rose. The rendering
moved to the prefetch thread still needs the GIL, and the render thread waits
for it (risk 5 below). The late rate did not improve overall. The 1–2 s
freezes appear in both builds and have a separate, not yet identified cause.
The GIL fix, measured on hdpi (90 px/s, `pwm_bits` 8, preview open, 8-minute
runs after a 2-minute warm-up, order A B C C B A, 2026-09-24). Each arm pools
two runs, about 81,000 frames:
| arm | late | by 1 | 2 | 3–5 | 6+ | 2+ late per 10k frames | freezes |
|---|---|---|---|---|---|---|---|
| A: step 1 as is | 0.90% | 575 | 64 | 91 | 9 | 20.1 | 0 |
| B: `switch_interval_ms` 1 | 0.78% | 510 | 105 | 23 | 2 | 15.8 | 0 |
| C: `prefetch_gate` | **0.60%** | 471 | 11 | 7 | 2 | **2.5** | 0 |
The gate removes the frames the render thread spent waiting for the GIL, and
it costs the prefetch nothing that shows: it parked the thread for 3–6 s per
run, and the next group was ready at every strip extension in every arm.
`prefetch_gate` is therefore on by default; `switch_interval_ms` stays an
off-by-default experiment. What is left is almost all one refresh late, which
is the per-frame budget (a 6.75 ms p50 blit in a refresh the panel holds at
83–85 Hz while rendering), not contention.
The runs restart the service, so the hourly sports refresh never fell inside
one. That refresh is its own case: about twenty ESPN chunk-fetch threads at
once, which the gate does not cover (it gates only the prefetch thread).
## The problem
Vegas mode builds its ticker from every plugin's content. Most of that work
already happens on a background prefetch thread
(`RenderPipeline.start_prefetch`). But any plugin whose content needs the
**shared display canvas** is deferred to the render thread
(`RenderPipeline.drain_deferred`), one plugin every two seconds. The code's
own comments put each of those at 40–600 ms, and the render thread presents no
frames while one runs.
On hdpi (Pi 4, 512×64) most plugins take that path: geochron, tide-display,
news, hockey-scoreboard, ledmatrix-stocks, incoming-packages, clock-simple,
countdown, birdnet-go, ledmatrix-music and odds-ticker. They arrive in bursts
("Whole group deferred; strip will extend as it drains") every minute or so,
12 fetches in five minutes. That is the "occasional pause" a viewer sees.
An 8-minute soak (`scripts/frame_soak.py --preview`) of the #628 build on
hdpi:
| late by | frames |
|---|---|
| 1 refresh | 238 |
| 2 | 32 |
| 3–5 | 30 |
| 6+ | 5 |
| freezes ≥ 250 ms | 2 (0.97 s total) |
The 3+ rows and the freezes are the pauses. The single-refresh row is a
separate problem: the blit is 6 ms of a 10 ms refresh, so there is little
slack. It is covered under *What this does not fix*.
## Why a plugin is canvas-bound
The plugin-facing canvas is a set of shared attributes on `DisplayManager`:
`image`, `draw`, `matrix`, and the `width`/`height` properties that read from
`matrix`. Three adapter paths (`src/vegas_mode/plugin_adapter.py`) need them,
and each returns `None` under `offscreen_only=True` so the plugin is queued for
the render thread:
1. **Display capture** (`_capture_display_content`): clear the canvas, call
`plugin.display()`, copy `display_manager.image`. Used by any plugin
without `get_vegas_content()` or a populated `scroll_helper`.
2. **Scroll-content generation** (`_trigger_scroll_content_generation`): a
ticker plugin whose `scroll_helper.cached_image` is empty is made to build
it by calling `display(force_clear=True)` or `_create_scrolling_display()`.
Both draw on the canvas.
3. **Narrowed rendering** (`DisplayManager.render_size`): swaps the shared
`matrix`, `image` and `draw` for a narrower set so the plugin lays out for
`render_width_pct`. The render thread would see the swap mid-frame.
The render thread keeps the canvas coherent only because nothing else touches
it at the same time. A background thread can't use it.
## The design: a per-thread render target
`capture_mode()` is already per-thread (#423 made its state a
`threading.local`, so a background capture no longer suppresses the render
loop's pushes). The same move applies to the canvas itself:
```python
with display_manager.offscreen(width=None, height=None) as surface:
plugin.display(force_clear=True)
content = surface.image.copy()
```
For the **calling thread only**, inside the block:
| accessor | resolves to |
|---|---|
| `display_manager.image`, `.draw` | the surface's own image and draw: a fresh black canvas, `fontmode = "1"` |
| `display_manager.matrix` | a logical proxy reporting the surface size, so `width`/`height` and plugins that read `matrix.width` follow it. Hardware calls through it (`SetImage`, `SwapOnVSync`, `Clear`, brightness writes) are inert. |
| `update_display()`, `clear()` | canvas-only: the block implies capture mode, which is already per-thread |
| `set_scrolling_state()`, `set_frame_hold()` | no-ops, so a plugin's `display()` cannot re-pace the live scroll. Today it can, when it is captured on the render thread. |
Every other thread sees the real canvas, unchanged. The render loop in
particular keeps presenting while a plugin draws elsewhere.
### Implementation sketch
- `image`, `draw` and `matrix` become properties over `_image`, `_draw` and
`_matrix`, plus a thread-local current surface. The getter returns the
surface's value when the calling thread has one, else the shared one; setters
mirror that. That costs about 0.1 µs per access, and `update_display()` reads
each a handful of times per frame. Every existing `self.image = ...` in
`DisplayManager` (`clear()`, setup, fallback) keeps working and becomes
thread-correct for free.
- `render_size()` is rebuilt on `offscreen()`: it creates or narrows the
calling thread's surface instead of swapping shared state.
- `offscreen()` nests and always restores on exit, including when the plugin
raises.
- `VisualDisplayManager` (the plugin test harness) gets the same method, for
parity.
### Adapter changes
- `get_content(offscreen_only=True)` stops returning `None` for the three
paths above. Each runs inside `display_manager.offscreen(render_width)`.
- `_capture_display_content` and `_trigger_scroll_content_generation` drop
their "copy the shared image, restore it afterwards" bookkeeping, since the
shared image is never touched.
- **Take the plugin's lock.** `PluginManager.get_plugin_lock()` keeps
`update()` and `display()` mutually exclusive in normal rotation, but Vegas
never takes it, so today's render-thread captures already race
`update()`. Off the render thread the adapter can afford to wait: blocking
acquire with a timeout (proposed 2 s). On timeout it keeps the cached segment
and tries again next group.
- `drain_deferred()` and the deferred queue are deleted. The only render-thread
fetch left is the inline fallback when no prepared group is ready, which in
practice is the first extension. Prefetching at start removes that too.
## Live elements: content that changes while it scrolls
Offscreen rendering is also what makes fresh content possible. A plugin's
segment is drawn when its group is prefetched, and the strip carries
7,000-10,000 px of content ahead of the viewport, so at ~100 px/s a score drawn
then reaches the screen 70-100 seconds later -- and once in the strip it never
changed: when a plugin reported new data, Vegas only dropped its caches, so the
change appeared on the plugin's *next* turn, minutes later.
A plugin can now hand Vegas **live elements** instead of pictures
(`BasePlugin.get_vegas_elements()`, see "Live Vegas elements" in
[PLUGIN_API_REFERENCE.md](PLUGIN_API_REFERENCE.md#live-vegas-elements)):
named, fixed-width pieces of content -- one per game card, one for a map. Vegas
records where each lands in the strip and, when the plugin's data changes,
redraws just the changed ones off the render thread and copies their pixels
over the old ones between two frames. A card already crossing the panel
changes; nothing next to it moves.
### Why not redraw every frame
On a Pi the render thread has about 4 ms of slack per refresh at 512x64 after
the ~6 ms blit, and a scoreboard card is ~29 ms of Pillow work that holds the
GIL. Drawing on the render thread is out of the question at any rate, so the
render thread only ever *copies* pixels that are already drawn. Measured on a
Pi 4 (ledpi): writing a 35 KB card into a 20,000 px strip takes 8.5 µs, a
101 KB map 17 µs, four cards (the per-frame cap) 34 µs -- against 124 µs for
the viewport slice every frame already does.
### How an update reaches the screen
1. A plugin's `update()` completes. The update worker calls
`PluginManager._note_update_completed`, which calls the update listeners
(`add_update_listener`) there and then, with the plugin's lock still held.
Vegas's listener moves the plugin's **epoch** on
(`src/vegas_mode/elements.py`, `LiveEpochs`) and wakes the live worker.
2. The **live worker** (`src/vegas_mode/live_worker.py`), the one background
thread that draws for the strip once it holds a live element, finds the
plugin's elements whose recorded epoch is older than its current one,
nearest the screen first, and calls `get_vegas_elements()` under the
plugin's lock (0.25 s wait, then a 1 s backoff). Elements whose `version`
is unchanged cost nothing; the rest are pinned and checksummed, and each
whose pixels changed becomes a patch in a one-per-element slot (the latest
wins).
3. Between two frames the render thread
(`RenderPipeline.apply_live_patches`, from `coordinator.run_frame`) pops at
most four patches or two screens of bytes and copies each into the strip
with `ScrollHelper.patch_columns`. It takes no lock and draws nothing. A
patch made for an older strip, for an element trimmed away or already
behind the screen, or from older data than the strip shows, is dropped.
End to end, a new score reaches a card already on screen within one poll of
the data source (30 s for live games) plus about a second: the listener is
immediate, and while live elements exist the update tick that schedules
plugins runs every second instead of every four.
Elements that change with **time** rather than data (an aircraft moving
between position reports) ask for `refresh_hz`; the worker calls
`redraw_vegas_element()` -- without the plugin's lock, from state the plugin
publishes in one assignment -- that often while the element is on or within
`live_lead_screens` of the screen, capped by `live_max_hz` (5), at 1 Hz
without the render gate, and halved for an element whose redraws average over
50 ms.
### Geometry
A live element is never trimmed to its ink: the adapter pads it with
`content_padding` black columns either side and pins its width, and a redraw
at any other width is refused (it shows the next time the plugin comes round).
Records keep **absolute** strip columns -- the strip column plus everything
trimmed off the front since the strip was composed -- so a trim moves one
origin rather than every record. Nothing on screen is ever moved, inserted or
resized; a game added to a slate appears on the plugin's next turn.
### Why not a SegmentStrip
The proposal here was to replace the single strip with a list of segments.
In-place patching of the single strip meets every goal without that: the
patch is O(element) and the strip layout never changes. What a segment list
would still buy is cheaper extensions, and most of that came from making the
strip's PIL copy lazy instead (`ScrollHelper.cached_image`: an extension used
to rebuild it twice, 1.7-3.8 ms each on a Pi 4). The `extend` row of
`frame_soak.py`'s "after work" table says whether the rest is worth it.
### When it is off
- `display.vegas_scroll.live_refresh: false` (the kill switch; also in the
web UI), or `vegas_live: false` in one plugin's section.
- Always under multi-display sync: the follower mirrors whole strips only, so
a patch would never reach it. (Continuous-mode sync has a separate problem:
the follower is not sent extensions or trims at all.)
- In swap mode (`continuous_scroll: false`) and with `offscreen_prefetch:
false`.
- For plugins without the hook, which are drawn and placed exactly as before,
and on the paths that fetch without the plugin's lock (the first strip of a
run, the render thread's fallback fetch), which use `get_vegas_content()`.
## Risks, and what was checked
1. **Plugins holding their own reference to the shared `draw` or `image`.**
They would keep drawing into the shared canvas, and routing by thread can't
redirect them. A grep of the 49 plugins installed on hdpi found none storing
`display_manager.draw` or `.image` in an attribute (a pattern search, so
indirect aliasing would slip past it). A plugin that did would
draw into an image nobody displays, which trims to a blank segment. That is
not corruption, and it is no worse than today.
2. **Plugins calling the matrix directly.** None in the audit. Inside
`offscreen()` the proxy makes it inert anyway.
3. **Font thread-safety.** `FontManager` shares font objects across plugins.
Measured on Pillow 12.3, two threads rendering text take 1.94× as long as
one, so text rendering holds the GIL and FreeType is never entered
concurrently. Re-check if Pillow changes that.
4. **Plugin thread-safety.** `display()` moves to the prefetch thread. The
plugin lock makes it exclusive with `update()`, which is more protection
than it has today. Threads a plugin starts itself are not covered, as today.
5. **The GIL.** Moving 40–600 ms of plugin rendering off the render thread
removes the pauses, but the work still needs the GIL. Pillow drawing holds
it, and a waiting thread only gets it back after the switch interval
(default 5 ms). Expect some single-refresh late frames while a prefetch
runs. Measure with the soak. A render process separate from plugin work
is the structural answer (the "native presenter" step). Two experiments
get most of the way first (results under Status, above):
- `vegas_scroll.switch_interval_ms` lowers the switch interval for a Vegas
run (1 ms is the obvious try), so the render thread waits at most that
long behind bytecode. It does nothing for a C call that keeps the GIL.
- `vegas_scroll.prefetch_gate` (`src/common/render_gate.py`) lets the
prefetch thread run Python only while the render thread is blocked in
`SwapOnVSync`, up to just before the refresh the swap returns on, and
parks it the rest of the time. That covers C calls too, since the gate is
checked before each one starts. It never parks the thread while it holds
a lock the render thread takes, and never for more than 50 ms. It needs
the rebuilt binding, which releases the GIL during the swap. On by
default.
## What this does not fix
- **The blit.** Copying a 512×64 frame into the matrix (`SetImage`) is ~6 ms at
8 PWM bits on a Pi 4, leaving ~4 ms of slack per refresh. That is the main
source of the single-refresh late frames. Holding frames for two refreshes
(≈50 px/s) doubles the budget. Cutting the blit itself is the native-presenter
step.
- **Multi-display sync in continuous mode.** The follower is sent the whole
strip only at a new cycle and on connect, never the extensions and trims of
continuous mode, so it drifts from the leader after the first extension.
Live elements stay off under sync for that reason.
## Test plan
- **Unit, `DisplayManager`:** one thread inside `offscreen()` draws while
another reads `image`/`draw`/`matrix`/`width`/`height` and sees the real
canvas. Also: `update_display()` and `set_scrolling_state()` are inert inside;
`render_size()` narrows only the calling thread; nesting and exceptions
restore state.
- **Unit, adapter:** a stub display-capture plugin and a stub scroll-helper
plugin both return content with `offscreen_only=True`, and nothing is queued
for the render thread. The plugin lock is taken, and a timeout keeps the cached
segment.
- **Emulator integration:** a stub canvas-bound plugin whose `display()` sleeps
300 ms. The Vegas render loop never goes a frame without presenting (frame
timing recorder: zero freezes).
- **Live elements** (`test/test_vegas_live_*.py`,
`test/test_vegas_elements_*.py`, `test/test_scroll_helper_patch.py`): every
record points at exactly its element's pixels through any sequence of
compose, extend and trim; a patch changes only its element's columns (a
property test against a twin strip that is never patched); the render
thread's apply takes no lock and draws nothing; the worker's priorities,
floors, backoff and hand-over; and, end to end on the emulator with the stub
plugin (`test/fixtures/plugins/vegas-live-stub`), an update changes a card
already in the strip and an animated element moves with no update at all.
- **Hardware:** an hdpi soak, A/B against the #628 build, alternating order.
Targets: no freezes, an empty 6+ bucket, the 3–5 bucket near zero, and the late
rate below 0.66%. Plus, for freshness: log each segment's age when it enters
the viewport, and compare the median and max before and after.
## Rollout
1. **Offscreen rendering** (shipped): `offscreen()`, the adapter on the
prefetch thread, and the plugin lock. Removed the render-thread pauses.
2. **Measurement and the lazy strip image:** late frames attributed to the
render-thread work before them (`FrameTimingRecorder.note_op`, the "after
work" table), and extensions no longer rebuilding the strip's PIL copy.
3. **Live elements:** the plugin API, the records, the worker and in-place
patches, with the sports scoreboards and the flight map adopting it.
4. **Live games in the ticker by default:** `live_in_ticker` true, so a live
game's cards update in the marquee instead of the full-screen scoreboard
replacing it; existing configs are switched once (`ConfigManager`).
`display.vegas_scroll.offscreen_prefetch` (default `true`) restores the
deferred path when `false`, and `display.vegas_scroll.live_refresh` (default
`true`) turns live elements off. Keep both for one release, then delete the
old paths.
## Open questions
1. Multi-display sync: is there a two-Pi rig to test on? Replaying strip
operations to the follower (append, trim, patch, in absolute columns) would
fix continuous-mode sync and let live elements run under it.
2. Is the `extend` cost worth a segment list after the lazy image? The soak's
"after work" table answers it per rig.
+2
View File
@@ -29,6 +29,8 @@ in again (services pick them up on restart).
| `assets/` | web user | dirs `755`, files `644` | Root writes downloaded logos regardless | | `assets/` | web user | dirs `755`, files `644` | Root writes downloaded logos regardless |
| `/var/cache/ledmatrix/` | `root:ledmatrix` | `2775` (setgid) | Shared cache: see below | | `/var/cache/ledmatrix/` | `root:ledmatrix` | `2775` (setgid) | Shared cache: see below |
| Cache files | creator : `ledmatrix` | `660` | | | Cache files | creator : `ledmatrix` | `660` | |
| `/run/ledmatrix/` | `root` | `755` | tmpfs; `RuntimeDirectory=` in `ledmatrix.service`, removed when the display stops |
| `/run/ledmatrix/control.sock` | `root` : cache directory's group (`ledmatrix`) | `660` | The display's control socket; only root and that group can connect. See [IPC_CONTROL_SOCKET.md](IPC_CONTROL_SOCKET.md#security-model) |
| `scripts/fix_perms/safe_plugin_rm.sh`, `safe_pip_install.sh` | `root:root` | `755` | Run as root through sudo, so the web user must not be able to edit them | | `scripts/fix_perms/safe_plugin_rm.sh`, `safe_pip_install.sh` | `root:root` | `755` | Run as root through sudo, so the web user must not be able to edit them |
| `/etc/sudoers.d/ledmatrix_web`, `ledmatrix_wifi` | `root` | `440` | | | `/etc/sudoers.d/ledmatrix_web`, `ledmatrix_wifi` | `root` | `440` | |
+198 -131
View File
@@ -26,7 +26,7 @@ fields:
| Check | Fields | What happens when one is missing | | Check | Fields | What happens when one is missing |
|---|---|---| |---|---|---|
| JSON schema, [`schema/manifest_schema.json`](../schema/manifest_schema.json) | `id`, `name`, `version`, `author`, `entry_point`, `class_name`, `compatible_versions` | Install from URL logs a warning (`PluginStoreManager._validate_manifest_schema()`); nothing is refused | | JSON schema, [`schema/manifest_schema.json`](../schema/manifest_schema.json) | `id`, `name`, `version`, `author`, `entry_point`, `class_name`, `compatible_versions` | Install from URL logs a warning (`PluginStoreManager._validate_manifest_schema()`); nothing is refused |
| Plugin Store install, [`src/plugin_system/store_manager.py`](../src/plugin_system/store_manager.py) | `id`, `name`, `class_name`, `display_modes` | Install is refused. A registry install first tries to detect a missing `class_name` from the entry-point file | | Plugin Store install, [`src/plugin_system/store_install.py`](../src/plugin_system/store_install.py) | `id`, `name`, `class_name`, `display_modes` | Install is refused. A registry install first tries to detect a missing `class_name` from the entry-point file |
| Plugin loader, [`src/plugin_system/plugin_loader.py`](../src/plugin_system/plugin_loader.py) | `class_name` | The plugin fails to load | | Plugin loader, [`src/plugin_system/plugin_loader.py`](../src/plugin_system/plugin_loader.py) | `class_name` | The plugin fails to load |
Defaults and other uses: Defaults and other uses:
@@ -149,15 +149,27 @@ Clean up resources when plugin is unloaded. Override to close connections, stop
#### `on_config_change(new_config: Dict[str, Any]) -> None` #### `on_config_change(new_config: Dict[str, Any]) -> None`
Called after plugin configuration is updated via web API. Called after the plugin's section of `config.json` changes -- a save in the
web UI, say. Every lifecycle hook runs in the display process, which is the
only process that runs plugins: the web interface writes `config.json`, and
the display's config watcher calls this with the prepared section. See
[ARCHITECTURE.md](ARCHITECTURE.md#web-and-display-processes-who-runs-plugins).
In the display service it runs on the config watcher thread while holding
the plugin's lock, so it never overlaps your `update()` or `display()`. If
the plugin stays busy for more than 5 seconds, the change is applied later
from the update thread: as soon as the plugin is free, and before its next
`update()` at the latest.
#### `on_enable() -> None` #### `on_enable() -> None`
Called when plugin is enabled. Called when the display loads the plugin enabled: at startup, or when it is
switched on in the web UI.
#### `on_disable() -> None` #### `on_disable() -> None`
Called when plugin is disabled. Called when the display unloads the plugin, e.g. when it is switched off in
the web UI.
#### `get_update_interval() -> Optional[float]` #### `get_update_interval() -> Optional[float]`
@@ -296,9 +308,9 @@ the core then falls back to its own live-content check — so a plugin whose
weight calculation is broken still gets `live_weight` for a game that really weight calculation is broken still gets `live_weight` for a game that really
is live, rather than being demoted to 1. is live, rather than being demoted to 1.
Only consulted when the user has set `vegas_scroll.live_in_ticker`. With the Only consulted while `vegas_scroll.live_in_ticker` is on (the default since
default (`false`) live content preempts Vegas entirely and there is no ticker 3.8.0). With it off live content preempts Vegas entirely and there is no
to be weighted within. See ticker to be weighted within. See
[ADVANCED_FEATURES.md](ADVANCED_FEATURES.md#live-content-in-the-ticker). [ADVANCED_FEATURES.md](ADVANCED_FEATURES.md#live-content-in-the-ticker).
### Vegas scroll hooks ### Vegas scroll hooks
@@ -308,6 +320,58 @@ rotating one at a time. Plugins control how their content appears via
these hooks. See [ADVANCED_FEATURES.md](ADVANCED_FEATURES.md) for the user these hooks. See [ADVANCED_FEATURES.md](ADVANCED_FEATURES.md) for the user
side of Vegas mode. side of Vegas mode.
#### Vegas participation
Each plugin takes part in Vegas mode in one of three ways:
| Participation | What Vegas does |
|---|---|
| `'scroll'` | The plugin's content (`get_vegas_content()`) scrolls by with everything else |
| `'pause'` | The scroll stops when the plugin's turn comes round; its `display()` draws it full screen for `get_display_duration()` seconds, then the scroll resumes |
| `'exclude'` | The plugin is left out of Vegas mode |
Declare the plugin's default in `manifest.json`:
```json
{
"id": "my-alerts",
"vegas_participation": "pause"
}
```
The user can override it per plugin with `vegas_participation` in that
plugin's config section (it is one of the core-owned properties, see
[PLUGIN_CONFIG_CORE_PROPERTIES.md](PLUGIN_CONFIG_CORE_PROPERTIES.md)).
Vegas resolves it in this order:
1. the user's `vegas_participation` config value;
2. the plugin's `get_vegas_participation()` — the default implementation
reads the manifest's `vegas_participation`, then derives a value from
the legacy hooks below;
3. derived from the legacy hooks: `get_vegas_display_mode()` returning
`VegasDisplayMode.STATIC` → `'pause'`; otherwise
`get_vegas_content_type()` returning `'none'` → `'exclude'`; everything
else → `'scroll'`.
Step 3 is exactly what Vegas did before participation existed, so a plugin
that declares nothing behaves as it always has. Manifest
`vegas_participation` is new in core 3.8.0; older cores ignore it and use
the legacy hooks.
#### `get_vegas_participation() -> str`
Returns `'scroll'`, `'pause'` or `'exclude'`. Override it only when the
answer depends on state — pause only while an alert is live, exclude while
there is nothing to show; for a fixed answer use the manifest. Vegas applies
the user's config value before calling an override, so an override does not
need to check it. A value that is not one of the three is ignored with a log
line and the legacy hooks decide.
```python
def get_vegas_participation(self):
return 'pause' if self._alert_is_live() else 'scroll'
```
#### `get_vegas_content() -> Optional[PIL.Image | List[PIL.Image] | None]` #### `get_vegas_content() -> Optional[PIL.Image | List[PIL.Image] | None]`
Return content to inject into the scroll. Multi-item plugins (sports, Return content to inject into the scroll. Multi-item plugins (sports,
@@ -315,26 +379,113 @@ odds, news) should return a *list* of PIL Images so each item scrolls
independently. Static plugins (clock, weather) can return a single image. independently. Static plugins (clock, weather) can return a single image.
Returning `None` falls back to capturing whatever `display()` produces. Returning `None` falls back to capturing whatever `display()` produces.
#### `get_vegas_content_type() -> str` #### `get_vegas_render_width() -> int`
`'multi'`, `'static'`, or `'none'`. Affects how Vegas mode treats the The width Vegas wants this plugin's content to occupy, from the plugin's
plugin. Default `'static'`. `vegas_width_pct` config value or the global
`display.vegas_scroll.render_width_pct`. Vegas also narrows
`display_manager` while it asks for content, so a plugin that sizes itself
from `display_manager.width` does not need to read this.
#### `get_vegas_display_mode() -> VegasDisplayMode` #### Live Vegas elements
Returns one of `VegasDisplayMode.SCROLL`, `FIXED_SEGMENT`, or `STATIC`. *New in core 3.8.0.* Content from `get_vegas_content()` is baked into the
Read from `config["vegas_mode"]` or override directly. ticker's strip when the plugin's turn is prefetched, so a score drawn then
scrolls past with that score however many goals are scored while it crosses
the panel. A plugin that returns **live elements** instead gets them updated
in place: after its `update()` the ticker asks again, compares each element
with what the strip holds, and swaps the changed ones in between two frames
-- on screen included -- without anything next to them moving.
#### `get_supported_vegas_modes() -> List[VegasDisplayMode]` ```python
try:
from src.plugin_system.vegas_elements import VegasElement
except ImportError: # core older than 3.8.0: the hook is never called
VegasElement = None
The set of Vegas modes this plugin can render. Used by the UI to populate class MyScoreboard(BasePlugin):
the mode selector for this plugin. def get_vegas_elements(self):
if VegasElement is None:
return None
return [VegasElement(key=f"game:{g['id']}",
image=self._card(g), # cache by fingerprint
version=self._fingerprint(g)) # changes iff pixels would
for g in self.games]
```
#### `get_vegas_segment_width() -> Optional[int]` `VegasElement(key, image, version=None, live=True, refresh_hz=0.0)`:
For `FIXED_SEGMENT` plugins, the number of *panels* the segment | Field | Meaning |
occupies in the scroll (pixel width = panels × `single_panel_width`, |---|---|
from `display.hardware.cols`). `None` uses the default of 1 panel. | `key` | Names the element across redraws; unique in the list, stable for the same logical item (`"game:nfl:401547417"`, `"map"`). |
| `image` | The element now, at the display's height. A live element's **width must not depend on its data**: a redraw at another width is never swapped in (it appears the next time the plugin comes round), because nothing on screen may move. |
| `version` | Anything hashable that changes exactly when the pixels would. Handed back with the **same image object** as last time, it lets the ticker skip converting the element; a new image is always converted and compared by its pixels, so a redraw for new settings is never missed. `None` means "compare pixels". |
| `live` | `False` places it as plain content (trimmed, never refreshed): separators, decoration. |
| `refresh_hz` | For content that changes with **time** rather than data (an aircraft moving between position reports): the ticker calls `redraw_vegas_element()` about this often while the element is on or near the screen, capped by `vegas_scroll.live_max_hz` and at 1 Hz without the rebuilt rgbmatrix binding. |
**`get_vegas_elements() -> Optional[List[VegasElement]]`** — called on the
ticker's background thread under the plugin's lock (never while `update()`
runs), on a canvas of its own and told its render width, exactly like
`get_vegas_content()`. It is called after every `update()` while any of the
plugin's elements is on or ahead of the screen, so it must be cheap when
nothing changed (cache images by version), idempotent, and must not fetch.
Return `None` to use `get_vegas_content()`, which a plugin must keep working
for older cores and for the paths that do not ask for elements (the ticker's
first strip, multi-display sync, the `live_refresh` switch).
**`redraw_vegas_element(key, width, height, at) -> Optional[PIL.Image]`** —
only for elements with `refresh_hz`. Called **without** the plugin's lock,
possibly while `update()` runs, so read only state `update()` replaces in one
assignment (an immutable snapshot), never state it mutates in place. `at` is
the `time.monotonic()` the pixels are expected on the panel: draw the element
as it should look then. Return exactly `width` x `height`, or `None` to skip
the tick.
**`notify_vegas_data_changed()`** — data that arrives outside `update()` (a
background thread, a push callback) calls this so the ticker redraws without
waiting for the next `update()`. Safe from any thread.
Live elements are never trimmed to their ink: the ticker pads each with
`content_padding` black columns either side, the margin trimming would have
left. A single element wider than the plugin's width budget
(`vegas_max_width_screens`, not counting that padding) is cropped like any
other content and scrolls by as plain, no longer live. The user can turn them off per plugin with `vegas_live: false` (a
core-owned property) or for the whole ticker with
`display.vegas_scroll.live_refresh: false`; they are always off under
multi-display sync.
`scripts/check_plugin.py` checks the contract for any plugin that implements
the hook (unique keys, height, width stable with no new data, redraw size,
slow calls) and prints a `vegas elements` row; the checks are in
`src/plugin_system/testing/vegas.py`. `test/fixtures/plugins/vegas-live-stub`
is a small working example.
#### Legacy: `get_vegas_content_type()` and `get_vegas_display_mode()`
Superseded by participation, and still read to derive it when neither the
user nor the manifest declares one (step 3 above). Only two answers ever
mattered: `get_vegas_content_type()` returning `'none'`, and
`get_vegas_display_mode()` returning `VegasDisplayMode.STATIC`.
- `get_vegas_content_type()` returns `'multi'`, `'static'` or `'none'`
(default `'static'`).
- `get_vegas_display_mode()` returns a `VegasDisplayMode` member (not a
string — the string `'static'` never paused anything). The default reads
the plugin's `vegas_mode` config value (`"scroll"`, `"fixed"` or
`"static"`), else maps content type `'multi'` to `SCROLL` and anything
else to `FIXED_SEGMENT`.
`SCROLL` and `FIXED_SEGMENT` (and `vegas_mode` `"scroll"` and `"fixed"`)
have always behaved identically: both scroll. The distinction is deprecated
and goes away in 3.9.0 — see [Deprecated APIs](#deprecated-apis).
#### Deprecated: `get_supported_vegas_modes()` and `get_vegas_segment_width()`
Never read by core, and removed in 3.9.0: calling the `BasePlugin`
implementation logs a deprecation warning. A plugin's own override keeps
working for the plugin itself. `get_vegas_segment_width()` read the
`vegas_panel_count` config value, which has never affected Vegas — a card's
width comes from `get_vegas_content()` and `vegas_width_pct`.
> The full source for `BasePlugin` lives in > The full source for `BasePlugin` lives in
> `src/plugin_system/base_plugin.py`. If a method here disagrees with the > `src/plugin_system/base_plugin.py`. If a method here disagrees with the
@@ -477,18 +628,6 @@ self.display_manager.update_display()
This is the canonical way to render arbitrary images. This is the canonical way to render arbitrary images.
### Weather Icons (deprecated)
> Deprecated, removed in 3.7.0 — draw your own icons (the weather plugin
> ships `WeatherIcons`). See [Deprecated APIs](#deprecated-apis).
- `draw_weather_icon(condition, x, y, size=16)` — icon for a condition
string such as `"clear"`, `"clouds"`, `"rain"`, `"snow"`, `"storm"`
- `draw_sun(x, y, size=16)`, `draw_cloud(x, y, size=16, color=(200, 200, 200))`,
`draw_rain(x, y, size=16)`, `draw_snow(x, y, size=16)`
- `draw_text_with_icons(text, icons=None, x=None, y=None, color=(255, 255, 255))`
— text plus a list of `(icon_type, x, y)` icons; calls `update_display()`
### Scrolling State Management ### Scrolling State Management
For plugins that implement scrolling content, use these methods to coordinate with the display system. For plugins that implement scrolling content, use these methods to coordinate with the display system.
@@ -579,20 +718,6 @@ Process any deferred updates if not currently scrolling. Called automatically by
**Note**: Plugins typically don't need to call this directly. **Note**: Plugins typically don't need to call this directly.
#### `get_scrolling_stats() -> dict`
> Deprecated, removed in 3.7.0. See [Deprecated APIs](#deprecated-apis).
Get current scrolling statistics for debugging.
**Returns**: Dictionary with scrolling state information
**Example**:
```python
stats = self.display_manager.get_scrolling_stats()
self.logger.debug(f"Scrolling: {stats['is_scrolling']}, Deferred: {stats['deferred_count']}")
```
### Available Fonts ### Available Fonts
The Display Manager provides several pre-loaded fonts: The Display Manager provides several pre-loaded fonts:
@@ -722,27 +847,6 @@ Get data with automatic strategy detection from cache key.
data = self.cache_manager.get_with_auto_strategy("nhl_live_scores") data = self.cache_manager.get_with_auto_strategy("nhl_live_scores")
``` ```
#### `get_background_cached_data(key: str, sport_key: Optional[str] = None) -> Optional[Dict[str, Any]]`
> Deprecated, removed in 3.7.0 — use `get()`. See [Deprecated APIs](#deprecated-apis).
Get background service cached data with sport-specific intervals.
**Parameters**:
- `key` (str): Cache key
- `sport_key` (str, optional): Sport identifier (e.g., 'nhl', 'nba') for live interval lookup
**Returns**: Cached data, or `None` if not found or stale
**Example**:
```python
# Uses sport-specific live_update_interval from config
games = self.cache_manager.get_background_cached_data(
"nhl_games",
sport_key="nhl"
)
```
### Strategy Methods ### Strategy Methods
#### `get_cache_strategy(data_type: str, sport_key: Optional[str] = None) -> Dict[str, Any]` #### `get_cache_strategy(data_type: str, sport_key: Optional[str] = None) -> Dict[str, Any]`
@@ -761,23 +865,6 @@ strategy = self.cache_manager.get_cache_strategy("sports_live", sport_key="nhl")
max_age = strategy['max_age'] # Get configured max age max_age = strategy['max_age'] # Get configured max age
``` ```
#### `get_sport_live_interval(sport_key: str) -> int`
> Deprecated, removed in 3.7.0. See [Deprecated APIs](#deprecated-apis).
Get the live_update_interval for a specific sport from config.
**Parameters**:
- `sport_key` (str): Sport identifier (e.g., 'nhl', 'nba')
**Returns**: Live update interval in seconds
**Example**:
```python
interval = self.cache_manager.get_sport_live_interval("nhl")
# Returns configured live_update_interval for NHL
```
#### `get_data_type_from_key(key: str) -> str` #### `get_data_type_from_key(key: str) -> str`
Extract data type from cache key to determine appropriate cache strategy. Extract data type from cache key to determine appropriate cache strategy.
@@ -787,17 +874,6 @@ Extract data type from cache key to determine appropriate cache strategy.
**Returns**: Inferred data type string **Returns**: Inferred data type string
#### `get_sport_key_from_cache_key(key: str) -> Optional[str]`
> Deprecated, removed in 3.7.0. See [Deprecated APIs](#deprecated-apis).
Extract sport key from cache key for sport-specific strategies.
**Parameters**:
- `key` (str): Cache key
**Returns**: Sport identifier, or `None` if not found
### Utility Methods ### Utility Methods
#### `clear_cache(key: Optional[str] = None) -> None` #### `clear_cache(key: Optional[str] = None) -> None`
@@ -835,30 +911,6 @@ for file_info in files:
self.logger.info(f"Cache: {file_info['key']}, Age: {file_info['age_display']}") self.logger.info(f"Cache: {file_info['key']}, Age: {file_info['age_display']}")
``` ```
### Metrics Methods (deprecated)
#### `get_cache_metrics() -> Dict[str, Any]`
> Deprecated, removed in 3.7.0. See [Deprecated APIs](#deprecated-apis).
Get cache performance metrics.
**Returns**: Dictionary with cache statistics (`total_requests`, `cache_hit_rate`, `background_hit_rate`, `api_calls_saved`, `average_fetch_time`, etc.)
**Example**:
```python
metrics = self.cache_manager.get_cache_metrics()
self.logger.info(f"Cache hit rate: {metrics['cache_hit_rate']:.2%}")
```
#### `get_memory_cache_stats() -> Dict[str, Any]`
> Deprecated, removed in 3.7.0. See [Deprecated APIs](#deprecated-apis).
Get memory cache statistics.
**Returns**: Dictionary with memory cache stats (size, max_size, etc.)
--- ---
## Plugin Manager ## Plugin Manager
@@ -897,14 +949,6 @@ for plugin_id, plugin in all_plugins.items():
self.logger.info(f"Plugin {plugin_id} is loaded") self.logger.info(f"Plugin {plugin_id} is loaded")
``` ```
#### `get_enabled_plugins() -> List[str]`
> Deprecated, removed in 3.7.0 — check `enabled` on the instances in `plugin_manager.plugins`. See [Deprecated APIs](#deprecated-apis).
Get list of enabled plugin IDs.
**Returns**: List of plugin identifier strings
#### `get_plugin_info(plugin_id: str) -> Optional[Dict[str, Any]]` #### `get_plugin_info(plugin_id: str) -> Optional[Dict[str, Any]]`
Get plugin information including manifest and runtime info. Get plugin information including manifest and runtime info.
@@ -1070,10 +1114,19 @@ if weather is not None and weather.enabled:
## Deprecated APIs ## Deprecated APIs
These still work in 3.6 but log a warning the first time they are called A deprecated method still works but logs a warning the first time it is
(`journalctl -u ledmatrix` shows which one), and are **removed in 3.7.0**. called (`journalctl -u ledmatrix` shows which one), until the release that
Nothing in core, the official plugins or the third-party plugins in the removes it. [DEPRECATIONS_3.8.md](DEPRECATIONS_3.8.md) is the usage scan
registry calls them. behind each removal: which of the deprecated methods the official plugins,
the registry's third-party plugins and core still call or override. Only
methods that scan reports unused are removed; the rest stay until their
callers migrate.
### Removed in 3.8.0
Deprecated in 3.5.0 with a warning on first call, and gone in 3.8.0:
the scan found no caller in any official or third-party plugin. Calling one
now raises `AttributeError`.
| Object | Methods | Instead | | Object | Methods | Instead |
|---|---|---| |---|---|---|
@@ -1086,3 +1139,17 @@ registry calls them.
| `font_manager` | `set_override`, `remove_override`, `get_overrides`, `add_font`, `remove_font`, `validate_font`, `get_size_tokens`, `get_performance_stats`, `get_manager_fonts`, `get_detected_fonts`, `get_plugin_fonts`, `unregister_plugin_fonts` | no replacement | | `font_manager` | `set_override`, `remove_override`, `get_overrides`, `add_font`, `remove_font`, `validate_font`, `get_size_tokens`, `get_performance_stats`, `get_manager_fonts`, `get_detected_fonts`, `get_plugin_fonts`, `unregister_plugin_fonts` | no replacement |
| `plugin_manager` | `get_enabled_plugins` | check `enabled` on the entries in `plugin_manager.plugins` | | `plugin_manager` | `get_enabled_plugins` | check `enabled` on the entries in `plugin_manager.plugins` |
### Removed in 3.9.0
The Vegas APIs that described a fixed-width segment, which Vegas never
implemented. Vegas participation (`'scroll'`, `'pause'`, `'exclude'`, see
[Vegas scroll hooks](#vegas-scroll-hooks)) replaces them. Calling one of the
methods, or setting `vegas_panel_count`, logs a warning once per process.
No official plugin calls them; calendar, olympics and blackjack override
`get_supported_vegas_modes()`, which keeps working for the plugin itself.
| What | Instead |
|---|---|
| `BasePlugin.get_supported_vegas_modes()` | declare `vegas_participation` in the manifest |
| `BasePlugin.get_vegas_segment_width()` and the `vegas_panel_count` config key | nothing: a card's width comes from `get_vegas_content()` and `vegas_width_pct` |
| `VegasDisplayMode.SCROLL` vs `FIXED_SEGMENT` (`vegas_mode` `"scroll"` vs `"fixed"`) | `'scroll'` participation; the two always behaved the same |
+9 -1
View File
@@ -124,6 +124,14 @@ Plugins are configured by adding their plugin ID as a top-level key in the confi
} }
``` ```
How often the core calls a plugin's `update()`: the plugin's
`get_update_interval()` if it returns a number, else `update_interval` in the
plugin's `manifest.json`, else `update_interval` in its `config.json` section
as above, else 60 seconds. A config `update_interval` therefore only sets the
scheduler's cadence for a plugin whose manifest does not; plugins that expose
it in their config schema typically also honour it themselves inside
`update()`. See [PLUGIN_API_REFERENCE.md](PLUGIN_API_REFERENCE.md#get_update_interval---optionalfloat).
### Plugin Display Durations ### Plugin Display Durations
Add plugin display modes to the `display_durations` section: Add plugin display modes to the `display_durations` section:
@@ -194,7 +202,7 @@ plugin-repos/
``` ```
The Plugin Store refuses a manifest that lacks any of `id`, `name`, The Plugin Store refuses a manifest that lacks any of `id`, `name`,
`class_name` or `display_modes` (`store_manager.py`); the loader itself `class_name` or `display_modes` (`store_install.py`); the loader itself
needs `class_name`. `version` is not required, but the store compares it needs `class_name`. `version` is not required, but the store compares it
with the registry's `latest_version` to offer updates, so set it. with the registry's `latest_version` to offer updates, so set it.
`entry_point` defaults to `manager.py` if omitted. The config schema is not `entry_point` defaults to `manager.py` if omitted. The config schema is not
+1 -1
View File
@@ -203,7 +203,7 @@ Forms are rendered on the server, not generated in the browser:
from the schema (widgets named by `x-widget` are rendered by the scripts in from the schema (widgets named by `x-widget` are rendered by the scripts in
`web_interface/static/v3/js/widgets/`) `web_interface/static/v3/js/widgets/`)
4. **Save Configuration** posts the form to `/api/v3/plugins/config` 4. **Save Configuration** posts the form to `/api/v3/plugins/config`
(`web_interface/blueprints/api_v3/plugins.py`), which validates it against (`web_interface/blueprints/api_v3/plugin_config.py`), which validates it against
the schema, writes `config.json` (secret fields go to the schema, writes `config.json` (secret fields go to
`config_secrets.json`) and shows a notification `config_secrets.json`) and shows a notification
+3 -3
View File
@@ -32,7 +32,7 @@
│ • masks x-secret fields │ │ • masks x-secret fields │
│ • renders partials/plugin_config.html (render_field macros) │ │ • renders partials/plugin_config.html (render_field macros) │
│ │ │ │
│ api_v3 blueprint (blueprints/api_v3/plugins.py) │ │ api_v3 blueprint (blueprints/api_v3/plugin_config.py) │
│ save_plugin_config() POST /api/v3/plugins/config │ │ save_plugin_config() POST /api/v3/plugins/config │
│ get_plugin_config() GET /api/v3/plugins/config │ │ get_plugin_config() GET /api/v3/plugins/config │
│ get_plugin_schema() GET /api/v3/plugins/schema │ │ get_plugin_schema() GET /api/v3/plugins/schema │
@@ -91,7 +91,7 @@ validatePluginConfigForm() (client-side checks)
POST /api/v3/plugins/config?plugin_id=<id> (form data, all fields of the form) POST /api/v3/plugins/config?plugin_id=<id> (form data, all fields of the form)
│ │
▼ ▼
save_plugin_config() (api_v3/plugins.py) save_plugin_config() (api_v3/plugin_config.py)
├─→ Start from the stored config.json[<id>] ├─→ Start from the stored config.json[<id>]
├─→ Apply form fields: dotted names → nested keys, "[]" checkbox ├─→ Apply form fields: dotted names → nested keys, "[]" checkbox
│ groups → lists, values coerced to the schema's types │ groups → lists, values coerced to the schema's types
@@ -175,7 +175,7 @@ Implement `on_config_change(new_config)` in the plugin (see
|---------|------| |---------|------|
| Tab partial loader | `web_interface/blueprints/pages_v3.py` (`_load_plugin_config_partial`) | | Tab partial loader | `web_interface/blueprints/pages_v3.py` (`_load_plugin_config_partial`) |
| Form template and field macros | `web_interface/templates/v3/partials/plugin_config.html` | | Form template and field macros | `web_interface/templates/v3/partials/plugin_config.html` |
| Save / get / schema / reset handlers | `web_interface/blueprints/api_v3/plugins.py` | | Save / get / schema / reset handlers | `web_interface/blueprints/api_v3/plugin_config.py` |
| Schema loading, defaults, validation | `src/plugin_system/schema_manager.py` | | Schema loading, defaults, validation | `src/plugin_system/schema_manager.py` |
| Secret masking and splitting | `src/web_interface/secret_helpers.py` | | Secret masking and splitting | `src/web_interface/secret_helpers.py` |
| Widgets | `web_interface/static/v3/js/widgets/` | | Widgets | `web_interface/static/v3/js/widgets/` |
+25
View File
@@ -33,6 +33,31 @@ is `CORE_PLUGIN_PROPERTIES` in `src/plugin_system/schema_manager.py`):
- Read by `src/vegas_mode/plugin_adapter.py` and `BasePlugin`, which - Read by `src/vegas_mode/plugin_adapter.py` and `BasePlugin`, which
validate the values themselves and ignore a bad one with a log line validate the values themselves and ignore a bad one with a log line
5. **`vegas_participation`** (string enum: `"scroll"`, `"pause"`,
`"exclude"`; no default)
- Description: how this plugin takes part in Vegas mode — its content
scrolls by, the scroll pauses for its turn and shows it full screen, or
it is left out
- Overrides the plugin's own default (its manifest's
`vegas_participation`, else what its legacy Vegas hooks say); unset
means "use the plugin's default"
- Deliberately has no default: one would be written into every plugin's
config and override what each plugin declares
- Read by `resolve_vegas_participation()` in
`src/plugin_system/base_plugin.py`; see
[PLUGIN_API_REFERENCE.md](PLUGIN_API_REFERENCE.md#vegas-participation)
6. **`vegas_live`** (boolean; no default, unset means on)
- Description: for a plugin with live Vegas elements (it implements
`get_vegas_elements()`), whether the ticker changes what is already
scrolling when the plugin's data changes. `false` shows each card as it
was when drawn, as before live elements existed
- Ignored by plugins without live elements, and whenever live elements
are off for the whole ticker (`display.vegas_scroll.live_refresh`)
- Read by `PluginAdapter.is_live_capable()` in
`src/vegas_mode/plugin_adapter.py`; see
[PLUGIN_API_REFERENCE.md](PLUGIN_API_REFERENCE.md#live-vegas-elements)
`skin` and `skin_options` were core properties until the skin system was `skin` and `skin_options` were core properties until the skin system was
removed. A plugin config saved with them still loads and saves; the keys are removed. A plugin config saved with them still loads and saves; the keys are
dropped on the next save (see `RETIRED_PLUGIN_KEYS` in `schema_manager.py`). dropped on the next save (see `RETIRED_PLUGIN_KEYS` in `schema_manager.py`).
+5 -7
View File
@@ -5,11 +5,9 @@
A plugin can name an icon for its tab in the web interface's second nav row A plugin can name an icon for its tab in the web interface's second nav row
(next to **Plugin Manager**) with the `icon` field in `manifest.json`. (next to **Plugin Manager**) with the `icon` field in `manifest.json`.
> **Status:** the tab code honors `icon`, but `GET /api/v3/plugins/installed` `GET /api/v3/plugins/installed` passes the manifest's `icon` through (a
> (`web_interface/blueprints/api_v3/plugins.py`) does not currently include non-string value comes back as `null`), and a plugin without one gets the
> the manifest's `icon` in its response, so every tab shows the default default puzzle piece.
> puzzle piece. Setting `icon` is harmless and will take effect once the API
> passes it through again.
## Font Awesome classes only ## Font Awesome classes only
@@ -55,8 +53,8 @@ With no `icon` (or an empty one) the tab shows `fas fa-puzzle-piece`.
or misspelled class renders as a blank space. or misspelled class renders as a blank space.
2. Include the style prefix (`fas`, `far` or `fab`) as well as the icon 2. Include the style prefix (`fas`, `far` or `fab`) as well as the icon
class. class.
3. See the status note above: the icon is currently not passed through by 3. The manifest is re-read on each plugin list load; reload the page after
the API. editing `icon`.
## Related Documentation ## Related Documentation
+2 -2
View File
@@ -25,7 +25,7 @@ which runs as root.** Anything installed only into another user's
The web interface is not root, so it installs through a narrow sudo helper: The web interface is not root, so it installs through a narrow sudo helper:
1. `PluginStoreManager._install_dependencies()` 1. `PluginStoreManager._install_dependencies()`
(`src/plugin_system/store_manager.py`) calls (`src/plugin_system/store_install.py`) calls
`install_requirements_file()` (`src/common/permission_utils.py`). `install_requirements_file()` (`src/common/permission_utils.py`).
2. That runs `sudo -n bash scripts/fix_perms/safe_pip_install.sh <plugin>/requirements.txt`. 2. That runs `sudo -n bash scripts/fix_perms/safe_pip_install.sh <plugin>/requirements.txt`.
The helper checks the path is the project's own `requirements.txt` or a The helper checks the path is the project's own `requirements.txt` or a
@@ -154,7 +154,7 @@ For more, see the [Plugin Dependency Troubleshooting Guide](PLUGIN_DEPENDENCY_TR
## Files to Reference ## Files to Reference
- Service units: `systemd/ledmatrix.service`, `systemd/ledmatrix-web.service` - Service units: `systemd/ledmatrix.service`, `systemd/ledmatrix-web.service`
- Store installs: `src/plugin_system/store_manager.py` (`_install_dependencies`) - Store installs: `src/plugin_system/store_install.py` (`_install_dependencies`)
- Root install helper: `src/common/permission_utils.py` (`install_requirements_file`), `scripts/fix_perms/safe_pip_install.sh` - Root install helper: `src/common/permission_utils.py` (`install_requirements_file`), `scripts/fix_perms/safe_pip_install.sh`
- Load-time installs: `src/plugin_system/plugin_loader.py` (`install_dependencies`) - Load-time installs: `src/plugin_system/plugin_loader.py` (`install_dependencies`)
- Sudo rules: `scripts/install/lib_sudoers.sh` (written by `first_time_install.sh` - Sudo rules: `scripts/install/lib_sudoers.sh` (written by `first_time_install.sh`
+2 -5
View File
@@ -519,15 +519,12 @@ When developing plugins, you'll need to use the APIs provided by the LEDMatrix s
- `draw_text()` - Text rendering. For images, paste directly onto - `draw_text()` - Text rendering. For images, paste directly onto
`display_manager.image` (a PIL Image) and call `update_display()`; `display_manager.image` (a PIL Image) and call `update_display()`;
there is no `draw_image()` helper method. there is no `draw_image()` helper method.
- `draw_weather_icon()`, `draw_sun()`, `draw_cloud()` - Weather icons
(deprecated, removed in 3.7.0 — draw your own icons)
- `get_text_width()`, `get_font_height()` - Text utilities - `get_text_width()`, `get_font_height()` - Text utilities
- `set_scrolling_state()`, `defer_update()` - Scrolling state management - `set_scrolling_state()`, `defer_update()` - Scrolling state management
**Cache Manager** (`self.cache_manager`): **Cache Manager** (`self.cache_manager`):
- `get()`, `set()`, `delete()` - Basic caching - `get()`, `set()`, `delete()` - Basic caching
- `get_cached_data_with_strategy()` - Advanced caching with strategies - `get_cached_data_with_strategy()` - Advanced caching with strategies
- `get_background_cached_data()` - deprecated, removed in 3.7.0 — use `get()`
**Plugin Manager** (`self.plugin_manager`): **Plugin Manager** (`self.plugin_manager`):
- `get_plugin()`, `get_all_plugins()` - Access other plugins - `get_plugin()`, `get_all_plugins()` - Access other plugins
@@ -535,7 +532,7 @@ When developing plugins, you'll need to use the APIs provided by the LEDMatrix s
See [PLUGIN_API_REFERENCE.md](PLUGIN_API_REFERENCE.md) for complete See [PLUGIN_API_REFERENCE.md](PLUGIN_API_REFERENCE.md) for complete
documentation, and its [Deprecated APIs](PLUGIN_API_REFERENCE.md#deprecated-apis) documentation, and its [Deprecated APIs](PLUGIN_API_REFERENCE.md#deprecated-apis)
table for everything removed in 3.7.0. table for everything removed in 3.8.0.
## 3rd Party Plugin Development ## 3rd Party Plugin Development
@@ -645,7 +642,7 @@ To have your plugin added to the official plugin store:
3. **Contact maintainers** (own-repository plugins): 3. **Contact maintainers** (own-repository plugins):
- Open a GitHub issue in the [ledmatrix-plugins](https://github.com/ChuckBuilds/ledmatrix-plugins) repository - Open a GitHub issue in the [ledmatrix-plugins](https://github.com/ChuckBuilds/ledmatrix-plugins) repository
- Or reach out on Discord: https://discord.gg/uW36dVAtcT - Or reach out on Discord: https://discord.gg/RdrC37rEag
- Include: Repository URL, plugin description, why it's useful - Include: Repository URL, plugin description, why it's useful
4. **Review process**: 4. **Review process**:
+4
View File
@@ -56,6 +56,8 @@ Going deeper:
- [ADVANCED_FEATURES.md](ADVANCED_FEATURES.md) — Vegas scroll, on-demand display, - [ADVANCED_FEATURES.md](ADVANCED_FEATURES.md) — Vegas scroll, on-demand display,
cache management, background services, permissions cache management, background services, permissions
- [FONT_MANAGER.md](FONT_MANAGER.md) — font system - [FONT_MANAGER.md](FONT_MANAGER.md) — font system
- [SCROLL_PERFORMANCE.md](SCROLL_PERFORMANCE.md) — how scrolling is paced, and how to make a plugin's marquee smooth
- [OFFSCREEN_RENDERING.md](OFFSCREEN_RENDERING.md) — rendering plugin content off the render thread
- [PERMISSIONS.md](PERMISSIONS.md) — file ownership, sudo rules, repair scripts - [PERMISSIONS.md](PERMISSIONS.md) — file ownership, sudo rules, repair scripts
- [MQTT bridge](../integrations/mqtt_bridge/README.md) — control the display from Home Assistant over MQTT - [MQTT bridge](../integrations/mqtt_bridge/README.md) — control the display from Home Assistant over MQTT
@@ -70,6 +72,8 @@ Going deeper:
## Contributing to LEDMatrix itself ## Contributing to LEDMatrix itself
- [ARCHITECTURE.md](ARCHITECTURE.md) — processes, display loop, plugin system, web UI; where to start reading - [ARCHITECTURE.md](ARCHITECTURE.md) — processes, display loop, plugin system, web UI; where to start reading
- [WEB_FRONTEND_ARCHITECTURE.md](WEB_FRONTEND_ARCHITECTURE.md) — the web UI's ES modules, page lifecycle and form model, and the page-by-page migration to them
- [IPC_CONTROL_SOCKET.md](IPC_CONTROL_SOCKET.md) — the display's control socket: protocol, security model, stage plan
- [DEVELOPMENT.md](DEVELOPMENT.md) — environment setup - [DEVELOPMENT.md](DEVELOPMENT.md) — environment setup
- [HOW_TO_RUN_TESTS.md](HOW_TO_RUN_TESTS.md) — running the test suite - [HOW_TO_RUN_TESTS.md](HOW_TO_RUN_TESTS.md) — running the test suite
- [MULTI_ROOT_WORKSPACE_SETUP.md](MULTI_ROOT_WORKSPACE_SETUP.md) — multi-repo workspace - [MULTI_ROOT_WORKSPACE_SETUP.md](MULTI_ROOT_WORKSPACE_SETUP.md) — multi-repo workspace
+340 -35
View File
@@ -18,6 +18,39 @@ top level instead of under `data` (install-from-url, registry-from-url, the
auth endpoints, upload endpoints, `system/git-info`, `system/check-update`), auth endpoints, upload endpoints, `system/git-info`, `system/check-update`),
the entry below says so. the entry below says so.
**Cross-site requests are refused.** A `POST`, `PUT`, `PATCH` or `DELETE`
carrying an `Origin` header (or, without one, a `Referer`) that is not the
host the request was sent to gets `403` with `"error_code":
"CROSS_SITE_REQUEST"`; so does `Origin: null`. This stops other websites from
driving the Pi through a LAN user's browser. Scripts, curl, Home Assistant and
the MQTT bridge send neither header and are unaffected. A browser page on
another origin (a dashboard you host elsewhere, say) can no longer call the
API; call it server-side instead. Behind a reverse proxy, pass the original
`Host` through, port included (nginx: `proxy_set_header Host $http_host;`;
`$host` drops the port) -- `X-Forwarded-Host` is not read.
**Authentication (optional, off by default).** With no web password set,
nothing below needs credentials. Once one is set (General > Security, or
[`POST /auth/password`](#web-login-and-api-tokens)), every route needs a login
session or an API token:
```bash
curl -H "Authorization: Bearer lmx_..." http://your-pi-ip:5000/api/v3/display/current
```
Without either, an API route answers `401` with
`{"status": "error", "error_code": "AUTH_REQUIRED", "message": ...}`, a
`WWW-Authenticate: Bearer realm="LEDMatrix"` header and the login page's URL
in `X-LEDMatrix-Login`; an unknown or revoked token gets `"error_code":
"INVALID_TOKEN"`. Browser page loads are redirected to `/login` instead, and
HTMX requests get `401` with `HX-Redirect: /login?...`. Never asked for
credentials: requests from the Pi itself (loopback, with no `X-Forwarded-For`,
`X-Real-IP`, `Forwarded` or `X-Forwarded-Host` header), `/static/*`, the
captive-portal probe URLs, `/login`, `/api/v3/health` (status only, see
[Health Check](#health-check)), and -- only while the Pi is in access-point
mode -- `/setup`, `GET /wifi/status`, `GET /wifi/scan` and
`POST /wifi/connect`.
## Table of Contents ## Table of Contents
- [Configuration](#configuration) - [Configuration](#configuration)
@@ -35,18 +68,20 @@ the entry below says so.
- [Health and Status](#health-and-status) - [Health and Status](#health-and-status)
- [Schedule (dim/power)](#schedule-dimpower) - [Schedule (dim/power)](#schedule-dimpower)
- [Integrations](#integrations) - [Integrations](#integrations)
- [Web login and API tokens](#web-login-and-api-tokens)
- [Plugin-specific endpoints](#plugin-specific-endpoints) - [Plugin-specific endpoints](#plugin-specific-endpoints)
- [Starlark Apps](#starlark-apps) - [Starlark Apps](#starlark-apps)
> The API blueprint is the `api_v3` package in > The API blueprint is the `api_v3` package in
> `web_interface/blueprints/api_v3/` (one module per area: `config.py`, > `web_interface/blueprints/api_v3/` (one module per area: `config.py`,
> `display.py`, `plugins.py`, `system.py`, `backup.py`, `fonts.py`, > `display.py`, `system.py`, `backup.py`, `fonts.py`, `misc.py`, `wifi.py`,
> `misc.py`, `wifi.py`, `starlark.py`). `web_interface/app.py` registers it > `starlark.py`, and `plugins.py` plus the `plugin_*.py` modules for the
> plugin routes). `web_interface/app.py` registers it
> at `/api/v3` (`app.register_blueprint(api_v3, url_prefix='/api/v3')`). > at `/api/v3` (`app.register_blueprint(api_v3, url_prefix='/api/v3')`).
> The three SSE endpoints (`/api/v3/stream/*`) are defined directly on the > The three SSE endpoints (`/api/v3/stream/*`) are defined directly on the
> Flask app in `app.py` (`stream_stats`, `stream_display`, `stream_logs`). > Flask app in `app.py` (`stream_stats`, `stream_display`, `stream_logs`).
> `test/fixtures/api_v3_url_map.json` is the canonical list of blueprint > `test/fixtures/api_v3_url_map.json` is the canonical list of blueprint
> routes (116 URL rules); a test fails if the code and that fixture differ. > routes; a test fails if the code and that fixture differ.
--- ---
@@ -119,10 +154,17 @@ there an unchecked checkbox — which the browser omits — is saved as
```json ```json
{ {
"status": "success", "status": "success",
"message": "Configuration saved successfully" "message": "Configuration saved successfully",
"restart_required": true
} }
``` ```
`restart_required` is always true here: display hardware, rotation,
durations and general settings take effect when the display restarts, and
the web UI shows its restart banner on the flag. (Plugin sections saved
through this route reach the running plugin live, like
`POST /plugins/config`.)
Invalid values (e.g. an out-of-range `target_fps`, a hardware option the Invalid values (e.g. an out-of-range `target_fps`, a hardware option the
Raspberry Pi 5 driver cannot use) are rejected with `400` and nothing is Raspberry Pi 5 driver cannot use) are rejected with `400` and nothing is
saved. saved.
@@ -212,7 +254,10 @@ times `07:00`-`23:00`. At least one day must be enabled.
Retrieve `config/config_secrets.json` with every set value replaced by eight Retrieve `config/config_secrets.json` with every set value replaced by eight
bullet characters (`"••••••••"`). Empty values and `YOUR_*` placeholders bullet characters (`"••••••••"`). Empty values and `YOUR_*` placeholders
are returned as-is, so a client can tell "set" from "not set". are returned as-is, so a client can tell "set" from "not set". The
`web_auth` section (the web login's password hash, API-token hashes and
cookie key) is left out entirely; [Get Main Configuration](#get-main-configuration)
leaves it out too.
**Response**: **Response**:
```json ```json
@@ -237,7 +282,8 @@ Replace `config/config.json` with the JSON body (advanced use only).
Save the secrets file (advanced use only). Masked values (`"••••••••"`) Save the secrets file (advanced use only). Masked values (`"••••••••"`)
and blank strings in the body are dropped, and the rest is merged onto the and blank strings in the body are dropped, and the rest is merged onto the
stored secrets, so posting back the GET response unchanged changes nothing. stored secrets, so posting back the GET response unchanged changes nothing.
A secret cannot be cleared by blanking it here. A secret cannot be cleared by blanking it here. A `web_auth` key in the body
is ignored; the stored login settings are kept.
--- ---
@@ -389,7 +435,7 @@ Request a specific plugin to display on-demand.
- `mode` (string, optional): Display mode name (plugin_id inferred if not provided) - `mode` (string, optional): Display mode name (plugin_id inferred if not provided)
- `duration` (number, optional): Duration in seconds (0 = until stopped) - `duration` (number, optional): Duration in seconds (0 = until stopped)
- `pinned` (boolean, optional): Pin display (pause rotation) - `pinned` (boolean, optional): Pin display (pause rotation)
- `start_service` (boolean, optional): (Re)start the display service so it picks the request up (default: true) - `start_service` (boolean, optional): Start the display service if it is not running (default: true). A running service is never restarted: it picks the request up within about a quarter of a second. When false and the service is stopped, the route returns 400.
**Response**: **Response**:
```json ```json
@@ -401,13 +447,23 @@ Request a specific plugin to display on-demand.
"mode": "nfl_live", "mode": "nfl_live",
"duration": 45, "duration": 45,
"pinned": true, "pinned": true,
"service": { "active": true, "returncode": 0, "stdout": "", "stderr": "" } "service": { "active": true, "returncode": 0, "stdout": "", "stderr": "" },
"transport": "socket"
} }
} }
``` ```
`service` is `null` when `start_service` is false. `service` is `null` when `start_service` is false.
`transport` says how the request reached the display: `"socket"` means the
display's control socket acknowledged it (it is queued for the render thread;
see [IPC_CONTROL_SOCKET.md](IPC_CONTROL_SOCKET.md)), `"mailbox"` means it was
written to the cache mailbox the display polls, as before the socket existed.
With `"mailbox"`, `socket_error` gives the reason the socket was not used
(`no_socket` when the display is stopped or predates the socket, `timeout`,
`refused`, `busy`, ...). Either way the request is applied the same way;
`request_id` is the same id in both.
### Stop On-Demand Display ### Stop On-Demand Display
**POST** `/api/v3/display/on-demand/stop` **POST** `/api/v3/display/on-demand/stop`
@@ -430,11 +486,14 @@ Stop the current on-demand display.
"status": "success", "status": "success",
"data": { "data": {
"request_id": "uuid-here", "request_id": "uuid-here",
"service": null "service": null,
"transport": "socket"
} }
} }
``` ```
`transport` and `socket_error` are as for start.
--- ---
## Plugins ## Plugins
@@ -464,21 +523,65 @@ List all installed plugins with their status and metadata.
"enabled": true, "enabled": true,
"verified": true, "verified": true,
"loaded": true, "loaded": true,
"state": "loaded", "state": "enabled",
"error_info": null, "error_info": null,
"loaded_version": "1.2.3",
"loaded_at": 1790000000.0,
"last_updated": "2025-01-15T10:30:00Z", "last_updated": "2025-01-15T10:30:00Z",
"last_commit": "abc1234", "last_commit": "abc1234",
"last_commit_message": "feat: Add live game updates", "last_commit_message": "feat: Add live game updates",
"branch": "main", "branch": "main",
"web_ui_actions": [], "web_ui_actions": [],
"vegas_mode": null, "vegas_mode": null,
"vegas_content_type": null "vegas_content_type": null,
"vegas_participation": "scroll",
"vegas_participation_source": "manifest"
} }
] ],
"runtime": {
"status": "live",
"published_at": 1790000030.0,
"age_seconds": 12.4,
"stale_after": 180.0
}
} }
} }
``` ```
Metadata comes from each plugin's files on disk; `enabled` is the plugin's
`enabled` flag in `config.json` (missing means disabled, as the display
reads it). `vegas_mode` is the plugin's configured `vegas_mode`, or `null`.
`loaded`, `state`, `error_info`, `loaded_version` and `loaded_at` come from
the runtime snapshot the display publishes (the web process runs no plugin
code). `state` is the display's lifecycle state (`loaded` while loading,
`enabled`, `disabled`, `error`, `unloaded`); `error_info` is `null` or
`{"type", "message", "at", "recoverable"}`, with the message redacted and at
most 200 characters (the full error is at `/errors/*`). `loaded_version` is
the version the display loaded, which differs from `version` after an update
until the display restarts. A plugin a live snapshot does not list is
`loaded: false`, `state: "unloaded"`.
`runtime.status` says whether to believe them: `live` (fresh snapshot from
a running display), `stale` (not refreshed within `stale_after` seconds: the
display is hung or died), `stopped` (the display shut down) or `unknown`
(nothing published yet). Unless it is `live`, every one of those fields is
`null`. Health and metrics are at [`/plugins/health`](#get-plugin-health)
and `/plugins/metrics`.
`vegas_participation` is what Vegas mode does with the plugin: `"scroll"`,
`"pause"` or `"exclude"` (see
[PLUGIN_API_REFERENCE.md](PLUGIN_API_REFERENCE.md#vegas-participation)),
and `vegas_participation_source` says where it came from. The web reads it
the way the display resolves it, as far as files can tell: the user's own
`vegas_participation` setting (`"config"`), else the manifest's declared
`vegas_participation` (`"manifest"`). Past those the display derives it from
the plugin's code -- a `get_vegas_participation()` override or the legacy
Vegas hooks -- which the web process never runs, so `vegas_participation`
is `null` and the source is `"runtime"`. A plugin that overrides
`get_vegas_participation()` decides at run time and can differ from its
manifest's declaration. `vegas_content_type` is always `null`.
### Get Plugin Configuration ### Get Plugin Configuration
**GET** `/api/v3/plugins/config?plugin_id=<plugin_id>` **GET** `/api/v3/plugins/config?plugin_id=<plugin_id>`
@@ -638,7 +741,19 @@ Install a plugin from the plugin store.
``` ```
When the operation queue is unavailable the install runs synchronously and When the operation queue is unavailable the install runs synchronously and
the response has only a `message`. the response has only a `message` and the restart fields below.
The finished operation's `result` (from `/plugins/operation/<operation_id>`)
carries `restart_required`: true when the plugin is already enabled in
`config.json`, because the running display does not load newly installed
files by itself; `restart_message` then holds the restart banner's wording.
A plugin that is not enabled needs no restart: enabling it loads it.
A plugin whose registry entry (or downloaded manifest) needs a newer
LEDMatrix is refused: the synchronous install answers `409` with a message
such as `Failed to install plugin x: X requires LEDMatrix 3.8.0 or newer…`,
and a queued one fails with that message. Nothing already installed is
changed.
### Uninstall Plugin ### Uninstall Plugin
@@ -663,6 +778,11 @@ Remove an installed plugin.
} }
``` ```
The finished operation's `result` carries `restart_required`. Removing the
plugin's config (the default) lets the display unload it by itself, so it is
false; with `preserve_config: true` an enabled plugin keeps running until
the display restarts, and it is true.
### Update Plugin ### Update Plugin
**POST** `/api/v3/plugins/update` **POST** `/api/v3/plugins/update`
@@ -683,11 +803,21 @@ Update a plugin to the latest version. Runs synchronously.
"message": "Plugin football-scoreboard updated ...", "message": "Plugin football-scoreboard updated ...",
"data": { "data": {
"last_updated": "2025-01-15T10:30:00Z", "last_updated": "2025-01-15T10:30:00Z",
"commit": "abc1234..." "commit": "abc1234...",
} "update_status": "updated"
},
"restart_required": true,
"restart_message": "Plugin updated — restart the display to run the new version"
} }
``` ```
`update_status` is `updated`, `up_to_date` or `local_only`.
`restart_required` is true when the plugin changed and is enabled: the
running display keeps the code it loaded until it restarts.
An update this core cannot run answers `409` with `Plugin update refused:`
and the reason; the installed version is left as it was.
### Install Plugin from URL ### Install Plugin from URL
**POST** `/api/v3/plugins/install-from-url` **POST** `/api/v3/plugins/install-from-url`
@@ -716,10 +846,13 @@ Install a plugin directly from a GitHub repository URL. Runs synchronously.
"message": "Plugin my-plugin installed successfully", "message": "Plugin my-plugin installed successfully",
"plugin_id": "my-plugin", "plugin_id": "my-plugin",
"name": "My Plugin", "name": "My Plugin",
"branch": "main" "branch": "main",
"restart_required": false
} }
``` ```
`restart_required` follows the same rule as `/plugins/install`.
### Load Registry from URL ### Load Registry from URL
**POST** `/api/v3/plugins/registry-from-url` **POST** `/api/v3/plugins/registry-from-url`
@@ -869,7 +1002,13 @@ Get a plugin's resource limits. `data` is `null` when none are configured.
**POST** `/api/v3/plugins/limits/<plugin_id>` **POST** `/api/v3/plugins/limits/<plugin_id>`
Set a plugin's resource limits. The body replaces all four limits: a key you Set a plugin's resource limits. The body replaces all four limits: a key you
omit is stored as no limit (`warning_threshold` defaults to `0.8`). omit is stored as no limit (`warning_threshold` defaults to `0.8`). Each value
must be a non-negative number or `null`; anything else is a 400.
The limits are stored in the shared cache. A display service that has already
read limits for the plugin keeps using those until it restarts; likewise the
health and metrics reset routes clear the stored record and the web process's
copy, not the display service's in-memory state.
**Request Body**: **Request Body**:
```json ```json
@@ -885,8 +1024,11 @@ omit is stored as no limit (`warning_threshold` defaults to `0.8`).
**GET** `/api/v3/plugins/state` **GET** `/api/v3/plugins/state`
Get the state manager's record for every plugin, keyed by plugin id. Pass Every plugin that is installed or configured, keyed by plugin id: desired
`?plugin_id=<id>` for one plugin (`data` is then that record). state from `config.json` and the plugins directory, observed state from the
display's runtime snapshot. Built per request; there is no state file.
Pass `?plugin_id=<id>` for one plugin (`data` is then that record; 404 if
it is neither installed nor configured).
**Response**: **Response**:
```json ```json
@@ -895,23 +1037,41 @@ Get the state manager's record for every plugin, keyed by plugin id. Pass
"data": { "data": {
"football-scoreboard": { "football-scoreboard": {
"plugin_id": "football-scoreboard", "plugin_id": "football-scoreboard",
"status": "loaded", "status": "enabled",
"installed": true,
"in_config": true,
"enabled": true, "enabled": true,
"version": "1.2.3", "version": "1.2.3",
"loaded": true,
"state": "enabled",
"error_info": null,
"loaded_version": "1.2.3",
"loaded_at": 1790000000.0,
"installed_at": "2025-01-15T10:30:00", "installed_at": "2025-01-15T10:30:00",
"last_updated": "2025-01-15T10:30:00", "last_updated": "2025-01-15T10:30:00"
"config_version": 1,
"metadata": {}
} }
} },
"runtime": {"status": "live", "published_at": 1790000030.0, "age_seconds": 12.4, "stale_after": 180.0}
} }
``` ```
`status` is `enabled` / `disabled` for an installed plugin, `unknown` for
one that is configured but not installed, and `error` when the display
reports its state as `error`. `installed_at` and `last_updated` are the
newest successful install, and install or update, in the operation history
(`null` when it has none). The runtime fields follow the same rule as
[`/plugins/installed`](#get-installed-plugins): `null` unless
`runtime.status` is `live`.
### Reconcile Plugin State ### Reconcile Plugin State
**POST** `/api/v3/plugins/state/reconcile` **POST** `/api/v3/plugins/state/reconcile`
Reconcile plugin state across config, disk and the state manager. Reconcile desired state (`config.json` plus the plugins on disk) with the
display's runtime snapshot. Desired-state gaps are fixed (a plugin on disk
with no config section is added disabled); observed-state gaps -- enabled
but not loaded, loaded at an older version than is installed -- are
reported with `fix_action: "no_action"`.
**Request Body** (optional): **Request Body** (optional):
```json ```json
@@ -1182,13 +1342,22 @@ searches.
"version": "1.2.3", "version": "1.2.3",
"branch": "main", "branch": "main",
"default_branch": "main", "default_branch": "main",
"plugin_path": "plugins/football-scoreboard" "plugin_path": "plugins/football-scoreboard",
"commit": "843588025a81197056f8d96779ccb2be19337ab8",
"ledmatrix_min_version": "3.7.0",
"aliases": [],
"incompatible_reason": null
} }
] ]
} }
} }
``` ```
`commit` (the monorepo commit that introduced `version`),
`ledmatrix_min_version` and `aliases` come from the registry entry and are
`null` / `[]` when an older registry lacks them. `incompatible_reason` is the
message an install would be refused with on this core, or `null`.
### Get GitHub Status ### Get GitHub Status
**GET** `/api/v3/plugins/store/github-status` **GET** `/api/v3/plugins/store/github-status`
@@ -1329,20 +1498,59 @@ Get LEDMatrix repository version.
**GET** `/api/v3/system/check-update` **GET** `/api/v3/system/check-update`
Whether `origin/main` has commits the checkout lacks. Cached briefly. Whether newer code is available on this device's update channel. On
Fields at the top level (no envelope): `stable` that is a newer release tag than the checkout (`target_version`
names it); on `beta`, and on `stable` while it waits on a branch for a
release that contains the current commit, it is commits on `origin/main`
the checkout lacks. A detached checkout newer than the newest release is
never offered an update: Update Code leaves it where it is until a release
includes it, and `channel_message` says so in the General tab's words.
Cached briefly. Fields at the top level (no envelope):
```json ```json
{ {
"update_available": true, "update_available": true,
"remote_sha": "abc123...", "remote_sha": "abc123...",
"commits_behind": 3 "commits_behind": 3,
"target_version": "v3.8.0",
"channel": "stable",
"configured_channel": "stable",
"waiting": false,
"newest_release": "v3.8.0",
"current_release": null,
"channel_message": "Stable: release v3.8.0 is available."
} }
``` ```
When git cannot run the check, the response also carries When git cannot run the check, the response also carries
`"check_failed": true` and an `error` explaining why. `"check_failed": true` and an `error` explaining why.
### Update Channel
**GET** `/api/v3/system/update-channel`
The update channel and what the next Update Code or weekly update would do
(in `data`): `configured` (`"stable"`, `"beta"` or `null` for a config from
before channels), `channel` (the one in effect), `waiting` (stable, but the
device is newer than the newest release, so it follows `main` for now),
`action` (`none`, `checkout_tag`, `pull` or `switch_to_beta`),
`newest_release`, `current_release`, `branch` (`""` when on a release tag),
`message`. Reads local refs; `?fetch=1` fetches from origin first.
**POST** `/api/v3/system/update-channel`
```json
{
"channel": "beta"
}
```
Saves `auto_update.channel`. The next update applies it; switching to
`stable` never installs an older version than the one running, and the
`message` says when the device keeps following `main` until a newer release.
400 for anything but `stable` or `beta`. The General tab form also accepts
`auto_update_channel` on `POST /api/v3/config/main`.
### Automatic Update Status ### Automatic Update Status
**GET** `/api/v3/system/auto-update` **GET** `/api/v3/system/auto-update`
@@ -1368,7 +1576,9 @@ Hide the current automatic-update alert until a new one replaces it.
Branch, dirty state, recent commits and remote for the Tools tab. Fields at Branch, dirty state, recent commits and remote for the Tools tab. Fields at
the top level: `branch`, `dirty`, `status`, `recent_commits`, `remote_url` the top level: `branch`, `dirty`, `status`, `recent_commits`, `remote_url`
(credentials scrubbed), `upstream`, `can_pull`. (credentials scrubbed), `upstream`, `can_pull`, and for the update channel
`detached`, `version` (`git describe`), `current_release` (the release tag
HEAD is exactly on, else `null`) and `channel_message` (detached only).
### Git Branches ### Git Branches
@@ -1381,7 +1591,10 @@ Fetches `origin` and lists branches to switch to: `current`, `upstream`,
**POST** `/api/v3/system/action` **POST** `/api/v3/system/action`
Execute system-level actions. JSON or form data. Execute system-level actions. Send JSON (`Content-Type: application/json`).
A form-encoded or `text/plain` body is accepted only with an `HX-Request`
header (HTMX sends it; a cross-site HTML form cannot) and is otherwise
refused with `415`.
**Request Body**: **Request Body**:
```json ```json
@@ -1942,6 +2155,18 @@ Health of the web interface, display service, config file, plugin system and
display snapshot. `data.status` is `healthy` or `degraded`, with display snapshot. `data.status` is `healthy` or `degraded`, with
`data.services` and `data.checks`. `data.services` and `data.checks`.
`data.checks.display_loop` is the display's render-loop heartbeat: `running`
(with `heartbeat_age_seconds`), `stalled` (no heartbeat for 60s: the panel is
frozen even if the service is active; the status turns `degraded`), or
`not_reported` when the display writes none (not started yet, the dev server,
Windows), which does not affect the status.
Open even when the web login is on, for uptime monitors; a caller that is not
logged in (and has no token) then gets only `{"status": "success", "data":
{"status": "healthy" | "degraded"}}`. A stalled render loop still shows there
as `degraded`; the `checks` detail is only for logged-in callers, tokens and
requests from the Pi itself.
### Hardware Status ### Hardware Status
**GET** `/api/v3/hardware/status` **GET** `/api/v3/hardware/status`
@@ -2007,20 +2232,100 @@ enabled.
Home Assistant MQTT bridge service state and settings: `data.service`, Home Assistant MQTT bridge service state and settings: `data.service`,
`data.config_exists`, `data.config_path`, `data.config` (password `data.config_exists`, `data.config_path`, `data.config` (password
omitted), `data.password_set`, `data.env_override_prefix`. omitted), `data.password_set`, `data.api_token_set`, `data.env_override_prefix`.
**PUT** `/api/v3/integrations/mqtt-bridge/config` **PUT** `/api/v3/integrations/mqtt-bridge/config`
Write `integrations/mqtt_bridge/bridge_config.json`. Only the keys you send Write `integrations/mqtt_bridge/bridge_config.json`. Only the keys you send
change. The password is write-only: omit `mqtt_password` to keep it, send a change. The password is write-only: omit `mqtt_password` to keep it, send a
value to replace it, or send `"clear_password": true`. A password with value to replace it, or send `"clear_password": true`. The web-login API token
the bridge sends (`ledmatrix_api_token`, needed only when login is on and the
bridge runs on another machine) is write-only the same way, cleared with
`"clear_api_token": true`. A password with
`mqtt_tls` off is refused unless `allow_insecure_mqtt` is true. Returns `mqtt_tls` off is refused unless `allow_insecure_mqtt` is true. Returns
`data.password_set` and `data.restart_required` (the bridge must be `data.password_set`, `data.api_token_set` and `data.restart_required` (the
restarted to pick up changes). See bridge must be restarted to pick up changes). See
[integrations/mqtt_bridge/README.md](../integrations/mqtt_bridge/README.md). [integrations/mqtt_bridge/README.md](../integrations/mqtt_bridge/README.md).
--- ---
## Web login and API tokens
The optional password and API tokens (`web_auth` in
`config/config_secrets.json`, `web_interface/auth.py`). No route here returns
the password hash, a token hash or the cookie key. A request authenticated by
an API token gets `403` `TOKEN_NOT_ALLOWED` from every route in this section:
tokens are for integrations, not for changing who can log in. Wrong current
passwords (`403` `WRONG_PASSWORD`) count against the same per-address limit as
the login page: 5 a minute, 30 an hour, then `429`.
Lost password: run `sudo python3 scripts/reset_web_password.py` on the Pi.
### Login status
**GET** `/api/v3/auth/status`
```json
{
"status": "success",
"data": {
"enabled": true,
"signed_in": true,
"access": "session",
"min_password_length": 8,
"tokens": [
{"id": "3f9c1a2b4d5e6f70", "name": "Home Assistant", "prefix": "lmx_Ab3d",
"created_at": "2026-09-29T20:14:03+00:00"}
]
}
}
```
`access` is how this request got in: `open` (login off), `session`,
`localhost`, `ap-setup` or `token`.
### Set or change the password
**POST** `/api/v3/auth/password`
Body: `{"new_password": "...", "current_password": "..."}`.
`current_password` is required once login is on. At least 8 characters, no
leading or trailing space (`400` `WEAK_PASSWORD`). Setting the first password
turns login on. Every existing login session ends; the caller's own browser
is signed in again with the answer.
### Turn login off
**POST** `/api/v3/auth/disable`
Body: `{"current_password": "..."}`. Removes the password; API tokens are
kept (and are needed again if login is turned back on).
### API tokens
**GET** `/api/v3/auth/tokens` — `data.tokens`, as in the status answer.
**POST** `/api/v3/auth/tokens` — body `{"name": "Home Assistant"}` (1-60
characters). Answers `201` with `data.token`, the token itself (`lmx_` plus 43
characters), and `data.record`. **The token is never shown again**; only its
SHA-256 is stored. At most 50 tokens.
**DELETE** `/api/v3/auth/tokens/<id>` — revoke; it stops working on the next
request. `404` for an unknown id.
Send a token as `Authorization: Bearer <token>`.
### Login page
`GET /login` shows the login form (and redirects home when login is off or
this browser is already signed in); `POST /login` with a form field `password`
(and optional `next`, a path on this server) signs in and redirects to `next`,
or answers `401` with the form again. `POST /logout` ends the session. Both
are outside `/api/v3` and go through the cross-site check like every other
`POST`.
---
## Plugin-specific endpoints ## Plugin-specific endpoints
A handful of endpoints belong to individual plugins. The music plugin's A handful of endpoints belong to individual plugins. The music plugin's
+7
View File
@@ -342,6 +342,7 @@ service's user.
| **wait** | Time blocked in `SwapOnVSync`, i.e. the slack left in each refresh. A p50 near zero means the rig has no headroom and anything extra lands a frame late. | | **wait** | Time blocked in `SwapOnVSync`, i.e. the slack left in each refresh. A p50 near zero means the rig has no headroom and anything extra lands a frame late. |
| **work** | Everything else between two frames: drawing, scrolling, and waiting for the GIL. A wide gap between its p50 and p99 is another thread getting in the way. | | **work** | Everything else between two frames: drawing, scrolling, and waiting for the GIL. A wide gap between its p50 and p99 is another thread getting in the way. |
| **Binding** | `STOCK` means the rgbmatrix binding holds the GIL through the vsync wait, which starves every other thread. See *Rebuilding the binding*. | | **Binding** | `STOCK` means the rgbmatrix binding holds the GIL through the vsync wait, which starves every other thread. See *Rebuilding the binding*. |
| **after work** | Frames presented straight after tagged render-thread work, with their own late rate: `extend` and `compose` (Vegas building its strip), `patch` (live elements, once they land). A kind whose late rate sits well above the overall one is the work making frames late. Shown only when something tagged its work. |
The refresh rate is estimated from the frames themselves (swaps that block on The refresh rate is estimated from the frames themselves (swaps that block on
vsync can only land on refresh boundaries). Cross-check it with vsync can only land on refresh boundaries). Cross-check it with
@@ -406,6 +407,10 @@ sudo python3 scripts/render_bench.py --speed 50 # a held (frame_hold 2) spe
sudo python3 scripts/render_bench.py --busy 2 # with threads imitating plugin updates sudo python3 scripts/render_bench.py --busy 2 # with threads imitating plugin updates
sudo python3 scripts/render_bench.py --json /tmp/pi4-512x64.json sudo python3 scripts/render_bench.py --json /tmp/pi4-512x64.json
# render-thread strip work, each tagged so the report gives it a late rate:
sudo python3 scripts/render_bench.py --patch-bytes 101376 --patch-every 25 # a live map patch
sudo python3 scripts/render_bench.py --strip-screens 30 --extend-every-screens 6 # Vegas extensions
sudo systemctl start ledmatrix sudo systemctl start ledmatrix
``` ```
@@ -468,6 +473,8 @@ refreshes" comes from.
| `duplicate` | frames that advanced no pixels. A crisp fixed-step scroll should show none; any at all means the loop is presenting faster than the strip is moving. | | `duplicate` | frames that advanced no pixels. A crisp fixed-step scroll should show none; any at all means the loop is presenting faster than the strip is moving. |
| `blank` | frames with no visible slice to draw: the helper had no content. Should be zero. | | `blank` | frames with no visible slice to draw: the helper had no content. Should be zero. |
| `restarts` | how many times the strip was scrolled through end to end. Informational: the bench restarts the strip where a plugin would hand over to the next one. | | `restarts` | how many times the strip was scrolled through end to end. Informational: the bench restarts the strip where a plugin would hand over to the next one. |
| `patches` | `--patch-bytes N --patch-every K`: N bytes of columns written into the strip in place every K frames, on screen or (`--patch-where ahead`) just past it -- what a live element update costs the render thread. Their frames are the `patch` row under *after work*. |
| `extensions` | `--extend-every-screens N`: a block appended and the scrolled-past columns trimmed every N screens, as continuous Vegas does. The cost is a copy of the whole strip, so size it like Vegas's with `--strip-screens` (8,000-20,000px). Their frames are the `extend` row. |
`--json` writes the full report plus the panel geometry, the solved speed and `--json` writes the full report plus the panel geometry, the solved speed and
these counters, so two rigs (or one rig before and after a change) can be these counters, so two rigs (or one rig before and after a change) can be
+325 -52
View File
@@ -35,10 +35,11 @@ defaults, or as capabilities they opt into.
### Reusability — write once, nine plugins benefit ### Reusability — write once, nine plugins benefit
Only code that is **identical in intent across all nine** moves into the base Only code that is **identical across every plugin that carries it** moves into
class. That set is small and knowable — it is exactly the methods present in every core. Stages 0–3 moved the copies that already were; what is left has drifted,
copy today (phase B1 below). Everything else stays where it is until it earns and earns promotion by being reconciled first — made identical in all nine
promotion. plugins, one method family per release, with every visible difference decided
rather than averaged away. See [Roadmap](#roadmap).
### Modularity — a change to one feature cannot reach a plugin that doesn't use it ### Modularity — a change to one feature cannot reach a plugin that doesn't use it
@@ -81,6 +82,15 @@ more. Shared sports code lives in `src/common`:
| `sports_shared.py` | 3.3.0 | `SportsCoreSharedMixin`, `SportsLiveSharedMixin`, `SportsRecentSharedMixin` — the sport-independent `sports.py` methods | | `sports_shared.py` | 3.3.0 | `SportsCoreSharedMixin`, `SportsLiveSharedMixin`, `SportsRecentSharedMixin` — the sport-independent `sports.py` methods |
| `sports_helpers.py` | 3.5.0 | clamp/logo/rotation free functions and `SportsHelpersMixin`, plus the `_favorite_key` seam | | `sports_helpers.py` | 3.5.0 | clamp/logo/rotation free functions and `SportsHelpersMixin`, plus the `_favorite_key` seam |
| `espn_dates.py` | 3.5.0 | ESPN date-range and `limit` workarounds | | `espn_dates.py` | 3.5.0 | ESPN date-range and `limit` workarounds |
| `favorite_team_check.py` | 3.6.0 | `FavoriteTeamCheck` — logs why a favourite team code shows nothing |
| `sports_timezone.py` | 3.6.0 | Which timezone start times are drawn in (`resolve_timezone_name`) |
| `sports_celebration.py` | 3.7.0 | `SportsCelebrationMixin` — draws the score/win takeover; colour helpers |
| `sports_fetch.py` | 3.7.0 | `SportsFetchMixin` — season fetch, live lookback and live-odds decisions |
| `sports_card_wrappers.py` | 3.7.0 | `SportsCardWrappersMixin` — the game renderer's `sports_card` delegations |
| `sports_plugin_host.py` | next release | `SportsPluginHostMixin` — the plugin class's (`manager.py`) identical helpers: Vegas weight, off-thread switch refresh |
| `sports_live_scroll.py` | next release | `SportsLiveScrollMixin` — rebuild a live scroll strip mid-cycle, keeping the marquee's place |
| `sports_display_rules.py` | next release | `SportsCardOptionsMixin`, `SportsGameRulesMixin` — scorebug date options, the no-favourites filter, non-favourite live dwell |
| `sports_font_path.py` | next release | `resolve_font_path` — what the plugins' `_resolve_font_path` copies return |
Each is described in [src/common/README.md](../src/common/README.md). Each is described in [src/common/README.md](../src/common/README.md).
@@ -92,9 +102,10 @@ modules taken from the plugin copies, each a **new module** rather than growth
on an existing one: a plugin that deletes a method copy and relies on an older on an existing one: a plugin that deletes a method copy and relies on an older
module having gained it fails at runtime with an `AttributeError`, while a module having gained it fails at runtime with an `AttributeError`, while a
missing module fails at load, where the version checks can see it. missing module fails at load, where the version checks can see it.
`sports_helpers.py` is the newest (it holds `_favorite_key`, the override point `sports_helpers.py` holds `_favorite_key`, the override point listed below.
listed below, for later phases); its parity test compares every body against Each promoted module has a parity test that compares its bodies against the
the plugin copies when `LEDMATRIX_PLUGINS` points at a checkout, and plugin copies when `LEDMATRIX_PLUGINS` points at a checkout
(`test_sports_helpers.py`, `test_sports_stage3_parity.py`), and
`test/test_common_is_hardware_free.py` keeps `src/common` free of `test/test_common_is_hardware_free.py` keeps `src/common` free of
`rgbmatrix`, `src.display_manager` and `src.plugin_system`. How a plugin adopts a `rgbmatrix`, `src.display_manager` and `src.plugin_system`. How a plugin adopts a
module and drops its copy is documented in the plugins repo's module and drops its copy is documented in the plugins repo's
@@ -166,6 +177,15 @@ Mix it in **before** the mode class — `class SoccerLive(CelebrationMixin,
SportsLive)` — so the celebration `display()` runs first and falls through to SportsLive)` — so the celebration `display()` runs first and falls through to
the scorebug via `super()`. the scorebug via `super()`.
What shipped is narrower. `src/common/sports_celebration.py`
(`SportsCelebrationMixin`) holds only the drawing, which is identical in the
five scoreboards that celebrate (afl, football, hockey, nrl, soccer — hockey
grew celebrations after this was written). Arming a celebration stays in each
plugin: the trigger bodies differ (nrl matches favourites by team id, football
folds a touchdown's extra point into one celebration and picks scenery by
points), and so does `display()`. The seams above were not needed to move the
drawing, so none was added.
**Rotation strategies.** The three "dialects" turned out to be one algorithm **Rotation strategies.** The three "dialects" turned out to be one algorithm
(Smooth Weighted Round-Robin) in two shapes: an incremental picker holding state (Smooth Weighted Round-Robin) in two shapes: an incremental picker holding state
across calls (afl/nrl/soccer) and a precomputed per-cycle list across calls (afl/nrl/soccer) and a precomputed per-cycle list
@@ -221,12 +241,284 @@ legacy compatibility rather than the mechanism.
> (`display_manager.refresh_hz`), and speed comes from > (`display_manager.refresh_hz`), and speed comes from
> `scroll_settings.scroll_speed` alone. See `docs/SCROLL_PERFORMANCE.md`. > `scroll_settings.scroll_speed` alone. See `docs/SCROLL_PERFORMANCE.md`.
## Phases ## Roadmap
B0–B3 are merged and shipping in core 3.2.0. Everything that remains is ### Done: stages 0–3
**rollout**, and it splits into three phases with very different risk profiles.
The original plan folded the last two together; they are separated here because The second project, after the B phases below: move what the nine `sports.py`
one of them cannot break a user on an old core and the other can. copies (and their support files) carried byte-identically into `src/common`,
one new module per stage, and delete the copies once the plugins floor on the
release that ships it.
| Stage | Core | Plugins (ledmatrix-plugins) | What moved |
|---|---|---|---|
| 0 | none needed; found #662 (odds `no_odds` marker) and #663 (a reloaded plugin's dir goes first on `sys.path`) | #562 | Deleted the bundled copies nothing could reach (`base_odds_manager`, `logo_downloader`, three unused data sources, ~4.2k lines); three UFC fixes |
| 1 | 3.5.0: `sports_helpers` (#583), `espn_dates`, `json_body` | #563 (1a, the eight team scoreboards), #564 (1b, ufc); floor 3.5.0 | The identical helpers and ESPN date-range handling; ufc also adopted the `sports_shared` mixins |
| 2 | 3.6.0: `favorite_team_check`, `sports_timezone`; fixes in 3.6.1 (#667) and 3.6.2 (#670) | #565 (guarded adoption), #567 (f1), #570 (sunset, floor 3.6.1), #571 (soccer, 3.6.2) | The favourite-team check (seven copies) and the timezone resolver (ten); each plugin keeps a thin timezone binding |
| 3 | 3.7.0 (#672): `sports_celebration`, `sports_fetch`, `sports_card_wrappers` | #572 (goldens first), #574 (floor 3.7.0, copies deleted) | Celebration drawing (five plugins), four fetch methods (nine), seventeen card delegations (eight renderers) |
Stage 3 was re-checked independently when this roadmap was written: #574's
parent and #574 itself, rendered through the core harness against core 3.7.0,
gave pixel-identical output for all 399 frames (192 harness screens across the
nine plugins at the eight default sizes, 72 scroll/Vegas cards, 135
celebration frames), with a parent-vs-parent rerun as the determinism control.
### Stage 4: the identical sweep (core done; adoption waits for a release)
Re-measured on ledmatrix-plugins `56c4f15` (2026-09-30) the report still
lists 58 families identical in every copy. Stage 4 moves the ones that are
identical across the nine, or across eight with the ninth lacking the
method, into four new modules: `sports_plugin_host` (ten `manager.py`
helpers, all nine), `sports_live_scroll` (eight `manager.py` methods, every
plugin with a live strip, so not ufc), `sports_display_rules` (four
`sports.py` methods, in two mixins because their carriers differ) and
`sports_font_path`. The parity test (`test/test_sports_stage4_parity.py`)
compares each with every plugin copy using this report's own normalisation,
plus decorators and constant values, which the normalisation drops.
`_resolve_font_path` was meant to be replaced by
`font_layout.resolve_asset_path`, but that never looks in the cwd, and the
plugins' copy does first, so the swap would change which font a process
started from another checkout loads. `resolve_font_path` is the copy's
behaviour on a core that ships it, checked path for path against all 17
copies (`test/test_sports_font_path.py`).
Left in the plugins, though identical:
- `_get_timezone`, `_extract_game_details`, `_fetch_data` (nine): a
per-plugin import and the abstract contract, as in stage 3.
- `_schema_font_size`, `_resolve_font_size` (eight renderers): they read the
plugin's own `_SCHEMA_PATH`, as in stage 3.
- The 29 families carried by seven plugins or fewer: the afl/nrl/soccer
lineage's own helpers (`_swrr_advance`, `_refresh_switch_mode_managers`,
`_initialize_logo_dir`, ...), the multi-league helpers
(`_resolve_managers_for_mode`, `_extract_mode_type`, ...), and eleven
two-plugin helpers. Each is one lineage's code; most go when
family 13 or 14 reconciles the code around them. `_odds_color` (seven
renderers) is already core's, in `SportsHelpersMixin`; a renderer that
wants it can inherit that.
### Why the method changes
Byte-identical promotion has nearly run dry. Measured on ledmatrix-plugins
`4327c2e` (2026-09-29, after stage 3) with `scripts/sports_drift_report.py`:
| File | Method families | In all nine | Method lines | Identical copies beyond the first | Drifted families |
|---|---:|---:|---:|---:|---:|
| `sports.py` | 94 | 30 | 30,976 | 1,479 lines | 19 |
| `manager.py` | 114 | 36 | 27,137 | 3,198 lines | 38 |
| `game_renderer.py` (8 plugins) | 89 | — | 6,830 | 546 lines | 11 |
"Drifted" means in at least seven plugins with at least three different
bodies. Everything still identical adds up to about 5,200 duplicated lines;
the rest of the ~65,000 method lines is drifted, one outlier away from
identical, or unique to one plugin. Drifted code cannot move unchanged, so
consolidation stalls unless the copies are made identical first.
`manager.py`, the largest copy of all and the layer the display controller and
Vegas talk to, was in no plan before this one.
### The method: reconcile, then promote
**Owner decision (2026-09-29):** each release, pick one drifted method family,
make all nine copies identical, then promote it to core. A *family* here is a
set of methods that share state and ship together (the rankings methods, the
game-over check); the report measures each method in it. The procedure:
1. **Measure.** `python scripts/sports_drift_report.py --family sports.py::<name> --diff`
lists which plugins share each body and diffs every variant against the
most common one. Put the grouping in the PR.
2. **Classify every difference**, and say which class in the PR:
- *A fix one copy has and the others lack* (a lock, a guard, a correct
season year). Port it. It is a behaviour change, so it gets a CHANGELOG
line in each plugin.
- *A per-sport fact* (hockey ends in period 3; a soccer clock counts up).
Make it a declared class constant or override point with a default, as
`FINAL_PERIOD`, `CLOCK_COUNTS_DOWN`, `COALESCE_SCORING_SEQUENCE` and
`_favorite_key` are, and add it to the tables above. Never a sport-name
branch: core must not learn sport names.
- *A product difference*: anything a user can see (which games show, a
colour, a date, a badge, how long a screen stays). The owner picks the
behaviour before the code changes; the decision goes in the PR and in a
test that pins it (as `test/test_sports_twins.py` pins the twins).
- *Noise*: comments, log wording, dead branches. Pick one.
3. **Pin the output first.** Before touching the family, its output must be
covered: the harness goldens (`test/golden`), the scroll cards
(`golden-cards`) and celebrations (`golden-celebration`) for drawing
families; for logic families, a table-driven test over the nine plugins'
fixture games. Missing coverage lands in its own PR first, as #572 did for
stage 3.
4. **Reconcile in the plugins** (a monorepo PR). The report must show one
variant per class for the family. Render every touched plugin before and
after through the harness and diff pixels, not hashes. Every differing
frame must match a recorded product decision; any other difference is a
bug. Bump each plugin's `version`, add a `versions[]` entry and a CHANGELOG
entry, and run `update_registry.py`.
5. **Promote in core**: a new `src/common` module per family (a new module, not
growth on an old one, for the reason under Converging on `src/common`), a
parity test against the plugin copies, and a CHANGELOG module entry naming
the release that ships it.
6. **Adopt** once that release is out: each plugin floors on it, inherits the
mixin, deletes its copy, gains a sunset guard (like the monorepo's
`scripts/test_stage3_mixin_copies.py`), and is pixel-diffed again; the
expected difference is zero.
7. **Re-measure** and update the numbers here.
A family is only reconciled when *all nine* agree. Leaving one plugin behind
recreates the drift the report exists to measure.
Soaks: pixel diffs prove the drawing, not the timing. A family that changes
when data arrives or which games are live (5, 7, 9, 13 and 14 below) needs a
live-game soak on a rig, and out-of-season sports wait for their season.
Before a soak, check the rig's `*_display_mode`: a board in `switch` mode tells
you nothing about the scroll path.
### Order
One family per release, in this order. Variant counts are from the report
above (per method: distinct bodies across the plugins that carry it, counted
per class role). Stage 4 needs no reconciliation and can ride along with any
release.
| # | Family | Methods (variants) | Why here |
|---|---|---|---|
| 4 | Identical sweep | `manager.py`: `_dispatch_switch_refresh`, `_favorite_team_is_live`, `get_vegas_priority_weight`, `_game_involves`, `_favorite_scan_targets`, `_favorite_scan_games`, `_get_total_games_for_manager` (all nine, 1); the live-scroll helpers `_preserving_scroll_position`, `_refresh_live_scroll_managers`, `_live_scroll_managers`, `_note_live_scroll_built`, `_live_scroll_needs_rebuild`, `_live_scroll_fields` (eight, 1). `sports.py`: `_card_option`, `_filtered_or_all`, `_effective_live_duration`, `_recent_date_text` (eight, 1). 58 identical families in all | Nothing to decide; brings `manager.py` into core as a `SportsPluginHostMixin`. `_resolve_font_path` (identical in nine `sports.py` and eight renderers) becomes `sports_font_path.resolve_font_path`, not `font_layout.resolve_asset_path`, which skips the cwd. Core side done; see [Stage 4](#stage-4-the-identical-sweep-core-done-adoption-waits-for-a-release) |
| 5 | Game-over check | `SportsLive._is_game_really_over` (5) | Pure logic, no pixels; its seams (`FINAL_PERIOD`, `CLOCK_COUNTS_DOWN`) were designed in B1. The pilot for the procedure |
| 6 | Favourite matching | `_is_favorite_game` (7 across three classes), `_select_games_for_display` (2: nrl), `_select_recent_games_for_display` (3) | Everything that asks "is this a favourite" goes through the 3.5.0 `_favorite_key` seam |
| 7 | Other-games rotation | `_by_importance`, `_other_games_window`, `_advance_other_games_if_due` (2 each: football), `_rotate_other_games_on_display` (2: ufc) | One outlier each; football carries two fixes the other eight lack |
| 8 | Rankings | `_fetch_team_rankings` (3), `_choose_poll` (3), `_load_division_team_ids`, `_passes_other_filters`, `_best_rank`, `_is_ranked_game` (2 each: football) | Needs 7; the rank badge and the "ranked only" filter read it |
| 9 | Live fetch and odds | `_fetch_todays_games` (5), `_fetch_odds` (3), `_attach_odds_to_rotated_games` (3) | The prerequisite for one shared ESPN poller across plugins |
| 10 | View model | `_extract_game_details_common` (9 of 9) | Every renderer reads it; its keys are additive-only, so reconcile to the superset and leave sport extras in `_extract_game_details` |
| 11 | Switch scorebug | `_load_fonts` (4), `_load_custom_font_from_element_config` (6), `_get_layout_offset` (5), `_fit_score_font` (2: football), `_load_and_resize_logo` (9), `_draw_dynamic_odds` (9), then `_draw_scorebug_layout` (20: Upcoming 9, Recent 8, Live 2, ufc's Core 1) | Needs 10 and the twin decisions below. Several releases: fonts and offsets, logos, odds, then one mode's layout per release |
| 12 | Scroll/Vegas card | `game_renderer.py`: `render_game_card` (6), `_load_and_resize_logo` (8), `_load_custom_font`, `preload_logos`, `__init__` (7 each), `_draw_records_or_rankings` (6), `_load_fonts`, `_draw_dynamic_odds`, `_draw_live_game_status`, `_draw_recent_game_status`, `_get_team_display_text` (5 each), `_draw_text_with_outline` (2: football) | Same decisions as 11; sequence after the scroll-performance work on pre-rendered strips lands |
| 13 | Mode lifecycle | `sports.py`: `update` (23), `__init__` (16), `display` (13), `_advance_live_game_if_due` (6) | Where per-sport behaviour lives; last in `sports.py`. Each difference becomes a seam or a strategy chosen by name (live rotation already has three) |
| 14 | `manager.py` host | Nine different bodies in nine plugins: `__init__`, `display`, `update`, `has_live_content`, `get_live_modes`, `get_vegas_content`, `on_config_change`, `_initialize_managers`, `_get_available_modes`, `_get_current_manager`, `_adapt_config_for_manager`. Dynamic duration: `_evaluate_dynamic_cycle_completion` (9), `_record_dynamic_progress` (8), `get_cycle_duration` (8), `get_dynamic_duration_cap` (6), `supports_dynamic_duration`, `is_cycle_complete` (5 each), `reset_cycle_state` (4). Scroll: `_display_scroll_mode` (8), `_ensure_scroll_content_for_vegas` (8), `_collect_games_for_scroll` (7), `_should_use_scroll_mode`, `_has_any_scroll_mode` (6 each) | See below |
**`manager.py`.** Reconciling it body by body would take a release per
method. The plan is a host class in core, `SportsScoreboardPlugin(BasePlugin)`,
that takes the plugin's leagues as data (key, label, ESPN path, Live, Recent
and Upcoming classes: basketball's `manager.py` already describes its leagues
as such a table) and a typed mode key instead of the mode-name string parsing
(`endswith('_live')`, `split('_')`) every copy repeats. Order within it:
stage 4's identical helpers first; then dynamic duration, then live priority (`has_live_content`,
`get_live_modes`, `has_live_priority`), then Vegas content, then mode
resolution, then the lifecycle methods. Pilot the whole host on nrl or afl,
the smallest copies (about 1,850 lines each), with a frame soak and a
live-game soak before a second plugin moves. `get_vegas_content` is also being
changed by the scroll-performance work: coordinate before touching it.
**Held:** `data_sources.py` (nine copies; soccer's matches core's) and
`dynamic_team_resolver.py` (eight true forks, a different constructor from
core's). Effort on data fetching is better spent on the shared poller that
family 9 prepares.
### Product decisions each family needs
Owner calls to make before (or while) reconciling. Items marked *verify* are
suspected behaviour that needs a payload or a rig to confirm first.
- **5, game-over check.** Which rule each sport gets: the clock never ends a
game in afl, nrl and soccer (`CLOCK_COUNTS_DOWN = False`); hockey ends at
0:00 from period 3, basketball, football and lacrosse from period 4.
baseball and ufc share a copy that reads a missing clock as "0:00": dormant
in baseball (its games carry no `period`), and not triggered by ufc's round
breaks either. ESPN sends a break as `STATUS_END_OF_ROUND` with displayClock
`-`, not `0:00` (verified against recorded payloads; ledmatrix-plugins#580
pins it). Whatever rule ufc gets must not read `-` as `0:00`. Decide ufc's
rule: no clock rule (ESPN's `STATUS_FINAL` is the only end signal it needs;
this also closes a ~1 s window at the horn when the ticking clock reads
`0:00`), or its own final period.
- **6, favourite matching.** NRL keeps matching favourites by team id
(abbreviations collide: NEW, CAN), through `_favorite_key` rather than its
own copies of the selection methods. Six plugins log the recent-games
selection at INFO; baseball, football and ufc do not.
- **7, other-games rotation.** football advances the rotation window under
`_games_lock` (update() and display() both advance it; interleaved, a
window of games is skipped) and fixes a favourites-only pool that recomposed
the list on every frame. Port both. ufc does not attach odds to fights
rotated in: decide whether rotated fights show odds.
- **8, rankings.** (a) afl, basketball, nrl and soccer turn a *standings*
payload into ranks (a pro league's standings position becomes the rank
badge); baseball, hockey, lacrosse, ufc and football do not. Which is
right is visible on every pro-league card with "show ranking" on.
(b) football also keys ranks by team id, so two schools sharing an
abbreviation across divisions cannot be confused: adopt for all.
(c) football asks for the division roster of the *season* year (July
onward is this year's season), which is right for football and wrong for
college basketball, hockey and lacrosse, whose ESPN season is the year it
ends: a per-sport seam, not football's constant. (d) baseball's
`_choose_poll` calls `dynamic_team_resolver.choose_top_division_poll`, the
others inline it: one home, in core.
- **9, live fetch and odds.** (a) afl, basketball, nrl and soccer cache the
live scoreboard for 30 s under `<sport_key>_scoreboard_current`; the others
do not (the harness fixtures seed that key, so the change shows up there).
(b) basketball fetches college games with no `dates` parameter, citing a
404 (*verify* now that `espn_dates` handles ranges). (c) odds are fetched
three ways: blocking (hockey, lacrosse), a thread waited on for 1.5–2 s
(six plugins), or fire-and-forget for upcoming games (basketball). This
sets how long `update()` takes and when an odds line appears. (d) nrl
guards on a missing odds manager; port it.
- **10, view model.** Per key, whether every sport emits it. Additive only:
no key is renamed or removed.
- **11 and 12, the scorebug and the card.** The pinned divergences in
`test/test_sports_twins.py`, where switch mode and scroll mode draw the
same game differently:
- weekday timezone: the card reads only `config["timezone"]` and falls back
to UTC, so a board with only the global zone labels an evening kickoff
with the next day. **Decided 2026-09-24: use the plugin's timezone
(fix); not yet implemented;**
- an out-of-range start time: the scorebug drops the weekday, the card
raises;
- favourite result on a nested payload, which score wins when flat and
nested disagree, and where the favourites come from (the manager's list
vs the game's stamped list plus config);
- the element vocabulary (`team_text` vs `team_name`; rank and odds in one
map, not the other), and its consequences: a `team_name` colour reaching
one team face and not the other, and the odds face shared with the score
face in scroll mode only;
- per-mode colour overrides, which apply in switch mode only;
- by design, kept unless the owner says otherwise: the date format
(`switch_date_format` "numeric" vs the card's "abbrev") and the upcoming
centre (`switch_upcoming_center` "date_time" vs "vs"), both with an
"inherit" opt-in; and the two schema-font caches (per class vs per path).
Also: football's `_fit_score_font` swaps to the narrow score face at any
panel height when the score overflows, where the other seven keep the
design face at or below the design height (a 64x32 board shows the
difference); and whether switch mode and the card become one renderer drawn
at two sizes.
- **13, mode lifecycle.** The live-rotation dialect per sport (incremental
SWRR in afl, nrl and soccer; a precomputed schedule elsewhere); which sports
arm celebrations and on what (stays in each plugin, as in stage 3).
- **14, `manager.py`.** Dynamic-duration semantics (what completes a cycle,
the floor and cap per mode), what counts as live content for live priority
(favourites only or any live game), and the order of Vegas content.
### Measuring progress
`scripts/sports_drift_report.py` prints the numbers above for any
ledmatrix-plugins checkout (`--plugins <path>` or `LEDMATRIX_PLUGINS`). CI runs
it on every push and PR against the monorepo's main (the "Sports drift report"
job in `.github/workflows/test.yml`): report only, never failing, with the
tables in the job summary and the full JSON as an artifact. The monorepo's
`scripts/check_sports_drift.py` is the gate: it fails when a function that
agrees across the plugins starts to differ. A stage is done when its family
shows one variant per class here and its copies are gone.
```
python scripts/sports_drift_report.py --plugins ../ledmatrix-plugins
python scripts/sports_drift_report.py --family sports.py::_is_game_really_over --diff
```
## Phases B0–B6 (history)
The first project: it moved the scroll orchestration into core and proved the
upgrade path (floors, the store's compatibility gate, the sunset). All seven
phases are done. They are kept because the reasoning in B4–B6 is what every
later stage relies on; the plan from here is [Roadmap](#roadmap).
B0–B3 shipped in core 3.2.0. The rollout after them split into three phases
with very different risk profiles, because one of them cannot break a user on
an old core and the other can.
| Phase | Scope | Status | Gate | | Phase | Scope | Status | Gate |
|---|---|---|---| |---|---|---|---|
@@ -261,8 +553,12 @@ a floor can be trusted against, and today it is not:
the update path that re-downloads. the update path that re-downloads.
`update_plugin`'s git branch pulls in place and re-downloads nothing, so it `update_plugin`'s git branch pulls in place and re-downloads nothing, so it
stayed ungated until `_gate_pulled_commit` closed it — checked after the pull stayed ungated until `_gate_pulled_commit` closed it — checked after the pull
(the registry carries no floor field, so the incoming floor is unknowable (the registry then carried no floor field, so the incoming floor was
before it) and undone with `git reset --hard` to the pre-pull commit. That unknowable before it) and undone with `git reset --hard` to the pre-pull
commit. The registry now publishes `ledmatrix_min_version`, and install and
update refuse on it before downloading or pulling; both post-download gates
remain as the fallback for older registries, other branches and
`compatible_versions`. That
route is rare in practice, since monorepo plugins install as archives; it was route is rare in practice, since monorepo plugins install as archives; it was
closed because the sunset rule in the plugins repo's closed because the sunset rule in the plugins repo's
`08-shared-sports-code.md` states as **condition 3** that the core enforces `08-shared-sports-code.md` states as **condition 3** that the core enforces
@@ -425,10 +721,13 @@ deprecated `ledmatrix_min`). See
order any floor-raising tool must reproduce — and note the name is **inverted** order any floor-raising tool must reproduce — and note the name is **inverted**
between the top level and `versions[]`. between the top level and `versions[]`.
**Still not adopted, deliberately:** `data_sources.py`, `game_renderer.py` and **The modules held back then** (`data_sources.py`, `game_renderer.py`,
`base_odds_manager.py`. The standing decision held them until B6 closed; it now `base_odds_manager.py`) have since gone different ways: the eight team
has, so they can be reconsidered — with B5's lesson applied, which is to build scoreboards import core's `base_odds_manager` (ufc keeps an MMA fork), the
the object and diff rendered output rather than trust a static check. game renderers inherit core's `SportsCardWrappersMixin` (3.7.0) but keep their
drawing, and `data_sources.py` is still copied. Their status is under
[Roadmap](#roadmap). B5's lesson applies to all of them: build the object and
diff rendered output rather than trust a static check.
### B5 retrospective — what the adoption actually cost ### B5 retrospective — what the adoption actually cost
@@ -471,40 +770,12 @@ and its one delivered user-visible gain was that adopted plugins honoured the
global `target_fps` instead of hardcoding ~100 FPS (since withdrawn: see the global `target_fps` instead of hardcoding ~100 FPS (since withdrawn: see the
note under the B3 design above). note under the B3 design above).
### Decision: stop adopting further modules until B6 closes ### Decision: stop adopting further modules until B6 closes (lifted)
`data_sources.py` (9 copies), `game_renderer.py` (8) and `base_odds_manager.py` Held from B5 until B6 ran on 2026-09-01: each adoption added a second copy to
are the obvious next candidates. **Do not adopt them yet.** Each adoption adds keep in step against a payoff that depended on the sunset. Once the store
carrying cost — a second copy to keep in step — against a payoff that is refused a too-new plugin on every route, adopting and sunsetting in one stage
contingent on B6, and B6 is gated on an installed base we cannot currently became safe, and stages 0–3 under [Roadmap](#roadmap) did exactly that.
measure. Consolidate what is already committed; revisit when B6 does.
## What's next
Steps 1–5 of the original plan are **done**: 3.2.0 is tagged and published with
a version number CI now asserts (#428), the compatibility gate is in
`install_plugin` and reads `compatible_versions` as well as the floor
(#431, #433), the newest manifest entry is required to use `ledmatrix_min_version`
(plugins #244), and all eight plugins have adopted the scroll orchestration
(plugins #245–#249, repaired in #251, tidied in #252).
What actually remains, smallest first:
1. **Soak the adoptions on hardware.** football and hockey have been run on a
live rig through real games; baseball was watched through one earlier. The
rest are proven by harness, unit tests and pixel comparison. Out-of-season
sports cannot be soaked until their season starts. When you do, **check the
rig's `*_display_mode` first** — a board in `switch` mode will happily load a
sunset plugin and tell you nothing about the scroll code the sunset changed.
2. **Cut 3.3.0.** Not required by B6 — its floors are 3.2.0, which is released —
but `calendar` 1.2.3 floors at 3.3.0 for the device-authorization endpoints
that landed after 3.2.0, so it is un-installable until the release exists.
3. **Reconsider the held modules** (`data_sources.py`, `game_renderer.py`,
`base_odds_manager.py`) now that the sunset has closed. `game_renderer.py` is
the largest single duplication left: ~11,500 lines across eight plugins, with
~36,500 more in the eight `sports.py`. The `src/base_classes/sports/`
package promoted in B1/B2 was never imported by a plugin and has been
removed, so the plugin copies are the only starting point.
## How to keep this project healthy ## How to keep this project healthy
@@ -531,7 +802,9 @@ Lessons this migration paid for, worth applying beyond it:
## Rules for contributors ## Rules for contributors
- **Promote on evidence, not intuition.** A method moves to core when every copy - **Promote on evidence, not intuition.** A method moves to core when every copy
has it and they agree on intent. Otherwise it stays in the plugins. that has it is identical. Drifted copies are reconciled first, one family
per release, with each visible difference an owner decision (see
[Roadmap](#roadmap)); until then they stay in the plugins.
- **Never add a sport name to core.** If core needs to know which sport it is, - **Never add a sport name to core.** If core needs to know which sport it is,
the design is wrong — add an override point instead. the design is wrong — add an override point instead.
- **A capability that is not opted into must not execute.** If you find yourself - **A capability that is not opted into must not execute.** If you find yourself
+108
View File
@@ -295,6 +295,42 @@ sudo systemctl cat ledmatrix-web | grep User
--- ---
#### Issue: Updates and the update channel
**Symptoms:**
- The General tab says "Stable: this device runs code newer than the newest
release ... keeps following main"
- Tools shows a version such as `v3.8.0` instead of a branch name, or `git
status` over SSH says `HEAD detached at v3.8.0`
- Update Code says "already up to date" while GitHub's `main` has newer commits
**Explanation:** these are the Stable update channel working as intended
(`auto_update.channel`, General → Update Channel). Stable installs the
newest release tag, which git checks out without a branch ("detached
HEAD"); that is normal and every update path handles it. Stable never
installs an older version than the one running, so a device that is ahead of
the newest release keeps following `main` until a release includes its
commit, then switches to releases on its own.
**Solutions:**
1. **Want the newest code instead?** Set Update Channel to **Beta** and click
Update Code. The device leaves the release for `main` and pulls it.
Or from SSH:
```bash
curl -X POST http://localhost:5000/api/v3/system/update-channel \
-H 'Content-Type: application/json' -d '{"channel": "beta"}'
```
2. **See what the next update will do:**
```bash
curl 'http://localhost:5000/api/v3/system/update-channel?fetch=1'
```
3. **Local changes after a channel switch:** edits that no longer fit the new
version are kept in the git stash rather than lost; `git stash list`
shows them as "LEDMatrix autostash before update".
---
### WiFi & AP Mode Issues ### WiFi & AP Mode Issues
#### AP Mode Not Activating #### AP Mode Not Activating
@@ -516,6 +552,64 @@ sudo systemctl cat ledmatrix-web | grep User
python3 scripts/check_plugin.py --plugin plugin-id python3 scripts/check_plugin.py --plugin plugin-id
``` ```
#### Panel Frozen, or the Display Restarts Every Few Minutes
**Symptoms:**
- The panel stops changing while `systemctl status ledmatrix` says `active`
- The display restarts on its own, a couple of minutes after it froze
- `/api/v3/health` shows `checks.display_loop.status` as `stalled`
The display's render loop checks in with systemd every few seconds
(`WatchdogSec=120` in `ledmatrix.service`) and writes a heartbeat to
`/run/ledmatrix/display-heartbeat.json`. When the loop gets stuck -- almost
always inside one plugin's `display()` -- the check-ins stop, and after two
minutes systemd kills and restarts the display. The kill dumps every thread's
stack into the log, so it says which plugin was stuck.
**Solutions:**
1. **Find the stuck plugin.** Look for the watchdog kill and the stack dump
after it. The render loop is the thread whose stack runs through
`display_controller.py` in `run` (usually the `Current thread` block);
the first `plugin-repos/...` file in it is the plugin:
```bash
sudo journalctl -u ledmatrix --since "1 hour ago" | grep -A40 "Watchdog timeout"
```
2. **Check the heartbeat by hand.** Its age should stay under about ten
seconds while the display runs:
```bash
cat /run/ledmatrix/display-heartbeat.json
curl -s http://localhost:5000/api/v3/health | python3 -m json.tool | grep -A3 display_loop
```
`not_reported` means the display writes no heartbeat: it has not drawn
its first frame yet, or it runs an older version.
3. **Disable the plugin** in the web UI and report it to its author with the
stack dump. Restarts that repeat back off from 10 seconds to two minutes
apart, so a plugin that hangs on every start does not restart the display
hundreds of times an hour.
4. **Is the watchdog installed?** Installs from before it keep their old unit
until the installer is re-run (a startup warning says the unit differs
from its template):
```bash
systemctl show -p WatchdogUSec ledmatrix # 2min once running; 0 = not installed
sudo ./scripts/install/install_service.sh
```
`WatchdogUSec` reads `15min` for the first minutes after a start: that is
the start-up allowance, narrowed to two minutes once the first frame is on
the panel.
5. **A plugin that legitimately blocks longer** than two minutes (it should
not; `display()` runs on the render thread) can be given more time with a
drop-in, `sudo systemctl edit ledmatrix`:
```ini
[Service]
WatchdogSec=300
```
`WatchdogSec=0` turns the watchdog off.
#### Stale Cache Data #### Stale Cache Data
**Symptoms:** **Symptoms:**
@@ -616,6 +710,15 @@ sudo systemctl cat ledmatrix-web | grep User
``` ```
**Note:** Minimum recommended: 300 seconds (5 minutes) **Note:** Minimum recommended: 300 seconds (5 minutes)
How often the core calls the plugin's `update()` comes from the plugin
itself first: its `get_update_interval()` if it has one, then
`update_interval` in its `manifest.json`. The `update_interval` in
`config.json` is used by the scheduler only when the manifest sets none.
Many plugins also read their own config `update_interval` and skip the
API call inside `update()` until it has elapsed, which is what makes the
setting above effective; check the plugin's settings form or
`config_schema.json` for the option it actually honours.
2. **Check current rate limit usage:** 2. **Check current rate limit usage:**
- OpenWeatherMap free tier: 1,000 calls/day, 60 calls/minute - OpenWeatherMap free tier: 1,000 calls/day, 60 calls/minute
- With 300s interval: 288 calls/day (well within limits) - With 300s interval: 288 calls/day (well within limits)
@@ -943,6 +1046,11 @@ git reset --hard HEAD~1
# Or rollback to specific commit # Or rollback to specific commit
git reset --hard <commit-hash> git reset --hard <commit-hash>
# On the Stable update channel HEAD is a release tag, not a branch:
# go back to an earlier release instead (the next update moves forward again)
git tag --list 'v*' --sort=-v:refname | head
git checkout --detach v3.7.0
# Restart all services # Restart all services
sudo systemctl restart ledmatrix sudo systemctl restart ledmatrix
sudo systemctl restart ledmatrix-web sudo systemctl restart ledmatrix-web
+284
View File
@@ -0,0 +1,284 @@
# Web frontend architecture
This page covers where the web UI's JavaScript is going and how it gets
there one page at a time. The UI is Flask + HTMX + Alpine.js. Templates
live in `web_interface/templates/v3/` and static files in
`web_interface/static/v3/`.
Two rules hold at every step:
- **The Pi never builds anything.** It serves the files that are committed.
CI builds the generated CSS (`scripts/build_css.py`, see #685) and checks
it. The JavaScript needs no build at all: it is native ES modules that the
browser loads as they are.
- **Every page keeps working, and so does every plugin.** Third-party plugin
forms, `x-widget` scripts and plugin web UIs use the existing `window.*`
names. Each name keeps working as an alias until a release announces that
it will be removed.
## Where it started
- About 195 `window.*` globals. Their load order is held together by comments
repeated in the headers of `app-early.js`, `app-shell.js` and
`plugins_manager.js`.
- About 5,700 lines of inline `<script>` in the tab partials.
`js/htmx-config.js` re-runs every one of them after every htmx swap, so
each partial's code had to cope with running twice.
- The installed-plugin list is kept in four places.
- Plugin config forms are drawn by the `render_field` macro in
`partials/plugin_config.html`, which is about 1,100 lines of Jinja. It
duplicates the JS widgets. The server then needs about 430 lines to
rebuild JSON from the flat dotted keys the form posts. The soccer form
renders to 1.2 MB of HTML.
- `plugins_manager.js` is 3,800 lines. The owner decided that it needs a
namespace refactor before it is split, which is what this plan provides.
## Target
```
static/v3/js/
core/ ES modules ("type": "module" in core/package.json)
boot.js entry point; base.html loads it with <script type="module">
registry.js page lifecycle: init/destroy on htmx swaps
api.js fetch wrapper for /api/v3 (JSON envelope, login redirect)
facade.js window.LEDMatrix and deprecated aliases
(later) escape.js, notify.js, dialog.js, streams.js, visibility.js,
store.js (the one installed-plugin store), form/renderer.js
pages/ one module per tab partial
cache.js export init(root, ctx), destroy(root, ctx)
...
```
### The page lifecycle
A converted partial has no `<script>`. Its root element names its page:
```html
<div class="..." data-page="cache"> ... </div>
```
`core/boot.js` registers each page with a loader:
`registry.register('cache', () => import('../pages/cache.js'))`. A page's
module is fetched only when its partial first appears.
`core/registry.js` handles the rest:
| Event | What the registry does |
|---|---|
| `htmx:beforeSwap` (on `document`, so it runs after the body-level handlers that can veto a swap) | If `detail.shouldSwap` is still true, destroys every mounted page inside the swap target |
| `htmx:afterSwap` | Destroys any mounted page whose root has left the document, then mounts every `data-page` root not mounted yet |
| `LEDMatrix.pages.refresh()` | Same as afterSwap. `loadPartialDirect` (the no-htmx fallback in `base.html`) calls it |
| `start()` | Mounts whatever is already on the page. Module scripts run deferred, so a partial may arrive first |
Mounting is idempotent: a root is never initialised twice.
Each mount gets a `ctx` object:
| Field | Contents |
|---|---|
| `ctx.root` | The page's root element |
| `ctx.name` | The page name |
| `ctx.signal` | An `AbortSignal` that is aborted after `destroy()` |
| `ctx.state` | A per-mount object for the page's own state |
| `ctx.api` | Shared service from `boot.js` |
| `ctx.notify` | Shared service from `boot.js` |
A page that passes `{ signal: ctx.signal }` to `addEventListener` and
`fetch` needs no teardown code. Its listeners and in-flight requests go
away when the partial is swapped out. `pages/cache.js` is the worked
example: its delete buttons use one delegated listener, rows are built with
`textContent` rather than markup strings, and a newer load supersedes an
older one.
### One facade
`window.LEDMatrix` is the only global the module code adds:
| Member | What it is |
|---|---|
| `api` | `core/api.js`: `get`/`post`/`put`/`del`. Resolves to the JSON body, rejects with an `ApiError` |
| `pages` | `register`, `refresh`, `list` |
| `notify(message, type)` | Calls `window.showNotification`, looked up at call time |
| `escape` | Read-through to `window.LEDEscape` |
| `widgets` | Read-through to `window.LEDMatrixWidgets` |
| `deprecate(name, target, replacement)` | Keeps an old `window.*` name working. It warns once in the console, then forwards |
`ApiError` carries `status`, `body`, `network` and `loginRequired`.
`escape`, `widgets` and `notify` are read at call time. The classic scripts
that define them are deferred, and a plugin may replace them.
Login: `base.html` wraps `window.fetch` before any other script runs, and
the wrapper sends a 401 with `X-LEDMatrix-Login` to the login page (#683).
`api.js` calls `window.fetch` at call time, so its requests get the same
redirect. It also rejects that answer quietly with `loginRequired`, so no
error message flashes up while the page navigates away.
### Serving modules from the Pi
- **MIME type.** A browser runs a module only when it is served with a
JavaScript MIME type. `app.py` pins `.js` and `.mjs` to `text/javascript`
rather than trusting the host's mimetypes table, and
`test/web_interface/test_es_modules.py` checks it.
- **Caching.** `url_for` adds `?v=<mtime>` to the entry script, but modules
import each other by plain relative URL, without the version. A static
`.js` request without `v` is therefore served `Cache-Control: no-cache`
(revalidated, so 304 when unchanged) instead of being cached as immutable
for a year. Versioned URLs keep the long cache. A later optimisation is an
import map that maps each module to its versioned URL.
- **Load order.** `boot.js` loads after every classic script. Modules are
deferred and run in document order with the deferred classic scripts.
Nothing classic may depend on a module at load time. A classic script that
needs a module service calls `window.LEDMatrix` at run time.
### One form model
`src/plugin_system/field_model.py` provides
`build_field_model(schema, config, plugin_id)`. It walks a plugin's schema
once and returns a JSON tree with these keys for each field:
- path, label, help, widget
- starting value, default
- constraints, options, secret flag
- the exact form controls the macro posts today (`inputs`)
- the JS widget it mounts (`mount`)
`test/test_field_model_parity.py` renders the real macro for every schema it
can find and checks that the model names the same controls, with the same
starting values, in the same order, and the same widget mounts. The schemas
come from `plugin-repos/`, `test/fixtures/plugins/`, the ledmatrix-plugins
monorepo when a checkout is present, and a synthetic schema that reaches
every branch of the macro. The test was mutation-checked when it was
written. Each of these deliberate model bugs makes it fail:
- dropping the checkbox-group sentinel
- dropping a table's `00:00` time default
- picking the first matching `<select>` option instead of the last
- dropping the `None` quirk
- missing all-hidden objects
The model mirrors the macro's quirks on purpose. The parity run surfaced
these:
- 83 number fields whose schema default is `null` render `value="None"`.
- Four array fields name an `x-widget` the core does not ship (`color`,
`tag-input`) and fall back to a comma-separated text box.
- A list-typed `type` uses its first entry, so `["null", "string"]` draws a
text box.
- Eleven objects with no properties and no widget render nothing.
These get fixed once, in the renderer, after the switch below.
## Switching forms to the model, behind a flag
Stage 1 (this change) only proves the model is complete. Rendering does not
change. The switch is staged so either path can be turned back on at any
point:
1. **Model endpoint.** `GET /api/v3/plugins/config/model?plugin_id=<id>`
returns `build_field_model(schema, prepared_masked_config)`. It uses the
same preparation as the partial: defaults merged, secrets masked.
2. **Renderer module.** `core/form/renderer.js` walks the model. It draws
plain fields itself and hands every widget to `LEDMatrixWidgets` through
one `mount(el, field)` adapter. The adapter keeps plugin widgets' existing
`render(container, config, value, options)` signature (the hard
constraint in PRODUCT.md). `getValue()` results are assembled into one
JSON object.
3. **Flag.** `plugin_config.html` renders the macro unless the form-model
flag is on. The flag is a `web_interface.form_model` setting in
`config.json` (default off), plus a per-browser override
(`localStorage.ledmatrixFormModel`) so a tester can compare both paths on
one device. With the flag on, the partial renders only a
`<div data-page="plugin-config" data-plugin-id="...">` root, and
`pages/plugin-config.js` fetches the model and renders it.
4. **JSON submit.** With the flag on, Save posts
`Content-Type: application/json` to the existing
`POST /api/v3/plugins/config` JSON path (`plugin_config.py`, `is_json`).
That path already validates against the schema and keeps secrets. No
dotted keys, no `__rendered_section`, no checkbox reconstruction.
5. **Save parity test.** This gates turning the flag on by default. For every
schema, posting the macro form's data and posting the renderer's JSON
must store the same config.
6. **Retire.** Once the flag has been on by default for a release with no
regressions, the macro shrinks to a no-JS fallback for plain fields, and
the form-encoded reconstruction (`_parse_form_value_with_schema`,
`_set_nested_value`, `_set_missing_booleans_to_false` and friends in
`api_v3/__init__.py`) is deleted. The settings search index is then built
from the model instead of from rendered HTML.
## Migration order
Smallest and most isolated first. `plugins_manager.js` goes last. Line counts
are the inline script in each partial today.
| # | Page | Inline JS | Why it is here |
|---|---|---|---|
| 1 | Cache (`cache.html`) | 163 lines, now 0 | **Done in stage 1.** One endpoint pair, no globals other pages use. The reference conversion |
| 2 | Rotation (`durations.html`) | 29 | Tiny. One htmx form |
| 3 | Operation History | 293 | No globals, read-only list |
| 4 | Config Editor (`raw_json.html`) | 212 | No globals. CodeMirror is set up and torn down in init/destroy |
| 5 | Backup & Restore | 232 | 5 globals used only by its own `onclick`s; these become delegated listeners plus deprecated aliases |
| 6 | Schedule | 193 | 2 globals used as `hx-on` response handlers. Moves `hx-on` handlers into page listeners |
| 7 | General | 147 | `webLogin` global and the security section. The first page that touches login |
| 8 | Display | 231 | First page with `LEDVisibility` timers: those move to a `ctx.visibility` service that stops on destroy |
| 9 | Overview | 410 (4 scripts) | First-run surface: Getting Started, update banner, live preview. Five globals |
| 10 | WiFi | 364 | `x-data="wifiSetup()"` is defined by its own script. Moves to `Alpine.data()` registered from the module. AP-mode first screen, so it needs the AP-mode test on a real device |
| 11 | Fonts | 681 | Large, but self-contained (6 globals) |
| 12 | Logs | 801 | 14 globals, a stream and timers. Uses the visibility service from step 8 |
| 13 | Tools | 1,022 | 21 globals, MQTT bridge, Pixlet editor, diagnostics polling |
| 14 | Starlark app config, plugin config (`plugin_config.html`) | 123 + 294 | Plugin panels sit inside an Alpine `x-if` that removes them without an htmx swap. The registry's sweep covers that on the next swap; this step adds a MutationObserver or an `x-if` hook. Then the form-model flag (above) |
| 15 | Plugin Manager (`plugins.html` + `plugins_manager.js`) | 3,836-line file | Last. Split along the seams that already exist (installed grid, store, registries, Starlark section, on-demand) into `pages/plugins/*.js`. Its 42 globals become aliases. The four installed-plugin stores merge into one `core/store.js`, and `window.installedPlugins` becomes a getter over it |
The shell moves in parallel, a service at a time, with no page depending on
the order:
| Service | Current home | New module |
|---|---|---|
| `showNotification` | 4 versions | `core/notify.js` |
| The modal helper | `utils/dialog.js` | `core/dialog.js` |
| SSE streams | `app-shell.js` | `core/streams.js` |
| `LEDVisibility` | `app-shell.js` | `core/visibility.js` |
Each move leaves the old global as an alias. When the last inline script is
gone, the script re-execution in `htmx-config.js` and the "HTMX never
loaded" fallbacks in `base.html` can go too (keep the captive-page path).
## How the tests cover each step
The JS suites live in `test/js` (see `test/js/README.md` and
`docs/HOW_TO_RUN_TESTS.md`). In CI, the **Web UI JS tests** job installs
jsdom, starts the web interface and runs `node test/js/run_all.js` with
`REQUIRE_DOM=1`, so a skipped DOM suite fails the job.
`test/test_js_unit_suites.py` also runs every unit suite under pytest.
Unit suites need only node. They import the shipped modules directly:
`core/package.json` and `pages/package.json` mark those directories
`"type": "module"`.
| Suite | Kind | What it covers |
|---|---|---|
| `unit/test_page_registry.js` | Unit, minimal DOM shim | The lifecycle: one init per root, destroy on swap, a veto keeps the page, swaps elsewhere leave it alone, the sweep, lazy loading, a destroy while loading, error containment |
| `unit/test_core_modules.js` | Unit | `api.js` (envelope, errors, abort, login redirect, path check) and `facade.js` (facade, aliases) |
| `dom/test_cache_page.js` | DOM: real partial, real API shape | No inline script; one request per swap and per Refresh after five swaps; a cancelled request draws nothing; hostile keys stay text; delete, empty, error, network and login states |
| `test/web_interface/test_es_modules.py` | pytest | MIME type; `no-cache` without `?v` and immutable with it; `boot.js` loads last; every import resolves inside `core/` and `pages/`; every registered page has its module and exactly one partial root; a converted partial has no `<script>` |
| `test/test_field_model_parity.py` | pytest | The model against the macro for every available schema |
What each future step adds:
- **A page conversion** adds `dom/test_<page>_page.js`, built like the cache
suite: the real partial from the server, the real API's payload shape, N
swaps followed by one action that must make exactly one request, the
destroy and cancel behaviour, and escaping. `test_es_modules.py` picks up
the new page automatically. A unit suite that today slices a function out
of a template or `plugins_manager.js` and `eval`s it is rewritten to
import the module once that code moves (stage C of the plan).
- **A shell service move** adds a unit suite for the module and an alias
test showing the old global still works.
- **The form switch** adds the save parity test (macro form data and
renderer JSON store the same config, for every schema) and a DOM suite
for `pages/plugin-config.js`. Both run with the flag on and off.
- **`plugins_manager.js`.** The existing DOM suites (`test_installed_dom.js`,
`test_store_dom.js`, `test_no_double_fetch.js`) already test the real
Plugin Manager in jsdom. They stay green throughout the split and are the
gate for it, alongside the unit suites that pin its card rendering and
escaping.
+68 -5
View File
@@ -78,7 +78,9 @@ The Overview tab provides at-a-glance information and quick actions:
- **Start Display** / **Stop Display** — control the display service - **Start Display** / **Stop Display** — control the display service
- **Restart Display Service** — apply configuration changes - **Restart Display Service** — apply configuration changes
- **Restart Web Service** — restart the web UI itself - **Restart Web Service** — restart the web UI itself
- **Update Code** — `git pull` the latest version (stashes local changes) - **Update Code** — update to the newest version on the update channel (the
newest release on Stable, the newest code on `main` on Beta; stashes local
changes). The channel is set on the General tab.
- **Reboot System** / **Shutdown System** — confirm-gated power controls - **Reboot System** / **Shutdown System** — confirm-gated power controls
**Display Preview:** **Display Preview:**
@@ -90,6 +92,11 @@ The Overview tab provides at-a-glance information and quick actions:
Configure basic system settings: Configure basic system settings:
- **Automatic Updates** — weekly updates with a health check and rollback
- **Update Channel** — **Stable** (default) installs releases; **Beta**
installs the newest code on `main` before it is released. Switching to
Stable never installs an older version: a device ahead of the newest
release keeps following `main` until a release includes it
- **Timezone** — used by all time/date displays - **Timezone** — used by all time/date displays
- **Location** — city/state/country for weather and other location-aware - **Location** — city/state/country for weather and other location-aware
plugins plugins
@@ -130,6 +137,34 @@ Configure basic system settings:
Click **Save** to write changes to `config/config.json`. Most changes Click **Save** to write changes to `config/config.json`. Most changes
require a display service restart from **Overview**. require a display service restart from **Overview**.
Below the settings, the **Security** section (its own buttons, not the Save
button) controls the optional login:
- **Web interface password** — off by default. Setting one turns login on:
browsers on your network then see a login page, and stay logged in for 30
days (across restarts). The browser you set it from stays logged in.
Changing the password logs every other browser out. **Turn login off**
needs the current password. A **Log out** button appears in the header
while you are logged in. Five wrong passwords in a minute (or 30 in an
hour) from one address make it wait.
- **API tokens** — for Home Assistant, scripts, or the MQTT bridge on another
machine. Give it a name, click **Create token**, and copy the token right
away: it is shown once. Revoke it here when it is no longer needed.
- Never asked for a password: a browser on the Pi itself, and the Wi-Fi setup
page while the Pi is in access-point mode (so you can always get it back on
a network).
**Forgot the password?** SSH into the Pi and run:
```bash
sudo python3 ~/LEDMatrix/scripts/reset_web_password.py
```
(use the folder LEDMatrix is installed in). Login is off again right away,
no restart needed, and you can set a new password. API tokens are kept; add
`--revoke-tokens` to delete them too. Alternatively, open
`http://localhost:5000` in a browser on the Pi itself.
### Display Tab ### Display Tab
Configure your LED matrix hardware: Configure your LED matrix hardware:
@@ -346,6 +381,14 @@ The API blueprint (`web_interface/blueprints/api_v3/`) is registered at
- `POST /api/v3/plugins/install` — Install a plugin from the store - `POST /api/v3/plugins/install` — Install a plugin from the store
- `POST /api/v3/plugins/install-from-url` — Install a plugin from a GitHub URL - `POST /api/v3/plugins/install-from-url` — Install a plugin from a GitHub URL
If the optional login is on, send an API token (General > Security):
```bash
curl -H "Authorization: Bearer lmx_..." http://your-pi-ip:5000/api/v3/display/current
```
Scripts running on the Pi itself need no token.
**Note:** See [REST_API_REFERENCE.md](REST_API_REFERENCE.md) for complete API documentation. **Note:** See [REST_API_REFERENCE.md](REST_API_REFERENCE.md) for complete API documentation.
--- ---
@@ -408,9 +451,27 @@ The API blueprint (`web_interface/blueprints/api_v3/`) is registered at
## Security Considerations ## Security Considerations
**Network Access:** **Network Access:**
- The interface is accessible to anyone on your local network - By default the interface is accessible to anyone on your local network
- No authentication is currently implemented - An optional password (General > Security) makes every page and API call
- Recommended for trusted networks only need a login or an API token; see [General Tab](#general-tab). Requests
from the Pi itself and the Wi-Fi setup flow in access-point mode stay open,
and `/api/v3/health` answers only its overall status without a login
- The interface speaks plain HTTP, so the password and tokens cross your
network unencrypted: still recommended for trusted networks only
- Behind a reverse proxy **on the Pi**, make it send `X-Forwarded-For`
(nginx: `proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;`).
Without it every proxied request looks like it comes from the Pi itself,
which is never asked to log in
**Other websites:**
- A web page you open elsewhere could otherwise make your browser send
commands to the Pi (reboot, update, config changes). The interface refuses
any change request whose `Origin`/`Referer` header names a different site
(403 `CROSS_SITE_REQUEST`), so use the interface from its own address.
- Scripts, curl, Home Assistant and the MQTT bridge send no such header and
keep working. Behind a reverse proxy, forward the original `Host` header
with its port (nginx: `proxy_set_header Host $http_host;` -- `$host`
drops the port).
**Best Practices:** **Best Practices:**
1. Run on a private network (not exposed to internet) 1. Run on a private network (not exposed to internet)
@@ -428,7 +489,9 @@ The web interface uses modern web technologies:
- **Backend:** Flask with Blueprint-based modular design - **Backend:** Flask with Blueprint-based modular design
- **Frontend:** HTMX for dynamic content, Alpine.js for reactive components - **Frontend:** HTMX for dynamic content, Alpine.js for reactive components
- **Styling:** Tailwind CSS for responsive design - **Styling:** Tailwind CSS utilities, generated at development time and
committed (the Pi never builds CSS; see
[`web_interface/README.md`](../web_interface/README.md#styling-tailwind-css))
- **Real-Time:** Server-Sent Events (SSE) for live updates - **Real-Time:** Server-Sent Events (SSE) for live updates
### File Locations ### File Locations
+4
View File
@@ -835,6 +835,10 @@ if [ ! -f "$PROJECT_ROOT_DIR/config/config.json" ]; then
cat > "$PROJECT_ROOT_DIR/config/config.json" <<'EOF' cat > "$PROJECT_ROOT_DIR/config/config.json" <<'EOF'
{ {
"web_display_autostart": true, "web_display_autostart": true,
"auto_update": {
"enabled": false,
"channel": "stable"
},
"timezone": "America/Chicago", "timezone": "America/Chicago",
"display": { "display": {
"hardware": { "hardware": {
+7
View File
@@ -96,6 +96,13 @@ environment as `LEDMATRIX_MQTT_<KEY>` (`LEDMATRIX_MQTT_MQTT_PASSWORD`, say),
which keeps a broker password out of a file on disk — put it in a systemd which keeps a broker password out of a file on disk — put it in a systemd
drop-in with `Environment=` or `EnvironmentFile=` instead. drop-in with `Environment=` or `EnvironmentFile=` instead.
**Web login.** If the web interface's optional login is on (General >
Security), a bridge running on the Pi itself still needs nothing: requests from
the Pi are never asked to log in. A bridge on another machine needs an API
token: create one under General > Security and set `"ledmatrix_api_token"`
(or `LEDMATRIX_MQTT_LEDMATRIX_API_TOKEN`, or the token field in the Tools tab's
bridge settings). It is sent as `Authorization: Bearer <token>`.
Set `mqtt_tls: true` for a broker with TLS. `mqtt_tls_insecure` skips Set `mqtt_tls: true` for a broker with TLS. `mqtt_tls_insecure` skips
certificate verification and exists only for a self-signed broker on a certificate verification and exists only for a self-signed broker on a
trusted LAN; it logs a warning when used. trusted LAN; it logs a warning when used.
@@ -66,6 +66,10 @@ DEFAULTS = {
"mqtt_tls": False, "mqtt_tls": False,
"mqtt_tls_insecure": False, "mqtt_tls_insecure": False,
"ledmatrix_api_base": "http://localhost:5000", "ledmatrix_api_base": "http://localhost:5000",
# Only needed when the web interface's optional login is on AND the bridge
# reaches it from another machine: requests from the Pi itself never need
# one. Create it under General > Security; sent as a Bearer token.
"ledmatrix_api_token": None,
"request_timeout": 15, "request_timeout": 15,
"on_demand_duration": None, "on_demand_duration": None,
"log_level": "INFO", "log_level": "INFO",
@@ -125,10 +129,13 @@ class LEDMatrixClient:
""" """
def __init__(self, api_base: str, timeout: int = 15, def __init__(self, api_base: str, timeout: int = 15,
session: Optional[requests.Session] = None): session: Optional[requests.Session] = None,
api_token: Optional[str] = None):
self.api_base = api_base.rstrip("/") self.api_base = api_base.rstrip("/")
self.timeout = timeout self.timeout = timeout
self.session = session or requests.Session() self.session = session or requests.Session()
if api_token:
self.session.headers["Authorization"] = f"Bearer {api_token}"
def _call(self, method: str, path: str, **kwargs) -> Dict[str, Any]: def _call(self, method: str, path: str, **kwargs) -> Dict[str, Any]:
url = f"{self.api_base}/api/v3{path}" url = f"{self.api_base}/api/v3{path}"
@@ -403,7 +410,8 @@ class Bridge:
self.status_topic = f"{self.command_topic}/status" self.status_topic = f"{self.command_topic}/status"
self.state_topic = f"{self.command_topic}/state" self.state_topic = f"{self.command_topic}/state"
self.availability_topic = f"{self.command_topic}/availability" self.availability_topic = f"{self.command_topic}/availability"
self.client = LEDMatrixClient(config["ledmatrix_api_base"], config["request_timeout"]) self.client = LEDMatrixClient(config["ledmatrix_api_base"], config["request_timeout"],
api_token=config.get("ledmatrix_api_token") or None)
self.handler = CommandHandler(self.client, config.get("on_demand_duration")) self.handler = CommandHandler(self.client, config.get("on_demand_duration"))
self._stop = threading.Event() self._stop = threading.Event()
self._mqtt = None self._mqtt = None
+102
View File
@@ -0,0 +1,102 @@
# The mypy ratchet: modules that type-check clean, one path per line, sorted.
#
# CI ("Type check (mypy ratchet)") runs `python scripts/check_types.py`, which
# runs mypy on exactly these files (imports followed silently, so errors in an
# unlisted module they import don't count) and fails on any error, so a listed
# module stays clean. Most of src/ isn't clean yet. When you make a module
# clean, add it here. Don't take a module off to get CI green -- fix the error
# (annotation-only where you can: hints, typing.cast, TYPE_CHECKING imports;
# widen an annotation rather than delete a defensive runtime check).
src/__init__.py
src/adaptive_images.py
src/auto_update_setup.py
src/backup_manager.py
src/base_odds_manager.py
src/cache/__init__.py
src/cache/cache_metrics.py
src/cache/cache_strategy.py
src/cache/memory_cache.py
src/common/__init__.py
src/common/api_helper.py
src/common/bdf_font.py
src/common/espn_dates.py
src/common/favorite_team_check.py
src/common/font_layout.py
src/common/frame_timing.py
src/common/json_body.py
src/common/logo_helper.py
src/common/path_safety.py
src/common/permission_utils.py
src/common/render_gate.py
src/common/scroll_config.py
src/common/snapshot_policy.py
src/common/sports_card.py
src/common/sports_card_wrappers.py
src/common/sports_celebration.py
src/common/sports_display_rules.py
src/common/sports_fetch.py
src/common/sports_font_path.py
src/common/sports_live_scroll.py
src/common/sports_plugin_host.py
src/common/sports_scroll.py
src/common/sports_timezone.py
src/common/sports_vegas.py
src/config_service.py
src/core_config_keys.py
src/deprecation.py
src/device_location.py
src/display_geometry.py
src/dynamic_team_resolver.py
src/exceptions.py
src/font_usage.py
src/ipc/__init__.py
src/ipc/client.py
src/ipc/contract.py
src/ipc/server.py
src/logging_config.py
src/logo_downloader.py
src/matrix_support.py
src/pi5_matrix_support.py
src/plugin_system/__init__.py
src/plugin_system/compatibility.py
src/plugin_system/field_model.py
src/plugin_system/operation_history.py
src/plugin_system/operation_queue.py
src/plugin_system/operation_types.py
src/plugin_system/plugin_catalog.py
src/plugin_system/plugin_dirs.py
src/plugin_system/plugin_executor.py
src/plugin_system/plugin_health.py
src/plugin_system/plugin_loader.py
src/plugin_system/plugin_runtime.py
src/plugin_system/plugin_state.py
src/plugin_system/repo_urls.py
src/plugin_system/resource_monitor.py
src/plugin_system/saved_repositories.py
src/plugin_system/schema_manager.py
src/plugin_system/state_reconciliation.py
src/plugin_system/testing/__init__.py
src/plugin_system/testing/bounds_display_manager.py
src/plugin_system/testing/loading.py
src/plugin_system/testing/mocks.py
src/plugin_system/testing/plugin_test_base.py
src/plugin_system/testing/sizes.py
src/plugin_system/testing/vegas.py
src/plugin_system/vegas_elements.py
src/redaction.py
src/scan_order.py
src/startup_validator.py
src/vegas_mode/__init__.py
src/vegas_mode/config.py
src/vegas_mode/coordinator.py
src/vegas_mode/elements.py
src/vegas_mode/geometry.py
src/vegas_mode/live_worker.py
src/vegas_mode/stream_manager.py
src/web_interface/api_helpers.py
src/web_interface/config_arrays.py
src/web_interface/error_handler.py
src/web_interface/errors.py
src/web_interface/secret_helpers.py
src/web_interface/validators.py
+25 -10
View File
@@ -1,6 +1,11 @@
[mypy] [mypy]
# Mypy configuration for LEDMatrix # Mypy configuration for LEDMatrix
# What a bare `mypy` checks. ini values can't span lines or carry trailing
# comments -- this file used to have both, so mypy refused to read it at all.
files = src
exclude = (^|/)(test|__pycache__)/
# Python version # Python version
python_version = 3.10 python_version = 3.10
@@ -25,11 +30,11 @@ warn_unreachable = True
# Strict optional checking # Strict optional checking
strict_optional = True strict_optional = True
# Disallow untyped definitions # Disallow untyped definitions (set to True once all code is typed)
disallow_untyped_defs = False # Set to True once all code is typed disallow_untyped_defs = False
# Disallow untyped calls # Disallow untyped calls (set to True once all code is typed)
disallow_untyped_calls = False # Set to True once all code is typed disallow_untyped_calls = False
# Check untyped definitions # Check untyped definitions
check_untyped_defs = True check_untyped_defs = True
@@ -96,10 +101,20 @@ ignore_missing_imports = True
[mypy-spotipy.*] [mypy-spotipy.*]
ignore_missing_imports = True ignore_missing_imports = True
# Exclude test files and generated files
exclude = (?x)(
^test/.*|
^.*/__pycache__/.*|
^.*\.pyc$
)
# numpy's own stubs (numpy>=2.3) use Python 3.12 `type` statements, which mypy
# refuses to parse under python_version = 3.10 -- and 3.10 is the floor this
# code has to run on, so it stays. Treat numpy as Any instead: skip it, and
# follow_imports_for_stubs makes the skip apply to its .pyi files too.
[mypy-numpy.*]
follow_imports = skip
follow_imports_for_stubs = True
# orjson is optional (see requirements.txt): the modules that use it fall back
# to the stdlib when `import orjson` fails. Whether mypy sees its stubs would
# otherwise depend on whether it happens to be installed -- installed, the
# `orjson = None` fallback is a type error and the stdlib branch "unreachable";
# not installed, silencing either is an unused ignore. Treat it as Any always.
[mypy-orjson.*]
follow_imports = skip
follow_imports_for_stubs = True
+2 -2
View File
@@ -1,10 +1,10 @@
# Test/dev-only dependencies (not needed on a running display). # Test/dev-only dependencies (not needed on a running display).
# Install alongside requirements.txt: pip install -r requirements.txt -r requirements-test.txt # Install alongside requirements.txt: pip install -r requirements.txt -r requirements-test.txt
pytest>=9.0.3,<10.0.0 pytest>=9.0.3,<10.0.0
pytest-cov>=4.1.0,<5.0.0 pytest-cov>=4.1.0,<8.0.0
pytest-mock>=3.11.0,<4.0.0 pytest-mock>=3.11.0,<4.0.0
freezegun>=1.2,<2 # deterministic time for golden-image tests freezegun>=1.2,<2 # deterministic time for golden-image tests
psutil>=6.0.0,<8.0.0 # optional at runtime; installed for tests so the psutil>=6.0.0,<7.0.0 # optional at runtime; installed for tests so the
# /system/status endpoint's real path is exercised # /system/status endpoint's real path is exercised
mypy>=1.5.0,<2.0.0 # static type checking (also pinned in .pre-commit-config.yaml) mypy>=1.5.0,<2.0.0 # static type checking (also pinned in .pre-commit-config.yaml)
PyYAML>=6.0.2,<7.0.0 # not a core dependency: test_starlark_pixlet_routes loads PyYAML>=6.0.2,<7.0.0 # not a core dependency: test_starlark_pixlet_routes loads
+1 -1
View File
@@ -7,7 +7,7 @@ Pillow>=12.2.0,<13.0.0
numpy>=1.24.0 # For fast array operations in ScrollHelper (compatible with 2.x) numpy>=1.24.0 # For fast array operations in ScrollHelper (compatible with 2.x)
# Timezone handling # Timezone handling
pytz>=2024.2,<2025.0 # Updated for latest timezone data pytz>=2024.2,<2027.0 # Updated for latest timezone data
# HTTP requests # HTTP requests
requests>=2.33.0,<3.0.0 requests>=2.33.0,<3.0.0
+8
View File
@@ -14,6 +14,14 @@ project_dir = os.path.dirname(os.path.abspath(__file__))
if project_dir not in sys.path: if project_dir not in sys.path:
sys.path.insert(0, project_dir) sys.path.insert(0, project_dir)
# Under systemd the watchdog clock is already running, and start-up (plugin
# loads, initial updates) takes far longer than the render loop's limit. Widen
# it before anything slow is imported; the render loop narrows it again once
# its first frame is on the panel. A no-op outside systemd. Standard library
# only -- see src/display_watchdog.py.
from src import display_watchdog
display_watchdog.watchdog.begin_startup()
# Parse command-line arguments BEFORE any imports # Parse command-line arguments BEFORE any imports
parser = argparse.ArgumentParser(description='LEDMatrix Display Controller') parser = argparse.ArgumentParser(description='LEDMatrix Display Controller')
parser.add_argument('-e', '--emulator', action='store_true', parser.add_argument('-e', '--emulator', action='store_true',
+5
View File
@@ -149,6 +149,11 @@
}, },
"description": "Array of display mode names this plugin provides" "description": "Array of display mode names this plugin provides"
}, },
"vegas_participation": {
"type": "string",
"enum": ["scroll", "pause", "exclude"],
"description": "How this plugin takes part in Vegas mode by default: 'scroll' (its content scrolls by), 'pause' (the scroll stops for its turn and display() draws it full screen) or 'exclude' (left out). A user's per-plugin vegas_participation setting overrides it. Omit it to derive the participation from get_vegas_display_mode() / get_vegas_content_type(). Cores before 3.8.0 ignore it."
},
"api_requirements": { "api_requirements": {
"type": "array", "type": "array",
"items": { "items": {
+5
View File
@@ -31,12 +31,17 @@ display; **diagnostic** — run by hand on a Pi when something is wrong.
| `diagnose_web_interface.sh` | diagnostic | Checks why the web interface is not reachable | | `diagnose_web_interface.sh` | diagnostic | Checks why the web interface is not reachable |
| `download_pixlet.sh` | keep | Downloads the bundled Pixlet binaries for Starlark apps (also run from the web UI) | | `download_pixlet.sh` | keep | Downloads the bundled Pixlet binaries for Starlark apps (also run from the web UI) |
| `emergency_reconnect.sh` | diagnostic | Reconnects to your WiFi network if captive-portal testing leaves the Pi offline | | `emergency_reconnect.sh` | diagnostic | Reconnects to your WiFi network if captive-portal testing leaves the Pi offline |
| `frame_soak.py` | diagnostic | Soaks a running display and reports how often frames reached the panel late (docs/SCROLL_PERFORMANCE.md) |
| `install_dependencies_apt.py` | keep | Dependency installer that tries apt packages first, then pip (installer Step 7, plugin loader) | | `install_dependencies_apt.py` | keep | Dependency installer that tries apt packages first, then pip (installer Step 7, plugin loader) |
| `install_plugin_dependencies.sh` | diagnostic | Installs plugin requirements by hand when the automatic install fails | | `install_plugin_dependencies.sh` | diagnostic | Installs plugin requirements by hand when the automatic install fails |
| `plugin_api_usage.py` | dev-only | Scans core, the plugin monorepo and the registry's third-party plugins for callers of every `@deprecated` core method; its output is [docs/DEPRECATIONS_3.8.md](../docs/DEPRECATIONS_3.8.md) |
| `prove_security.py` | keep | Security property checks run by pre-commit | | `prove_security.py` | keep | Security property checks run by pre-commit |
| `render_bench.py` | diagnostic | Benchmarks the render loop against the panel's real refresh rate on a synthetic strip |
| `render_plugin.py` | dev-only | Runs a plugin's `update()` + `display()` and saves the frame as a PNG | | `render_plugin.py` | dev-only | Runs a plugin's `update()` + `display()` and saves the frame as a PNG |
| `reset_web_password.py` | keep | Turns the optional web login off when the password is lost (`sudo python3 scripts/reset_web_password.py`; docs/WEB_INTERFACE_GUIDE.md) |
| `run_plugin_tests.py` | dev-only | Discovers and runs plugin test suites | | `run_plugin_tests.py` | dev-only | Discovers and runs plugin test suites |
| `scroll_speeds.py` | keep | Shows and tries the scroll speeds your panel can display cleanly | | `scroll_speeds.py` | keep | Shows and tries the scroll speeds your panel can display cleanly |
| `sports_drift_report.py` | keep | Counts the different bodies of each method across the nine scoreboards in a `ledmatrix-plugins` checkout (report-only CI job; docs/SPORTS_UNIFICATION.md) |
| `troubleshoot_captive_portal.sh` | diagnostic | Troubleshoots captive-portal WiFi setup after you can SSH back in | | `troubleshoot_captive_portal.sh` | diagnostic | Troubleshoots captive-portal WiFi setup after you can SSH back in |
| `update_plugin_repos.py` | dev-only | Pulls the latest `ledmatrix-plugins` monorepo | | `update_plugin_repos.py` | dev-only | Pulls the latest `ledmatrix-plugins` monorepo |
| `verify_installation.sh` | diagnostic | Checks that an installation completed correctly | | `verify_installation.sh` | diagnostic | Checks that an installation completed correctly |
+1 -1
View File
@@ -194,7 +194,7 @@ def process_schema_file(schema_path: Path) -> bool:
print(f" ✓ Modified {len(modified_fields)} fields") print(f" ✓ Modified {len(modified_fields)} fields")
return True return True
else: else:
print(f" ✓ No changes needed") print(" ✓ No changes needed")
return False return False
+2 -3
View File
@@ -87,7 +87,6 @@ def find_duplicate_fields(schema: Dict[str, Any], path: str = "") -> List[str]:
def validate_schema_syntax(schema_path: Path) -> tuple[bool, List[str]]: def validate_schema_syntax(schema_path: Path) -> tuple[bool, List[str]]:
"""Validate JSON Schema syntax.""" """Validate JSON Schema syntax."""
errors = []
try: try:
with open(schema_path, 'r', encoding='utf-8') as f: with open(schema_path, 'r', encoding='utf-8') as f:
schema = json.load(f) schema = json.load(f)
@@ -164,7 +163,7 @@ def analyze_schema(schema_path: Path) -> Dict[str, Any]:
if "update_interval_seconds" in properties: if "update_interval_seconds" in properties:
analysis["update_interval_variant"] = "update_interval_seconds" analysis["update_interval_variant"] = "update_interval_seconds"
analysis["naming_issues"].append( analysis["naming_issues"].append(
f"Uses 'update_interval_seconds' instead of 'update_interval'" "Uses 'update_interval_seconds' instead of 'update_interval'"
) )
else: else:
analysis["missing_common_fields"].append(field_name) analysis["missing_common_fields"].append(field_name)
@@ -239,7 +238,7 @@ def main():
print(f" Missing common fields: {', '.join(result['missing_common_fields'])}") print(f" Missing common fields: {', '.join(result['missing_common_fields'])}")
if result['naming_issues']: if result['naming_issues']:
print(f" Naming issues:") print(" Naming issues:")
for issue in result['naming_issues']: for issue in result['naming_issues']:
print(f" - {issue}") print(f" - {issue}")
+239
View File
@@ -0,0 +1,239 @@
#!/usr/bin/env python3
"""Build the web UI's Tailwind CSS with the pinned standalone Tailwind CLI.
The generated files are committed, so the Pi never builds anything. Run this
on a dev machine (or let CI run it) after changing a template, a static JS
file, or anything under ``web_interface/tailwind/``:
python3 scripts/build_css.py # rebuild the committed CSS
python3 scripts/build_css.py --check # exit 1 if the committed CSS is stale
No Node or npm: the script downloads Tailwind's standalone CLI (a single
executable) for this OS and CPU from the Tailwind GitHub release, checks it
against the SHA-256 pinned below, and caches it outside the repo
(``$LEDMATRIX_TAILWIND_CACHE``, else the per-user cache directory).
Outputs (see ``BUILDS``):
- ``web_interface/static/v3/tailwind.css``: the utilities the templates and
static JS use. Linked before ``app.css`` in ``base.html``.
- ``web_interface/static/v3/plugin-frame.css``: preflight plus a broad set of
common utilities, for plugin ``web_ui/`` fragments served in an iframe.
Their markup lives in plugin repos, so it can't be scanned; the safelist in
``plugin-frame.config.js`` stands in for it.
To move to a new Tailwind v3 release, change ``TAILWIND_VERSION`` and every
hash in ``TAILWIND_ASSETS`` (the release's ``sha256sums.txt``, or the digests
from ``gh api repos/tailwindlabs/tailwindcss/releases/tags/<tag>``), rebuild,
and review the diff of the generated CSS.
"""
from __future__ import annotations
import argparse
import hashlib
import os
import platform
import shutil
import stat
import subprocess # nosec B404 - list-form argv only, no shell # nosemgrep
import sys
import tempfile
import urllib.request
from pathlib import Path
PROJECT_ROOT = Path(__file__).resolve().parent.parent
TAILWIND_DIR = PROJECT_ROOT / "web_interface" / "tailwind"
STATIC_V3 = PROJECT_ROOT / "web_interface" / "static" / "v3"
TAILWIND_VERSION = "3.4.19"
# asset name -> SHA-256, from the v3.4.19 release.
TAILWIND_ASSETS = {
"tailwindcss-linux-arm64": "e5b2d27694daa80cc52ec29553ba2c6bd43d86bd51a9d633ed24058b9c05a676",
"tailwindcss-linux-armv7": "e3610b109a64720295e1c00a18dd2d6d79d3cddc618219aa0830de97a55429a4",
"tailwindcss-linux-x64": "4af3198c015616ea7d6617974ec3d70d987ecc00c1ca8463b0a30fd65cc7c06e",
"tailwindcss-macos-arm64": "7fdeb00818b6214a337383063282b2361ecb08bbc08f8c8a7ba97ee1e2eaa4fe",
"tailwindcss-macos-x64": "a597f407e0f1f03535731f5b42f1576a8152cb5fffc2f38e754722bc0c280045",
"tailwindcss-windows-arm64.exe": "f2b6b999747aa0ae31999d59db117b1ba1e4e15e17675d7108e30aac4b680686",
"tailwindcss-windows-x64.exe": "a15158c4c5e0e7a75f7229bfe4986fe7710d2edc468b6f96c8981f78ab211347",
}
DOWNLOAD_URL = (
"https://github.com/tailwindlabs/tailwindcss/releases/download/v{version}/{asset}"
)
# (input CSS, config, output) -- all relative to the project root.
BUILDS = (
(
"web_interface/tailwind/app.input.css",
"web_interface/tailwind/tailwind.config.js",
"web_interface/static/v3/tailwind.css",
),
(
"web_interface/tailwind/plugin-frame.input.css",
"web_interface/tailwind/plugin-frame.config.js",
"web_interface/static/v3/plugin-frame.css",
),
)
def asset_name() -> str:
"""The release asset for this OS and CPU."""
system = platform.system()
machine = platform.machine().lower()
if machine in ("x86_64", "amd64"):
arch = "x64"
elif machine in ("aarch64", "arm64"):
arch = "arm64"
elif machine.startswith("armv7") or machine == "armv8l":
arch = "armv7"
else:
raise SystemExit(f"No standalone Tailwind CLI for CPU {machine!r}.")
if system == "Linux":
name = f"tailwindcss-linux-{arch}"
elif system == "Darwin":
name = f"tailwindcss-macos-{arch}"
elif system == "Windows":
name = f"tailwindcss-windows-{arch}.exe"
else:
raise SystemExit(f"No standalone Tailwind CLI for {system!r}.")
if name not in TAILWIND_ASSETS:
raise SystemExit(f"No standalone Tailwind CLI for {system} {machine}.")
return name
def cache_dir() -> Path:
override = os.environ.get("LEDMATRIX_TAILWIND_CACHE")
if override:
return Path(override)
if platform.system() == "Windows":
base = Path(os.environ.get("LOCALAPPDATA", Path.home() / "AppData" / "Local"))
elif platform.system() == "Darwin":
base = Path.home() / "Library" / "Caches"
else:
base = Path(os.environ.get("XDG_CACHE_HOME", Path.home() / ".cache"))
return base / "ledmatrix" / "tailwindcss"
def sha256_of(path: Path) -> str:
digest = hashlib.sha256()
with open(path, "rb") as fh:
for chunk in iter(lambda: fh.read(1 << 20), b""):
digest.update(chunk)
return digest.hexdigest()
def ensure_cli() -> Path:
"""Path to the verified CLI, downloading it on first use."""
name = asset_name()
expected = TAILWIND_ASSETS[name]
target = cache_dir() / f"v{TAILWIND_VERSION}" / name
if target.is_file():
if sha256_of(target) == expected:
return target
print(f"Cached {target} fails its SHA-256 check; downloading it again.")
target.unlink()
target.parent.mkdir(parents=True, exist_ok=True)
url = DOWNLOAD_URL.format(version=TAILWIND_VERSION, asset=name)
if not url.startswith("https://"):
raise SystemExit(f"Refusing to download the Tailwind CLI over a non-https URL: {url}")
print(f"Downloading Tailwind CLI v{TAILWIND_VERSION} ({name})...")
fd, tmp_name = tempfile.mkstemp(dir=target.parent, prefix=".download-")
tmp = Path(tmp_name)
try:
with os.fdopen(fd, "wb") as out, urllib.request.urlopen(url, timeout=120) as resp: # nosec B310 - https only, checked above
shutil.copyfileobj(resp, out)
actual = sha256_of(tmp)
if actual != expected:
raise SystemExit(
f"SHA-256 mismatch for {url}\n expected {expected}\n got {actual}"
)
tmp.chmod(tmp.stat().st_mode | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH)
os.replace(tmp, target)
finally:
if tmp.exists():
tmp.unlink()
return target
def run_build(
cli: Path, input_css: str, config: str, output: Path, work_dir: Path
) -> None:
# The minifier's rule merging depends on the input file's line endings,
# so a Windows checkout (core.autocrlf, CRLF) would build different bytes
# than CI's Linux one and --check would fail. Feed the CLI an LF copy.
# (Content files' line endings don't matter; @import isn't used, so the
# copy's location doesn't either.)
lf_input = work_dir / (Path(input_css).name)
lf_input.write_bytes(
(PROJECT_ROOT / input_css).read_bytes().replace(b"\r\n", b"\n")
)
cmd = [
str(cli),
"--input", str(lf_input),
"--config", str(PROJECT_ROOT / config),
"--output", str(output),
"--minify",
]
# NODE_ENV=production and no browserslist lookup keep the output the
# same on every machine.
env = dict(os.environ, NODE_ENV="production", BROWSERSLIST_IGNORE_OLD_DATA="1")
# The CLI path is computed here (cache dir + pinned asset name) and the
# binary was SHA-256-verified by ensure_cli(); env is os.environ plus two
# fixed values.
result = subprocess.run(cmd, cwd=PROJECT_ROOT, env=env, capture_output=True, text=True) # nosec B603 - list-form argv, no shell # nosemgrep
if result.returncode != 0:
sys.stderr.write(result.stdout + result.stderr)
raise SystemExit(f"Tailwind build failed for {input_css}")
# The CLI writes without a trailing newline; add one so the committed
# file is a well-formed text file and editors leave it alone.
text = output.read_text(encoding="utf-8").replace("\r\n", "\n")
if not text.endswith("\n"):
text += "\n"
output.write_bytes(text.encode("utf-8"))
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
parser.add_argument(
"--check",
action="store_true",
help="build to a temp dir and fail if the committed CSS differs",
)
args = parser.parse_args(argv)
cli = ensure_cli()
stale = []
with tempfile.TemporaryDirectory(prefix="ledmatrix-css-") as tmp:
for input_css, config, output in BUILDS:
committed = PROJECT_ROOT / output
built = Path(tmp) / Path(output).name if args.check else committed
run_build(cli, input_css, config, built, Path(tmp))
if args.check:
old = (
committed.read_bytes().replace(b"\r\n", b"\n")
if committed.is_file()
else None
)
if old != built.read_bytes():
stale.append(output)
else:
print(f"Wrote {output} ({committed.stat().st_size:,} bytes)")
if stale:
print(
"The committed CSS is out of date: " + ", ".join(stale) + "\n"
"Run `python3 scripts/build_css.py` and commit the result."
)
return 1
if args.check:
print("Committed CSS is up to date.")
return 0
if __name__ == "__main__":
sys.exit(main())
+21
View File
@@ -72,6 +72,7 @@ from src.plugin_system.testing.harness import ( # noqa: E402
from src.plugin_system.testing.sizes import ( # noqa: E402 from src.plugin_system.testing.sizes import ( # noqa: E402
parse_size_token, resolve_test_sizes, safe_mode_filename, size_label, parse_size_token, resolve_test_sizes, safe_mode_filename, size_label,
) )
from src.plugin_system.testing.vegas import check_plugin_vegas_elements # noqa: E402
logger = get_logger("[Check Plugin]") logger = get_logger("[Check Plugin]")
@@ -193,6 +194,24 @@ def check_one(plugin_id: str, search_dirs: List[str], sizes, mock_data: Dict,
all_run_results.extend(results) all_run_results.extend(results)
# Live Vegas elements, for a plugin that has them: checked once, at the
# first size, with the base config.
width, height = effective_sizes[0]
try:
vegas = check_plugin_vegas_elements(
plugin_id, plugin_dir, full_config, effective_mock_data, width, height,
run_update=effective_run_update)
except Exception as exc: # noqa: BLE001 - one plugin must not end an --all run
all_run_results.append(RenderResult(
plugin_id, width, height, "vegas elements",
error=f"the element check itself failed: {exc!r}"))
return all_run_results
if vegas.implemented:
all_run_results.append(RenderResult(
plugin_id, width, height, "vegas elements",
error="; ".join(vegas.errors) or None,
notes=[f"{vegas.elements} element(s), {vegas.live} live"] + vegas.warnings))
return all_run_results return all_run_results
@@ -236,6 +255,8 @@ def print_report(all_results: Dict[str, List[RenderResult]]) -> bool:
f" controller skips the mode") f" controller skips the mode")
else: else:
status, detail = "FAIL", "" status, detail = "FAIL", ""
if r.notes:
detail += f" ({'; '.join(r.notes)})"
print(f" [{status}] {r.size_label:>7} {r.mode}{detail}") print(f" [{status}] {r.size_label:>7} {r.mode}{detail}")
print() print()
return everything_ok return everything_ok
+8 -7
View File
@@ -112,15 +112,13 @@ if command -v python3 >/dev/null 2>&1; then
echo "Python: $PYTHON_VERSION" echo "Python: $PYTHON_VERSION"
if [ "$PYTHON_MAJOR" -eq "3" ]; then if [ "$PYTHON_MAJOR" -eq "3" ]; then
if [ "$PYTHON_MINOR" -ge "10" ] && [ "$PYTHON_MINOR" -le "12" ]; then if [ "$PYTHON_MINOR" -ge "10" ] && [ "$PYTHON_MINOR" -le "13" ]; then
print_success "Python version is fully supported (3.10-3.12)" print_success "Python version is supported (3.10-3.13)"
elif [ "$PYTHON_MINOR" -eq "13" ]; then
print_warning "Python 3.13 detected - most packages compatible, but some may have limited testing"
print_warning "Please report any compatibility issues you encounter"
elif [ "$PYTHON_MINOR" -ge "14" ]; then elif [ "$PYTHON_MINOR" -ge "14" ]; then
print_warning "Python 3.${PYTHON_MINOR} is very new - some packages may not be compatible yet" print_warning "Python 3.${PYTHON_MINOR} is very new - some packages may not be compatible yet"
else else
print_warning "Python 3.${PYTHON_MINOR} is outdated - upgrade to 3.10+ recommended" # Pillow 12 and the pinned test tools need 3.10+, so this won't install.
print_error "Python 3.${PYTHON_MINOR} is too old - Python 3.10+ is required"
fi fi
else else
print_error "Python 2.x detected - Python 3.10+ is required" print_error "Python 2.x detected - Python 3.10+ is required"
@@ -155,7 +153,10 @@ ESSENTIAL_PACKAGES=(
for pkg_info in "${ESSENTIAL_PACKAGES[@]}"; do for pkg_info in "${ESSENTIAL_PACKAGES[@]}"; do
IFS=':' read -r pkg desc <<< "$pkg_info" IFS=':' read -r pkg desc <<< "$pkg_info"
if dpkg -l | grep -q "^ii $pkg "; then # dpkg-query rather than `dpkg -l | grep -q`: under pipefail, grep -q
# exiting on its first match kills dpkg with SIGPIPE and fails the pipeline,
# which reported installed packages as missing.
if [ "$(dpkg-query -W -f='${Status}' "$pkg" 2>/dev/null)" = "install ok installed" ]; then
print_success "$desc ($pkg) is installed" print_success "$desc ($pkg) is installed"
else else
print_warning "$desc ($pkg) not installed - will be installed during setup" print_warning "$desc ($pkg) not installed - will be installed during setup"
+99
View File
@@ -0,0 +1,99 @@
#!/usr/bin/env python3
"""Type-check the modules listed in mypy-clean.txt (the mypy ratchet).
Most of src/ still has mypy errors, so CI can't require a clean `mypy src`.
Instead mypy-clean.txt lists the modules that *are* clean, and this script
fails if any of them regresses. When you make another module clean, add it to
the list; nothing ever comes off it.
Imports are followed silently: a listed module is checked against the types of
everything it imports, but errors inside those imported modules are not
reported, so a clean file isn't failed by an unlisted neighbour.
Usage:
python scripts/check_types.py # check the listed modules
python scripts/check_types.py --list # print the list and exit
Extra arguments after ``--`` are passed to mypy.
Exit status: 0 clean, 1 mypy errors, 2 a bad list (missing file, duplicate,
unsorted, or empty).
"""
import argparse
import subprocess # nosec B404 - list-form argv only, no shell # nosemgrep
import sys
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parent.parent
LIST_FILE = REPO_ROOT / "mypy-clean.txt"
def read_list(path: Path = LIST_FILE) -> list:
"""The listed paths, in file order, with comments and blank lines dropped."""
entries = []
for raw in path.read_text(encoding="utf-8").splitlines():
line = raw.split("#", 1)[0].strip()
if line:
entries.append(line)
return entries
def list_problems(entries: list, root: Path = REPO_ROOT) -> list:
"""Why the list can't be used as-is; empty when it is fine."""
problems = []
if not entries:
problems.append(f"{LIST_FILE.name} lists no modules")
seen = set()
for entry in entries:
if entry in seen:
problems.append(f"listed twice: {entry}")
seen.add(entry)
if "\\" in entry:
problems.append(f"use forward slashes: {entry}")
elif not (root / entry).is_file():
problems.append(f"listed but not found (renamed or deleted? update the list): {entry}")
if entries != sorted(entries):
problems.append(f"{LIST_FILE.name} is not sorted")
return problems
def main(argv=None) -> int:
parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
parser.add_argument("--list", action="store_true", help="print the listed modules and exit")
parser.add_argument("mypy_args", nargs="*", help="extra mypy arguments (after --)")
args = parser.parse_args(argv)
entries = read_list()
problems = list_problems(entries)
if problems:
for problem in problems:
print(f"check_types: {problem}", file=sys.stderr)
return 2
if args.list:
print("\n".join(entries))
return 0
cmd = [
sys.executable, "-m", "mypy",
"--config-file", str(REPO_ROOT / "mypy.ini"),
"--follow-imports=silent",
*args.mypy_args,
*entries,
]
print(f"check_types: mypy on {len(entries)} modules from {LIST_FILE.name}", flush=True)
# This interpreter's mypy, fixed flags, and paths from the checked-in list.
result = subprocess.run(cmd, cwd=REPO_ROOT) # nosec B603 - list-form argv, no shell # nosemgrep
if result.returncode > 1: # mypy itself failed (bad config, crash)
return result.returncode
if result.returncode != 0:
print(
"check_types: a module on the mypy ratchet has type errors. Fix them "
f"(annotation-only where possible) rather than taking it off {LIST_FILE.name}.",
file=sys.stderr,
)
return 1
return 0
if __name__ == "__main__":
sys.exit(main())
+55 -4
View File
@@ -200,12 +200,21 @@ def api_plugin_defaults(plugin_id):
return jsonify({'defaults': defaults}) return jsonify({'defaults': defaults})
#: /api/render "vegas" values: the plugin's block of the Vegas strip, built
#: from its live elements (falling back to its Vegas content, as the ticker
#: does) or from its ordinary Vegas content only.
VEGAS_VIEWS = ('live', 'plain')
def _render_once(plugin_id, plugin_dir, manifest, config, mock_data, width, height, def _render_once(plugin_id, plugin_dir, manifest, config, mock_data, width, height,
skip_update): skip_update, vegas=None):
"""Render one plugin at one size. Returns the /api/render response dict. """Render one plugin at one size. Returns the /api/render response dict.
A fresh plugin instance per call, mirroring the safety harness, so sizes A fresh plugin instance per call, mirroring the safety harness, so sizes
never share state. never share state. With ``vegas`` set ('live' or 'plain') the image is the
plugin's block of the Vegas strip instead of its display(), laid out by
the ticker's own code (src/plugin_system/testing/vegas.py), and the
response lists where each live element sits in it.
""" """
from src.plugin_system.testing import VisualTestDisplayManager, MockCacheManager, MockPluginManager from src.plugin_system.testing import VisualTestDisplayManager, MockCacheManager, MockPluginManager
from src.plugin_system.plugin_loader import PluginLoader from src.plugin_system.plugin_loader import PluginLoader
@@ -243,6 +252,10 @@ def _render_once(plugin_id, plugin_dir, manifest, config, mock_data, width, heig
logger.warning("update() raised for plugin %s", plugin_id, exc_info=True) logger.warning("update() raised for plugin %s", plugin_id, exc_info=True)
warnings.append(f"update() raised: {type(e).__name__} — see server log") warnings.append(f"update() raised: {type(e).__name__} — see server log")
if vegas:
return _vegas_response(plugin_id, plugin_instance, display_manager, vegas,
start_time, errors, warnings)
# Run display() # Run display()
try: try:
plugin_instance.display(force_clear=True) plugin_instance.display(force_clear=True)
@@ -262,6 +275,40 @@ def _render_once(plugin_id, plugin_dir, manifest, config, mock_data, width, heig
} }
def _vegas_response(plugin_id, plugin_instance, display_manager, vegas, start_time,
errors, warnings):
"""The /api/render response for the Vegas strip view."""
import base64
import io
from src.plugin_system.testing.vegas import render_vegas_strip
block, layout = None, []
try:
block, layout = render_vegas_strip(plugin_instance, plugin_id, display_manager,
live=(vegas == 'live'))
except Exception as e:
logger.warning("Vegas render raised for plugin %s", plugin_id, exc_info=True)
errors.append(f"Vegas render raised: {type(e).__name__} — see server log")
if block is None:
if not errors:
errors.append("The plugin has no Vegas content")
block = display_manager.image
elif vegas == 'live' and not layout:
warnings.append("No live elements: this is the plugin's ordinary Vegas content")
buffer = io.BytesIO()
block.convert('RGB').save(buffer, format='PNG')
return {
'image': 'data:image/png;base64,' + base64.b64encode(buffer.getvalue()).decode('ascii'),
'width': block.width,
'height': block.height,
'render_time_ms': round((time.time() - start_time) * 1000, 1),
'errors': errors,
'warnings': warnings,
'live_elements': [{'key': key, 'x': x, 'width': width} for x, key, width in layout],
}
def _trusted_plugin_dir(plugin_dir: Path) -> Optional[Path]: def _trusted_plugin_dir(plugin_dir: Path) -> Optional[Path]:
"""Re-derive a plugin directory from the search dirs' own listings. """Re-derive a plugin directory from the search dirs' own listings.
@@ -333,6 +380,10 @@ def api_render():
if not (MIN_HEIGHT <= height <= MAX_HEIGHT): if not (MIN_HEIGHT <= height <= MAX_HEIGHT):
return jsonify({'error': f'height must be between {MIN_HEIGHT} and {MAX_HEIGHT}'}), 400 return jsonify({'error': f'height must be between {MIN_HEIGHT} and {MAX_HEIGHT}'}), 400
vegas = data.get('vegas') or None
if vegas is not None and vegas not in VEGAS_VIEWS:
return jsonify({'error': f'vegas must be one of {", ".join(VEGAS_VIEWS)}'}), 400
try: try:
plugin_dir, manifest, config, mock_data, skip_update = _parse_render_request(data) plugin_dir, manifest, config, mock_data, skip_update = _parse_render_request(data)
except LookupError: except LookupError:
@@ -345,7 +396,7 @@ def api_render():
try: try:
result = _render_once(data['plugin_id'], plugin_dir, manifest, config, result = _render_once(data['plugin_id'], plugin_dir, manifest, config,
mock_data, width, height, skip_update) mock_data, width, height, skip_update, vegas=vegas)
except Exception: except Exception:
app.logger.exception('plugin load failed during render') app.logger.exception('plugin load failed during render')
return jsonify({'error': 'Failed to load plugin; see server log'}), 500 return jsonify({'error': 'Failed to load plugin; see server log'}), 500
@@ -423,7 +474,7 @@ def main():
global _extra_dirs global _extra_dirs
_extra_dirs = args.extra_dir _extra_dirs = args.extra_dir
print(f"LEDMatrix Dev Preview Server") print("LEDMatrix Dev Preview Server")
print(f"Open http://{args.host}:{args.port} in your browser") print(f"Open http://{args.host}:{args.port} in your browser")
print(f"Plugin search dirs: {[str(d) for d in get_search_dirs()]}") print(f"Plugin search dirs: {[str(d) for d in get_search_dirs()]}")
print() print()
+42
View File
@@ -35,6 +35,9 @@ blit copying the frame into the matrix canvas (rgbmatrix SetImage).
wait blocked in SwapOnVSync, i.e. slack before the refresh. wait blocked in SwapOnVSync, i.e. slack before the refresh.
work everything else between two frames: drawing, scrolling, and work everything else between two frames: drawing, scrolling, and
waiting for the GIL. waiting for the GIL.
after work frames presented straight after tagged render-thread work
(Vegas strip extensions, live-element patches), with their own
late rate. Shown only when something tagged its work.
""" """
from __future__ import annotations from __future__ import annotations
@@ -126,6 +129,33 @@ def _edge(index: int, bucket_ms: float):
return round((index + 1) * bucket_ms, 2) return round((index + 1) * bucket_ms, 2)
def op_rows(totals: Dict[str, Any]) -> Dict[str, Dict[str, Any]]:
"""Per kind of noted render-thread work: how often its frame was late.
A kind's frames are the ones presented straight after that work ran (see
"Operations" in src/common/frame_timing.py). Stats from a recorder that
predates the counters have none, and give an empty table.
"""
frames = totals.get("op_frames") or {}
late = totals.get("late_op_frames") or {}
freezes = totals.get("op_freezes") or {}
moved = totals.get("op_bytes") or {}
rows = {}
for kind in sorted(set(frames) | set(freezes)):
count = frames.get(kind, 0)
if not count and not freezes.get(kind, 0):
continue
rows[kind] = {
"frames": count,
"late": late.get(kind, 0),
"late_pct": (round(100.0 * late.get(kind, 0) / count, 3)
if count else None),
"freezes": freezes.get(kind, 0),
"bytes": moved.get(kind, 0),
}
return rows
def build_report(before, after, preview: bool) -> Dict[str, Any]: def build_report(before, after, preview: bool) -> Dict[str, Any]:
delta = diff(before, after) delta = diff(before, after)
totals = delta["totals"] totals = delta["totals"]
@@ -159,6 +189,7 @@ def build_report(before, after, preview: bool) -> Dict[str, Any]:
if totals["worst_interval_ms"] else None), if totals["worst_interval_ms"] else None),
"timing_ms": {name: percentiles(h, bucket_ms) "timing_ms": {name: percentiles(h, bucket_ms)
for name, h in delta["histograms"].items()}, for name, h in delta["histograms"].items()},
"ops": op_rows(totals),
} }
# The rate the panel held while rendering: the typical frame's interval # The rate the panel held while rendering: the typical frame's interval
# per refresh held. A few percent under the idle rate is normal (the Pi is # per refresh held. A few percent under the idle rate is normal (the Pi is
@@ -216,6 +247,17 @@ def print_report(report: Dict[str, Any], limit: float) -> None:
print(f"{name:<18}" + "".join(f"{str(row.get(k, '-')):>8}" print(f"{name:<18}" + "".join(f"{str(row.get(k, '-')):>8}"
for k in ("p50", "p95", "p99", "max"))) for k in ("p50", "p95", "p99", "max")))
print() print()
ops = report.get("ops") or {}
if ops:
# Frames presented straight after render-thread work of each kind. A
# late rate well above the overall one points at that work.
print(f"{'after work':<18}{'frames':>8}{'late':>8}{'late %':>8}"
f"{'freezes':>9}{'MB moved':>10}")
for kind, row in ops.items():
pct = "-" if row["late_pct"] is None else f"{row['late_pct']:g}"
print(f"{kind:<18}{row['frames']:>8}{row['late']:>8}{pct:>8}"
f"{row['freezes']:>9}{row['bytes'] / 1e6:>10.2f}")
print()
if report["late_pct"] is None: if report["late_pct"] is None:
print("RESULT nothing scrolled - no verdict") print("RESULT nothing scrolled - no verdict")
elif not locked(report, limit): elif not locked(report, limit):
+20 -5
View File
@@ -26,11 +26,26 @@ fi
# Get the full paths to commands and validate each one # Get the full paths to commands and validate each one
MISSING_CMDS=() MISSING_CMDS=()
SYSTEMCTL_PATH=$(command -v systemctl) || true # Full path of a command, also looking in the sbin directories. This script runs
REBOOT_PATH=$(command -v reboot) || true # as the web user, whose PATH usually lacks /usr/sbin and /sbin -- where reboot
POWEROFF_PATH=$(command -v poweroff) || true # and poweroff live -- so `command -v` alone silently dropped their rules.
BASH_PATH=$(command -v bash) || true find_command() {
JOURNALCTL_PATH=$(command -v journalctl) || true local found
found=$(command -v "$1" 2>/dev/null) && { printf '%s\n' "$found"; return 0; }
for dir in /usr/sbin /sbin /usr/bin /bin; do
if [ -x "$dir/$1" ]; then
printf '%s\n' "$dir/$1"
return 0
fi
done
return 1
}
SYSTEMCTL_PATH=$(find_command systemctl) || true
REBOOT_PATH=$(find_command reboot) || true
POWEROFF_PATH=$(find_command poweroff) || true
BASH_PATH=$(find_command bash) || true
JOURNALCTL_PATH=$(find_command journalctl) || true
SAFE_RM_PATH="$PROJECT_ROOT/scripts/fix_perms/safe_plugin_rm.sh" SAFE_RM_PATH="$PROJECT_ROOT/scripts/fix_perms/safe_plugin_rm.sh"
SAFE_PIP_INSTALL_PATH="$PROJECT_ROOT/scripts/fix_perms/safe_pip_install.sh" SAFE_PIP_INSTALL_PATH="$PROJECT_ROOT/scripts/fix_perms/safe_pip_install.sh"
+778
View File
@@ -0,0 +1,778 @@
#!/usr/bin/env python3
"""Who still calls or overrides the core methods marked ``@deprecated``?
A deprecated plugin-facing method may only be removed once nothing uses it,
and plugins live in other repositories. This script answers the question for
every method ``src/deprecation.py``'s decorator marks in core:
1. it lists the markers by parsing ``src/`` (so the list can never drift from
the code);
2. it scans, with the ``ast`` module, core itself (``src/``,
``web_interface/``, ``scripts/``, the top-level ``*.py``; ``test/``
separately), the official monorepo's ``plugins/`` directory, and every
third-party plugin the monorepo's ``plugins.json`` lists with its own repo
URL (shallow-cloned read-only into a cache directory);
3. it reports, per method and per plugin, the calls and overrides it found,
and a verdict: unused (safe to remove in the marker's release), still used
(keep or migrate those plugins first), or needs review.
Matching is by method name, so it has to separate real uses from unrelated
methods that happen to share the name (the weather plugin's own ``draw_sun``,
say). Each hit is classified by what it is attached to:
* **call** -- ``<receiver>.name`` where the receiver is named like the owning
object (``self.cache_manager``, ``display_manager``, ``plugin_manager`` ...,
or a local alias assigned from one), or ``self``/``super()`` inside a class
that subclasses the owner. Attribute references that are not called
(``callback=cm.get_cache_metrics``) count too.
* **override** -- ``def name`` in a class that subclasses the owner.
* **review** -- ``<receiver>.name`` where the receiver says nothing about its
type, or ``getattr(obj, "name")``. Possibly a real use; read the listed line.
* **unrelated** -- ``self.name`` inside a class that defines ``name`` itself
and does not subclass the owner, ``Klass.name`` where the same tree defines
``Klass.name``, or ``def name`` in such a class: a name collision, not a use.
* **internal** -- a hit inside the body of another deprecated core method
(``draw_rain`` calling ``draw_cloud``): it keeps the method only as long as
that caller is kept.
Only calls and overrides make a method "still used"; review hits make it
"needs review"; hits in test files are listed but never block removal (a test
that mocks a method does not need it to exist).
python3 scripts/plugin_api_usage.py # clone everything, print Markdown
python3 scripts/plugin_api_usage.py --monorepo ../ledmatrix-plugins
python3 scripts/plugin_api_usage.py --output docs/DEPRECATIONS_3.8.md
python3 scripts/plugin_api_usage.py --format json
Nothing is ever written to the repositories it scans: the monorepo path is only
read, and clones live in ``--cache-dir``.
"""
from __future__ import annotations
import argparse
import ast
import json
import os
import re
import subprocess # nosec B404 - list-form argv only, no shell # nosemgrep
import sys
import tempfile
from collections import defaultdict
from dataclasses import dataclass, field
from datetime import datetime, timezone
from pathlib import Path
from typing import Dict, Iterable, Iterator, List, Optional, Set, Tuple
REPO_ROOT = Path(__file__).resolve().parent.parent
MONOREPO_URL = "https://github.com/ChuckBuilds/ledmatrix-plugins"
MONOREPO_SLUG = "chuckbuilds/ledmatrix-plugins"
#: Receiver names that mean "this is the owning core object". Compared against
#: the last name in the receiver (``self.plugin_manager.cache_manager`` ->
#: ``cache_manager``), lower-cased with leading underscores stripped.
OWNER_RECEIVERS: Dict[str, Tuple[str, ...]] = {
"CacheManager": ("cache_manager", "cache_mgr", "cachemanager", "cache", "cm"),
"DisplayManager": ("display_manager", "display_mgr", "displaymanager", "display", "dm"),
"FontManager": ("font_manager", "font_mgr", "fontmanager", "fonts", "fm"),
"PluginManager": ("plugin_manager", "plugin_mgr", "pluginmanager", "pm"),
}
#: Directories never scanned (vendored environments, VCS metadata, caches).
SKIP_DIRS = {".git", "__pycache__", "node_modules", ".venv", "venv", "env",
"site-packages", ".tox", ".mypy_cache", ".pytest_cache"}
CORE_DIRS = ("src", "web_interface", "scripts")
CORE_TEST_DIRS = ("test",)
# --------------------------------------------------------------------------
# Markers
@dataclass
class Marker:
owner: str # class name, e.g. "CacheManager"
method: str
removal: str
alternative: Optional[str]
module: str # e.g. "src.cache_manager"
line: int
@property
def key(self) -> str:
return f"{self.owner}.{self.method}"
def _decorator_name(node: ast.expr) -> Optional[str]:
target = node.func if isinstance(node, ast.Call) else node
if isinstance(target, ast.Name):
return target.id
if isinstance(target, ast.Attribute):
return target.attr
return None
def find_markers(core_root: Path) -> List[Marker]:
"""Every ``@deprecated(...)`` method under ``core_root/src``."""
markers: List[Marker] = []
for path in sorted((core_root / "src").rglob("*.py")):
if path.name == "deprecation.py":
continue
tree = _parse(path)
if tree is None:
continue
module = ".".join(path.relative_to(core_root).with_suffix("").parts)
for cls in (n for n in ast.walk(tree) if isinstance(n, ast.ClassDef)):
for fn in cls.body:
if not isinstance(fn, (ast.FunctionDef, ast.AsyncFunctionDef)):
continue
for dec in fn.decorator_list:
if _decorator_name(dec) != "deprecated" or not isinstance(dec, ast.Call):
continue
args = [a.value if isinstance(a, ast.Constant) else None for a in dec.args]
kw = {k.arg: k.value.value for k in dec.keywords
if isinstance(k.value, ast.Constant)}
removal = args[0] if args else kw.get("removal")
alternative = args[1] if len(args) > 1 else kw.get("alternative")
markers.append(Marker(cls.name, fn.name, str(removal), alternative,
module, fn.lineno))
return markers
# --------------------------------------------------------------------------
# Scanning
@dataclass
class Hit:
kind: str # call | override | review | unrelated | internal
path: str
line: int
code: str
test: bool
via: Optional[str] = None # internal: the deprecated core method it sits in
@dataclass
class Source:
"""One plugin (or core) tree to scan."""
name: str
group: str # core | core-tests | monorepo | third-party
root: Optional[Path]
error: Optional[str] = None
hits: Dict[str, List[Hit]] = field(default_factory=lambda: defaultdict(list))
files: int = 0 # Python files scanned
def _parse(path: Path) -> Optional[ast.AST]:
try:
return ast.parse(path.read_text(encoding="utf-8", errors="replace"), str(path))
except (SyntaxError, ValueError):
return None
def _iter_py(root: Path) -> Iterator[Path]:
for dirpath, dirnames, filenames in os.walk(root):
dirnames[:] = [d for d in dirnames if d not in SKIP_DIRS]
for name in filenames:
if name.endswith(".py"):
yield Path(dirpath) / name
def _is_test_path(rel: Path) -> bool:
parts = [p.lower() for p in rel.parts]
return (any(p in ("test", "tests") for p in parts[:-1])
or parts[-1].startswith("test_") or parts[-1].endswith("_test.py")
or parts[-1] == "conftest.py")
def _terminal(node: ast.expr) -> Optional[str]:
"""The last name in a receiver expression, or None if it has none."""
if isinstance(node, ast.Name):
return node.id
if isinstance(node, ast.Attribute):
return node.attr
if isinstance(node, ast.Call):
return _terminal(node.func)
if isinstance(node, ast.Subscript):
return _terminal(node.value)
return None
def _norm(name: Optional[str]) -> str:
return (name or "").lstrip("_").lower()
def _base_names(cls: ast.ClassDef) -> List[str]:
return [t for t in (_terminal(b) for b in cls.bases) if t]
class _Scanner(ast.NodeVisitor):
"""Collect hits for every marked method name in one file."""
def __init__(self, markers: Dict[str, List[Marker]], lines: List[str],
rel: str, test: bool, core_modules: Dict[str, str], module: Optional[str],
local_definers: Dict[str, Set[str]], built: Dict[str, str]):
self.markers = markers # method name -> markers with that name
self.local_definers = local_definers # method name -> this tree's own classes/modules defining it
self.built = built # ``x``/``self.x`` -> class it was built from in this file
self.lines = lines
self.rel = rel
self.test = test
self.core_modules = core_modules # owner class -> defining module (core only)
self.module = module # this file's module when scanning core
self.classes: List[ast.ClassDef] = []
self.scope: List[ast.AST] = [] # enclosing classes and functions
self.aliases: List[Dict[str, str]] = [{}] # local name -> owner class
self.out: Dict[str, List[Hit]] = defaultdict(list)
# -- helpers
def _code(self, node: ast.AST) -> str:
line = self.lines[node.lineno - 1] if 0 < node.lineno <= len(self.lines) else ""
return line.strip()[:160]
def _add(self, marker: Marker, kind: str, node: ast.AST) -> None:
via = self._inside_deprecated()
if via and kind != "unrelated":
# Only reached through another deprecated method: goes when that does.
kind = "internal"
self.out[marker.key].append(Hit(kind, self.rel, node.lineno, self._code(node),
self.test, via if kind == "internal" else None))
def _inside_deprecated(self) -> Optional[str]:
"""``Owner.method`` when this node sits in a deprecated core method's body."""
for i in range(len(self.scope) - 2, -1, -1):
cls, fn = self.scope[i], self.scope[i + 1]
if isinstance(cls, ast.ClassDef):
if isinstance(fn, (ast.FunctionDef, ast.AsyncFunctionDef)):
for m in self.markers.get(fn.name, ()):
if self._is_owner_class(cls, m.owner):
return m.key
return None
return None
def _owner_for_receiver(self, name: Optional[str]) -> Optional[str]:
n = _norm(name)
for scope in reversed(self.aliases):
if name in scope:
return scope[name]
for owner, receivers in OWNER_RECEIVERS.items():
if n in receivers:
return owner
return None
def _is_owner_class(self, cls: ast.ClassDef, owner: str) -> bool:
"""True for the real core class (only when scanning its own module)."""
return (self.module is not None and cls.name == owner
and self.core_modules.get(owner) == self.module)
def _subclasses(self, cls: ast.ClassDef, owner: str) -> bool:
return owner in _base_names(cls)
def _class_defines(self, cls: ast.ClassDef, name: str) -> bool:
return any(isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef)) and n.name == name
for n in cls.body)
# -- scopes
def visit_ClassDef(self, node: ast.ClassDef) -> None:
for fn in node.body:
if isinstance(fn, (ast.FunctionDef, ast.AsyncFunctionDef)) and fn.name in self.markers:
for m in self.markers[fn.name]:
if self._is_owner_class(node, m.owner):
continue # the definition itself
kind = "override" if self._subclasses(node, m.owner) else "unrelated"
self._add(m, kind, fn)
self.classes.append(node)
self.scope.append(node)
self.generic_visit(node)
self.scope.pop()
self.classes.pop()
def _visit_function(self, node) -> None:
self.aliases.append({})
self.scope.append(node)
self.generic_visit(node)
self.scope.pop()
self.aliases.pop()
visit_FunctionDef = _visit_function
visit_AsyncFunctionDef = _visit_function
def visit_Assign(self, node: ast.Assign) -> None:
# ``dm = self.display_manager`` makes ``dm.draw_sun()`` a call.
owner = self._owner_for_receiver(_terminal(node.value))
for target in node.targets:
if isinstance(target, ast.Name) and owner:
self.aliases[-1][target.id] = owner
self.generic_visit(node)
# -- uses
def visit_Attribute(self, node: ast.Attribute) -> None:
if node.attr in self.markers:
for m in self.markers[node.attr]:
self._add(m, self._classify(node, m), node)
self.generic_visit(node)
def _classify(self, node: ast.Attribute, m: Marker) -> str:
recv = node.value
cls = self.classes[-1] if self.classes else None
is_self = isinstance(recv, ast.Name) and recv.id in ("self", "cls")
is_super = (isinstance(recv, ast.Call) and isinstance(recv.func, ast.Name)
and recv.func.id == "super")
if is_self or is_super:
if cls is not None and (self._is_owner_class(cls, m.owner) or self._subclasses(cls, m.owner)):
return "call"
if cls is not None and self._class_defines(cls, m.method):
return "unrelated"
return "review"
name = _terminal(recv)
if self._owner_for_receiver(name) == m.owner:
return "call"
definers = self.local_definers.get(m.method, ())
if name in definers or self.built.get(name or "") in definers:
# e.g. the weather plugin's WeatherIcons.draw_sun, or
# self._strategy_component = CacheStrategy(); ...get_sport_live_interval()
return "unrelated"
return "review"
def visit_Call(self, node: ast.Call) -> None:
func = node.func
if (isinstance(func, ast.Name) and func.id in ("getattr", "hasattr", "setattr", "delattr")
and len(node.args) >= 2 and isinstance(node.args[1], ast.Constant)
and node.args[1].value in self.markers):
for m in self.markers[node.args[1].value]:
owner = self._owner_for_receiver(_terminal(node.args[0]))
self._add(m, "call" if owner == m.owner else "review", node)
self.generic_visit(node)
def _built_from(tree: ast.AST) -> Dict[str, str]:
"""``{name: Class}`` for every ``name = Class(...)`` / ``self.name = Class(...)``."""
built: Dict[str, str] = {}
for node in ast.walk(tree):
if isinstance(node, ast.Assign) and isinstance(node.value, ast.Call):
cls = _terminal(node.value.func)
for target in node.targets:
name = _terminal(target) if isinstance(target, (ast.Name, ast.Attribute)) else None
if name and cls:
built[name] = cls
return built
def scan_tree(source: Source, roots: Iterable[Path], base: Path, markers: List[Marker],
core: bool, test_override: Optional[bool] = None,
definer_roots: Iterable[Path] = ()) -> None:
by_name: Dict[str, List[Marker]] = defaultdict(list)
for m in markers:
by_name[m.method].append(m)
core_modules = {m.owner: m.module for m in markers}
owners = {m.owner for m in markers}
files: List[Tuple[Path, str, Optional[ast.AST]]] = []
for root in roots:
if not root.exists():
continue
for path in ([root] if root.is_file() else sorted(_iter_py(root))):
source.files += 1
text = path.read_text(encoding="utf-8", errors="replace")
if any(name in text for name in by_name):
files.append((path, text, _parse(path)))
# Classes (and modules) in this tree with their own method of a marked
# name, so ``WeatherIcons.draw_sun()`` is recognised as theirs.
local_definers: Dict[str, Set[str]] = defaultdict(set)
definer_files = list(files)
for root in definer_roots:
for path in (sorted(_iter_py(root)) if root.is_dir() else ()):
text = path.read_text(encoding="utf-8", errors="replace")
if any(name in text for name in by_name):
definer_files.append((path, text, _parse(path)))
for path, _, tree in definer_files:
for node in (tree.body if tree is not None else ()):
if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) and node.name in by_name:
local_definers[node.name].add(path.stem)
for cls in (n for n in ast.walk(tree) if isinstance(n, ast.ClassDef)) if tree else ():
if cls.name in owners and core:
continue
if owners & set(_base_names(cls)):
continue
for fn in cls.body:
if isinstance(fn, (ast.FunctionDef, ast.AsyncFunctionDef)) and fn.name in by_name:
local_definers[fn.name].add(cls.name)
for path, text, tree in files:
rel = path.relative_to(base)
test = _is_test_path(rel) if test_override is None else test_override
if tree is None:
# Unparseable (Python 2, a template ...): fall back to text, as review.
for no, line in enumerate(text.splitlines(), 1):
for name in by_name:
if re.search(rf"{re.escape(name)}", line):
for m in by_name[name]:
source.hits[m.key].append(
Hit("review", rel.as_posix(), no, line.strip()[:160], test))
continue
module = ".".join(rel.with_suffix("").parts) if core else None
scanner = _Scanner(by_name, text.splitlines(), rel.as_posix(), test,
core_modules, module, local_definers, _built_from(tree))
scanner.visit(tree)
for key, hits in scanner.out.items():
source.hits[key].extend(hits)
# --------------------------------------------------------------------------
# Fetching plugin trees (read-only)
def _git(*args: str, cwd: Optional[Path] = None) -> subprocess.CompletedProcess:
# Never stop to ask for credentials: a deleted or private plugin repo
# should be reported as not scanned, not hang the scan.
env = {**os.environ, "GIT_TERMINAL_PROMPT": "0"}
return subprocess.run( # nosec B603 B607 - list-form git argv, no shell; URLs follow "--" # nosemgrep
["git", *args], cwd=cwd, capture_output=True, text=True,
encoding="utf-8", errors="replace", timeout=300, env=env)
def _rmtree(path: Path) -> None:
"""Delete a clone; git marks pack files read-only, which Windows refuses to delete."""
import shutil
import stat
def retry(func, target, _exc):
os.chmod(target, stat.S_IWRITE)
func(target)
if sys.version_info >= (3, 12):
shutil.rmtree(path, onexc=retry)
else:
shutil.rmtree(path, onerror=retry)
def shallow_clone(url: str, branch: Optional[str], dest: Path, reuse: bool) -> Optional[str]:
"""Clone ``url`` into ``dest`` (depth 1), replacing any earlier clone.
Returns an error string, or None on success.
"""
if reuse and (dest / ".git").exists():
return None
if dest.exists():
_rmtree(dest)
dest.parent.mkdir(parents=True, exist_ok=True)
args = ["-c", "core.longpaths=true", "clone", "--quiet", "--depth", "1"]
if branch:
args += ["--branch", branch]
# "--" ends option parsing: a registry URL starting with "-" (for example
# "--upload-pack=...") is then only ever a repository argument.
result = _git(*args, "--", url, str(dest))
if result.returncode != 0 and branch:
result = _git("-c", "core.longpaths=true", "clone", "--quiet", "--depth", "1",
"--", url, str(dest))
if result.returncode != 0:
lines = (result.stderr or result.stdout).strip().splitlines()
return lines[-1] if lines else "git clone failed"
return None
def _head(path: Path, branch: bool = True) -> str:
"""Commit (and branch) of a checkout, via read-only git calls."""
rev = _git("--no-optional-locks", "rev-parse", "--short=8", "HEAD", cwd=path)
if rev.returncode != 0:
return "unknown revision"
if not branch:
return rev.stdout.strip()
ref = _git("--no-optional-locks", "rev-parse", "--abbrev-ref", "HEAD", cwd=path)
return f"{ref.stdout.strip()} @ {rev.stdout.strip()}"
def _plugin_id(plugin_dir: Path) -> str:
try:
return json.loads((plugin_dir / "manifest.json").read_text(encoding="utf-8"))["id"]
except (OSError, ValueError, KeyError, TypeError):
return plugin_dir.name
def _is_monorepo(url: str) -> bool:
return MONOREPO_SLUG in url.lower().rstrip("/").removesuffix(".git")
# --------------------------------------------------------------------------
# Report
def verdicts(markers: List[Marker], sources: List[Source]) -> Dict[str, Tuple[str, str]]:
"""``{Owner.method: (status, text)}`` across every source.
An *internal* hit (a call from inside another deprecated method) keeps a
method only while that caller is itself kept, so statuses are resolved
until they stop changing.
"""
failed = [s.name for s in sources if s.error]
status: Dict[str, Tuple[str, str]] = {}
for _ in range(len(markers) + 1):
changed = False
for m in markers:
used, review = [], []
for s in sources:
live = [h for h in s.hits.get(m.key, []) if not h.test]
if any(h.kind in ("call", "override") for h in live) or any(
h.kind == "internal" and status.get(h.via, ("",))[0] == "used"
for h in live):
used.append(s.name)
elif any(h.kind == "review" for h in live) or any(
h.kind == "internal" and status.get(h.via, ("",))[0] == "review"
for h in live):
review.append(s.name)
if used:
new = ("used", f"still used by {', '.join(used)} — keep or migrate first")
elif review:
new = ("review", f"needs review: possible use in {', '.join(review)}")
elif failed:
new = ("unknown", f"not proven unused: {len(failed)} plugin(s) could not be scanned")
else:
new = ("unused", f"unused — safe to remove in {m.removal}")
if status.get(m.key) != new:
status[m.key] = new
changed = True
if not changed:
break
return status
def _counts(hits: List[Hit]) -> Dict[str, int]:
c: Dict[str, int] = defaultdict(int)
for h in hits:
c[("test " if h.test else "") + h.kind] += 1
return c
def _usage_cell(marker: Marker, sources: List[Source], kinds: Tuple[str, ...]) -> str:
parts = []
for s in sources:
c = _counts(s.hits.get(marker.key, []))
bits = [f"{c[k]} {k}{'s' if c[k] != 1 else ''}" for k in kinds if c[k]]
if bits:
parts.append(f"{s.name} ({', '.join(bits)})")
return "; ".join(parts) or "—"
def render_markdown(markers: List[Marker], sources: List[Source], meta: Dict[str, str]) -> str:
out: List[str] = []
w = out.append
w("# Deprecated plugin APIs: usage scan")
w("")
w("Generated by `scripts/plugin_api_usage.py` — do not edit by hand; re-run it "
"(see [How to re-run](#how-to-re-run)).")
w("")
w(f"- Scanned: {meta['date']}, core {meta['core_version']}")
w(f"- Monorepo: {meta['monorepo']}")
w(f"- Third-party plugins: {meta['third_party']}")
failed = [s for s in sources if s.error]
if failed:
w("- **Not scanned:** " + "; ".join(f"{s.name} ({s.error})" for s in failed))
w("")
status = verdicts(markers, sources)
tally: Dict[str, int] = defaultdict(int)
for st, _ in status.values():
tally[st] += 1
w(f"**{len(markers)} deprecated methods: {tally['unused']} unused, "
f"{tally['used']} still used, {tally['review']} need review"
+ (f", {tally['unknown']} not proven" if tally["unknown"] else "") + ".**")
w("")
w("Counted per plugin: a *call* is `<receiver>.method` on an object named like "
"the owner (`cache_manager`, `display_manager`, `font_manager`, `plugin_manager`), "
"or on `self`/`super()` in a subclass; an *override* is `def method` in a subclass "
"of the owner. *Review* hits are `.method` on a receiver whose type the scan cannot "
"tell. *Internal* hits sit inside another deprecated core method and go with it. "
"*Unrelated* hits are a different class's own method with the same name "
"(a name collision), and never block removal; neither do hits in test files.")
w("")
w("| Method | Removal | Core | Plugins (calls / overrides) | Name collisions & tests | Verdict |")
w("|---|---|---|---|---|---|")
core_sources = [s for s in sources if s.group in ("core", "core-tests")]
plugin_sources = [s for s in sources if s.group not in ("core", "core-tests")]
for m in markers:
core = _usage_cell(m, core_sources, ("call", "override", "review", "internal",
"test call", "test override", "test review",
"test internal"))
plugins = _usage_cell(m, plugin_sources, ("call", "override", "review"))
other = _usage_cell(m, plugin_sources, ("unrelated", "test call", "test override",
"test review", "test unrelated"))
w(f"| `{m.key}` | {m.removal} | {core} | {plugins} | {other} | {status[m.key][1]} |")
w("")
groups = [("unused", "Unused — safe to remove"), ("used", "Still used — keep or migrate first"),
("review", "Needs review"), ("unknown", "Not proven unused")]
for st, title in groups:
names = [m.key for m in markers if status[m.key][0] == st]
if names:
w(f"## {title} ({len(names)})")
w("")
w(", ".join(f"`{n}`" for n in names))
w("")
detail = [(m, s, h) for m in markers for s in sources
for h in s.hits.get(m.key, []) if h.kind != "unrelated" or not h.test]
if detail:
w("## Every hit")
w("")
w("File paths are relative to the plugin's directory (core: the repo root).")
w("")
w("| Method | Where | File:line | Kind | Code |")
w("|---|---|---|---|---|")
for m, s, h in detail:
kind = ("test " if h.test else "") + h.kind
if h.via:
kind += f" (in `{h.via}`)"
code = h.code.replace("|", "\\|").replace("`", "'")
w(f"| `{m.key}` | {s.name} | {h.path}:{h.line} | {kind} | `{code}` |")
w("")
w("## Sources scanned")
w("")
w("| Source | Group | Python files | Hits |")
w("|---|---|---|---|")
for s in sources:
n = sum(len(v) for v in s.hits.values())
files = f"not scanned: {s.error}" if s.error else str(s.files)
w(f"| {s.name} | {s.group} | {files} | {n} |")
w("")
w("## How to re-run")
w("")
w("```bash")
w("# Clones the monorepo and each third-party plugin (depth 1) into a temp cache:")
w("python3 scripts/plugin_api_usage.py --output docs/DEPRECATIONS_3.8.md")
w("# Or scan a local monorepo checkout (read only) instead of cloning it:")
w("python3 scripts/plugin_api_usage.py --monorepo ../ledmatrix-plugins")
w("```")
w("")
w("Before removing a method in its release, re-run the scan against the current "
"monorepo and registry: a plugin added since this file was generated may have "
"started calling it. Remove only methods the fresh scan reports unused; move "
"the rest to a later release (the test in `test/test_deprecation.py` fails "
"while a marker names a release at or below `src.__version__`).")
w("")
return "\n".join(out)
def render_json(markers: List[Marker], sources: List[Source], meta: Dict[str, str]) -> str:
data = {"meta": meta, "sources": [{"name": s.name, "group": s.group, "error": s.error}
for s in sources], "methods": []}
status = verdicts(markers, sources)
for m in markers:
st, text = status[m.key]
data["methods"].append({
"method": m.key, "module": m.module, "removal": m.removal,
"alternative": m.alternative, "status": st, "verdict": text,
"hits": [{"source": s.name, **h.__dict__} for s in sources
for h in s.hits.get(m.key, [])],
})
return json.dumps(data, indent=2)
# --------------------------------------------------------------------------
def main(argv: Optional[List[str]] = None) -> int:
parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
parser.add_argument("--monorepo", type=Path,
help="local ledmatrix-plugins checkout to scan (read only); "
"default: shallow-clone its main branch")
parser.add_argument("--registry", type=Path,
help="plugins.json to read third-party plugins from "
"(default: the monorepo's)")
parser.add_argument("--cache-dir", type=Path,
default=Path(tempfile.gettempdir()) / "ledmatrix-plugin-api-usage",
help="where clones go (default: %(default)s)")
parser.add_argument("--reuse-cache", action="store_true",
help="scan clones already in --cache-dir instead of re-cloning "
"(offline re-runs; the report may then be stale)")
parser.add_argument("--no-third-party", action="store_true",
help="skip third-party plugins (the report then cannot prove anything unused)")
parser.add_argument("--format", choices=("md", "json"), default="md")
parser.add_argument("--output", type=Path, help="write the report here instead of stdout")
args = parser.parse_args(argv)
if hasattr(sys.stdout, "reconfigure"):
sys.stdout.reconfigure(encoding="utf-8")
markers = find_markers(REPO_ROOT)
if not markers:
print("No @deprecated markers found in src/.", file=sys.stderr)
return 0
sys.path.insert(0, str(REPO_ROOT))
try:
from src import __version__ as core_version
except Exception: # noqa: BLE001 -- reporting only
core_version = "unknown"
sources: List[Source] = []
core = Source("core", "core", REPO_ROOT)
scan_tree(core, [REPO_ROOT / d for d in CORE_DIRS] + sorted(REPO_ROOT.glob("*.py")),
REPO_ROOT, markers, core=True)
core_tests = Source("core tests", "core-tests", REPO_ROOT)
scan_tree(core_tests, [REPO_ROOT / d for d in CORE_TEST_DIRS], REPO_ROOT, markers,
core=True, test_override=True,
definer_roots=[REPO_ROOT / d for d in CORE_DIRS])
sources += [core, core_tests]
# Monorepo
if args.monorepo:
mono = args.monorepo.resolve()
mono_desc = f"local checkout `{mono.name}` ({_head(mono)})"
else:
mono = args.cache_dir / "ledmatrix-plugins"
err = shallow_clone(MONOREPO_URL, "main", mono, args.reuse_cache)
if err:
print(f"Could not clone the monorepo: {err}", file=sys.stderr)
return 1
mono_desc = f"[ChuckBuilds/ledmatrix-plugins]({MONOREPO_URL}) ({_head(mono)})"
plugins_dir = mono / "plugins"
mono_dirs = sorted(p for p in plugins_dir.iterdir() if p.is_dir()) if plugins_dir.is_dir() else []
for d in mono_dirs:
s = Source(_plugin_id(d), "monorepo", d)
scan_tree(s, [d], d, markers, core=False)
sources.append(s)
mono_desc += f", {len(mono_dirs)} plugins"
# Third-party plugins from the registry
registry = args.registry or (mono / "plugins.json")
third: List[dict] = []
try:
reg = json.loads(registry.read_text(encoding="utf-8"))
entries = reg["plugins"] if isinstance(reg, dict) else reg
third = [e for e in entries if e.get("repo") and not _is_monorepo(e["repo"])]
except (OSError, ValueError, KeyError) as exc:
print(f"Could not read {registry}: {exc}", file=sys.stderr)
return 1
if args.no_third_party:
tp_desc = "skipped (--no-third-party)"
else:
for e in third:
dest = args.cache_dir / "third-party" / re.sub(r"[^\w.-]", "_", e["id"])
err = shallow_clone(e["repo"], e.get("branch") or None, dest, args.reuse_cache)
root = dest / e["plugin_path"] if e.get("plugin_path") else dest
s = Source(e["id"], "third-party", root, error=err)
if not err:
scan_tree(s, [root], root, markers, core=False)
sources.append(s)
tp_desc = (f"{len(third)} with their own repo in `plugins.json` "
f"({', '.join(e['id'] for e in third)})")
meta = {
"date": datetime.now(timezone.utc).strftime("%Y-%m-%d"),
"core_version": core_version,
"core_rev": _head(REPO_ROOT, branch=False),
"monorepo": mono_desc,
"third_party": tp_desc,
}
report = (render_json if args.format == "json" else render_markdown)(markers, sources, meta)
if args.output:
args.output.write_text(report, encoding="utf-8", newline="\n")
print(f"Wrote {args.output}", file=sys.stderr)
else:
sys.stdout.write(report)
return 0
if __name__ == "__main__":
sys.exit(main())
+192 -7
View File
@@ -25,6 +25,14 @@ against another) and for A/B testing a change to the render path.
sudo python3 scripts/render_bench.py --busy 2 # with background load sudo python3 scripts/render_bench.py --busy 2 # with background load
sudo python3 scripts/render_bench.py --json /tmp/pi4.json sudo python3 scripts/render_bench.py --json /tmp/pi4.json
# the cost of changing pixels under a moving strip (Vegas live elements):
# a 101KB write into the visible columns every 25 frames
sudo python3 scripts/render_bench.py --patch-bytes 101376 --patch-every 25
# the cost of extending a Vegas-sized strip on the render thread: a 30-screen
# strip, extended by 6 screens (and trimmed) every 6 screens scrolled
sudo python3 scripts/render_bench.py --strip-screens 30 --extend-every-screens 6
sudo systemctl start ledmatrix sudo systemctl start ledmatrix
Like scripts/scroll_speeds.py, this never starts or stops the service itself, Like scripts/scroll_speeds.py, this never starts or stops the service itself,
@@ -92,13 +100,17 @@ def load_config() -> dict:
return config return config
def build_strip(width: int, height: int, label: str): def build_strip(width: int, height: int, label: str, screens: float = 4.0):
"""A marquee strip a few screens wide, with text and colour. """A marquee strip about ``screens`` screens wide, with text and colour.
Deliberately not plain white text on black: how long ``SetImage`` takes Deliberately not plain white text on black: how long ``SetImage`` takes
depends on how many subpixels are lit, so a strip that is mostly dark depends on how many subpixels are lit, so a strip that is mostly dark
flatters the panel and hides exactly the regression this benchmark exists flatters the panel and hides exactly the regression this benchmark exists
to catch. to catch.
The width matters to the extension mode, whose cost is a copy of the whole
strip: Vegas carries 8,000-20,000 columns, so measure extension against a
strip that wide (``--strip-screens``), not the four-screen default.
""" """
from PIL import Image, ImageDraw, ImageFont from PIL import Image, ImageDraw, ImageFont
@@ -123,7 +135,7 @@ def build_strip(width: int, height: int, label: str):
text_width = max(1, box[2] - box[0]) text_width = max(1, box[2] - box[0])
text_height = box[3] - box[1] text_height = box[3] - box[1]
reps = max(2, (width * 4) // text_width + 1) reps = max(2, int(width * screens) // text_width + 1)
strip = Image.new("RGB", (text_width * reps, height), (0, 0, 0)) strip = Image.new("RGB", (text_width * reps, height), (0, 0, 0))
draw = ImageDraw.Draw(strip) draw = ImageDraw.Draw(strip)
draw.fontmode = "1" # the panel has no partial brightness; see DisplayManager draw.fontmode = "1" # the panel has no partial brightness; see DisplayManager
@@ -179,6 +191,119 @@ class BackgroundLoad:
zlib.compress(image.tobytes(), 1) zlib.compress(image.tobytes(), 1)
class StripWork:
"""Render-thread work a Vegas strip does between frames, on a schedule.
Patching writes a block of columns into the strip in place, as a live
element update does. Extending appends a block and trims what has scrolled
past, as continuous Vegas does (render_pipeline.extend_scroll_content).
Both run where Vegas runs them -- on the frame loop, before the next frame
is drawn -- and are tagged with ``FrameTimingRecorder.note_op``, so the
report shows how often the frame straight after each one was late.
The content comes from the benchmark's own strip, prepared before the run:
in Vegas it is drawn off the render thread, so drawing it here would time
work the render thread never does.
"""
def __init__(self, helper, recorder, source, *, patch_bytes: int = 0,
patch_every: int = 25, patch_where: str = "visible",
extend_every_screens: float = 0.0, extend_width: int = 0,
separator: int = 32) -> None:
import numpy as np
from PIL import Image
self.helper = helper
self.recorder = recorder
self.width = helper.display_width
self.height = helper.display_height
self.patch_every = max(1, int(patch_every))
self.patch_where = patch_where
self.separator = max(0, int(separator))
self.patches = 0
self.patched_bytes = 0
self.extensions = 0
self._frames = 0
self._position_at_extend = 0.0
pixels = np.asarray(source.convert("RGB"))
source_width = pixels.shape[1]
def columns(count: int, offset: int):
# Wraps around the source, so any width can be cut from it.
return np.ascontiguousarray(
pixels[:, (np.arange(count) + offset) % source_width])
# Two versions to alternate between, so every patch changes pixels.
self._patches = []
if patch_bytes > 0:
count = max(1, int(patch_bytes) // (self.height * 3))
self._patches = [columns(count, 0), columns(count, count)]
self.extend_every = (int(extend_every_screens * self.width)
if extend_every_screens > 0 else 0)
self._blocks = []
if self.extend_every:
# By default each append (block plus its separator) replaces
# exactly what scrolled past since the last one, so the strip
# holds its width, as Vegas's does in the steady state.
count = int(extend_width) or max(1, self.extend_every - self.separator)
self._blocks = [Image.fromarray(columns(count, 0)),
Image.fromarray(columns(count, count))]
def reset(self) -> None:
"""The strip was restarted from the beginning."""
self._position_at_extend = self.helper.scroll_position
def before_frame(self) -> None:
"""Do whatever work is due before the next frame is drawn."""
if self._blocks:
self._extend_if_due()
if self._patches:
self._frames += 1
if self._frames % self.patch_every == 0:
self._patch()
def _extend_if_due(self) -> None:
helper = self.helper
if helper.scroll_position < self._position_at_extend:
self._position_at_extend = helper.scroll_position
# Due on a fixed cadence rather than N screens after the last one ran,
# which would drift by the overshoot of a multi-pixel step each time.
due_at = self._position_at_extend + self.extend_every
if helper.scroll_position < due_at:
return
block = self._blocks[self.extensions % 2]
helper.append_content([block], item_gap=self.separator, element_gap=0)
moved = helper.cached_array.nbytes
# One screen kept behind the viewport, as Vegas does. The trim shifts
# every strip coordinate, the cadence's included.
cut = helper.drop_scrolled_prefix(keep_before=self.width)
if cut:
moved += helper.cached_array.nbytes
self._position_at_extend = due_at - cut
self.extensions += 1
self.recorder.note_op("extend", moved)
def _patch(self) -> None:
patch = self._patches[self.patches % 2]
strip = self.helper.cached_array
count = patch.shape[1]
if strip is None or count > strip.shape[1]:
return
start = int(self.helper.scroll_position)
if self.patch_where == "visible":
x = start + max(0, (self.width - count) // 2)
else:
# Just past the right edge: a change to content not yet on screen.
x = start + self.width + 16
x = max(0, min(x, strip.shape[1] - count))
strip[:, x:x + count] = patch
self.patches += 1
self.patched_bytes += patch.nbytes
self.recorder.note_op("patch", patch.nbytes)
def main(argv=None) -> int: def main(argv=None) -> int:
parser = argparse.ArgumentParser( parser = argparse.ArgumentParser(
@@ -202,8 +327,36 @@ def main(argv=None) -> int:
help="also write the report as JSON, for comparing rigs") help="also write the report as JSON, for comparing rigs")
parser.add_argument("--label", default=None, parser.add_argument("--label", default=None,
help="name for this run in the JSON report (default: hostname)") help="name for this run in the JSON report (default: hostname)")
parser.add_argument("--strip-screens", type=float, default=4.0, metavar="S",
help="strip width in screens (default 4). Vegas strips are "
"8,000-20,000px; the extension cost scales with it")
parser.add_argument("--patch-bytes", type=int, default=0, metavar="N",
help="write N bytes of columns into the strip in place "
"every --patch-every frames, as a Vegas live element "
"update does (a 150x64 card is ~29KB, a 512x64 map "
"~100KB)")
parser.add_argument("--patch-every", type=int, default=25, metavar="K",
help="frames between patches (default 25; 1 = every frame)")
parser.add_argument("--patch-where", choices=("visible", "ahead"),
default="visible",
help="patch the columns on screen, or just past its right "
"edge (default visible)")
parser.add_argument("--extend-every-screens", type=float, default=0.0,
metavar="N",
help="append a block and trim the strip every N screens "
"scrolled, as continuous Vegas does")
parser.add_argument("--extend-width", type=int, default=0, metavar="W",
help="width of each appended block in px (default: N "
"screens less the separator, so the strip holds its "
"width)")
args = parser.parse_args(argv) args = parser.parse_args(argv)
if args.extend_every_screens > 0 and args.strip_screens < args.extend_every_screens + 3:
# The strip must stay ahead of the viewport between extensions.
args.strip_screens = args.extend_every_screens + 3
print(f"strip widened to {args.strip_screens:g} screens so extensions "
"keep ahead of the viewport")
# Everything the display service logs would otherwise land in the middle of # Everything the display service logs would otherwise land in the middle of
# the report; the benchmark's own output is the point. The stall watchdog # the report; the benchmark's own output is the point. The stall watchdog
# is the exception: a stack dump naming what held a frame up belongs here. # is the exception: a stack dump naming what held a frame up belongs here.
@@ -272,8 +425,9 @@ def main(argv=None) -> int:
print(f"asked for {requested:.1f} px/s -> {choice.describe()}") print(f"asked for {requested:.1f} px/s -> {choice.describe()}")
helper.set_sub_pixel_scrolling(False) helper.set_sub_pixel_scrolling(False)
helper.set_scrolling_image( strip = build_strip(width, height, f"{choice.pixels_per_second:.0f} px/s",
build_strip(width, height, f"{choice.pixels_per_second:.0f} px/s")) screens=args.strip_screens)
helper.set_scrolling_image(strip)
# The display service's own recorder, owned outright here: never flushed to # The display service's own recorder, owned outright here: never flushed to
# the service's stats file, drained exactly at the start and end of the # the service's stats file, drained exactly at the start and end of the
@@ -288,8 +442,23 @@ def main(argv=None) -> int:
recorder.scrolling_now = display._scrolling_now # pylint: disable=protected-access recorder.scrolling_now = display._scrolling_now # pylint: disable=protected-access
display.frame_timing = recorder display.frame_timing = recorder
print(f"scrolling {width}x{height} for {args.seconds:.0f}s" work = StripWork(helper, recorder, strip,
+ (f" with {args.busy} background worker(s)" if args.busy else "") patch_bytes=args.patch_bytes, patch_every=args.patch_every,
patch_where=args.patch_where,
extend_every_screens=args.extend_every_screens,
extend_width=args.extend_width)
doing = []
if args.busy:
doing.append(f"{args.busy} background worker(s)")
if args.patch_bytes > 0:
doing.append(f"a {args.patch_bytes}B {args.patch_where} patch every "
f"{args.patch_every} frame(s)")
if work.extend_every:
doing.append(f"an extension every {args.extend_every_screens:g} screens")
print(f"scrolling {width}x{height} ({helper.cached_array.shape[1]}px strip) "
f"for {args.seconds:.0f}s"
+ (" with " + ", ".join(doing) if doing else "")
+ " ...", flush=True) + " ...", flush=True)
frames = 0 frames = 0
@@ -309,8 +478,11 @@ def main(argv=None) -> int:
before = recorder.snapshot() before = recorder.snapshot()
run_started = now run_started = now
frames = duplicates = blanks = restarts = 0 frames = duplicates = blanks = restarts = 0
work.patches = work.patched_bytes = work.extensions = 0
if run_started is not None and now - run_started >= args.seconds: if run_started is not None and now - run_started >= args.seconds:
break break
# Where Vegas does its strip work: before the frame is drawn.
work.before_frame()
helper.update_scroll_position() helper.update_scroll_position()
if helper.is_scroll_complete(): if helper.is_scroll_complete():
# The helper parks at the end of the strip and stops # The helper parks at the end of the strip and stops
@@ -320,6 +492,7 @@ def main(argv=None) -> int:
# the benchmark measures a still image for the rest of the # the benchmark measures a still image for the rest of the
# run and reports a smoothness it never demonstrated. # run and reports a smoothness it never demonstrated.
helper.reset_scroll() helper.reset_scroll()
work.reset()
restarts += 1 restarts += 1
visible = helper.get_visible_portion() visible = helper.get_visible_portion()
column = int(helper.scroll_position) column = int(helper.scroll_position)
@@ -375,6 +548,11 @@ def main(argv=None) -> int:
if restarts: if restarts:
print(f"restarts {restarts} (the strip was scrolled through " print(f"restarts {restarts} (the strip was scrolled through "
f"{restarts} time{'s' if restarts != 1 else ''})") f"{restarts} time{'s' if restarts != 1 else ''})")
if work.patches:
print(f"patches {work.patches} ({work.patched_bytes / 1e6:.1f} MB "
"written into the strip)")
if work.extensions:
print(f"extensions {work.extensions}")
if args.json_path: if args.json_path:
report.update({ report.update({
@@ -388,6 +566,13 @@ def main(argv=None) -> int:
"duplicate_frames": duplicates, "duplicate_frames": duplicates,
"blank_frames": blanks, "blank_frames": blanks,
"strip_restarts": restarts, "strip_restarts": restarts,
"strip_screens": args.strip_screens,
"patch_bytes": args.patch_bytes,
"patch_every": args.patch_every,
"patch_where": args.patch_where,
"patches": work.patches,
"extend_every_screens": args.extend_every_screens,
"extensions": work.extensions,
"max_late_pct": args.max_late_pct, "max_late_pct": args.max_late_pct,
"passed": frame_soak.passed(report, args.max_late_pct), "passed": frame_soak.passed(report, args.max_late_pct),
}) })
+57
View File
@@ -56,9 +56,33 @@ def main() -> int:
help='Display mode to render, for plugins that declare ' help='Display mode to render, for plugins that declare '
'more than one in their manifest (e.g. nrl_live). ' 'more than one in their manifest (e.g. nrl_live). '
'Omitted, the plugin picks its own default.') 'Omitted, the plugin picks its own default.')
parser.add_argument('--vegas', action='store_true',
help="Render the plugin's block of the Vegas ticker strip "
"instead of display(): its live elements if it has "
"them, else its Vegas content, laid out as the "
"ticker lays them out. Also writes the live "
"elements' keys and columns to <output>.json")
parser.add_argument('--no-live', action='store_true',
help="With --vegas: ignore live elements and render the "
"plugin's ordinary Vegas content (for before/after)")
parser.add_argument('--timeline', type=int, default=0, metavar='ROWS',
help="With --vegas: render ROWS rows, each the block a "
"--timeline-step later as the ticker would update it "
"in place (animated elements redrawn for that moment)")
parser.add_argument('--timeline-step', type=float, default=0.25, metavar='SECONDS',
help="Seconds between --timeline rows (default 0.25)")
parser.add_argument('--timeline-update', action='store_true',
help="With --timeline: run update() before each row and "
"redraw every live element from the new data")
args = parser.parse_args() args = parser.parse_args()
if args.timeline > 1 and args.no_live:
# A timeline shows live elements changing; plain content never does.
parser.error("--timeline shows live elements; it cannot be combined with --no-live")
if (args.timeline or args.no_live) and not args.vegas:
parser.error("--timeline and --no-live need --vegas")
if not (MIN_DIMENSION <= args.width <= MAX_DIMENSION): if not (MIN_DIMENSION <= args.width <= MAX_DIMENSION):
print(f"Error: --width must be between {MIN_DIMENSION} and {MAX_DIMENSION} (got {args.width})") print(f"Error: --width must be between {MIN_DIMENSION} and {MAX_DIMENSION} (got {args.width})")
raise SystemExit(1) raise SystemExit(1)
@@ -145,6 +169,39 @@ def main() -> int:
except Exception as e: except Exception as e:
logger.warning("update() raised: %s — continuing to display()", e) logger.warning("update() raised: %s — continuing to display()", e)
if args.vegas:
Path(args.output).parent.mkdir(parents=True, exist_ok=True)
if args.vegas and args.timeline > 1:
from src.plugin_system.testing.vegas import render_vegas_timeline
image, rows = render_vegas_timeline(
plugin_instance, args.plugin, display_manager, steps=args.timeline,
step_seconds=args.timeline_step, run_update=args.timeline_update)
if image is None:
logger.error("Plugin '%s' has no Vegas content", args.plugin)
return 1
image.save(args.output)
logger.info("Saved a %d-row Vegas timeline (%dx%d) to %s",
rows, image.width, image.height, args.output)
return 0
if args.vegas:
from src.plugin_system.testing.vegas import render_vegas_strip
block, layout = render_vegas_strip(
plugin_instance, args.plugin, display_manager, live=not args.no_live)
if block is None:
logger.error("Plugin '%s' has no Vegas content", args.plugin)
return 1
block.save(args.output)
sidecar = Path(args.output).with_suffix('.json')
sidecar.write_text(json.dumps(
{"width": block.width, "height": block.height,
"live_elements": [{"key": k, "x": x, "width": w} for x, k, w in layout]},
indent=2) + "\n", encoding="utf-8")
logger.info("Saved Vegas strip %dx%d (%d live element(s)) to %s and %s",
block.width, block.height, len(layout), args.output, sidecar)
return 0
# A plugin that declares several display modes usually renders nothing # A plugin that declares several display modes usually renders nothing
# useful without being told which one to draw: the scoreboards keep their # useful without being told which one to draw: the scoreboards keep their
# state on per-mode sub-managers and their no-argument path returns False. # state on per-mode sub-managers and their no-argument path returns False.
+104
View File
@@ -0,0 +1,104 @@
#!/usr/bin/env python3
"""
Turn the web interface's optional login off, for when the password is lost.
Removes the password (and the key that signs login cookies) from the
``web_auth`` section of ``config/config_secrets.json``. The interface is then
open again, as it is before a password is ever set, and a new password can be
set under General > Security. API tokens are kept unless ``--revoke-tokens``
is given. Nothing else in the secrets file is touched, and the web service
does not need a restart: it notices the change on the next request.
Run it on the Pi, from any directory:
sudo python3 ~/LEDMatrix/scripts/reset_web_password.py
``sudo`` because the secrets file is not readable by every user. The file
keeps its owner and permissions.
Another way in without the password: open the interface from the Pi itself
(http://localhost:5000). Requests from the Pi are never asked to log in.
"""
import argparse
import json
import sys
from pathlib import Path
PROJECT_ROOT = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(PROJECT_ROOT))
from src.config_manager_atomic import atomic_write_json # noqa: E402
SECTION = 'web_auth' # web_interface/auth.py; not imported to keep Flask out
LOGIN_KEYS = ('password_hash', 'session_secret', 'password_set_at')
def reset(settings_file: Path, revoke_tokens: bool = False) -> str:
"""Clear the login from ``settings_file`` (config_secrets.json).
Returns what was done. The message names the file and counts tokens; it
never includes anything read from the file.
"""
if not settings_file.exists():
return f'{settings_file} does not exist, so no password is set. Nothing to do.'
with open(settings_file, 'r', encoding='utf-8') as fh:
data = json.load(fh)
if not isinstance(data, dict):
raise ValueError(f'{settings_file} does not hold a JSON object')
section = data.get(SECTION)
if not isinstance(section, dict):
return 'No web login password is set. Nothing to do.'
had_password = bool(section.get('password_hash'))
token_count = len(section.get('tokens') or [])
for key in LOGIN_KEYS:
section.pop(key, None)
if revoke_tokens:
section.pop('tokens', None)
if section:
data[SECTION] = section
else:
data.pop(SECTION, None)
if not had_password and not (revoke_tokens and token_count):
return 'No web login password is set. Nothing to do.'
atomic_write_json(settings_file, data)
done = []
if had_password:
done.append('Web login is off: the interface opens without a password. '
'Set a new one under General > Security.')
if revoke_tokens and token_count:
done.append(f'Revoked {token_count} API token(s).')
elif token_count:
done.append(f'{token_count} API token(s) kept (use --revoke-tokens to remove them).')
return ' '.join(done)
def main(argv=None) -> int:
parser = argparse.ArgumentParser(
description='Turn the LEDMatrix web login off (lost password recovery).')
parser.add_argument('--secrets', dest='settings_file', type=Path,
default=PROJECT_ROOT / 'config' / 'config_secrets.json',
help='secrets file (default: config/config_secrets.json '
'in this LEDMatrix checkout)')
parser.add_argument('--revoke-tokens', action='store_true',
help='also delete every API token')
args = parser.parse_args(argv)
settings_file = args.settings_file
try:
outcome = reset(settings_file, revoke_tokens=args.revoke_tokens)
except PermissionError:
print(f'Permission denied reading or writing {settings_file}. Run it with sudo.',
file=sys.stderr)
return 1
except (OSError, ValueError) as err:
print(f'Could not reset the web login: {err}', file=sys.stderr)
return 1
print(outcome)
return 0
if __name__ == '__main__':
sys.exit(main())
-1
View File
@@ -299,6 +299,5 @@ def main():
if __name__ == '__main__': if __name__ == '__main__':
import importlib.util import importlib.util
from typing import Optional
sys.exit(main()) sys.exit(main())
+492
View File
@@ -0,0 +1,492 @@
#!/usr/bin/env python3
"""Report how far apart the nine scoreboards' copies of each method are.
The sports consolidation (docs/SPORTS_UNIFICATION.md) moves shared code from
the scoreboard plugins into ``src/common``. Byte-identical copies have mostly
been moved; what is left has drifted, and is promoted one *method family* at a
time by first making every copy identical ("reconcile, then promote"). This
report is the progress measure for that: for every method in the tracked
files it counts the copies and the distinct bodies among them, so a stage can
say "``_is_game_really_over``: 5 variants -> 1" instead of remembering it.
It reads a ledmatrix-plugins checkout and never fails a build: it is a report,
not a gate. The monorepo's own ``scripts/check_sports_drift.py`` is the gate
(it fails when a function that agrees across the plugins starts to differ).
Definitions
-----------
family
One method name in one tracked file, across every class that defines it
and every plugin. ``sports.py::update`` covers ``SportsLive.update``,
``SportsRecent.update`` and ``SportsUpcoming.update`` in all nine plugins.
Module-level functions are families too.
copies
How many definitions the family has (plugin x class).
plugins
How many of the nine plugins define it at least once.
variants
Distinct bodies among the copies, compared as ASTs with docstrings,
comments, formatting, decorators and annotations ignored. A family is
reconciled when every class in it is down to one variant.
per-class variants
The same count within one class role (``SportsLive.update`` across the
plugins). Class names are folded the way the plugins name them
(``SoccerScoreboardPlugin`` and ``UFCScoreboardPlugin`` are both
``SScoreboardPlugin``), so manager.py lines up across sports.
folded
Variants left after sport and league names are folded to a placeholder
(``self.nfl_live`` == ``self.nhl_live``, ``"NFL"`` == ``"NHL"``). The gap
between ``variants`` and ``folded`` is drift that is only naming.
Usage
-----
python scripts/sports_drift_report.py --plugins ../ledmatrix-plugins
python scripts/sports_drift_report.py --markdown # for a CI summary
python scripts/sports_drift_report.py --json out.json # machine-readable
python scripts/sports_drift_report.py --family sports.py::update
``--plugins`` defaults to ``$LEDMATRIX_PLUGINS`` (a checkout root or its
``plugins/`` directory, the same variable the core parity tests read). With no
checkout it says so and exits 0.
"""
from __future__ import annotations
import argparse
import ast
import collections
import difflib
import hashlib
import json
import os
import re
import sys
from pathlib import Path
from typing import Dict, Iterable, List, Optional, Tuple
#: The nine scoreboards the consolidation covers, by directory prefix.
SPORTS = ("afl", "baseball", "basketball", "football", "hockey", "lacrosse",
"nrl", "soccer", "ufc")
#: Files every scoreboard carries a copy of. sports.py and game_renderer.py
#: are the consolidation's subject; manager.py (the BasePlugin host, the
#: largest copy of all) joined the plan with the reconcile-then-promote
#: method. ufc has no game_renderer.py (it draws fights in fight_renderer.py).
DEFAULT_FILES = ("sports.py", "manager.py", "game_renderer.py")
#: A family is "drifted" when it is widespread and has several bodies. The
#: defaults match the review that introduced this report (at least 7 plugins,
#: at least 3 variants).
DEFAULT_MIN_PLUGINS = 7
DEFAULT_MIN_VARIANTS = 3
#: Sport, league and competition names that legitimately differ between the
#: plugins. Only used for the ``folded`` column.
SPORT_TOKENS = (
"afl", "nrl", "baseball", "basketball", "football", "hockey", "soccer",
"lacrosse", "ufc", "mma", "mlb", "milb", "nhl", "nfl", "nba", "wnba",
"ncaa", "ncaafb", "ncaam", "ncaaw", "ncaa_fb", "ncaa_baseball",
"ncaa_basketball", "ncaam_hockey", "ncaaw_hockey", "ncaam_lacrosse",
"ncaaw_lacrosse", "ncaam_basketball", "ncaaw_basketball", "epl",
"uefa", "mls", "laliga", "bundesliga", "seriea", "ligue1",
)
_TOKEN_RE = re.compile(
r"(?<![A-Za-z0-9])(" + "|".join(sorted(SPORT_TOKENS, key=len, reverse=True))
+ r")(?![A-Za-z0-9])", re.IGNORECASE)
_TOKEN_SET = {t.lower() for t in SPORT_TOKENS}
_CAMEL_RE = re.compile(r"[A-Z]+(?![a-z])|[A-Z][a-z0-9]*|[a-z0-9]+|_")
def fold(name: str) -> str:
"""Replace sport and league names in an identifier or string with ``S``.
Both spellings the plugins use: snake_case (``nfl_live`` -> ``S_live``)
and CamelCase (``UFCScoreboardPlugin`` -> ``SScoreboardPlugin``).
"""
name = _TOKEN_RE.sub("S", name)
# sub, not findall + join: characters between words (spaces, dots,
# braces in a log string) must survive, or distinct text folds together.
return _CAMEL_RE.sub(
lambda m: "S" if m.group(0).lower() in _TOKEN_SET else m.group(0), name)
def _strip_docstring(body: List[ast.stmt]) -> List[ast.stmt]:
if (body and isinstance(body[0], ast.Expr)
and isinstance(body[0].value, ast.Constant)
and isinstance(body[0].value.value, str)):
return body[1:] or [ast.Pass()]
return body
class _Canonical(ast.NodeTransformer):
"""Drop what is not behaviour: docstrings, decorators, annotations."""
def _func(self, node):
self.generic_visit(node)
node.body = _strip_docstring(node.body)
node.decorator_list = []
node.returns = None
return node
visit_FunctionDef = _func
visit_AsyncFunctionDef = _func
def visit_ClassDef(self, node):
self.generic_visit(node)
node.body = _strip_docstring(node.body)
return node
def visit_arg(self, node):
node.annotation = None
return node
def visit_AnnAssign(self, node):
# ``x: T = v`` is ``x = v``; a bare ``x: T`` does nothing at runtime.
self.generic_visit(node)
if node.value is None:
return None
return ast.copy_location(
ast.Assign(targets=[node.target], value=node.value), node)
class _Folded(_Canonical):
"""Canonical, plus sport names folded out of identifiers and strings."""
def visit_Name(self, node):
node.id = fold(node.id)
return node
def visit_Attribute(self, node):
self.generic_visit(node)
node.attr = fold(node.attr)
return node
def visit_arg(self, node):
node = super().visit_arg(node)
node.arg = fold(node.arg)
return node
def visit_keyword(self, node):
self.generic_visit(node)
if node.arg:
node.arg = fold(node.arg)
return node
def visit_Constant(self, node):
if isinstance(node.value, str):
node.value = fold(node.value)
return node
def _func(self, node):
node = super()._func(node)
node.name = fold(node.name)
return node
visit_FunctionDef = _func
visit_AsyncFunctionDef = _func
def _digest(node: ast.AST, transformer: ast.NodeTransformer) -> str:
# Re-parse a copy so the transformers never mutate the tree being walked.
clone = ast.parse(ast.unparse(node)).body[0]
clone = transformer.visit(clone)
# The function's own name is the family key, not part of its body.
if isinstance(clone, (ast.FunctionDef, ast.AsyncFunctionDef)):
clone.name = "_"
return hashlib.sha256(ast.dump(clone).encode()).hexdigest()[:12]
class Copy:
"""One definition of a method (or module-level function) in one plugin."""
__slots__ = ("plugin", "cls", "name", "lines", "exact", "folded", "source")
def __init__(self, plugin, cls, name, lines, exact, folded, source=""):
self.plugin = plugin
self.cls = cls
self.name = name
self.lines = lines
self.exact = exact
self.folded = folded
self.source = source
def collect_file(path: Path, plugin: str) -> List[Copy]:
"""Every top-level function and class method in one file."""
text = path.read_text(encoding="utf-8", errors="replace")
try:
tree = ast.parse(text)
except SyntaxError as exc:
print(f" ! {path}: {exc}", file=sys.stderr)
return []
out = []
def add(node, cls):
out.append(Copy(plugin, cls, node.name,
node.end_lineno - node.lineno + 1,
_digest(node, _Canonical()), _digest(node, _Folded()),
ast.get_source_segment(text, node) or ""))
for node in tree.body:
if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
add(node, "<module>")
elif isinstance(node, ast.ClassDef):
for child in node.body:
if isinstance(child, (ast.FunctionDef, ast.AsyncFunctionDef)):
add(child, fold(node.name))
return out
def resolve_plugins_dir(raw: Optional[str]) -> Optional[Path]:
"""A checkout root or its plugins/ directory; None when there is neither."""
if not raw:
return None
root = Path(raw)
if (root / "plugins").is_dir():
root = root / "plugins"
if not any((root / f"{s}-scoreboard").is_dir() for s in SPORTS):
return None
return root
def build(plugins_dir: Path, files: Iterable[str]) -> Dict[Tuple[str, str], List[Copy]]:
"""{(file, method name): [Copy, ...]} across the nine scoreboards."""
families: Dict[Tuple[str, str], List[Copy]] = collections.defaultdict(list)
for fname in files:
for sport in SPORTS:
path = plugins_dir / f"{sport}-scoreboard" / fname
if path.is_file():
for copy in collect_file(path, sport):
families[(fname, copy.name)].append(copy)
return families
def summarise(key: Tuple[str, str], copies: List[Copy]) -> dict:
"""The numbers for one family."""
per_class = collections.defaultdict(list)
for c in copies:
per_class[c.cls].append(c)
classes = []
for cls, members in sorted(per_class.items()):
groups = collections.defaultdict(list)
for c in members:
groups[c.exact].append(c.plugin)
classes.append({
"class": cls,
"copies": len(members),
"variants": len(groups),
"folded": len({c.folded for c in members}),
"groups": sorted((sorted(p) for p in groups.values()),
key=lambda g: (-len(g), g)),
})
total_lines = sum(c.lines for c in copies)
# What promotion would remove: every copy but one per class role.
one_each = sum(max(c.lines for c in members) for members in per_class.values())
return {
"file": key[0],
"family": key[1],
"plugins": len({c.plugin for c in copies}),
"copies": len(copies),
"variants": len({(c.cls, c.exact) for c in copies}),
"folded": len({(c.cls, c.folded) for c in copies}),
"worst_class_variants": max(k["variants"] for k in classes),
"lines": total_lines,
"duplicated_lines": total_lines - one_each,
"classes": classes,
}
def report(families, min_plugins: int, min_variants: int) -> dict:
rows = [summarise(k, v) for k, v in families.items()]
by_file = collections.defaultdict(list)
for r in rows:
by_file[r["file"]].append(r)
files = {}
for fname, frows in sorted(by_file.items()):
files[fname] = {
"families": len(frows),
"in_all_plugins": sum(1 for r in frows if r["plugins"] == len(SPORTS)),
"lines": sum(r["lines"] for r in frows),
"identical_duplicated_lines": sum(
r["duplicated_lines"] for r in frows if r["worst_class_variants"] == 1),
}
drifted = sorted(
(r for r in rows
if r["plugins"] >= min_plugins and r["variants"] >= min_variants),
key=lambda r: (-r["variants"], -r["lines"], r["file"], r["family"]))
identical = sorted(
(r for r in rows if r["copies"] >= 2 and r["worst_class_variants"] == 1),
key=lambda r: (-r["duplicated_lines"], r["file"], r["family"]))
# One body shared by every plugin but one: the cheapest reconciliations.
# "One" across the whole family: a class role whose odd one out is a
# different plugin from another role's is two outliers, not one.
one_outlier = sorted(
(r for r in rows
if r["plugins"] >= min_plugins and r["worst_class_variants"] == 2
and all(len(k["groups"]) < 2 or len(k["groups"][1]) == 1
for k in r["classes"])
and len(_minorities(r)) == 1),
key=lambda r: (-r["duplicated_lines"], r["file"], r["family"]))
return {"files": files, "drifted": drifted, "identical": identical,
"one_outlier": one_outlier,
"rows": rows, "thresholds": {"min_plugins": min_plugins,
"min_variants": min_variants}}
def _minorities(r) -> set:
"""Every plugin in a minority body, across the family's class roles."""
return {p for k in r["classes"] for g in k["groups"][1:] for p in g}
def _outlier(r) -> str:
"""The plugin whose body differs, for a one-outlier family."""
return ", ".join(sorted(_minorities(r)))
def _text(rep, top_identical: int) -> str:
out = []
out.append("Per file (all methods and module functions):")
for fname, f in rep["files"].items():
out.append(f" {fname:<18} {f['families']:>4} families, "
f"{f['in_all_plugins']:>3} in all {len(SPORTS)} plugins, "
f"{f['lines']:>6} lines; identical copies beyond the first: "
f"{f['identical_duplicated_lines']} lines")
t = rep["thresholds"]
out.append("")
out.append(f"Drifted families (in >= {t['min_plugins']} plugins, "
f">= {t['min_variants']} variants): {len(rep['drifted'])}")
out.append(f" {'file::family':<58} {'plug':>4} {'copies':>6} {'var':>4} "
f"{'fold':>4} {'worst':>5} {'lines':>6}")
for r in rep["drifted"]:
name = f"{r['file']}::{r['family']}"
out.append(f" {name:<58} {r['plugins']:>4} {r['copies']:>6} "
f"{r['variants']:>4} {r['folded']:>4} "
f"{r['worst_class_variants']:>5} {r['lines']:>6}")
out.append("")
out.append(f"One outlier (in >= {t['min_plugins']} plugins, every plugin but "
f"one agrees): {len(rep['one_outlier'])}")
for r in rep["one_outlier"]:
name = f"{r['file']}::{r['family']}"
out.append(f" {name:<58} {r['plugins']:>4} plugins, differs in "
f"{_outlier(r)}; {r['lines']} lines")
out.append("")
out.append(f"Identical in every copy (promote as-is), top {top_identical} "
f"by duplicated lines, of {len(rep['identical'])}:")
for r in rep["identical"][:top_identical]:
name = f"{r['file']}::{r['family']}"
out.append(f" {name:<58} {r['plugins']:>4} plugins "
f"{r['duplicated_lines']:>5} duplicated lines")
return "\n".join(out)
def _markdown(rep, top_identical: int, source: str) -> str:
t = rep["thresholds"]
out = ["## Sports drift report", "",
f"Scoreboard copies read from `{source}`. Report only: this never fails "
"the build. See docs/SPORTS_UNIFICATION.md.", "",
"| File | Families | In all 9 | Lines | Identical duplicated lines |",
"|---|---:|---:|---:|---:|"]
for fname, f in rep["files"].items():
out.append(f"| `{fname}` | {f['families']} | {f['in_all_plugins']} | "
f"{f['lines']} | {f['identical_duplicated_lines']} |")
out += ["", f"### Drifted families (in >= {t['min_plugins']} plugins, "
f">= {t['min_variants']} variants): {len(rep['drifted'])}", "",
"| Family | Plugins | Copies | Variants | Folded | Worst class | Lines |",
"|---|---:|---:|---:|---:|---:|---:|"]
for r in rep["drifted"]:
out.append(f"| `{r['file']}::{r['family']}` | {r['plugins']} | {r['copies']} | "
f"{r['variants']} | {r['folded']} | {r['worst_class_variants']} | "
f"{r['lines']} |")
out += ["", f"### One outlier (every plugin but one agrees): "
f"{len(rep['one_outlier'])}", "",
"| Family | Plugins | Differs in | Lines |", "|---|---:|---|---:|"]
for r in rep["one_outlier"]:
out.append(f"| `{r['file']}::{r['family']}` | {r['plugins']} | "
f"{_outlier(r)} | {r['lines']} |")
out += ["", f"### Identical in every copy: {len(rep['identical'])} "
f"(top {top_identical} by duplicated lines)", "",
"| Family | Plugins | Duplicated lines |", "|---|---:|---:|"]
for r in rep["identical"][:top_identical]:
out.append(f"| `{r['file']}::{r['family']}` | {r['plugins']} | "
f"{r['duplicated_lines']} |")
return "\n".join(out) + "\n"
def _family_detail(rep, families, wanted: str, show_diff: bool) -> str:
"""Which plugins share each body of one family; optionally the diffs.
The diff is against the body most plugins share (the first group), which
is where a reconciliation usually starts.
"""
fname, _, family = wanted.partition("::")
for r in rep["rows"]:
if r["file"] == fname and r["family"] == family:
out = [f"{wanted}: {r['plugins']} plugins, {r['copies']} copies, "
f"{r['variants']} variants ({r['folded']} after folding sport "
f"names), {r['lines']} lines"]
copies = families[(fname, family)]
for k in r["classes"]:
out.append(f" {k['class']}: {k['variants']} variant(s) "
f"({k['folded']} folded)")
for g in k["groups"]:
out.append(f" {', '.join(g)}")
if not show_diff or len(k["groups"]) < 2:
continue
by_plugin = {c.plugin: c for c in copies if c.cls == k["class"]}
base = by_plugin[k["groups"][0][0]]
for g in k["groups"][1:]:
other = by_plugin[g[0]]
out.extend(difflib.unified_diff(
base.source.splitlines(), other.source.splitlines(),
f"{base.plugin}-scoreboard/{fname}",
f"{other.plugin}-scoreboard/{fname}", lineterm="", n=2))
return "\n".join(out)
return f"{wanted}: no such family"
def main(argv: Optional[List[str]] = None) -> int:
ap = argparse.ArgumentParser(description=__doc__.split("\n")[0])
ap.add_argument("--plugins", default=os.environ.get("LEDMATRIX_PLUGINS"),
help="ledmatrix-plugins checkout (default: $LEDMATRIX_PLUGINS)")
ap.add_argument("--files", default=",".join(DEFAULT_FILES),
help="comma-separated files to compare (default: %(default)s)")
ap.add_argument("--min-plugins", type=int, default=DEFAULT_MIN_PLUGINS)
ap.add_argument("--min-variants", type=int, default=DEFAULT_MIN_VARIANTS)
ap.add_argument("--top-identical", type=int, default=15)
ap.add_argument("--markdown", action="store_true",
help="print a Markdown summary (for $GITHUB_STEP_SUMMARY)")
ap.add_argument("--json", metavar="PATH",
help="also write the full report as JSON")
ap.add_argument("--family", action="append", default=[],
help="show which plugins share each body, e.g. sports.py::update")
ap.add_argument("--diff", action="store_true",
help="with --family, also diff each variant against the most common one")
args = ap.parse_args(argv)
plugins_dir = resolve_plugins_dir(args.plugins)
if plugins_dir is None:
msg = ("No ledmatrix-plugins checkout: pass --plugins or set "
"LEDMATRIX_PLUGINS. Nothing to report.")
print(f"_{msg}_\n" if args.markdown else msg)
return 0
files = [f.strip() for f in args.files.split(",") if f.strip()]
families = build(plugins_dir, files)
rep = report(families, args.min_plugins, args.min_variants)
if args.json:
with open(args.json, "w", encoding="utf-8") as fh:
json.dump(rep, fh, indent=2)
fh.write("\n")
if args.markdown:
print(_markdown(rep, args.top_identical, str(plugins_dir)), end="")
else:
print(_text(rep, args.top_identical))
for wanted in args.family:
print()
print(_family_detail(rep, families, wanted, args.diff))
return 0
if __name__ == "__main__":
sys.exit(main())
+18 -1
View File
@@ -247,6 +247,19 @@
</div> </div>
</details> </details>
<!-- What to render: the plugin's screen, or its block of the Vegas strip -->
<div class="flex items-center gap-2">
<label for="viewSelect" class="text-xs whitespace-nowrap" style="color: var(--text-secondary);">View</label>
<select id="viewSelect" onchange="onConfigChange()"
class="flex-1 px-2 py-1.5 rounded-lg text-xs"
style="background: var(--bg-primary); color: var(--text-primary); border: 1px solid var(--border-color);"
title="Vegas strip: the plugin's block of the Vegas ticker, laid out as the ticker lays it out">
<option value="">Display</option>
<option value="live">Vegas strip (live elements)</option>
<option value="plain">Vegas strip (plain Vegas content)</option>
</select>
</div>
<!-- Render buttons --> <!-- Render buttons -->
<div class="flex gap-2"> <div class="flex gap-2">
<button onclick="renderPlugin()" id="renderBtn" <button onclick="renderPlugin()" id="renderBtn"
@@ -489,6 +502,7 @@
width: width, width: width,
height: height, height: height,
mock_data: mockData, mock_data: mockData,
vegas: document.getElementById('viewSelect').value || null,
}), }),
}); });
@@ -510,8 +524,11 @@
updateZoom(); updateZoom();
// Show render time // Show render time
const live = data.live_elements;
document.getElementById('renderTimeText').textContent = document.getElementById('renderTimeText').textContent =
`${data.render_time_ms}ms`; `${data.render_time_ms}ms` + (live ? ` · ${data.width}px strip, ` +
`${live.length} live element(s)` +
(live.length ? `: ${live.map(e => e.key).join(', ')}` : '') : '');
// Show warnings/errors // Show warnings/errors
showMessages(data.errors || [], data.warnings || []); showMessages(data.errors || [], data.warnings || []);
+73 -3
View File
@@ -14,12 +14,20 @@ the same reason: the rollback cannot depend on packages the update changed.
The updater leaves data/auto_update_pending.json: The updater leaves data/auto_update_pending.json:
{"status": "pending", "old_head": ..., "new_head": ..., {"status": "pending", "old_head": ..., "new_head": ...,
"old_ref": "main" | "" (detached) | absent (older updaters),
"display_was_active": bool, "dependency_failures": [...], "created_at": ...} "display_was_active": bool, "dependency_failures": [...], "created_at": ...}
This moves its status to "verifying" and then to one of "success", This moves its status to "verifying" and then to one of "success",
"rolled_back" or "rollback_failed", with "reason" and "detail" saying why. "rolled_back" or "rollback_failed", with "reason" and "detail" saying why.
The web interface reports that outcome and raises a banner for anything but The web interface reports that outcome and raises a banner for anything but
success. success.
"The display service is active" does not mean the panel is drawing: a render
loop stuck inside a plugin leaves the service active and the panel frozen.
Where the display writes a heartbeat (/run/ledmatrix, see
src/display_watchdog.py), the display also has to keep it fresh, from the
restarted process, to count as healthy. Where it never wrote one -- the code
being updated predates it -- the check is what it always was.
""" """
import json import json
import os import os
@@ -36,6 +44,14 @@ from pathlib import Path
PENDING_NAME = 'auto_update_pending.json' PENDING_NAME = 'auto_update_pending.json'
REQUIREMENT_FILES = ('requirements.txt', 'web_interface/requirements.txt') REQUIREMENT_FILES = ('requirements.txt', 'web_interface/requirements.txt')
WEB_HEALTH_URL = 'http://127.0.0.1:5000/api/v3/system/version' WEB_HEALTH_URL = 'http://127.0.0.1:5000/api/v3/system/version'
#: Written by the display's render loop every few seconds. A copy of
#: src/display_watchdog.HEARTBEAT_PATH, not an import: this file runs as a
#: copy made before the update and must not depend on the code it checks.
HEARTBEAT_PATH = '/run/ledmatrix/display-heartbeat.json'
#: How old the heartbeat may be. Well under STABLE_SECONDS: a display that
#: draws its first frame and then freezes must go stale inside the window it
#: has to stay healthy for, or the check would pass it.
HEARTBEAT_FRESH_SECONDS = 30
#: How long the services get to come up after a restart... #: How long the services get to come up after a restart...
HEALTH_TIMEOUT_SECONDS = 180 HEALTH_TIMEOUT_SECONDS = 180
#: ...and how long they must then stay up. Restart=on-failure makes a crash #: ...and how long they must then stay up. Restart=on-failure makes a crash
@@ -113,16 +129,35 @@ def _short(sha):
return (sha or 'unknown')[:7] return (sha or 'unknown')[:7]
def _read_heartbeat(path=HEARTBEAT_PATH):
"""The display's heartbeat, or None when there is none (or it is unreadable)."""
try:
with open(path, 'r', encoding='utf-8') as f:
data = json.load(f)
except (OSError, ValueError):
return None
return data if isinstance(data, dict) else None
class Verifier: class Verifier:
def __init__(self, project_root, run=subprocess.run, sleep=time.sleep, def __init__(self, project_root, run=subprocess.run, sleep=time.sleep,
clock=time.monotonic, web_responds=_web_responds, log=None): clock=time.monotonic, web_responds=_web_responds, log=None,
read_heartbeat=_read_heartbeat):
self.project_root = Path(project_root) self.project_root = Path(project_root)
self.pending_file = pending_path(project_root) self.pending_file = pending_path(project_root)
self.run = run self.run = run
self.sleep = sleep self.sleep = sleep
# Monotonic, and compared with the heartbeat's own monotonic stamp:
# CLOCK_MONOTONIC is one clock for every process on the machine.
self.clock = clock self.clock = clock
self.web_responds = web_responds self.web_responds = web_responds
self.log = log or (lambda msg: print(f'[auto-update-verify] {msg}', flush=True)) self.log = log or (lambda msg: print(f'[auto-update-verify] {msg}', flush=True))
self.read_heartbeat = read_heartbeat
#: Whether the display was writing a heartbeat before the update.
self.expect_heartbeat = False
#: When the display was last restarted; an older heartbeat is the
#: previous process's, not proof the new one draws.
self.display_restarted_at = None
def _run(self, args, timeout=GIT_TIMEOUT_SECONDS): def _run(self, args, timeout=GIT_TIMEOUT_SECONDS):
try: try:
@@ -154,18 +189,32 @@ class Verifier:
ok = True ok = True
# A display the user had stopped stays stopped. # A display the user had stopped stays stopped.
if display: if display:
self.display_restarted_at = self.clock()
ok = self.restart('ledmatrix') and ok ok = self.restart('ledmatrix') and ok
return self.restart('ledmatrix-web') and ok return self.restart('ledmatrix-web') and ok
def display_drawing(self):
"""True while the restarted display keeps its heartbeat fresh."""
data = self.read_heartbeat()
mono = data.get('mono') if data else None
if not isinstance(mono, (int, float)) or isinstance(mono, bool):
return False
if self.display_restarted_at is not None and mono < self.display_restarted_at:
return False # still the process from before the restart
return self.clock() - mono <= HEARTBEAT_FRESH_SECONDS
def wait_healthy(self, display): def wait_healthy(self, display):
"""None once the services are up and stay up, else what went wrong.""" """None once the services are up and stay up, else what went wrong."""
deadline = self.clock() + HEALTH_TIMEOUT_SECONDS + STABLE_SECONDS deadline = self.clock() + HEALTH_TIMEOUT_SECONDS + STABLE_SECONDS
healthy_since = baseline = None healthy_since = baseline = None
web = disp = False web = disp = False
active = drawing = True
count_known = True count_known = True
while self.clock() < deadline: while self.clock() < deadline:
web = self.web_responds() web = self.web_responds()
disp = self.service_active('ledmatrix') if display else True active = self.service_active('ledmatrix') if display else True
drawing = self.display_drawing() if (display and self.expect_heartbeat) else True
disp = active and drawing
restarts = self.restart_count('ledmatrix') if display else None restarts = self.restart_count('ledmatrix') if display else None
# Without a restart count a crash loop looks healthy between # Without a restart count a crash loop looks healthy between
# attempts, so an unreadable count never counts as stable. # attempts, so an unreadable count never counts as stable.
@@ -181,8 +230,11 @@ class Verifier:
problems = [] problems = []
if not web: if not web:
problems.append('the web interface did not respond') problems.append('the web interface did not respond')
if not disp: if not active:
problems.append('the display service did not stay running') problems.append('the display service did not stay running')
elif not drawing:
problems.append('the display service is running but its panel is not '
'being drawn (no fresh heartbeat)')
if web and disp and not count_known: if web and disp and not count_known:
problems.append("the display service's restart count could not be read") problems.append("the display service's restart count could not be read")
return '; '.join(problems) or 'the display service kept restarting' return '; '.join(problems) or 'the display service kept restarting'
@@ -225,6 +277,20 @@ class Verifier:
if not old: if not old:
return False, 'the commit to roll back to is unknown' return False, 'the commit to roll back to is unknown'
requirements = self.changed_requirements(old, new) if new else list(REQUIREMENT_FILES) requirements = self.changed_requirements(old, new) if new else list(REQUIREMENT_FILES)
# An update may have moved HEAD between main and a detached release
# tag (the stable/beta channels). Go back to where HEAD was -- the
# branch, or detached -- before resetting, or resetting would drag
# the wrong ref: main onto a release commit, or leave a device that
# was following main stuck on a detached one. No old_ref (an older
# updater wrote this file) means HEAD never moved between refs.
old_ref = pending.get('old_ref')
if old_ref is not None:
move = (['git', 'checkout', '--quiet', '--force', old_ref] if old_ref
else ['git', 'checkout', '--quiet', '--force', '--detach', old])
result = self._run(move, timeout=GIT_RESET_TIMEOUT_SECONDS)
if result.returncode != 0:
return False, (f'"{" ".join(move)}" failed: '
f'{(result.stderr or result.stdout or "").strip()}')
# --hard: the updater refuses to run with local edits to tracked core # --hard: the updater refuses to run with local edits to tracked core
# files (web_interface/auto_update.local_changes), so outside the # files (web_interface/auto_update.local_changes), so outside the
# plugin folders the only thing this discards is the update. Edits # plugin folders the only thing this discards is the update. Edits
@@ -258,6 +324,10 @@ class Verifier:
write_pending(self.pending_file, pending) write_pending(self.pending_file, pending)
display = bool(pending.get('display_was_active')) display = bool(pending.get('display_was_active'))
# Read before anything restarts: the display still running is the
# pre-update code, and whether it writes a heartbeat decides whether
# the updated one must.
self.expect_heartbeat = display and self.read_heartbeat() is not None
dependency_failures = pending.get('dependency_failures') or [] dependency_failures = pending.get('dependency_failures') or []
if dependency_failures: if dependency_failures:
# Never restart onto code whose packages did not install. # Never restart onto code whose packages did not install.
+31 -2
View File
@@ -42,6 +42,8 @@ class WiFiMonitorDaemon:
""" """
self.check_interval = check_interval self.check_interval = check_interval
self.wifi_manager = WiFiManager() self.wifi_manager = WiFiManager()
# mtime of wifi_config.json as last loaded; see _reload_config_if_changed.
self._config_mtime = self._config_file_mtime()
self.running = True self.running = True
self.last_state = None self.last_state = None
# Counts consecutive checks where nmcli says "connected" but internet is unreachable. # Counts consecutive checks where nmcli says "connected" but internet is unreachable.
@@ -57,7 +59,32 @@ class WiFiMonitorDaemon:
"""Handle shutdown signals""" """Handle shutdown signals"""
logger.info(f"Received signal {signum}, shutting down...") logger.info(f"Received signal {signum}, shutting down...")
self.running = False self.running = False
def _config_file_mtime(self):
try:
return self.wifi_manager.config_path.stat().st_mtime_ns
except OSError:
return None
def _reload_config_if_changed(self):
"""Re-read wifi_config.json when it has changed on disk.
The web UI's auto-enable toggle (POST /api/v3/wifi/ap/auto-enable)
only writes the file; this process read it once at startup, so the
toggle did nothing until the daemon restarted. One stat per check.
"""
mtime = self._config_file_mtime()
if mtime is None or mtime == self._config_mtime:
return
before = self.wifi_manager.config.get("auto_enable_ap_mode", True)
self.wifi_manager._load_config()
# _load_config can itself save (it fills in missing keys), so take
# the mtime after it, or that save would trigger another reload.
self._config_mtime = self._config_file_mtime()
after = self.wifi_manager.config.get("auto_enable_ap_mode", True)
if after != before:
logger.info(f"wifi_config.json changed: auto_enable_ap_mode={after}")
def run(self): def run(self):
"""Main daemon loop""" """Main daemon loop"""
logger.info("WiFi Monitor Daemon started") logger.info("WiFi Monitor Daemon started")
@@ -78,6 +105,8 @@ class WiFiMonitorDaemon:
while self.running: while self.running:
try: try:
self._reload_config_if_changed()
# One combined check that also returns the state it observed — # One combined check that also returns the state it observed —
# the previous flow fetched status before AND after the check # the previous flow fetched status before AND after the check
# on top of the check's own internal fetch, each one several # on top of the check's own internal fetch, each one several
@@ -219,7 +248,7 @@ def main():
parser.add_argument( parser.add_argument(
'--foreground', '--foreground',
action='store_true', action='store_true',
help='Run in foreground (for debugging)' help='Accepted for compatibility; the daemon always runs in the foreground'
) )
args = parser.parse_args() args = parser.parse_args()
+1 -1
View File
@@ -4,5 +4,5 @@ LEDMatrix Display System
Core source package for the LED Matrix Display project. Core source package for the LED Matrix Display project.
""" """
__version__ = "3.5.0" __version__ = "3.8.0"
+25 -9
View File
@@ -70,7 +70,14 @@ def _read(path):
def is_enabled(config): def is_enabled(config):
return bool((config.get('auto_update') or {}).get('enabled', False)) # Only the {"enabled": true} object turns this on. A hand-edited
# non-dict (e.g. "auto_update": true) raised AttributeError here and
# aborted startup setup; the web UI's save replaces such a value with {}
# (disabled), so read it the same way.
section = config.get('auto_update')
if not isinstance(section, dict):
return False
return bool(section.get('enabled', False))
class UpdateHelperSetup: class UpdateHelperSetup:
@@ -201,17 +208,26 @@ class UpdateHelperSetup:
try: try:
self.result_file.parent.mkdir(parents=True, exist_ok=True) self.result_file.parent.mkdir(parents=True, exist_ok=True)
fd, tmp = tempfile.mkstemp(dir=str(self.result_file.parent), prefix='.auto_update_setup_') fd, tmp = tempfile.mkstemp(dir=str(self.result_file.parent), prefix='.auto_update_setup_')
with os.fdopen(fd, 'w', encoding='utf-8') as f: try:
json.dump(result, f, indent=2) with os.fdopen(fd, 'w', encoding='utf-8') as f:
os.chmod(tmp, 0o644) json.dump(result, f, indent=2)
if self._web_ids and hasattr(os, 'chown'): os.chmod(tmp, 0o644)
# Only root can give the file away; the result is readable if self._web_ids and hasattr(os, 'chown'):
# (0644) either way, so a failed chown must not lose it. # Only root can give the file away; the result is readable
# (0644) either way, so a failed chown must not lose it.
try:
os.chown(tmp, *self._web_ids)
except OSError:
pass
os.replace(tmp, self.result_file)
except BaseException:
# Don't leave a .auto_update_setup_* file behind in the
# project dir every time the write fails.
try: try:
os.chown(tmp, *self._web_ids) os.unlink(tmp)
except OSError: except OSError:
pass pass
os.replace(tmp, self.result_file) raise
except OSError as e: except OSError as e:
logger.warning("Could not record automatic update setup result: %s", e) logger.warning("Could not record automatic update setup result: %s", e)
return result return result
+48 -12
View File
@@ -103,6 +103,32 @@ class FetchResult:
# FAILED, which turns "you cancelled this" into "this errored". # FAILED, which turns "you cancelled this" into "this errored".
final_status: Optional[FetchStatus] = None final_status: Optional[FetchStatus] = None
class _ConnectionRetryingSession:
"""``session.get`` that retries a connection error a few times.
For ESPN date chunks, which bypass _make_request_with_retry: a failed
chunk is logged and skipped, so a brief network blip would otherwise drop
a month from a cached season. That protection used to come from the
session adapter's own retries, which every other request stacked with the
retry loop.
"""
ATTEMPTS = 3
DELAY = 0.5
def __init__(self, session):
self._session = session
def get(self, *args, **kwargs):
for attempt in range(self.ATTEMPTS):
try:
return self._session.get(*args, **kwargs)
except requests.ConnectionError:
if attempt == self.ATTEMPTS - 1:
raise
time.sleep(self.DELAY * (attempt + 1))
class BackgroundDataService: class BackgroundDataService:
""" """
Background data service for fetching season data without blocking the main thread. Background data service for fetching season data without blocking the main thread.
@@ -163,10 +189,16 @@ class BackgroundDataService:
'average_fetch_time': 0.0 'average_fetch_time': 0.0
} }
# Session for HTTP requests # Session for HTTP requests. No retries at the adapter: a fetch goes
# through _make_request_with_retry (max_retries + 1 attempts with
# exponential backoff, logged), and date-range chunks through
# _ConnectionRetryingSession. With the adapter also retrying
# connection errors three times, a dead network cost up to 16
# connection attempts per request and held one of the few worker
# threads for all of them.
self.session = requests.Session() self.session = requests.Session()
self.session.mount('http://', requests.adapters.HTTPAdapter(max_retries=3)) self.session.mount('http://', requests.adapters.HTTPAdapter(max_retries=0))
self.session.mount('https://', requests.adapters.HTTPAdapter(max_retries=3)) self.session.mount('https://', requests.adapters.HTTPAdapter(max_retries=0))
# Default headers: core's shared set (real User-Agent, no hand-set # Default headers: core's shared set (real User-Agent, no hand-set
# Accept-Encoding) -- see src/common/api_helper.py. # Accept-Encoding) -- see src/common/api_helper.py.
@@ -247,15 +279,19 @@ class BackgroundDataService:
# same object the dict holds. # same object the dict holds.
self.completed_requests[request_id] = result self.completed_requests[request_id] = result
if callback: # The callback runs outside the lock, as on the worker path: it is
try: # plugin code, and holding the service lock through it blocked
callback(result) # every worker's result bookkeeping (and any other thread's
except Exception as e: # submit) for as long as the callback took.
logger.error(f"Error in callback for request {request_id}: {e}") if callback:
self._release_payload(result) try:
callback(result)
except Exception as e:
logger.error(f"Error in callback for request {request_id}: {e}")
self._release_payload(result)
logger.debug(f"Cache hit for {sport} {year} data") logger.debug(f"Cache hit for {sport} {year} data")
return request_id return request_id
# limit above 500 makes an ESPN *scoreboard* return a truncated list # limit above 500 makes an ESPN *scoreboard* return a truncated list
# (src/common/espn_dates.py). Other endpoints need more: /teams has 762 # (src/common/espn_dates.py). Other endpoints need more: /teams has 762
@@ -560,7 +596,7 @@ class BackgroundDataService:
""" """
logger.info("Recovering %s %s from a rejected date range", request.sport, request.year) logger.info("Recovering %s %s from a rejected date range", request.sport, request.year)
return fetch_espn_date_chunks( return fetch_espn_date_chunks(
self.session, _ConnectionRetryingSession(self.session),
request.url, request.url,
params=request.params, params=request.params,
headers=request.headers, headers=request.headers,
+73 -40
View File
@@ -88,7 +88,6 @@ _WIFI_REL = Path("config/wifi_config.json")
_YTM_REL = Path("config/ytm_auth.json") _YTM_REL = Path("config/ytm_auth.json")
_FONTS_REL = Path("assets/fonts") _FONTS_REL = Path("assets/fonts")
_PLUGIN_UPLOADS_REL = Path("assets/plugins") _PLUGIN_UPLOADS_REL = Path("assets/plugins")
_STATE_REL = Path("data/plugin_state.json")
#: The sections that are one file each: (section name, path, the #: The sections that are one file each: (section name, path, the
#: RestoreOptions flag that restores it). create, preview, validate and #: RestoreOptions flag that restores it). create, preview, validate and
@@ -102,6 +101,12 @@ _SINGLE_FILE_SECTIONS: Tuple[Tuple[str, Path, str], ...] = (
("ytm_auth", _YTM_REL, "restore_wifi"), ("ytm_auth", _YTM_REL, "restore_wifi"),
) )
#: Sections holding credentials. Restored onto a device that has no copy yet,
#: they would otherwise take the extracted temp file's umask mode (0o644,
#: world-readable); 0o640 matches what config_manager_atomic gives secrets.
_PRIVATE_SECTION_RELS = frozenset({_SECRETS_REL, _WIFI_REL, _YTM_REL})
_PRIVATE_FILE_MODE = 0o640
MANIFEST_NAME = "manifest.json" MANIFEST_NAME = "manifest.json"
PLUGINS_MANIFEST_NAME = "plugins.json" PLUGINS_MANIFEST_NAME = "plugins.json"
@@ -173,20 +178,27 @@ def _build_manifest(contents: List[str]) -> Dict[str, Any]:
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
def _plugins_directory(project_root: Path) -> Path: def _read_config(project_root: Path) -> Dict[str, Any]:
"""The plugin install directory: ``plugin_system.plugins_directory`` from """config/config.json as a dict; empty when missing or unreadable."""
config/config.json (relative to ``project_root`` unless absolute), or
``plugin-repos`` when the config does not say or cannot be read."""
configured: Any = None
try: try:
with (project_root / _CONFIG_REL).open("r", encoding="utf-8") as f: with (project_root / _CONFIG_REL).open("r", encoding="utf-8") as f:
config = json.load(f) config = json.load(f)
if isinstance(config, dict):
plugin_system = config.get("plugin_system")
if isinstance(plugin_system, dict):
configured = plugin_system.get("plugins_directory")
except (OSError, json.JSONDecodeError): except (OSError, json.JSONDecodeError):
pass return {}
return config if isinstance(config, dict) else {}
def _plugins_directory(project_root: Path,
config: Optional[Dict[str, Any]] = None) -> Path:
"""The plugin install directory: ``plugin_system.plugins_directory`` from
config/config.json (relative to ``project_root`` unless absolute), or
``plugin-repos`` when the config does not say or cannot be read."""
if config is None:
config = _read_config(project_root)
configured: Any = None
plugin_system = config.get("plugin_system")
if isinstance(plugin_system, dict):
configured = plugin_system.get("plugins_directory")
if not isinstance(configured, str) or not configured.strip(): if not isinstance(configured, str) or not configured.strip():
configured = "plugin-repos" configured = "plugin-repos"
path = Path(configured) path = Path(configured)
@@ -196,33 +208,23 @@ def _plugins_directory(project_root: Path) -> Path:
def list_installed_plugins(project_root: Path) -> List[Dict[str, Any]]: def list_installed_plugins(project_root: Path) -> List[Dict[str, Any]]:
""" """
Return a list of currently-installed plugins suitable for the backup Return a list of currently-installed plugins suitable for the backup
manifest. Each entry has ``plugin_id`` and ``version``. manifest. Each entry has ``plugin_id``, ``version`` and ``enabled``.
Reads ``data/plugin_state.json`` if present, then adds any plugin it The plugins are the ``manifest.json`` files in the configured plugin
does not list from the ``manifest.json`` files in the configured plugin directory (see :func:`_plugins_directory`), with the manifest's version;
directory (see :func:`_plugins_directory`). ``enabled`` is config.json's flag by the display's rule (a missing flag
is disabled). A restore reinstalls every listed plugin and takes enabled
state from the restored config.json, so ``enabled`` is informational.
``data/plugin_state.json`` is not read: it only ever repeated config's
enabled flags and the manifests' versions, and is retired (nothing
writes it any more). An old backup that listed a plugin only from that
file still restores it, since restore reads ``plugins.json`` as written.
""" """
plugins: Dict[str, Dict[str, Any]] = {} plugins: Dict[str, Dict[str, Any]] = {}
config = _read_config(project_root)
state_file = project_root / _STATE_REL plugins_root = _plugins_directory(project_root, config)
if state_file.exists():
try:
with state_file.open("r", encoding="utf-8") as f:
state = json.load(f)
raw_plugins = state.get("states", {}) if isinstance(state, dict) else {}
if isinstance(raw_plugins, dict):
for plugin_id, info in raw_plugins.items():
if not isinstance(info, dict):
continue
plugins[plugin_id] = {
"plugin_id": plugin_id,
"version": info.get("version") or "",
"enabled": bool(info.get("enabled", True)),
}
except (OSError, json.JSONDecodeError) as e:
logger.warning("Could not read plugin_state.json: %s", e)
plugins_root = _plugins_directory(project_root)
if plugins_root.exists(): if plugins_root.exists():
for entry in sorted(plugins_root.iterdir()): for entry in sorted(plugins_root.iterdir()):
if not entry.is_dir(): if not entry.is_dir():
@@ -235,12 +237,17 @@ def list_installed_plugins(project_root: Path) -> List[Dict[str, Any]]:
data = json.load(f) data = json.load(f)
except (OSError, json.JSONDecodeError): except (OSError, json.JSONDecodeError):
continue continue
# Valid JSON that is not an object (a list, a bare string) would
# raise AttributeError on .get() and abort the whole export.
if not isinstance(data, dict):
continue
plugin_id = data.get("id") or entry.name plugin_id = data.get("id") or entry.name
if plugin_id not in plugins: if plugin_id not in plugins:
section = config.get(plugin_id)
plugins[plugin_id] = { plugins[plugin_id] = {
"plugin_id": plugin_id, "plugin_id": plugin_id,
"version": data.get("version", ""), "version": data.get("version", ""),
"enabled": True, "enabled": isinstance(section, dict) and bool(section.get("enabled", False)),
} }
return sorted(plugins.values(), key=lambda p: p["plugin_id"]) return sorted(plugins.values(), key=lambda p: p["plugin_id"])
@@ -310,7 +317,11 @@ def create_backup(
contents: List[str] = [] contents: List[str] = []
# Stream directly to a temp file so we never hold the whole ZIP in memory. # Stream directly to a temp file so we never hold the whole ZIP in memory.
tmp_path = zip_path.with_suffix(".zip.tmp") # The name is unique per call: a fixed "<zip>.tmp" was shared by two
# exports started in the same second, which then wrote the same file.
fd, tmp_name = tempfile.mkstemp(dir=str(output_dir), prefix=f".{zip_name}.", suffix=".tmp")
os.close(fd)
tmp_path = Path(tmp_name)
try: try:
with zipfile.ZipFile(tmp_path, "w", compression=zipfile.ZIP_DEFLATED) as zf: with zipfile.ZipFile(tmp_path, "w", compression=zipfile.ZIP_DEFLATED) as zf:
for section, rel, _flag in _SINGLE_FILE_SECTIONS: for section, rel, _flag in _SINGLE_FILE_SECTIONS:
@@ -347,7 +358,24 @@ def create_backup(
manifest = _build_manifest(contents) manifest = _build_manifest(contents)
zf.writestr(MANIFEST_NAME, json.dumps(manifest, indent=2)) zf.writestr(MANIFEST_NAME, json.dumps(manifest, indent=2))
os.replace(tmp_path, zip_path) # Same-second exports share a timestamp; number the later one rather
# than replacing the backup the first one just returned. The name is
# claimed with an exclusive create (O_EXCL fails if it exists), so two
# exports finishing together can't both pick the same free name; the
# replace then swaps the finished archive in over our own placeholder.
suffix = 2
while True:
try:
os.close(os.open(zip_path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600))
break
except FileExistsError:
zip_path = output_dir / f"{Path(zip_name).stem}-{suffix}.zip"
suffix += 1
try:
os.replace(tmp_path, zip_path)
except BaseException:
zip_path.unlink(missing_ok=True)
raise
except Exception: except Exception:
tmp_path.unlink(missing_ok=True) tmp_path.unlink(missing_ok=True)
raise raise
@@ -450,7 +478,8 @@ def validate_backup(zip_path: Path) -> Tuple[bool, str, Dict[str, Any]]:
): ):
detected.append("plugin_uploads") detected.append("plugin_uploads")
plugins: List[Dict[str, Any]] = [] # Whatever the archive's manifest holds; checked below.
plugins: Any = []
if PLUGINS_MANIFEST_NAME in names: if PLUGINS_MANIFEST_NAME in names:
try: try:
plugins = json.loads(zf.read(PLUGINS_MANIFEST_NAME).decode("utf-8")) plugins = json.loads(zf.read(PLUGINS_MANIFEST_NAME).decode("utf-8"))
@@ -493,7 +522,7 @@ def _extract_zip_safe(zip_path: Path, dest_dir: Path) -> None:
shutil.copyfileobj(src, dst, length=64 * 1024) shutil.copyfileobj(src, dst, length=64 * 1024)
def _copy_file(src: Path, dst: Path) -> None: def _copy_file(src: Path, dst: Path, new_mode: Optional[int] = None) -> None:
"""Replace ``dst`` with ``src``, atomically, without needing to own ``dst``. """Replace ``dst`` with ``src``, atomically, without needing to own ``dst``.
``shutil.copy2`` opens the destination for writing, so it needs write ``shutil.copy2`` opens the destination for writing, so it needs write
@@ -509,6 +538,7 @@ def _copy_file(src: Path, dst: Path) -> None:
The destination's existing mode is preserved when there is one, so The destination's existing mode is preserved when there is one, so
restoring secrets does not silently widen them to the umask default. restoring secrets does not silently widen them to the umask default.
When there is none, ``new_mode`` (if given) is used instead of ``src``'s.
""" """
dst.parent.mkdir(parents=True, exist_ok=True) dst.parent.mkdir(parents=True, exist_ok=True)
@@ -530,6 +560,8 @@ def _copy_file(src: Path, dst: Path) -> None:
shutil.copyfile(src, tmp_path) shutil.copyfile(src, tmp_path)
if existing_mode is not None: if existing_mode is not None:
os.chmod(tmp_path, existing_mode) os.chmod(tmp_path, existing_mode)
elif new_mode is not None:
os.chmod(tmp_path, new_mode)
else: else:
shutil.copymode(src, tmp_path) shutil.copymode(src, tmp_path)
if existing_owner is not None and hasattr(os, 'chown'): if existing_owner is not None and hasattr(os, 'chown'):
@@ -593,7 +625,8 @@ def restore_backup(
result.skipped.append(section) result.skipped.append(section)
continue continue
try: try:
_copy_file(tmp_dir / rel, project_root / rel) _copy_file(tmp_dir / rel, project_root / rel,
new_mode=_PRIVATE_FILE_MODE if rel in _PRIVATE_SECTION_RELS else None)
result.restored.append(section) result.restored.append(section)
except OSError as e: except OSError as e:
logger.error("[Backup] Failed to restore %s: %s", rel.name, e, exc_info=True) logger.error("[Backup] Failed to restore %s: %s", rel.name, e, exc_info=True)
+33 -12
View File
@@ -16,11 +16,16 @@ import time
import requests import requests
import json import json
from typing import Dict, Any, Optional, List from typing import Dict, Any, Optional, List, cast
from src.common.api_helper import DEFAULT_HTTP_HEADERS from src.common.api_helper import DEFAULT_HTTP_HEADERS
def _is_no_odds_marker(data: Any) -> bool:
"""Whether a cached odds entry is the "ESPN had none" marker, not odds."""
return isinstance(data, dict) and bool(data.get("no_odds"))
class BaseOddsManager: class BaseOddsManager:
""" """
Base class for odds data fetching and management. Base class for odds data fetching and management.
@@ -100,7 +105,7 @@ class BaseOddsManager:
_FAILURE_COOLDOWN = 60.0 _FAILURE_COOLDOWN = 60.0
def get_odds(self, sport: str | None, league: str | None, event_id: str, def get_odds(self, sport: str | None, league: str | None, event_id: str,
update_interval_seconds: int = None) -> Optional[Dict[str, Any]]: update_interval_seconds: Optional[int] = None) -> Optional[Dict[str, Any]]:
""" """
Fetch odds data for a specific game. Fetch odds data for a specific game.
@@ -121,10 +126,20 @@ class BaseOddsManager:
cache_key = f"odds_espn_{sport}_{league}_{event_id}" cache_key = f"odds_espn_{sport}_{league}_{event_id}"
# Check cache first # Check cache first
cached_data = self.cache_manager.get_with_auto_strategy(cache_key) cached_data: Optional[Dict[str, Any]] = self.cache_manager.get_with_auto_strategy(cache_key)
# Per-game chatter, logged on every update of every game on the
# slate: debug, not the journal.
if cached_data: if cached_data:
self.logger.info(f"Using cached odds from ESPN for {cache_key}") # A game ESPN had no odds for is cached as {"no_odds": True} so it
# isn't re-requested every update. That marker is a cache hit --
# its ttl decides when to ask again -- but it is not odds: returned
# as-is, a caller saw a truthy dict and treated the game as having
# odds. The plugins' bundled copies already did this.
if _is_no_odds_marker(cached_data):
self.logger.debug("Cached no-odds marker for %s", cache_key)
return None
self.logger.debug(f"Using cached odds from ESPN for {cache_key}")
return cached_data return cached_data
if time.monotonic() < self._skip_network_until: if time.monotonic() < self._skip_network_until:
@@ -137,7 +152,7 @@ class BaseOddsManager:
self._skip_network_until - time.monotonic()) self._skip_network_until - time.monotonic())
return None return None
self.logger.info(f"Cache miss - fetching fresh odds from ESPN for {cache_key}") self.logger.debug(f"Cache miss - fetching fresh odds from ESPN for {cache_key}")
try: try:
# Map league names to ESPN API format # Map league names to ESPN API format
@@ -151,7 +166,7 @@ class BaseOddsManager:
espn_league = league_mapping.get(league, league) espn_league = league_mapping.get(league, league)
url = f"{self.base_url}/{sport}/leagues/{espn_league}/events/{event_id}/competitions/{event_id}/odds" url = f"{self.base_url}/{sport}/leagues/{espn_league}/events/{event_id}/competitions/{event_id}/odds"
self.logger.info(f"Requesting odds from URL: {url}") self.logger.debug(f"Requesting odds from URL: {url}")
response = self.session.get(url, timeout=self.request_timeout) response = self.session.get(url, timeout=self.request_timeout)
response.raise_for_status() response.raise_for_status()
@@ -163,9 +178,9 @@ class BaseOddsManager:
odds_data = self._extract_espn_data(raw_data) odds_data = self._extract_espn_data(raw_data)
if odds_data: if odds_data:
self.logger.info(f"Successfully extracted odds data: {odds_data}") self.logger.debug(f"Successfully extracted odds data: {odds_data}")
self.cache_manager.set(cache_key, odds_data, ttl=interval) self.cache_manager.set(cache_key, odds_data, ttl=interval)
self.logger.info(f"Saved odds data to cache for {cache_key} with TTL {interval}s") self.logger.debug(f"Saved odds data to cache for {cache_key} with TTL {interval}s")
else: else:
self.logger.debug(f"No odds data available for {cache_key}") self.logger.debug(f"No odds data available for {cache_key}")
# Cache the absence too, so the game is not re-requested # Cache the absence too, so the game is not re-requested
@@ -174,16 +189,22 @@ class BaseOddsManager:
return odds_data return odds_data
# Before RequestException: requests' JSONDecodeError subclasses it, so
# listed second this branch never ran and a bad body was reported as a
# failed fetch. It holds off like a failed fetch did, so only the
# message changes.
except (json.JSONDecodeError, requests.exceptions.JSONDecodeError):
self._skip_network_until = time.monotonic() + self._FAILURE_COOLDOWN
self.logger.error(f"Error decoding JSON response from ESPN API for {cache_key}.")
except requests.exceptions.RequestException as e: except requests.exceptions.RequestException as e:
self._skip_network_until = time.monotonic() + self._FAILURE_COOLDOWN self._skip_network_until = time.monotonic() + self._FAILURE_COOLDOWN
self.logger.error( self.logger.error(
"Error fetching odds from ESPN API for %s: %s. Holding off on odds " "Error fetching odds from ESPN API for %s: %s. Holding off on odds "
"for %.0fs so a slate of games does not pay this timeout each.", "for %.0fs so a slate of games does not pay this timeout each.",
cache_key, e, self._FAILURE_COOLDOWN) cache_key, e, self._FAILURE_COOLDOWN)
except json.JSONDecodeError:
self.logger.error(f"Error decoding JSON response from ESPN API for {cache_key}.") cached = self.cache_manager.get_with_auto_strategy(cache_key)
return None if _is_no_odds_marker(cached) else cast(Optional[Dict[str, Any]], cached)
return self.cache_manager.get_with_auto_strategy(cache_key)
def _extract_espn_data(self, data: Dict[str, Any]) -> Optional[Dict[str, Any]]: def _extract_espn_data(self, data: Dict[str, Any]) -> Optional[Dict[str, Any]]:
""" """
+6 -3
View File
@@ -8,7 +8,7 @@ import os
import time import time
import threading import threading
import logging import logging
from typing import Dict, Any, Optional from typing import Dict, Any, Optional, Union
# Historical fixed ceiling, kept as the fallback when RAM cannot be read. # Historical fixed ceiling, kept as the fallback when RAM cannot be read.
DEFAULT_MAX_SIZE = 1000 DEFAULT_MAX_SIZE = 1000
@@ -70,13 +70,15 @@ class MemoryCache:
""" """
self.logger = logging.getLogger(__name__) self.logger = logging.getLogger(__name__)
self._cache: Dict[str, Dict[str, Any]] = {} self._cache: Dict[str, Dict[str, Any]] = {}
self._timestamps: Dict[str, float] = {} # Values are time.time() floats; get()/cleanup also accept a numeric
# string, as a timestamp may have been restored from serialized data.
self._timestamps: Dict[str, Union[float, str]] = {}
self._lock = threading.Lock() self._lock = threading.Lock()
self._max_size = max_size self._max_size = max_size
self._cleanup_interval = cleanup_interval self._cleanup_interval = cleanup_interval
self._last_cleanup = time.time() self._last_cleanup = time.time()
def get(self, key: str, max_age: Optional[int] = None) -> Optional[Dict[str, Any]]: def get(self, key: str, max_age: Optional[float] = None) -> Optional[Dict[str, Any]]:
""" """
Get value from memory cache. Get value from memory cache.
@@ -200,6 +202,7 @@ class MemoryCache:
max_age_for_cleanup = 3600 # 1 hour max_age_for_cleanup = 3600 # 1 hour
expired_keys = [] expired_keys = []
timestamp: Optional[Union[float, str]]
for key, timestamp in list(self._timestamps.items()): for key, timestamp in list(self._timestamps.items()):
if isinstance(timestamp, str): if isinstance(timestamp, str):
try: try:
+1 -256
View File
@@ -37,7 +37,6 @@ from src.cache.disk_cache import DiskCache
from src.cache.cache_strategy import CacheStrategy from src.cache.cache_strategy import CacheStrategy
from src.cache.cache_metrics import CacheMetrics from src.cache.cache_metrics import CacheMetrics
from src.logging_config import get_logger from src.logging_config import get_logger
from src.deprecation import deprecated
# Canonical implementation lives in src.cache.disk_cache; re-exported here # Canonical implementation lives in src.cache.disk_cache; re-exported here
# because this module's docstring documents it and external code may import # because this module's docstring documents it and external code may import
@@ -408,122 +407,6 @@ class CacheManager:
"""Get the cache directory path.""" """Get the cache directory path."""
return self.cache_dir return self.cache_dir
@deprecated("3.7.0")
def has_data_changed(self, data_type: str, new_data: Dict[str, Any]) -> bool:
"""Check if data has changed from cached version."""
cached_data = self.load_cache(data_type)
if not cached_data:
return True
if data_type == 'weather':
return self._has_weather_changed(cached_data, new_data)
elif data_type == 'stocks':
return self._has_stocks_changed(cached_data, new_data)
elif data_type == 'stock_news':
return self._has_news_changed(cached_data, new_data)
elif data_type == 'nhl':
return self._has_nhl_changed(cached_data, new_data)
elif data_type == 'mlb':
return self._has_mlb_changed(cached_data, new_data)
return True
def _has_weather_changed(self, cached: Dict[str, Any], new: Dict[str, Any]) -> bool:
"""Check if weather data has changed."""
# Handle new cache structure where data is nested under 'data' key
if 'data' in cached:
cached = cached['data']
# Handle case where cached data might be the weather data directly
if 'current' in cached:
# This is the new structure with 'current' and 'forecast' keys
current_weather = cached.get('current', {})
if current_weather and 'main' in current_weather and 'weather' in current_weather:
cached_temp = round(current_weather['main']['temp'])
cached_condition = current_weather['weather'][0]['main']
return (cached_temp != new.get('temp') or
cached_condition != new.get('condition'))
# Handle old structure where temp and condition are directly accessible
return (cached.get('temp') != new.get('temp') or
cached.get('condition') != new.get('condition'))
def _has_stocks_changed(self, cached: Dict[str, Any], new: Dict[str, Any]) -> bool:
"""Check if stock data has changed."""
if not self._is_market_open():
return False
return cached.get('price') != new.get('price')
def _has_news_changed(self, cached: Dict[str, Any], new: Dict[str, Any]) -> bool:
"""Check if news data has changed."""
# Handle both dictionary and list formats
if isinstance(new, list):
# If new data is a list, cached data should also be a list
if not isinstance(cached, list):
return True
# Compare lengths and content
if len(cached) != len(new):
return True
# Compare titles since they're unique enough for our purposes
cached_titles = set(item.get('title', '') for item in cached)
new_titles = set(item.get('title', '') for item in new)
return cached_titles != new_titles
else:
# Original dictionary format handling
cached_headlines = set(h.get('id') for h in cached.get('headlines', []))
new_headlines = set(h.get('id') for h in new.get('headlines', []))
return not cached_headlines.issuperset(new_headlines)
def _has_nhl_changed(self, cached: Dict[str, Any], new: Dict[str, Any]) -> bool:
"""Check if NHL data has changed."""
return (cached.get('game_status') != new.get('game_status') or
cached.get('score') != new.get('score'))
def _has_mlb_changed(self, cached: Dict[str, Any], new: Dict[str, Any]) -> bool:
"""Check if MLB game data has changed."""
if not cached or not new:
return True
# Check if any games have changed status or score
for game_id, new_game in new.items():
cached_game = cached.get(game_id)
if not cached_game:
return True
# Check for score changes
if (new_game['away_score'] != cached_game['away_score'] or
new_game['home_score'] != cached_game['home_score']):
return True
# Check for status changes
if new_game['status'] != cached_game['status']:
return True
# For live games, check inning and count
if new_game['status'] == 'in':
if (new_game['inning'] != cached_game['inning'] or
new_game['inning_half'] != cached_game['inning_half'] or
new_game['balls'] != cached_game['balls'] or
new_game['strikes'] != cached_game['strikes'] or
new_game['bases_occupied'] != cached_game['bases_occupied']):
return True
return False
def _is_market_open(self) -> bool:
"""Check if the US stock market is currently open."""
return self._strategy_component.is_market_open()
@deprecated("3.7.0", "use set()")
def update_cache(self, data_type: str, data: Dict[str, Any]) -> bool:
"""Update cache with new data."""
cache_data = {
# Header first; see DiskCache's stale check.
'timestamp': time.time(),
'data': data,
}
return self.save_cache(data_type, cache_data)
def get(self, key: str, max_age: Optional[int] = 300, def get(self, key: str, max_age: Optional[int] = 300,
memory_ttl: Optional[int] = None) -> Optional[Dict[str, Any]]: memory_ttl: Optional[int] = None) -> Optional[Dict[str, Any]]:
"""Get data from cache if it exists and is not stale. """Get data from cache if it exists and is not stale.
@@ -564,42 +447,6 @@ class CacheManager:
cache_data['data'] = data cache_data['data'] = data
self.save_cache(key, cache_data) self.save_cache(key, cache_data)
@deprecated("3.7.0")
def setup_persistent_cache(self) -> bool:
"""
Set up a persistent cache directory with proper permissions.
This should be run once with sudo to create the directory.
"""
try:
# Try to create /var/cache/ledmatrix with proper permissions
from pathlib import Path
from src.common.permission_utils import (
ensure_directory_permissions,
get_cache_dir_mode
)
cache_dir = '/var/cache/ledmatrix'
cache_dir_path = Path(cache_dir)
ensure_directory_permissions(cache_dir_path, get_cache_dir_mode())
# Set ownership to the real user (not root)
real_user = os.environ.get('SUDO_USER')
if real_user:
import pwd
try:
uid = pwd.getpwnam(real_user).pw_uid
gid = pwd.getpwnam(real_user).pw_gid
os.chown(cache_dir, uid, gid)
self.logger.info(f"Set ownership of {cache_dir} to {real_user}")
except (OSError, KeyError) as e:
self.logger.warning(f"Could not set ownership for {cache_dir}: {e}", exc_info=True)
self.logger.info(f"Successfully set up persistent cache directory: {cache_dir}")
return True
except (OSError, IOError, PermissionError) as e:
self.logger.error(f"Failed to set up persistent cache directory {cache_dir}: {e}", exc_info=True)
return False
def cleanup_disk_cache(self, force: bool = False) -> Dict[str, Any]: def cleanup_disk_cache(self, force: bool = False) -> Dict[str, Any]:
""" """
Clean up expired disk cache files based on retention policies. Clean up expired disk cache files based on retention policies.
@@ -776,14 +623,6 @@ class CacheManager:
else: else:
self.logger.info("Disk cache cleanup thread stopped successfully") self.logger.info("Disk cache cleanup thread stopped successfully")
@deprecated("3.7.0")
def get_sport_live_interval(self, sport_key: str) -> int:
"""
Get the live_update_interval for a specific sport from config.
Falls back to default values if config is not available.
"""
return self._strategy_component.get_sport_live_interval(sport_key)
def get_cache_strategy(self, data_type: str, sport_key: Optional[str] = None) -> Dict[str, Any]: def get_cache_strategy(self, data_type: str, sport_key: Optional[str] = None) -> Dict[str, Any]:
""" """
Get cache strategy for different data types. Get cache strategy for different data types.
@@ -798,13 +637,6 @@ class CacheManager:
""" """
return self._strategy_component.get_data_type_from_key(key) return self._strategy_component.get_data_type_from_key(key)
@deprecated("3.7.0")
def get_sport_key_from_cache_key(self, key: str) -> Optional[str]:
"""
Extract sport key from cache key to determine appropriate live_update_interval.
"""
return self._strategy_component.get_sport_key_from_cache_key(key)
def get_cached_data_with_strategy(self, key: str, data_type: str = 'default') -> Optional[Dict[str, Any]]: def get_cached_data_with_strategy(self, key: str, data_type: str = 'default') -> Optional[Dict[str, Any]]:
""" """
Get data from cache using data-type-specific strategy. Get data from cache using data-type-specific strategy.
@@ -838,58 +670,6 @@ class CacheManager:
data_type = self.get_data_type_from_key(key) data_type = self.get_data_type_from_key(key)
return self.get_cached_data_with_strategy(key, data_type) return self.get_cached_data_with_strategy(key, data_type)
@deprecated("3.7.0", "use get()")
def get_background_cached_data(self, key: str, sport_key: Optional[str] = None) -> Optional[Dict[str, Any]]:
"""
Get data from background service cache with appropriate strategy.
This method is specifically designed for Recent/Upcoming managers
to use data cached by the background service.
Args:
key: Cache key to retrieve
sport_key: Sport key for determining appropriate cache strategy
Returns:
Cached data if available and fresh, None otherwise
"""
# Determine the appropriate cache strategy
data_type = self.get_data_type_from_key(key)
strategy = self.get_cache_strategy(data_type, sport_key)
# For Recent/Upcoming managers, we want to use the background service cache
# which should have longer TTLs than the individual manager caches
max_age = strategy['max_age']
memory_ttl = strategy.get('memory_ttl', max_age)
# Get the cached data
cached_data = self.get_cached_data(key, max_age, memory_ttl)
if cached_data:
# Record cache hit for performance monitoring
self.record_cache_hit('background')
# Unwrap if stored in { 'data': ..., 'timestamp': ... } format
if isinstance(cached_data, dict) and 'data' in cached_data:
return cached_data['data']
return cached_data
# Record cache miss for performance monitoring
self.record_cache_miss('background')
return None
@deprecated("3.7.0", "use get()")
def is_background_data_available(self, key: str, sport_key: Optional[str] = None) -> bool:
"""
Check if background service has fresh data available.
This helps Recent/Upcoming managers determine if they should
wait for background data or fetch immediately.
"""
data_type = self.get_data_type_from_key(key)
strategy = self.get_cache_strategy(data_type, sport_key)
# Check if we have data that's still fresh according to background service TTL
cached_data = self.get_cached_data(key, strategy['max_age'])
return cached_data is not None
def generate_sport_cache_key(self, sport: str, date_str: Optional[str] = None) -> str: def generate_sport_cache_key(self, sport: str, date_str: Optional[str] = None) -> str:
""" """
Centralized cache key generation for sports data. Centralized cache key generation for sports data.
@@ -906,44 +686,9 @@ class CacheManager:
date_str = datetime.now(pytz.utc).strftime('%Y%m%d') date_str = datetime.now(pytz.utc).strftime('%Y%m%d')
return f"{sport}_{date_str}" return f"{sport}_{date_str}"
@deprecated("3.7.0")
def record_cache_hit(self, cache_type: str = 'regular') -> None:
"""Record a cache hit for performance monitoring."""
self._metrics_component.record_hit(cache_type)
@deprecated("3.7.0")
def record_cache_miss(self, cache_type: str = 'regular') -> None:
"""Record a cache miss for performance monitoring."""
self._metrics_component.record_miss(cache_type)
@deprecated("3.7.0")
def record_fetch_time(self, duration: float) -> None:
"""Record fetch operation duration for performance monitoring."""
self._metrics_component.record_fetch_time(duration)
@deprecated("3.7.0")
def get_cache_metrics(self) -> Dict[str, Any]:
"""Get current cache performance metrics."""
return self._metrics_component.get_metrics()
@deprecated("3.7.0")
def log_cache_metrics(self) -> None:
"""Log current cache performance metrics."""
self._metrics_component.log_metrics()
@deprecated("3.7.0")
def get_memory_cache_stats(self) -> Dict[str, Any]:
"""
Get statistics about the memory cache.
Returns:
Dictionary with memory cache statistics
"""
return self._memory_cache_component.get_stats()
def log_memory_cache_stats(self) -> None: def log_memory_cache_stats(self) -> None:
"""Log current memory cache statistics.""" """Log current memory cache statistics."""
stats = self.get_memory_cache_stats() stats = self._memory_cache_component.get_stats()
self.logger.info(f"Memory Cache - Size: {stats['size']}/{stats['max_size']} " self.logger.info(f"Memory Cache - Size: {stats['size']}/{stats['max_size']} "
f"({stats['usage_percent']:.1f}%), " f"({stats['usage_percent']:.1f}%), "
f"Last cleanup: {time.time() - stats['last_cleanup']:.1f}s ago") f"Last cleanup: {time.time() - stats['last_cleanup']:.1f}s ago")
+138 -2
View File
@@ -24,24 +24,37 @@ Rules for the package:
| Module | For | Plugins import it? | Since | | Module | For | Plugins import it? | Since |
|---|---|---|---| |---|---|---|---|
| [`api_helper`](#api_helper) | HTTP GET/POST with caching and rate limiting | Yes | — | | [`api_helper`](#api_helper) | HTTP GET/POST with caching and rate limiting | Yes | — |
| [`bdf_font`](#bdf_font) | Load and draw BDF bitmap fonts | Yes, if drawing BDF text directly | Unreleased | | [`bdf_font`](#bdf_font) | Load and draw BDF bitmap fonts | Yes, if drawing BDF text directly | 3.5.0 |
| [`espn_dates`](#espn_dates) | Fetch ESPN scoreboards across a date range | Yes (scoreboards) | 3.5.0 | | [`espn_dates`](#espn_dates) | Fetch ESPN scoreboards across a date range | Yes (scoreboards) | 3.5.0 |
| [`favorite_team_check`](#favorite_team_check) | Log why a favourite team code shows nothing | Yes (scoreboards) | 3.6.0 |
| [`font_layout`](#font_layout) | Reproducible TrueType loading, crisp sizes | Yes | 3.4.0 | | [`font_layout`](#font_layout) | Reproducible TrueType loading, crisp sizes | Yes | 3.4.0 |
| [`frame_timing`](#frame_timing) | Timing of every presented frame, stall watchdog | No, core-internal | n/a |
| [`json_body`](#json_body) | Parse a response body as JSON, with orjson if installed | Optional (large payloads) | 3.5.0 |
| [`logo_helper`](#logo_helper) | Load, resize and cache team logos | Yes | — | | [`logo_helper`](#logo_helper) | Load, resize and cache team logos | Yes | — |
| [`path_safety`](#path_safety) | Turn request-supplied names into safe paths | No, core-internal | n/a | | [`path_safety`](#path_safety) | Turn request-supplied names into safe paths | No, core-internal | n/a |
| [`permission_utils`](#permission_utils) | File modes and shared-group ownership | Rarely | — | | [`permission_utils`](#permission_utils) | File modes and shared-group ownership | Rarely | — |
| [`render_gate`](#render_gate) | Keep background Python off the GIL while the panel swaps | No, core-internal | n/a |
| [`scroll_config`](#scroll_config) | Plugin scroll config → configured `ScrollHelper` | Yes (scrollers) | 3.4.0 | | [`scroll_config`](#scroll_config) | Plugin scroll config → configured `ScrollHelper` | Yes (scrollers) | 3.4.0 |
| [`scroll_helper`](#scroll_helper) | Pre-rendered horizontal scrolling | Yes | — | | [`scroll_helper`](#scroll_helper) | Pre-rendered horizontal scrolling | Yes | — |
| [`snapshot_policy`](#snapshot_policy) | When to write the web preview frame | No, core-internal | n/a | | [`snapshot_policy`](#snapshot_policy) | When to write the web preview frame | No, core-internal | n/a |
| [`sports_card`](#sports_card) | Scoreboard card settings, colours, fonts, dates | Yes (scoreboards) | 3.3.0 | | [`sports_card`](#sports_card) | Scoreboard card settings, colours, fonts, dates | Yes (scoreboards) | 3.3.0 |
| [`sports_card_wrappers`](#sports_card_wrappers) | The game renderer's `sports_card` delegations | Yes (scoreboards) | 3.7.0 |
| [`sports_celebration`](#sports_celebration) | Draw a scoreboard's score/win celebration | Yes (scoreboards) | 3.7.0 |
| [`sports_display_rules`](#sports_display_rules) | Which games a scoreboard shows, for how long, and its scorebug date line | Yes (scoreboards) | 3.8.0 |
| [`sports_fetch`](#sports_fetch) | Scoreboard season fetch, lookback and live-odds decisions | Yes (scoreboards) | 3.7.0 |
| [`sports_font_path`](#sports_font_path) | Find a scoreboard's bundled font whatever the cwd | Yes (scoreboards) | 3.8.0 |
| [`sports_game_renderer`](#sports_game_renderer) | Scoreboard scroll/Vegas card geometry | Yes (scoreboards) | 3.3.0 | | [`sports_game_renderer`](#sports_game_renderer) | Scoreboard scroll/Vegas card geometry | Yes (scoreboards) | 3.3.0 |
| [`sports_helpers`](#sports_helpers) | Small helpers every scoreboard `sports.py` copies | Yes (scoreboards) | 3.5.0 | | [`sports_helpers`](#sports_helpers) | Small helpers every scoreboard `sports.py` copies | Yes (scoreboards) | 3.5.0 |
| [`sports_live_scroll`](#sports_live_scroll) | Rebuild a live scroll strip mid-cycle without moving it | Yes (scoreboards) | 3.8.0 |
| [`sports_plugin_host`](#sports_plugin_host) | Helpers of a scoreboard's plugin class (`manager.py`) | Yes (scoreboards) | 3.8.0 |
| [`sports_scroll`](#sports_scroll) | Scoreboard scroll-display orchestration | Yes (scoreboards) | 3.2.0 | | [`sports_scroll`](#sports_scroll) | Scoreboard scroll-display orchestration | Yes (scoreboards) | 3.2.0 |
| [`sports_shared`](#sports_shared) | Sport-independent `sports.py` methods | Yes (scoreboards) | 3.3.0 | | [`sports_shared`](#sports_shared) | Sport-independent `sports.py` methods | Yes (scoreboards) | 3.3.0 |
| [`sports_vegas`](#sports_vegas) | Live Vegas cards: keys, card cache, sticky odds, finished games | Yes (scoreboards) | 3.8.0 |
| [`sports_timezone`](#sports_timezone) | Which timezone a scoreboard draws start times in | Yes (scoreboards) | 3.6.0 |
| [`sync_manager`](#sync_manager) | Leader/follower sync between two displays | No, core-internal | n/a | | [`sync_manager`](#sync_manager) | Leader/follower sync between two displays | No, core-internal | n/a |
| [`text_helper`](#text_helper) | Outlined text, wrapping, measurement | Yes | — | | [`text_helper`](#text_helper) | Outlined text, wrapping, measurement | Yes | — |
The four `sports_*` mixin and card modules hold code the scoreboard plugins The `sports_*` mixin and card modules hold code the scoreboard plugins
used to carry as identical copies. Each module docstring lists what a host used to carry as identical copies. Each module docstring lists what a host
class must provide. The plan behind them is in class must provide. The plan behind them is in
[docs/SPORTS_UNIFICATION.md](../../docs/SPORTS_UNIFICATION.md). [docs/SPORTS_UNIFICATION.md](../../docs/SPORTS_UNIFICATION.md).
@@ -97,6 +110,17 @@ results; `espn_date_chunks()`, `fetch_espn_date_chunks()`,
`clamp_espn_limit()` and `merge_scoreboard_payloads()` are the pieces. `clamp_espn_limit()` and `merge_scoreboard_payloads()` are the pieces.
Scoreboard plugins also bundle a copy for older cores. Scoreboard plugins also bundle a copy for older cores.
### favorite_team_check
[`favorite_team_check.py`](favorite_team_check.py).
`FavoriteTeamCheck(logger, leagues)`, where `leagues` maps a league key to
`(display name, ESPN sport/league path)`. `schedule(league_key, favorites)`
checks the configured favourite team codes against ESPN's team list once per
league, on a daemon thread, and logs a bad code with the nearest real one, or
says the league has nothing on yet; `reset()` re-arms it after a config edit.
Diagnostics only: every failure is swallowed. Scoreboard plugins also bundle
a copy for older cores.
### font_layout ### font_layout
[`font_layout.py`](font_layout.py). `load_truetype(path, size)` is [`font_layout.py`](font_layout.py). `load_truetype(path, size)` is
@@ -107,6 +131,24 @@ the size a bundled face renders on whole pixels at. `resolve_asset_path()`
resolves `assets/fonts/...` against the install root rather than the resolves `assets/fonts/...` against the install root rather than the
working directory. working directory.
### frame_timing
[`frame_timing.py`](frame_timing.py). Core-internal. `DisplayManager`
records every presented frame in a `FrameTimingRecorder`, which writes
cumulative late-frame counters and histograms to `/dev/shm` for
`scripts/frame_soak.py` and `scripts/render_bench.py`. `StallWatchdog` logs
the stack of whatever holds up a scroll. See
[docs/SCROLL_PERFORMANCE.md](../../docs/SCROLL_PERFORMANCE.md).
### json_body
[`json_body.py`](json_body.py). `response_json(response)` is
`response.json()` parsed by orjson when it is installed, falling back to the
stdlib parser (and requests' own error) otherwise. For multi-MB payloads such
as a season schedule, where the parse holds the GIL and freezes the display.
A plugin that also runs on older cores should guard the import, as
`espn_dates` does.
### logo_helper ### logo_helper
[`logo_helper.py`](logo_helper.py). `LogoHelper(display_width, [`logo_helper.py`](logo_helper.py). `LogoHelper(display_width,
@@ -134,6 +176,14 @@ let the root display service and the web user share files:
already call these; a plugin needs them only when it creates its own files already call these; a plugin needs them only when it creates its own files
outside the cache. See [docs/PERMISSIONS.md](../../docs/PERMISSIONS.md). outside the cache. See [docs/PERMISSIONS.md](../../docs/PERMISSIONS.md).
### render_gate
[`render_gate.py`](render_gate.py). Core-internal. `RenderGate` is opened by
the render thread around each vsync swap; a background thread inside
`gate.yielding()` (Vegas's prefetch) parks while the gate is closed, so the
render thread finds the GIL free when its refresh arrives. It never parks a
thread holding a guarded lock or inside logging, threading or import code.
### scroll_config ### scroll_config
[`scroll_config.py`](scroll_config.py). `configure(scroll_helper, [`scroll_config.py`](scroll_config.py). `configure(scroll_helper,
@@ -174,6 +224,50 @@ dates (`format_game_date()`, `format_game_time()`, `card_tzinfo()`) and font
sizes (`schema_font_size()`, `resolve_font_size()`). A plugin keeps its own sizes (`schema_font_size()`, `resolve_font_size()`). A plugin keeps its own
method and delegates the body. method and delegates the body.
### sports_card_wrappers
[`sports_card_wrappers.py`](sports_card_wrappers.py).
`SportsCardWrappersMixin`: the one-line methods a scoreboard's game renderer
uses to call `sports_card` with its own `config` and `logger`
(`_vs_text()`, `_element_color()`, `_format_game_date()`, ... seventeen in
all), under their existing names. They are what `sports_game_renderer`'s
mixin expects its host to provide. No `__init__` and no state.
### sports_celebration
[`sports_celebration.py`](sports_celebration.py). `SportsCelebrationMixin`
draws the full-screen takeover a scoreboard shows when a team scores or wins
(`_draw_celebration_layout(celebration)`): a backdrop in the scoring team's
colours read off its crest, scenery, confetti, the headline and the score.
The colour helpers are free functions (`logo_palette()`, `lift_color()`,
`mix_color()`, ...). Deciding *when* to celebrate stays in the plugin, which
builds the celebration dict the docstring describes.
### sports_display_rules
[`sports_display_rules.py`](sports_display_rules.py). Two `SportsCore`
mixins: `SportsCardOptionsMixin` (`_card_option()`, which never lets the
upcoming scorebug lose both its date and time, and `_recent_date_text()`;
list it before `SportsCoreSharedMixin`) and `SportsGameRulesMixin`
(`_filtered_or_all()`, the no-favourites quality filter that fails open, and
`_effective_live_duration()`, the shorter dwell for a non-favourite live
game).
### sports_fetch
[`sports_fetch.py`](sports_fetch.py). `SportsFetchMixin`: the `SportsCore`
methods that decide which requests a scoreboard makes --
`_fetch_season_directly()` (a season, in chunks ESPN accepts),
`_background_fetches_espn_ranges()`, `_needs_previous_day()` (the live
lookback) and `_wants_live_odds()` (odds only for games near the screen).
### sports_font_path
[`sports_font_path.py`](sports_font_path.py). `resolve_font_path(path)`: the
path as given when it exists (relative to the cwd), else
`font_layout.resolve_asset_path(path)`. What the scoreboards'
`_resolve_font_path` copies return on a core that ships it.
### sports_game_renderer ### sports_game_renderer
[`sports_game_renderer.py`](sports_game_renderer.py). [`sports_game_renderer.py`](sports_game_renderer.py).
@@ -191,6 +285,25 @@ what differs.
`_odds_color` and `_upcoming_date_and_time_text` under their existing names. `_odds_color` and `_upcoming_date_and_time_text` under their existing names.
Nothing in core uses it. Nothing in core uses it.
### sports_live_scroll
[`sports_live_scroll.py`](sports_live_scroll.py). `SportsLiveScrollMixin`:
keeps a live scroll strip current. It fingerprints the live games (the clock
and the display pipeline's own keys excluded, via the host's
`LIVE_VOLATILE_FIELDS`), rebuilds when they change, rate-limited by what a
rebuild costs, and `_preserving_scroll_position()` keeps the marquee where
it was. Pairs with `SportsPluginHostMixin`, whose `_dispatch_switch_refresh()`
it uses.
### sports_plugin_host
[`sports_plugin_host.py`](sports_plugin_host.py). `SportsPluginHostMixin`:
helpers of a scoreboard's `BasePlugin` subclass. `get_vegas_priority_weight()`
(more Vegas slots while a favourite plays, found across every plugin's data
shape), `_dispatch_switch_refresh()` (a manager refresh on a daemon thread, so
`display()` never waits on the network), `get_vegas_content_type()` and small
dynamic-duration helpers. List it before `BasePlugin`.
### sports_scroll ### sports_scroll
[`sports_scroll.py`](sports_scroll.py). `SportsScrollDisplay` and [`sports_scroll.py`](sports_scroll.py). `SportsScrollDisplay` and
@@ -208,6 +321,29 @@ fonts, colours, dates, the switch-mode upcoming card). The docstring lists
the attributes the host class must have and the three methods deliberately the attributes the host class must have and the three methods deliberately
left out. left out.
### sports_vegas
[`sports_vegas.py`](sports_vegas.py). What a scoreboard needs for live Vegas
cards (one element per game, swapped in place while it scrolls):
`game_key()`, `game_fingerprint()`, `dedupe_games()`, `VegasCardCache` (draws
a card only when its fingerprint changes), `StickyOdds` (keeps a card's odds
through a live poll that left them out), and `finished_games()` /
`with_finished_games()` (a game that just went final keeps its card, showing
FINAL). `SportsScrollDisplay.build_vegas_elements()` in `sports_scroll` puts
them together; a scoreboard not built on it (UFC) uses them directly.
### sports_timezone
[`sports_timezone.py`](sports_timezone.py).
`resolve_timezone_name(config, plugin_manager, cache_manager, log, *,
plugin_label, writeback_fixed_in=None)` and `resolve_timezone(...)` (the same
as a pytz zone): the plugin's own `timezone`, then the global one via either
manager's `config_manager`, then the host's zone (`system_timezone_name()`),
then UTC. `plugin_label` names the plugin in the warning logged when nothing
resolves; `writeback_fixed_in` is for a plugin that once wrote `"UTC"` into
the saved config (a bare plugin-level `"UTC"` is then ignored when another
source disagrees). Scoreboard plugins also bundle a copy for older cores.
### sync_manager ### sync_manager
[`sync_manager.py`](sync_manager.py). Core-internal. `DisplaySyncManager` [`sync_manager.py`](sync_manager.py). Core-internal. `DisplaySyncManager`
+27 -15
View File
@@ -11,12 +11,16 @@ import time
from datetime import datetime from datetime import datetime
from types import MappingProxyType from types import MappingProxyType
from src.common.espn_dates import ESPN_MAX_LIMIT from src.common.espn_dates import ESPN_MAX_LIMIT
from typing import Any, Dict, Mapping, Optional from typing import TYPE_CHECKING, Any, Dict, Mapping, Optional, cast
import requests import requests
from requests.adapters import HTTPAdapter from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry from urllib3.util.retry import Retry
if TYPE_CHECKING:
# What Session() puts in .headers; the stubs only promise a MutableMapping.
from requests.structures import CaseInsensitiveDict
#: The User-Agent core sends to ESPN and other data APIs. It names the client #: The User-Agent core sends to ESPN and other data APIs. It names the client
#: and links to it: around 2026-08-04 ESPN began 403ing bare custom tokens #: and links to it: around 2026-08-04 ESPN began 403ing bare custom tokens
@@ -84,7 +88,11 @@ class APIHelper:
self.session.headers.update({**DEFAULT_HTTP_HEADERS, 'Connection': 'keep-alive'}) self.session.headers.update({**DEFAULT_HTTP_HEADERS, 'Connection': 'keep-alive'})
# Rate limiting # Rate limiting
self._last_request_time = 0 self._last_request_time: float = 0 # wall clock, reported by get_request_stats()
# The interval is measured on time.monotonic(): a wall-clock step
# back (NTP correcting a Pi with no RTC) made time_since_last
# negative and the "remaining interval" sleep as long as the step.
self._last_request_monotonic: Optional[float] = None
self._min_request_interval = 1.0 # Minimum seconds between requests self._min_request_interval = 1.0 # Minimum seconds between requests
def get(self, url: str, params: Optional[Dict] = None, def get(self, url: str, params: Optional[Dict] = None,
@@ -108,14 +116,14 @@ class APIHelper:
cached = self._get_from_cache(cache_key, cache_ttl) cached = self._get_from_cache(cache_key, cache_ttl)
if cached is not None: if cached is not None:
self.logger.debug(f"Using cached response for {cache_key}") self.logger.debug(f"Using cached response for {cache_key}")
return cached return cast(Dict[Any, Any], cached)
# Rate limiting # Rate limiting
self._enforce_rate_limit() self._enforce_rate_limit()
try: try:
# Prepare request # Prepare request
request_headers = self.session.headers.copy() request_headers = cast('CaseInsensitiveDict[Any]', self.session.headers).copy()
if headers: if headers:
request_headers.update(headers) request_headers.update(headers)
@@ -129,7 +137,7 @@ class APIHelper:
response.raise_for_status() response.raise_for_status()
# Parse JSON response # Parse JSON response
data = response.json() data: Dict[Any, Any] = response.json()
# Cache response if cache key provided # Cache response if cache key provided
if cache_key and self.cache_manager: if cache_key and self.cache_manager:
@@ -243,7 +251,7 @@ class APIHelper:
self._enforce_rate_limit() self._enforce_rate_limit()
try: try:
request_headers = self.session.headers.copy() request_headers = cast('CaseInsensitiveDict[Any]', self.session.headers).copy()
if headers: if headers:
request_headers.update(headers) request_headers.update(headers)
@@ -256,7 +264,7 @@ class APIHelper:
) )
response.raise_for_status() response.raise_for_status()
return response.json() return cast(Optional[Dict[Any, Any]], response.json())
except requests.exceptions.RequestException as e: except requests.exceptions.RequestException as e:
self.logger.error(f"POST request failed for {url}: {e}") self.logger.error(f"POST request failed for {url}: {e}")
@@ -333,13 +341,14 @@ class APIHelper:
def _enforce_rate_limit(self) -> None: def _enforce_rate_limit(self) -> None:
"""Enforce rate limiting between requests.""" """Enforce rate limiting between requests."""
current_time = time.time() if self._last_request_monotonic is not None:
time_since_last = current_time - self._last_request_time time_since_last = time.monotonic() - self._last_request_monotonic
if time_since_last < self._min_request_interval: if time_since_last < self._min_request_interval:
sleep_time = self._min_request_interval - time_since_last sleep_time = self._min_request_interval - time_since_last
time.sleep(sleep_time) time.sleep(sleep_time)
self._last_request_monotonic = time.monotonic()
self._last_request_time = time.time() self._last_request_time = time.time()
def set_rate_limit(self, min_interval: float) -> None: def set_rate_limit(self, min_interval: float) -> None:
@@ -362,5 +371,8 @@ class APIHelper:
return { return {
'min_request_interval': self._min_request_interval, 'min_request_interval': self._min_request_interval,
'last_request_time': self._last_request_time, 'last_request_time': self._last_request_time,
'time_since_last_request': time.time() - self._last_request_time 'time_since_last_request': (
time.monotonic() - self._last_request_monotonic
if self._last_request_monotonic is not None
else time.time() - self._last_request_time),
} }
+4 -4
View File
@@ -37,7 +37,7 @@ import time
from concurrent.futures import ThreadPoolExecutor from concurrent.futures import ThreadPoolExecutor
from datetime import date, timedelta from datetime import date, timedelta
from functools import partial from functools import partial
from typing import Any, Dict, List, Optional, Tuple from typing import Any, Dict, List, Optional, Tuple, cast
try: try:
from src.common.json_body import response_json from src.common.json_body import response_json
@@ -159,7 +159,7 @@ def espn_date_chunks(start: date, end: date) -> List[str]:
return chunks return chunks
def merge_scoreboard_payloads(payloads: List[Dict[str, Any]]) -> Dict[str, Any]: def merge_scoreboard_payloads(payloads: List[Any]) -> Dict[str, Any]:
"""Fold chunk responses into one scoreboard payload. """Fold chunk responses into one scoreboard payload.
Events are de-duplicated by id and keep first-seen order. Non-event keys Events are de-duplicated by id and keep first-seen order. Non-event keys
@@ -202,7 +202,7 @@ def _fetch_one_chunk(
timeout=timeout, timeout=timeout,
) )
response.raise_for_status() response.raise_for_status()
return response_json(response) return cast(Optional[Dict[str, Any]], response_json(response))
except Exception as exc: # noqa: BLE001 - see docstring except Exception as exc: # noqa: BLE001 - see docstring
if logger: if logger:
logger.warning("ESPN chunk %s failed, skipping it: %s", chunk, exc) logger.warning("ESPN chunk %s failed, skipping it: %s", chunk, exc)
@@ -379,4 +379,4 @@ def fetch_espn_scoreboard(
if data is not None: if data is not None:
return data return data
response.raise_for_status() response.raise_for_status()
return response_json(response) return cast(Dict[str, Any], response_json(response))
+409
View File
@@ -0,0 +1,409 @@
"""
Explain an empty screen: a wrong team code, or a season that has not started.
Favourite teams are matched by exact ESPN abbreviation, so a plausible-looking
code silently matches nothing and the plugin shows an empty screen with no hint
that the code is at fault. The codes are not always guessable — ESPN calls
Alabama ``ALA`` rather than ``BAMA``, and Golden State ``GS`` rather than
``GSW``. Between seasons a perfectly correct code produces the same empty
screen for a completely different reason, and the two were indistinguishable
from the logs.
This module is diagnostics only. It runs on a daemon thread, once per league per
process, and every failure is swallowed: it must never delay a frame or change
what is displayed.
"""
import difflib
import logging
import re
import threading
from datetime import datetime, timezone
from typing import Dict, Iterable, List, Optional, Set, Tuple
TEAMS_URL = "https://site.api.espn.com/apis/site/v2/sports/{path}/teams?limit=1000"
SCOREBOARD_URL = "https://site.api.espn.com/apis/site/v2/sports/{path}/scoreboard"
REQUEST_TIMEOUT = 15
class FavoriteTeamCheck:
"""
Validates configured favourite team codes against ESPN, and says so in the log.
``leagues`` maps the plugin's own league key to a
``(human readable name, ESPN sport/league path)`` pair, e.g.
``{'nhl': ('NHL', 'hockey/nhl')}``.
"""
# How far out the next fixture has to be before it is worth mentioning.
# An off day or two is normal mid-season and saying so would just be noise.
GAP_DAYS = 3
def __init__(self, logger: Optional[logging.Logger],
leagues: Dict[str, Tuple[str, str]]) -> None:
self.logger = logger or logging.getLogger(__name__)
self.leagues = leagues
self._checked: Set[str] = set()
self._lock = threading.Lock()
def reset(self) -> None:
"""Re-check on the next call, e.g. after the user edits the config."""
with self._lock:
self._checked.clear()
def schedule(self, league_key: str, favorites: Iterable[str]) -> None:
"""Check one league in the background, at most once per process."""
try:
favorites = [str(f) for f in (favorites or []) if str(f).strip()]
if not favorites or league_key not in self.leagues:
return
with self._lock:
if league_key in self._checked:
return
self._checked.add(league_key)
threading.Thread(
target=self._run, args=(league_key, favorites),
name="favorite-team-check", daemon=True,
).start()
except Exception:
pass # nosec B110 - a diagnostic must never be the reason an update fails # nosemgrep
def _run(self, league_key: str, favorites) -> None:
try:
self._check(league_key, favorites)
except Exception as exc:
self.logger.debug("Favorite team check failed for %s: %s",
league_key, exc)
def _check(self, league_key: str, favorites) -> None:
name, path = self.leagues[league_key]
try:
teams = self._fetch_teams(path)
except Exception as exc:
self.logger.debug("Could not verify %s favorite teams: %s", name, exc)
return
if not teams:
# Some ESPN endpoints (college lacrosse) return no teams at all.
# Nothing can be concluded, so say nothing.
return
# Dynamic groups like AP_TOP_25 are expanded elsewhere; they are not
# team codes and must not be reported as bad ones.
codes = [f for f in favorites if not self._is_dynamic(f)]
recognised = [f for f in codes if f in teams]
unknown = [f for f in codes if f not in teams]
for code in unknown:
self.logger.warning(
"%s favorite team %r is not a %s team code.%s "
"Every code this league accepts is listed at %s.",
name, code, name, self._suggest(code, teams),
TEAMS_URL.format(path=path),
)
if codes and not recognised:
self.logger.warning(
"%s has no recognised favorite teams, so nothing will be shown "
"for it. Codes must be ESPN abbreviations, e.g. %s.",
name, ", ".join("{} ({})".format(a, n)
for a, n in list(sorted(teams.items()))[:3]),
)
return
if not recognised:
return
# Codes are fine, so check the other cause of an empty screen.
try:
note = self._schedule_note(path)
except Exception as exc:
self.logger.debug("Could not check the %s schedule: %s", name, exc)
return
if note:
self.logger.info(
"%s favorite teams %s look correct, but %s. An empty display "
"until then is expected, not a configuration problem.",
name, ", ".join(recognised), note,
)
else:
self.logger.info("%s favorite teams recognised: %s",
name, ", ".join(recognised))
@staticmethod
def _is_dynamic(code: str) -> bool:
upper = (code or "").strip().upper()
return upper.startswith("AP_") or upper.startswith("TOP_") or "TOP_" in upper
@staticmethod
def _fetch_teams(path: str) -> Dict[str, str]:
"""ESPN's {abbreviation: display name} for a league.
``limit=1000`` is required: the default page size truncates the NCAA
responses to roughly half their teams, which makes valid codes look wrong.
"""
import requests
payload = requests.get(TEAMS_URL.format(path=path),
timeout=REQUEST_TIMEOUT).json()
entries = payload['sports'][0]['leagues'][0]['teams']
return {
t['team']['abbreviation']: t['team']['displayName']
for t in entries if t.get('team', {}).get('abbreviation')
}
@classmethod
def _schedule_note(cls, path: str) -> Optional[str]:
"""
Why the league has nothing to show, as a clause, or ``None`` if it does.
Two things make this harder than reading ``events``:
* An out-of-season league does not come back empty. ESPN rolls the
scoreboard forward to the next day that has fixtures, so in July the
NHL endpoint returns seven September games. Emptiness cannot be the
signal; the date of those games is, and it is more useful anyway.
* A *finished* season rolls nowhere and returns its last game instead,
months in the past — so dates have to be filtered to the future
before the soonest one means anything.
"""
import requests
payload = requests.get(SCOREBOARD_URL.format(path=path),
timeout=REQUEST_TIMEOUT).json()
event_dates = [cls._parse_date(e.get('date'))
for e in payload.get('events') or []]
calendar_dates = []
for entry in (payload.get('leagues') or [{}])[0].get('calendar') or []:
calendar_dates.append(cls._parse_date(
entry if isinstance(entry, str) else entry.get('startDate')))
# Count the last day as current, rather than filtering on "later than
# right now": a game that began a few hours ago still means the league
# has something on, and dropping it would report a live slate as a
# finished season. A day's grace also keeps this correct whatever the
# user's timezone, since these timestamps are UTC.
now = datetime.now(timezone.utc)
def future(candidates):
return sorted(d for d in candidates if d and (now - d).days < 1)
# Events are fixtures; the calendar is week and phase boundaries,
# which routinely open days before their first game (an NFL week 1
# calendar entry starts the weekend before the Thursday opener).
# Reading the two together reported the earliest boundary as a game
# date -- "nothing on until 06 September" for a league whose first
# snap is the 10th. The calendar only gets a say when the scoreboard
# has no events at all to roll forward to: events that exist but are
# all in the past mean the season is over, and an offseason calendar
# phase must not be dressed up as its next game.
#
# The exception is a calendar of match days. With calendarType "day"
# and calendarIsWhitelist true, every entry is a day that has games,
# so a future entry is a real next fixture. Soccer needs it: between
# matchdays the scoreboard keeps showing the last one, so on
# 2026-09-29 every Premier League event was from 20 September and the
# next games (10 October) were only in the calendar. A day calendar
# that is not a whitelist (MLB's) lists days *without* games.
if any(event_dates):
upcoming = future(event_dates)
if not upcoming and cls._calendar_is_match_days(payload):
upcoming = future(calendar_dates)
else:
upcoming = future(calendar_dates)
if not upcoming:
if not any(event_dates) and not any(calendar_dates):
return None # Nothing published either way; draw no conclusion.
if cls._moved_to_later_phase(payload):
return None # e.g. postseason under way; see the method.
if cls._later_round_scheduled(payload, now):
return None # e.g. Europa League between matchdays.
return ("the season has finished and the next one's fixtures are "
"not published yet")
# A day or two out is just an off day, and saying so would be noise.
if (upcoming[0] - now).days < cls.GAP_DAYS:
return None
return "the league has nothing on until {}".format(
upcoming[0].strftime('%d %B %Y'))
@staticmethod
def _calendar_is_match_days(payload) -> bool:
"""Whether the league calendar lists the days that have games."""
league = (payload.get('leagues') or [{}])[0] or {}
return (league.get('calendarType') == 'day'
and league.get('calendarIsWhitelist') is True)
@staticmethod
def _moved_to_later_phase(payload) -> bool:
"""
Whether the league is in a later in-season phase than its events.
ESPN does not roll the scoreboard forward into a postseason. The day
after MLB's regular season ended, the default scoreboard still returned
that last regular-season day, while ``leagues[0].season`` already said
Postseason and the wild-card games were two days out. Past events alone
then read as a finished season while the same process's upcoming
manager was showing the favourite's playoff games.
Only regular season (2) and postseason (3) count as "later". The
offseason (4) follows the postseason too, and there past events really
do mean the season is over.
"""
season = ((payload.get('leagues') or [{}])[0] or {}).get('season') or {}
league_type = (season.get('type') or {}).get('type')
if league_type not in (2, 3):
return False
event_types = [(e.get('season') or {}).get('type')
for e in payload.get('events') or []]
known = [t for t in event_types if isinstance(t, int)]
return bool(known) and all(t < league_type for t in known)
@classmethod
def _later_round_scheduled(cls, payload, now: datetime) -> bool:
"""
Whether a "list" calendar has a round that has not started yet.
Competitions with a list calendar (the UEFA club competitions, the
World Cup, AFL, NFL) give each phase its rounds as ``entries`` with
start and end dates. Between matchdays the Europa League scoreboard
keeps showing the last one: on 2026-09-29 every event was from 17
September, the next matchday was only days away, and the rounds from
the knockout play-offs to the final were all still to come. A round
that starts later means the season is not over, even though the
date of the next fixture is not known.
Only a round's *start* counts. End dates are padded well past the
last game -- the World Cup's final round ran to 1 August for a 19 July
final -- so a future end date is also true of a finished season.
Rounds in an offseason phase (the college football All-Star week)
are not games for the favourites and do not count either.
"""
league = (payload.get('leagues') or [{}])[0] or {}
for phase in league.get('calendar') or []:
if not isinstance(phase, dict) or cls._is_offseason(phase.get('label')):
continue
for entry in phase.get('entries') or []:
if not isinstance(entry, dict) or cls._is_offseason(entry.get('label')):
continue
start = cls._parse_date(entry.get('startDate'))
if start and start > now:
return True
return False
@staticmethod
def _is_offseason(label) -> bool:
"""'Off Season', 'Offseason', 'Off-season' ..."""
return isinstance(label, str) and 'offseason' in re.sub(
r'[^a-z]', '', label.lower())
@staticmethod
def _parse_date(raw) -> Optional[datetime]:
if not raw or not isinstance(raw, str):
return None
try:
return datetime.fromisoformat(raw.replace('Z', '+00:00'))
except ValueError:
return None
@classmethod
def _suggest(cls, code: str, teams: Dict[str, str]) -> str:
"""Nearest matching code for a typo, as a ready-to-log clause."""
upper = (code or "").strip().upper()
if not upper or code in teams:
return ""
# Right code, wrong case — matching is case-sensitive. Guard on the case
# actually differing, so a valid code never draws this message.
for abbr in teams:
if abbr.upper() == upper:
return " Codes are case-sensitive; use {!r} ({}).".format(
abbr, teams[abbr])
ranked = cls._rank(upper, (a for a, n in teams.items()
if cls._abbreviates(upper, n)), teams)
if not ranked:
# Nicknames are often a fragment of a word rather than its initials:
# 'BAMA' sits inside 'Alabama' but abbreviates nothing in it. Require
# three characters, since shorter fragments match far too much.
if len(upper) >= 3:
ranked = cls._rank(
upper,
(a for a, n in teams.items()
if any(upper in w for w in cls._words(n))),
teams)
if len(ranked) == 1:
return " Closest match is {!r} ({}).".format(
ranked[0], teams[ranked[0]])
if ranked:
return " Did you mean {}?".format(", ".join(
"{!r} ({})".format(a, teams[a]) for a in ranked[:3]))
# Otherwise fall back to similarity, against names before codes: a name
# gives more characters to compare and so produces fewer ties.
names = {n.upper(): a for a, n in teams.items()}
hits = difflib.get_close_matches(upper, list(names), n=1, cutoff=0.6)
if hits:
abbr = names[hits[0]]
return " Closest match is {!r} ({}).".format(abbr, teams[abbr])
code_hits = difflib.get_close_matches(upper, list(teams), n=1, cutoff=0.6)
if code_hits:
return " Closest match is {!r} ({}).".format(
code_hits[0], teams[code_hits[0]])
return ""
@staticmethod
def _words(name: str):
return [w for w in re.split(r'[^A-Za-z0-9]+', (name or '').upper()) if w]
@classmethod
def _rank(cls, code: str, candidates, teams: Dict[str, str]):
"""
Order candidate codes best-first.
A code that picks up the *first* word of the name wins, because that is
how people shorten team names: 'SCAR' for South Carolina starts at
'South', whereas for Rutgers Scarlet Knights it starts mid-name. Without
this the tie is broken alphabetically and the obvious answer can land
third in the list.
"""
def key(abbr):
words = cls._words(teams.get(abbr, ''))
first_word_hit = bool(words) and words[0].startswith(code[:1])
return (not first_word_hit, len(abbr), abbr)
return sorted(set(candidates), key=key)
@staticmethod
def _abbreviates(code: str, name: str) -> bool:
"""
Whether ``code`` reads as an abbreviation of ``name``.
Each part of the code must be a prefix of one of the name's words, taken
in order — which is how people actually shorten team names. Plain string
similarity is no use for three-letter codes: 'MUN' scores identically
against 'MAN' and 'SUN', so Manchester United and Sunderland tie and the
suggestion is a coin flip. This rule separates them, because 'MUN'
splits as M-anchester UN-ited while Sunderland has no word starting M.
"""
words = [w for w in re.split(r'[^A-Za-z0-9]+', (name or '').upper()) if w]
def consume(rest: str, remaining: List[str]) -> bool:
if not rest:
return True
if not remaining:
return False
head, tail = remaining[0], remaining[1:]
# Skip this word entirely, as in "Manchester United" -> "UTD".
if consume(rest, tail):
return True
for size in range(1, min(len(rest), len(head)) + 1):
if head.startswith(rest[:size]) and consume(rest[size:], tail):
return True
return False
return consume((code or "").strip().upper(), words)
+77 -14
View File
@@ -66,6 +66,17 @@ faster than the panel (every frame early) or sits at half its rate (every
frame late), both of which look self-consistent to an estimate taken from frame late), both of which look self-consistent to an estimate taken from
their own intervals. their own intervals.
Operations
----------
The late count says how often, not which work did it. Render-thread work that
happens between two frames -- extending the Vegas strip, patching a live
element into it -- calls :meth:`FrameTimingRecorder.note_op` first, and the
next presented frame carries the tag: the interval that frame ends is the one
the work landed in. ``op_frames`` counts timed frames per kind,
``late_op_frames`` the late ones among them, ``op_freezes`` those that were a
freeze instead, and ``op_bytes`` what the work moved. A kind whose late rate
sits well above the overall one is the work to look at.
Stall watchdog Stall watchdog
-------------- --------------
Counting a freeze says that it happened, not why. ``StallWatchdog`` watches the Counting a freeze says that it happened, not why. ``StallWatchdog`` watches the
@@ -93,7 +104,7 @@ import tempfile
import threading import threading
import time import time
import traceback import traceback
from typing import Any, Callable, Dict, List, Optional, Tuple from typing import Any, Callable, Dict, List, Optional, Tuple, TypedDict
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -153,11 +164,20 @@ def default_stats_path() -> str:
return os.path.join(base, STATS_FILENAME) return os.path.join(base, STATS_FILENAME)
#: One presented frame's interval: (interval, blit, wait, hold, ops), where
#: ops is the work noted before it (kind -> bytes) or None.
_Frame = Tuple[float, float, float, int, Optional[Dict[str, int]]]
def _bucket(seconds: float) -> int: def _bucket(seconds: float) -> int:
index = int(seconds * 1000.0 / BUCKET_MS) index = int(seconds * 1000.0 / BUCKET_MS)
return min(max(index, 0), BUCKET_COUNT - 1) return min(max(index, 0), BUCKET_COUNT - 1)
def _bump(counter: Dict[str, int], key: str, by: int = 1) -> None:
counter[key] = counter.get(key, 0) + by
def binding_releases_gil() -> Optional[bool]: def binding_releases_gil() -> Optional[bool]:
"""Whether the loaded rgbmatrix binding releases the GIL, or None. """Whether the loaded rgbmatrix binding releases the GIL, or None.
@@ -250,12 +270,14 @@ class FrameTimingRecorder:
self.info = dict(info or {}) self.info = dict(info or {})
# Render-thread state. # Render-thread state.
self._pending: List[Tuple[float, float, float, int]] = [] self._pending: List[_Frame] = []
self._static_frames = 0 self._static_frames = 0
self._previous: Optional[Tuple[float, bool, int]] = None self._previous: Optional[Tuple[float, bool, int]] = None
# The interval ended by a static frame that followed a scrolling one, # The interval ended by a static frame that followed a scrolling one,
# until the next frame shows whether the scroll went on. # until the next frame shows whether the scroll went on.
self._unsure: Optional[Tuple[float, float, float, int]] = None self._unsure: Optional[_Frame] = None
# Work noted since the last frame (kind -> bytes), for the next one.
self._ops: Optional[Dict[str, int]] = None
self._last_flush: Optional[float] = None self._last_flush: Optional[float] = None
self._queue: "queue.SimpleQueue" = queue.SimpleQueue() self._queue: "queue.SimpleQueue" = queue.SimpleQueue()
self._worker: Optional[threading.Thread] = None self._worker: Optional[threading.Thread] = None
@@ -281,6 +303,11 @@ class FrameTimingRecorder:
"freeze_seconds": 0.0, "freeze_seconds": 0.0,
"freeze_by": {label: 0 for _, label in FREEZE_BUCKETS}, "freeze_by": {label: 0 for _, label in FREEZE_BUCKETS},
"worst_interval_ms": 0.0, "worst_interval_ms": 0.0,
# Per kind of noted render-thread work; see "Operations".
"op_frames": {},
"late_op_frames": {},
"op_freezes": {},
"op_bytes": {},
} }
self.histograms: Dict[str, Dict[int, int]] = { self.histograms: Dict[str, Dict[int, int]] = {
"blit": {}, "wait": {}, "work": {}, "interval_per_hold": {}, "blit": {}, "wait": {}, "work": {}, "interval_per_hold": {},
@@ -305,6 +332,21 @@ class FrameTimingRecorder:
# -- render thread ------------------------------------------------------ # -- render thread ------------------------------------------------------
def note_op(self, kind: str, nbytes: int = 0) -> None:
"""Tag the next presented frame with work done before it.
Render thread only, like :meth:`record`, which consumes the tag: the
interval the next frame ends is the one this work landed in. Several
notes before one frame accumulate, per kind. See "Operations".
:param kind: a short name for the work, e.g. ``"extend"``, ``"patch"``.
:param nbytes: how much the work moved, summed into ``op_bytes``.
"""
ops = self._ops
if ops is None:
ops = self._ops = {}
ops[kind] = ops.get(kind, 0) + int(nbytes)
def record(self, blit: float, wait: float, hold: int, scrolling: bool, def record(self, blit: float, wait: float, hold: int, scrolling: bool,
presented_at: float) -> None: presented_at: float) -> None:
"""One frame reached the panel. """One frame reached the panel.
@@ -318,6 +360,7 @@ class FrameTimingRecorder:
previous = self._previous previous = self._previous
self._previous = (presented_at, scrolling, hold) self._previous = (presented_at, scrolling, hold)
self.last_frame = (presented_at, scrolling, threading.get_ident()) self.last_frame = (presented_at, scrolling, threading.get_ident())
ops, self._ops = self._ops, None
if not scrolling: if not scrolling:
self._static_frames += 1 self._static_frames += 1
# The scroll ended, or its state went missing for this frame: the # The scroll ended, or its state went missing for this frame: the
@@ -325,7 +368,8 @@ class FrameTimingRecorder:
# state, so the interval is due at the scroll's own. # state, so the interval is due at the scroll's own.
self._unsure = None self._unsure = None
if previous is not None and previous[1]: if previous is not None and previous[1]:
self._unsure = (presented_at - previous[0], blit, wait, previous[2]) self._unsure = (presented_at - previous[0], blit, wait,
previous[2], ops)
elif self.watchdog is None and self.scrolling_now is not None \ elif self.watchdog is None and self.scrolling_now is not None \
and os.environ.get("LEDMATRIX_STALL_WATCHDOG", "1") != "0": and os.environ.get("LEDMATRIX_STALL_WATCHDOG", "1") != "0":
self.watchdog = StallWatchdog(self, **watchdog_settings()) self.watchdog = StallWatchdog(self, **watchdog_settings())
@@ -335,14 +379,14 @@ class FrameTimingRecorder:
unsure, self._unsure = self._unsure, None unsure, self._unsure = self._unsure, None
if previous[1]: if previous[1]:
if interval < GAP_SECONDS: if interval < GAP_SECONDS:
self._pending.append((interval, blit, wait, hold)) self._pending.append((interval, blit, wait, hold, ops))
elif unsure is not None and interval < RESUME_SECONDS: elif unsure is not None and interval < RESUME_SECONDS:
# One static frame between two scrolling ones: the scroll never # One static frame between two scrolling ones: the scroll never
# stopped, only its state did. Both intervals were motion. # stopped, only its state did. Both intervals were motion.
self._static_frames -= 1 self._static_frames -= 1
if unsure[0] < GAP_SECONDS: if unsure[0] < GAP_SECONDS:
self._pending.append(unsure) self._pending.append(unsure)
self._pending.append((interval, blit, wait, hold)) self._pending.append((interval, blit, wait, hold, ops))
if self._last_flush is None: if self._last_flush is None:
self._last_flush = presented_at self._last_flush = presented_at
@@ -382,15 +426,17 @@ class FrameTimingRecorder:
except Exception: # never let telemetry take anything down except Exception: # never let telemetry take anything down
logger.debug("Frame timing flush failed", exc_info=True) logger.debug("Frame timing flush failed", exc_info=True)
def aggregate(self, batch: List[Tuple[float, float, float, int]], def aggregate(self, batch: List[_Frame], static: int) -> None:
static: int) -> None: """Fold one window of frames into the running totals.
"""Fold one window of frames into the running totals."""
Each frame is ``(interval, blit, wait, hold, ops)``; ``ops`` (the work
noted before it, or None) may be left off.
"""
totals = self.totals totals = self.totals
totals["static_frames"] += static totals["static_frames"] += static
per_hold = sorted(interval / max(1, hold) per_hold = sorted(frame[0] / max(1, frame[3]) for frame in batch
for interval, _, _, hold in batch if frame[0] < FREEZE_SECONDS)
if interval < FREEZE_SECONDS)
if len(per_hold) >= MIN_FRAMES_FOR_REFRESH: if len(per_hold) >= MIN_FRAMES_FOR_REFRESH:
estimate = per_hold[len(per_hold) // 10] estimate = per_hold[len(per_hold) // 10]
current = self.refresh_period current = self.refresh_period
@@ -412,15 +458,22 @@ class FrameTimingRecorder:
period = self.refresh_period period = self.refresh_period
histograms = self.histograms histograms = self.histograms
for interval, blit, wait, hold in batch: for frame in batch:
interval, blit, wait, hold = frame[:4]
ops = frame[4] if len(frame) > 4 else None
totals["worst_interval_ms"] = max(totals["worst_interval_ms"], totals["worst_interval_ms"] = max(totals["worst_interval_ms"],
interval * 1000.0) interval * 1000.0)
if ops:
for kind, nbytes in ops.items():
_bump(totals["op_bytes"], kind, nbytes)
if interval >= FREEZE_SECONDS: if interval >= FREEZE_SECONDS:
totals["freezes"] += 1 totals["freezes"] += 1
totals["freeze_seconds"] += interval totals["freeze_seconds"] += interval
label = next(name for limit, name in FREEZE_BUCKETS label = next(name for limit, name in FREEZE_BUCKETS
if interval < limit) if interval < limit)
totals["freeze_by"][label] += 1 totals["freeze_by"][label] += 1
for kind in ops or ():
_bump(totals["op_freezes"], kind)
continue continue
totals["scroll_frames"] += 1 totals["scroll_frames"] += 1
for name, value in (("blit", blit), ("wait", wait), for name, value in (("blit", blit), ("wait", wait),
@@ -432,6 +485,10 @@ class FrameTimingRecorder:
if period: if period:
totals["timed_frames"] += 1 totals["timed_frames"] += 1
missed = round(interval / period) - hold missed = round(interval / period) - hold
for kind in ops or ():
_bump(totals["op_frames"], kind)
if missed >= 1:
_bump(totals["late_op_frames"], kind)
if missed >= 1: if missed >= 1:
totals["late_frames"] += 1 totals["late_frames"] += 1
totals["missed_refreshes"] += missed totals["missed_refreshes"] += missed
@@ -483,7 +540,13 @@ class FrameTimingRecorder:
raise raise
def watchdog_settings() -> Dict[str, float]: class _WatchdogSettings(TypedDict, total=False):
"""The StallWatchdog keyword arguments watchdog_settings() may set."""
threshold: float
poll: float
def watchdog_settings() -> _WatchdogSettings:
"""StallWatchdog arguments from ``LEDMATRIX_STALL_WATCHDOG_MS``, if set. """StallWatchdog arguments from ``LEDMATRIX_STALL_WATCHDOG_MS``, if set.
The poll comes down with the threshold, or a stall shorter than one poll The poll comes down with the threshold, or a stall shorter than one poll
+42 -17
View File
@@ -8,7 +8,7 @@ Extracted from LEDMatrix core to provide reusable functionality for plugins.
import logging import logging
import time import time
from pathlib import Path from pathlib import Path
from typing import Dict, List, Optional, Union from typing import Dict, List, Optional, Tuple, Union
import requests import requests
from PIL import Image, ImageDraw from PIL import Image, ImageDraw
@@ -80,6 +80,11 @@ class LogoHelper:
# Time-bounded rather than permanent so a logo that appears later (the # Time-bounded rather than permanent so a logo that appears later (the
# downloader writes them at runtime) is still picked up. # downloader writes them at runtime) is still picked up.
self._missing_logos: Dict[str, float] = {} self._missing_logos: Dict[str, float] = {}
# Failed downloads by logo path. A logo that is absent (not a stale
# placeholder) has no on-disk timestamp to back off on, so without this
# every call retried the download -- up to a 30s timeout each time.
self._download_failures: Dict[str, float] = {}
# Session for HTTP requests # Session for HTTP requests
self.session = requests.Session() self.session = requests.Session()
@@ -120,17 +125,7 @@ class LogoHelper:
# Resolve the effective target size BEFORE the cache lookup so the # Resolve the effective target size BEFORE the cache lookup so the
# key is size-qualified — a panel-size change must not return a # key is size-qualified — a panel-size change must not return a
# logo resized for the old dimensions. # logo resized for the old dimensions.
if max_width is None: max_width, max_height, scale = self._scaled_box(max_width, max_height, scale)
max_width = int(self.display_width * DEFAULT_LOGO_BOX_FACTOR)
if max_height is None:
max_height = int(self.display_height * DEFAULT_LOGO_BOX_FACTOR)
# Imported here: src.element_style imports src.common (for bdf_font),
# whose __init__ imports this module.
from src.element_style import coerce_scale
scale = coerce_scale(scale, 1.0)
if scale != 1.0:
max_width = max(1, int(round(max_width * scale)))
max_height = max(1, int(round(max_height * scale)))
# The key carries the scaled box, so two elements scaled differently # The key carries the scaled box, so two elements scaled differently
# cannot be served each other's image. # cannot be served each other's image.
cache_key = f"{team_abbr}_{logo_path}_{max_width}x{max_height}" cache_key = f"{team_abbr}_{logo_path}_{max_width}x{max_height}"
@@ -159,7 +154,7 @@ class LogoHelper:
return None return None
# Load image # Load image
logo = Image.open(logo_path) logo: Image.Image = Image.open(logo_path)
if logo.mode != 'RGBA': if logo.mode != 'RGBA':
logo = logo.convert('RGBA') logo = logo.convert('RGBA')
@@ -204,7 +199,12 @@ class LogoHelper:
return self.load_logo(team_abbr, logo_path, max_width, max_height, return self.load_logo(team_abbr, logo_path, max_width, max_height,
scale) scale)
# Download if URL provided and file doesn't exist # Download if URL provided and file doesn't exist, unless the last
# attempt for this path failed recently.
failed_at = self._download_failures.get(str(logo_path))
if (logo_url and failed_at is not None
and time.time() - failed_at < MISSING_LOGO_RECHECK_SECONDS):
logo_url = None
if logo_url: if logo_url:
try: try:
self.logger.info(f"Downloading logo for {team_abbr} from {logo_url}") self.logger.info(f"Downloading logo for {team_abbr} from {logo_url}")
@@ -218,6 +218,7 @@ class LogoHelper:
scale) scale)
except Exception as e: except Exception as e:
self.logger.error(f"Failed to download logo for {team_abbr}: {e}") self.logger.error(f"Failed to download logo for {team_abbr}: {e}")
self._download_failures[str(logo_path)] = time.time()
# The retry failed, so restart the back-off. The stale # The retry failed, so restart the back-off. The stale
# placeholder is still on disk with its old timestamp, and # placeholder is still on disk with its old timestamp, and
# leaving it there means the next call retries immediately -- # leaving it there means the next call retries immediately --
@@ -225,8 +226,30 @@ class LogoHelper:
# exists to prevent. # exists to prevent.
self._refresh_stale_placeholder(logo_path) self._refresh_stale_placeholder(logo_path)
# Create placeholder if all else fails # Create placeholder if all else fails. Sized to the same scaled box
return self._create_placeholder_logo(team_abbr, max_width, max_height) # a real logo gets, so a scaled element doesn't jump in size while
# its logo is missing.
box_width, box_height, _ = self._scaled_box(max_width, max_height, scale)
return self._create_placeholder_logo(team_abbr, box_width, box_height)
def _scaled_box(self, max_width: Optional[int], max_height: Optional[int],
scale: float) -> Tuple[int, int, float]:
"""The logo box after defaults and the user's scale are applied.
Returns ``(width, height, coerced_scale)``.
"""
if max_width is None:
max_width = int(self.display_width * DEFAULT_LOGO_BOX_FACTOR)
if max_height is None:
max_height = int(self.display_height * DEFAULT_LOGO_BOX_FACTOR)
# Imported here: src.element_style imports src.common (for bdf_font),
# whose __init__ imports this module.
from src.element_style import coerce_scale
scale = coerce_scale(scale, 1.0)
if scale != 1.0:
max_width = max(1, int(round(max_width * scale)))
max_height = max(1, int(round(max_height * scale)))
return max_width, max_height, scale
def _invalidate_cached_logo(self, team_abbr: str, logo_path: Path) -> None: def _invalidate_cached_logo(self, team_abbr: str, logo_path: Path) -> None:
"""Drop every cached size of one logo after its file changed on disk.""" """Drop every cached size of one logo after its file changed on disk."""
@@ -240,6 +263,7 @@ class LogoHelper:
# leaving it would hide a logo we just downloaded. # leaving it would hide a logo we just downloaded.
for key in [k for k in self._missing_logos if k.startswith(prefix)]: for key in [k for k in self._missing_logos if k.startswith(prefix)]:
del self._missing_logos[key] del self._missing_logos[key]
self._download_failures.pop(str(logo_path), None)
@staticmethod @staticmethod
def _refresh_stale_placeholder(logo_path: Path) -> None: def _refresh_stale_placeholder(logo_path: Path) -> None:
@@ -332,9 +356,10 @@ class LogoHelper:
self._logo_cache.clear() self._logo_cache.clear()
self._cache_order.clear() self._cache_order.clear()
self._missing_logos.clear() self._missing_logos.clear()
self._download_failures.clear()
self.logger.debug("Logo cache cleared") self.logger.debug("Logo cache cleared")
def get_cache_stats(self) -> Dict[str, int]: def get_cache_stats(self) -> Dict[str, float]:
""" """
Get cache statistics. Get cache statistics.
+41 -24
View File
@@ -290,6 +290,26 @@ def get_cache_dir_mode() -> int:
return 0o2775 # rwxrwsr-x (setgid + group writable) return 0o2775 # rwxrwsr-x (setgid + group writable)
def _sudo_bash_candidates() -> list:
"""Bash paths to try, in order, when running a vetted helper via sudo.
sudoers matches the exact argv, so ``sudo -n <bash> <helper> ...`` only
works if <bash> is the same path configure_web_sudo.sh wrote into the
rule -- whatever ``command -v bash`` said on the machine that ran it.
On merged-/usr systems /usr/bin/bash and /bin/bash are the same file but
different strings to sudo, and the web user's PATH can differ from the
installer's, so no single guess is reliable. Callers try each in turn and
move on only when sudo refused the command line (SUDO_REFUSAL_PHRASES).
The helper is invoked through bash rather than its shebang for the same
reason: the rule names bash, not the script.
"""
candidates = []
for candidate in ("/usr/bin/bash", "/bin/bash", _shutil.which("bash")):
if candidate and candidate not in candidates:
candidates.append(candidate)
return candidates
def sudo_remove_directory(path: Path, allowed_bases: Optional[list] = None) -> bool: def sudo_remove_directory(path: Path, allowed_bases: Optional[list] = None) -> bool:
""" """
Remove a directory using sudo as a last resort. Remove a directory using sudo as a last resort.
@@ -350,22 +370,25 @@ def sudo_remove_directory(path: Path, allowed_bases: Optional[list] = None) -> b
logger.error(f"Safe removal helper not found: {helper_script}") logger.error(f"Safe removal helper not found: {helper_script}")
return False return False
bash_path = _shutil.which('bash') or '/bin/bash'
try: try:
result = subprocess.run( for bash_path in _sudo_bash_candidates():
['sudo', '-n', bash_path, str(helper_script), str(resolved)], result = subprocess.run(
capture_output=True, ['sudo', '-n', bash_path, str(helper_script), str(resolved)],
text=True, capture_output=True,
timeout=30 text=True,
) timeout=30
if result.returncode == 0 and not resolved.exists(): )
logger.info(f"Successfully removed {path} via sudo helper") if result.returncode == 0 and not resolved.exists():
return True logger.info(f"Successfully removed {path} via sudo helper")
else: return True
stderr = result.stderr.strip() # Only a refused command line is worth another bash path; if the
logger.error(f"sudo helper failed for {path}: {stderr}") # helper itself ran and failed, a retry would just repeat it.
return False if result.returncode == 0 or not any(
phrase in (result.stderr or '') for phrase in SUDO_REFUSAL_PHRASES):
break
stderr = (result.stderr or '').strip()
logger.error(f"sudo helper failed for {path}: {stderr}")
return False
except subprocess.TimeoutExpired: except subprocess.TimeoutExpired:
logger.error(f"sudo helper timed out for {path}") logger.error(f"sudo helper timed out for {path}")
return False return False
@@ -417,16 +440,10 @@ def install_requirements_file(req_file: Path, timeout: int = 300) -> subprocess.
wrapper = project_root / "scripts" / "fix_perms" / "safe_pip_install.sh" wrapper = project_root / "scripts" / "fix_perms" / "safe_pip_install.sh"
if wrapper.exists(): if wrapper.exists():
# See sudo_remove_directory / configure_web_sudo.sh for why bash must # See _sudo_bash_candidates for why bash is invoked by explicit path
# be invoked with an explicit, known path rather than relying on the # and why there is more than one to try.
# wrapper's shebang: sudoers matches the exact command line.
bash_candidates = []
for candidate in ("/usr/bin/bash", "/bin/bash", _shutil.which("bash")):
if candidate and candidate not in bash_candidates:
bash_candidates.append(candidate)
result = None result = None
for bash_path in bash_candidates: for bash_path in _sudo_bash_candidates():
# bash_path and wrapper are fixed, known-good paths, and # bash_path and wrapper are fixed, known-good paths, and
# safe_pip_install.sh independently re-validates req_file is an # safe_pip_install.sh independently re-validates req_file is an
# allowed requirements.txt before installing anything as root. # allowed requirements.txt before installing anything as root.
+238
View File
@@ -0,0 +1,238 @@
"""Let a background thread run Python only while the render thread waits on vsync.
With plugin rendering moved to Vegas's prefetch thread (DisplayManager.offscreen,
#630) the render thread no longer stops for it, but it still shares the GIL
with it. The render thread spends most of each refresh inside SwapOnVSync,
which releases the GIL, and needs it back the moment the swap returns. If the
prefetch thread is running Python right then, the render thread waits: up to
the switch interval (5ms) behind bytecode, and for as long as a C call that
keeps the GIL takes. On hdpi that showed up as frames 2-5 refreshes late while
a group was being prepared.
The gate turns that around. The display manager opens it just before each swap,
with a deadline shortly ahead of the refresh the swap will return on, and
closes it when the swap returns. A thread inside ``gate.yielding()`` checks it on
every Python and C call through a profile hook, and once the window has closed
it parks -- blocked on a condition, GIL released -- until the next swap opens
it. The render thread then finds the GIL free when its refresh arrives, and the
background work runs in time the render thread was only spending waiting.
Parking a thread is only safe if nothing the render thread needs is stuck
behind it, so it is never parked:
* while it holds a lock registered with ``guard()`` (the Vegas buffers and
caches the render thread also takes);
* inside logging, threading, importlib or the cache, all of which take locks the
render thread can take too;
* when there is no render loop to protect -- no swap for ``STALE_SECONDS``, as
on a static screen or a stalled frame.
And a parked thread is never held more than ``MAX_WAIT_SECONDS`` at a time, so
whatever the gate gets wrong costs a frame, not a freeze. The render thread
itself is never gated, whatever it calls.
It gates the prefetch thread only. Gating the ESPN fetch threads as well was
tried for the hourly sports refresh, twenty-odd of them at once, and measured
worse on hdpi (0.85% late frames without it, 1.14% with it, across a burst every
five minutes): each parked thread has to take the GIL again just to park at the
end of every window, and the fetches ran two to three times as long.
"""
from __future__ import annotations
import math
import sys
import threading
import time
from collections import deque
from typing import Any, Callable, Deque, List, Optional, cast
from src.common.frame_timing import binding_releases_gil
#: Park background threads this long before the refresh a swap will return on,
#: so a short C call already under way has finished by then.
MARGIN_SECONDS = 0.002
#: The longest a background thread is parked in one go.
MAX_WAIT_SECONDS = 0.05
#: No swap for this long means there is no render loop running to protect.
STALE_SECONDS = 0.05
#: Swaps needed before the refresh period is trusted enough to open a window.
MIN_SAMPLES = 8
#: Parking inside any of these modules could hold a lock the render thread
#: takes: logging handler locks, Condition and Event internals, the module
#: import locks, and the disk and memory cache locks. Matched by module name,
#: not file path: a path can say "cache" or "logging" for reasons of its own --
#: a virtualenv under ~/.cache, or GitHub's /opt/hostedtoolcache, where every
#: stdlib frame would otherwise count and the gate would never park anything.
_UNSAFE_MODULES = frozenset({
"logging", "threading", "importlib", "src.cache_manager", "src.cache",
})
_UNSAFE_PREFIXES = ("logging.", "importlib.", "_frozen_importlib", "src.cache.")
def _unsafe(frame: Any, base: Any) -> bool:
"""True if a frame above ``base`` comes from somewhere parking could deadlock.
``base`` is the frame that entered ``yielding()``; what lies below it (the
thread's own bootstrap in threading.py) holds nothing.
"""
while frame is not None and frame is not base:
name = frame.f_globals.get("__name__") or ""
if name in _UNSAFE_MODULES or name.startswith(_UNSAFE_PREFIXES):
return True
frame = frame.f_back
return False
def swap_releases_gil() -> Optional[bool]:
"""Whether the loaded rgbmatrix binding releases the GIL, or None if none is loaded.
A thin delegate to src.common.frame_timing.binding_releases_gil (#629),
which this used to duplicate line for line. The name stays because the
coordinator calls it here and tests replace it here.
"""
return binding_releases_gil()
def _held(lock: Any) -> bool:
"""Is ``lock`` held? RLocks report this thread's ownership; plain locks, anyone's."""
is_owned = getattr(lock, "_is_owned", None)
if is_owned is not None:
return cast(bool, is_owned())
return cast(bool, lock.locked())
class RenderGate:
"""Opened by the render thread around each swap; honoured by background threads."""
def __init__(self, clock: Callable[[], float] = time.monotonic):
self.clock = clock
self._cond = threading.Condition()
self._generation = 0
self._open_until = 0.0
self._last_return: Optional[float] = None
self._periods: Deque[float] = deque(maxlen=64)
self._period: Optional[float] = None
self._guarded: List[Any] = []
self._local = threading.local()
self._render_ident: Optional[int] = None
#: How often, and for how long in all, background threads were parked.
self.parks = 0
self.parked_seconds = 0.0
def guard(self, *locks: Any) -> None:
"""Never park a thread while it holds (or, for a plain Lock, anyone holds) these."""
self._guarded.extend(lock for lock in locks if lock is not None)
# -- render thread -----------------------------------------------------
def refresh_period(self) -> Optional[float]:
"""The panel's refresh period from recent swaps, or None until known.
The 10th percentile of the gaps between swap returns, each divided by
the hold: a late frame only ever lengthens a gap, so the low end is
the panel's own period.
"""
return self._period
def before_swap(self, hold: int) -> None:
"""The render thread is about to block in SwapOnVSync: open the window."""
hold = max(1, int(hold))
period = self._period
now = self.clock()
last = self._last_return
if period and last is not None and now - last < STALE_SECONDS:
# The swap returns on the first refresh boundary after both the
# current frame's hold is up and this frame has been handed over;
# boundaries fall a whole period apart from the last return.
refreshes = max(hold, math.ceil((now - last) / period))
open_until = last + refreshes * period - MARGIN_SECONDS
else:
open_until = 0.0 # no rhythm to predict from: leave threads be
with self._cond:
self._open_until = open_until
self._generation += 1
self._cond.notify_all()
def after_swap(self, hold: int) -> None:
"""The swap returned and the render thread needs the GIL: close the window."""
now = self.clock()
self._open_until = 0.0
if self._render_ident is None:
# The first thread to swap is the render loop. A plugin pushing a
# live refresh from its update thread swaps too, but must not take
# over its exemption.
self._render_ident = threading.get_ident()
last = self._last_return
if last is not None and now - last < STALE_SECONDS:
self._periods.append((now - last) / max(1, int(hold)))
if len(self._periods) >= MIN_SAMPLES:
ordered = sorted(self._periods)
self._period = ordered[len(ordered) // 10]
self._last_return = now
# -- background threads ------------------------------------------------
def _should_park(self, frame: Any, now: float) -> bool:
if now < self._open_until:
return False # inside the window
last = self._last_return
if last is None or now - last > STALE_SECONDS or self._period is None:
return False # no render loop to protect
for lock in self._guarded:
if _held(lock):
return False
return not _unsafe(frame, getattr(self._local, "base", None))
def _hook(self, frame: Any, _event: str, _arg: Any) -> None:
now = self.clock()
if not self._should_park(frame, now):
return
generation = self._generation
with self._cond:
self._cond.wait_for(lambda: self._generation != generation,
timeout=MAX_WAIT_SECONDS)
self.parks += 1
self.parked_seconds += self.clock() - now
def yielding(self) -> "_Yielding":
"""``with gate.yielding():`` runs the block giving way to the render thread."""
return _Yielding(self)
class _Yielding:
"""Installs a gate's profile hook on the thread for the length of a block."""
def __init__(self, gate: RenderGate):
self.gate = gate
self._previous: Any = None
self._previous_base: Any = None
self._skipped = False
def __enter__(self) -> RenderGate:
gate = self.gate
# pylint: disable=protected-access
if threading.get_ident() == gate._render_ident:
self._skipped = True # parking the render thread parks the display
return gate
local = gate._local
self._previous_base = getattr(local, "base", None)
if self._previous_base is None:
# Nested blocks keep the outermost frame, so everything the thread
# entered since it first gave way is still checked for locks.
local.base = sys._getframe(1)
self._previous = sys.getprofile()
sys.setprofile(gate._hook)
return gate
def __exit__(self, *_exc: Any) -> None:
if self._skipped:
return
sys.setprofile(self._previous)
self.gate._local.base = self._previous_base # pylint: disable=protected-access
+10 -6
View File
@@ -49,7 +49,9 @@ from __future__ import annotations
import logging import logging
from dataclasses import dataclass, replace from dataclasses import dataclass, replace
from typing import Any, Dict, Optional from typing import Any, Dict, List, Optional, cast
from src.matrix_support import DEFAULT_REFRESH_LIMIT_HZ
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -63,8 +65,9 @@ MIN_PIXELS_PER_SECOND = 1.0
MAX_PIXELS_PER_SECOND = 500.0 MAX_PIXELS_PER_SECOND = 500.0
#: Assumed refresh when the caller does not say. Matches the usual #: Assumed refresh when the caller does not say. Matches the usual
#: ``display.hardware.limit_refresh_rate_hz``. #: ``display.hardware.limit_refresh_rate_hz``, and is the cap DisplayManager
DEFAULT_REFRESH_HZ = 100.0 #: applies when that key is missing.
DEFAULT_REFRESH_HZ = float(DEFAULT_REFRESH_LIMIT_HZ)
#: How far px/s may sit from a whole number of pixels per refresh before it is #: How far px/s may sit from a whole number of pixels per refresh before it is
#: worth warning about. 0.05px per frame is invisible; a third of a pixel is not. #: worth warning about. 0.05px per frame is invisible; a third of a pixel is not.
@@ -129,14 +132,14 @@ def crisp_ladder(
refresh_hz: float = DEFAULT_REFRESH_HZ, refresh_hz: float = DEFAULT_REFRESH_HZ,
max_frame_hold: int = MAX_FRAME_HOLD, max_frame_hold: int = MAX_FRAME_HOLD,
max_pixels_per_frame: int = MAX_PIXELS_PER_FRAME, max_pixels_per_frame: int = MAX_PIXELS_PER_FRAME,
): ) -> List[CrispSpeed]:
"""Every whole-pixel speed this panel can show, slowest first. """Every whole-pixel speed this panel can show, slowest first.
Duplicates are collapsed keeping the gentlest option: 100 px/s is reachable Duplicates are collapsed keeping the gentlest option: 100 px/s is reachable
as 1px every refresh or 2px every 2nd refresh, and the former moves in as 1px every refresh or 2px every 2nd refresh, and the former moves in
smaller increments, so that is the one worth offering. smaller increments, so that is the one worth offering.
""" """
best = {} best: Dict[float, CrispSpeed] = {}
for hold in range(1, max_frame_hold + 1): for hold in range(1, max_frame_hold + 1):
for ppf in range(1, max_pixels_per_frame + 1): for ppf in range(1, max_pixels_per_frame + 1):
pps = refresh_hz / hold * ppf pps = refresh_hz / hold * ppf
@@ -431,7 +434,8 @@ def configure(
# they start scrolling. configure() only reports what is needed. # they start scrolling. configure() only reports what is needed.
if choice: if choice:
requested = settings.requested_pixels_per_second # Set whenever there is a crisp choice (see the replace() above).
requested = cast(float, settings.requested_pixels_per_second)
if abs(requested - applied) > 0.05: if abs(requested - applied) > 0.05:
log.info( log.info(
"Scroll configured: %s (asked for %.1f px/s from %s; " "Scroll configured: %s (asked for %.1f px/s from %s; "
+240 -37
View File
@@ -18,7 +18,7 @@ Features:
import logging import logging
import math import math
import time import time
from typing import Optional, Dict, Any from typing import Optional, Dict, Any, List, Tuple
from PIL import Image from PIL import Image
import numpy as np import numpy as np
@@ -29,6 +29,15 @@ import numpy as np
FPS_LOG_INTERVAL = 5.0 FPS_LOG_INTERVAL = 5.0
def _rgb_pixels(item) -> np.ndarray:
"""An appended item's pixels as an RGB array, as pasting it would draw them."""
if isinstance(item, np.ndarray):
return item
if item.mode != 'RGB':
item = item.convert('RGB')
return np.asarray(item)
def frame_stats(frame_times: list) -> Dict[str, Any]: def frame_stats(frame_times: list) -> Dict[str, Any]:
"""Summary statistics over one window of frame durations (seconds). """Summary statistics over one window of frame durations (seconds).
@@ -111,9 +120,21 @@ class ScrollHelper:
self.total_distance_scrolled = 0.0 # Track total distance including wrap-arounds self.total_distance_scrolled = 0.0 # Track total distance including wrap-arounds
self.scroll_speed = 1.0 self.scroll_speed = 1.0
self.scroll_delay = 0.001 # Minimal delay for high FPS (1ms) self.scroll_delay = 0.001 # Minimal delay for high FPS (1ms)
self.cached_image: Optional[Image.Image] = None self.cached_image = None # see the property below
self.cached_array: Optional[np.ndarray] = None # Numpy array cache for fast operations self.cached_array: Optional[np.ndarray] = None # Numpy array cache for fast operations
self.total_scroll_width = 0 self.total_scroll_width = 0
# An extended strip lives in a buffer with spare room after it, and
# cached_array is a view of the buffer's live columns: an append writes
# only the new columns, and a trim only moves the view's start. See
# append_content. _strip_view is the view this helper last made; a
# cached_array that is anything else was set from outside and is not
# written through.
self._strip_buffer: Optional[np.ndarray] = None
self._strip_view: Optional[np.ndarray] = None
self._strip_start = 0
#: Bytes the last append_content / drop_scrolled_prefix copied, for
#: frame-timing attribution (src/common/frame_timing.py note_op).
self.last_copy_bytes = 0
# Pre-allocated buffer for output frame (reused to avoid allocations) # Pre-allocated buffer for output frame (reused to avoid allocations)
self._frame_buffer: Optional[np.ndarray] = None self._frame_buffer: Optional[np.ndarray] = None
@@ -172,7 +193,53 @@ class ScrollHelper:
# Scrolling state management # Scrolling state management
self.is_scrolling = False self.is_scrolling = False
self.scroll_complete = False self.scroll_complete = False
# -- the strip as a PIL image ---------------------------------------------
#
# Every frame is cut from cached_array; nothing on the frame path reads the
# PIL image's pixels. Extending and trimming a strip (append_content,
# drop_scrolled_prefix) used to rebuild that image in full each time
# anyway: Image.fromarray of a Vegas-sized strip is 1.7-3.8ms on a Pi 4,
# twice per extension, on the render thread. Those two now leave it to be
# built from the array on first read, which in Vegas means only by a
# multi-display sync push -- and the strip is not held twice in memory.
#
# Assigning cached_image still stores exactly what was assigned; a lazy
# image is only ever one the helper derived from its own array.
@property
def cached_image(self) -> Optional[Image.Image]:
"""The strip as a PIL image, built from ``cached_array`` if deferred."""
image = self.__dict__.get('_cached_image')
if image is not None:
return image
source = self.__dict__.get('_image_source')
if source is None:
return None
# Built from the array this read started with. Another thread (the
# sync push) may read while the render thread extends the strip; it
# then gets the strip as it was, as it did when the image was built
# eagerly, and the stale build is not kept.
image = Image.fromarray(source)
if self.__dict__.get('_image_source') is source:
self._cached_image = image
return image
@cached_image.setter
def cached_image(self, image: Optional[Image.Image]) -> None:
self._cached_image = image
self._image_source = None
def _defer_image(self) -> None:
"""The array just changed under the image: rebuild it only if read."""
self._cached_image = None
self._image_source = self.cached_array
def has_strip(self) -> bool:
"""Whether there is a strip (an image, or one deferred), not reading it."""
return (self.__dict__.get('_cached_image') is not None
or self.__dict__.get('_image_source') is not None)
def create_scrolling_image(self, content_items: list, def create_scrolling_image(self, content_items: list,
item_gap: int = 32, item_gap: int = 32,
element_gap: int = 16, element_gap: int = 16,
@@ -203,6 +270,7 @@ class ScrollHelper:
self.total_scroll_width = 0 self.total_scroll_width = 0
self.cached_image = Image.new('RGB', (self.display_width, self.display_height), (0, 0, 0)) self.cached_image = Image.new('RGB', (self.display_width, self.display_height), (0, 0, 0))
self.cached_array = np.array(self.cached_image) self.cached_array = np.array(self.cached_image)
self._forget_strip_buffer()
self.scroll_position = 0.0 self.scroll_position = 0.0
self.total_distance_scrolled = 0.0 self.total_distance_scrolled = 0.0
self.scroll_complete = False self.scroll_complete = False
@@ -238,6 +306,7 @@ class ScrollHelper:
self.cached_image = full_image self.cached_image = full_image
# Convert to numpy array for fast operations # Convert to numpy array for fast operations
self.cached_array = np.array(full_image) self.cached_array = np.array(full_image)
self._forget_strip_buffer()
actual_image_width = full_image.width actual_image_width = full_image.width
self.total_scroll_width = actual_image_width self.total_scroll_width = actual_image_width
@@ -254,6 +323,9 @@ class ScrollHelper:
now = time.time() now = time.time()
self.scroll_start_time = now self.scroll_start_time = now
self.last_progress_log_time = now self.last_progress_log_time = now
# The position just went back to 0; the first update must not advance
# it by however long the helper sat idle (off-screen) before this.
self.last_update_time = now
self.logger.info( self.logger.info(
"Dynamic duration target set to %ds (min=%ds, max=%ds, buffer=%.2f)", "Dynamic duration target set to %ds (min=%ds, max=%ds, buffer=%.2f)",
self.calculated_duration, self.calculated_duration,
@@ -280,7 +352,7 @@ class ScrollHelper:
Otherwise the position advances by elapsed time at the configured Otherwise the position advances by elapsed time at the configured
speed. speed.
""" """
if not self.cached_image: if not self.has_strip():
return return
# Calculate frame time for consistent scroll speed regardless of FPS # Calculate frame time for consistent scroll speed regardless of FPS
@@ -424,7 +496,7 @@ class ScrollHelper:
Returns: Returns:
PIL Image showing the visible portion, or None if no cached image PIL Image showing the visible portion, or None if no cached image
""" """
if not self.cached_image or self.cached_array is None: if self.cached_array is None or not self.has_strip():
return None return None
start_x_int = int(self.scroll_position) start_x_int = int(self.scroll_position)
@@ -498,7 +570,7 @@ class ScrollHelper:
slices (128×32 = 12 KB) used here. slices (128×32 = 12 KB) used here.
""" """
_size = (self.display_width, self.display_height) _size = (self.display_width, self.display_height)
img_w = self.cached_image.width img_w = self.cached_array.shape[1]
if end_x <= img_w: if end_x <= img_w:
# Normal case: single contiguous slice (fastest path) # Normal case: single contiguous slice (fastest path)
@@ -631,7 +703,10 @@ class ScrollHelper:
strip also defers completion, which is the intent. strip also defers completion, which is the intent.
Args: Args:
content_items: Images to append, in order content_items: Images to append, in order. An item may instead be
its pixels already as an RGB array (``np.asarray`` of an RGB
image), so a caller can do that conversion off the render
thread (Vegas prepares its blocks with the group).
item_gap: Gap between appended items, and between the existing item_gap: Gap between appended items, and between the existing
content and the first appended item content and the first appended item
element_gap: Extra gap after each item, mirroring element_gap: Extra gap after each item, mirroring
@@ -643,42 +718,103 @@ class ScrollHelper:
if not content_items: if not content_items:
return False return False
if self.cached_image is None or self.cached_array is None: if self.cached_array is None or not self.has_strip():
# Nothing to extend yet — this is just the first build. # Nothing to extend yet — this is just the first build.
self.create_scrolling_image( self.create_scrolling_image(
content_items, item_gap=item_gap, element_gap=element_gap, lead_gap=0) [Image.fromarray(item) if isinstance(item, np.ndarray) else item
for item in content_items],
item_gap=item_gap, element_gap=element_gap, lead_gap=0)
return True return True
gap = max(0, item_gap) gap = max(0, item_gap)
addition_width = ( pieces = []
sum(img.width for img in content_items)
+ gap * len(content_items) # one leading gap per item
+ element_gap * len(content_items)
)
addition = Image.new('RGB', (addition_width, self.display_height), (0, 0, 0))
x = 0 x = 0
for img in content_items: for item in content_items:
x += gap # separate from whatever precedes x += gap # separate from whatever precedes
addition.paste(img, (x, 0)) pixels = _rgb_pixels(item)
x += img.width + element_gap pieces.append((x, pixels))
x += pixels.shape[1] + element_gap
addition_width = x
# numpy concatenate then one conversion back, rather than allocating a # Each item is written straight into the spare room after the strip,
# full-width PIL image and pasting twice: the strip can be tens of # when there is some: the strip can be tens of thousands of columns
# thousands of columns wide and this runs on the render path. # wide and this runs on the render thread, where copying all of it
self.cached_array = np.concatenate( # (2-3 ms at 512x64 on a Pi 4) -- or even laying the items out in an
(self.cached_array, np.array(addition)), axis=1) # image of their own first (another 4-5 ms) -- cost the frame after
self.cached_image = Image.fromarray(self.cached_array) # every extension. The PIL image is built from the array only if
self.total_scroll_width = self.cached_image.width # something reads it (see cached_image).
self.cached_array = self._extended_strip(pieces, addition_width)
self._defer_image()
self.total_scroll_width = self.cached_array.shape[1]
self.scroll_complete = False self.scroll_complete = False
self.logger.info( # Debug: this runs on the render thread, and the caller (Vegas) logs
# each extension itself.
self.logger.debug(
"Appended %d item(s) (%dpx) to scroll strip: now %dpx, position %.0f", "Appended %d item(s) (%dpx) to scroll strip: now %dpx, position %.0f",
len(content_items), addition_width, self.total_scroll_width, len(content_items), addition_width, self.total_scroll_width,
self.scroll_position self.scroll_position
) )
return True return True
#: Room an extended strip's buffer is given, as a multiple of what it
#: holds when (re)allocated. Trims free columns at the front and appends
#: use them at the back, so with 3x the buffer is reallocated -- the one
#: full copy -- about once every two strip-lengths scrolled.
STRIP_SPARE_FACTOR = 3.0
def _extended_strip(self, pieces: List[Tuple[int, np.ndarray]], added: int) -> np.ndarray:
"""The strip with ``added`` black columns after it, ``pieces`` drawn in.
Each piece is ``(x, pixels)``, x counted from the old strip's end;
written in place when the buffer has the room.
"""
live = self.cached_array
if live is None:
# append_content builds a first strip itself and never comes here.
raise RuntimeError("no strip to extend")
width = live.shape[1]
buffer = self._strip_buffer
if (live is self._strip_view and buffer is not None
and self._strip_start + width + added <= buffer.shape[1]):
end = self._strip_start + width
self.last_copy_bytes = 0
else:
total = width + added
buffer = np.empty((live.shape[0], max(total + 1, int(total * self.STRIP_SPARE_FACTOR)))
+ live.shape[2:], dtype=live.dtype)
buffer[:, :width] = live
self._strip_buffer = buffer
self._strip_start = 0
end = width
self.last_copy_bytes = live.nbytes
rows = buffer.shape[0]
region = buffer[:, end:end + added]
# Black only where no piece lands -- the gaps, and below a short
# piece: blanking the whole region first cost as much again as
# writing the pieces (1.8 ms at 512x64 on a Pi 4).
covered = 0
for x, pixels in pieces:
pixels = pixels[:rows]
cols = pixels.shape[1]
if x > covered:
region[:, covered:x] = 0
region[:pixels.shape[0], x:x + cols] = pixels
if pixels.shape[0] < rows:
region[pixels.shape[0]:, x:x + cols] = 0
covered = max(covered, x + cols)
if covered < added:
region[:, covered:] = 0
self.last_copy_bytes += region.nbytes
self._strip_view = buffer[:, self._strip_start:self._strip_start + width + added]
return self._strip_view
def _forget_strip_buffer(self) -> None:
"""A new strip replaces the extended one: let its buffer go."""
self._strip_buffer = None
self._strip_view = None
self._strip_start = 0
def drop_scrolled_prefix(self, keep_before: int = 0) -> int: def drop_scrolled_prefix(self, keep_before: int = 0) -> int:
""" """
Discard columns that have already scrolled past, to bound memory. Discard columns that have already scrolled past, to bound memory.
@@ -696,14 +832,15 @@ class ScrollHelper:
Returns: Returns:
Number of columns actually removed Number of columns actually removed
""" """
if self.cached_image is None or self.cached_array is None: if self.cached_array is None or not self.has_strip():
return 0 return 0
strip_width = self.cached_array.shape[1]
# While the viewport wraps, get_visible_portion fills its right-hand side # While the viewport wraps, get_visible_portion fills its right-hand side
# from the *head* of the strip, so trimming the head would change what # from the *head* of the strip, so trimming the head would change what
# is on screen. Continuous mode extends before ever reaching that state; # is on screen. Continuous mode extends before ever reaching that state;
# refusing here keeps "trimming is invisible" true unconditionally. # refusing here keeps "trimming is invisible" true unconditionally.
if self.scroll_position + self.display_width > self.cached_image.width: if self.scroll_position + self.display_width > strip_width:
return 0 return 0
cut = int(self.scroll_position) - max(0, keep_before) cut = int(self.scroll_position) - max(0, keep_before)
@@ -711,15 +848,24 @@ class ScrollHelper:
return 0 return 0
# Never trim so far that the remaining strip is narrower than the # Never trim so far that the remaining strip is narrower than the
# viewport, or get_visible_portion has nothing to slice. # viewport, or get_visible_portion has nothing to slice.
cut = min(cut, max(0, self.cached_image.width - self.display_width)) cut = min(cut, max(0, strip_width - self.display_width))
if cut <= 0: if cut <= 0:
return 0 return 0
# .copy() so the original buffer is released rather than kept alive by if self.cached_array is self._strip_view:
# a numpy view. # Only the view's start moves; the columns behind it are reused
self.cached_array = self.cached_array[:, cut:].copy() # when the buffer is next reallocated (append_content).
self.cached_image = Image.fromarray(self.cached_array) self._strip_view = self.cached_array[:, cut:]
self.total_scroll_width = self.cached_image.width self._strip_start += cut
self.cached_array = self._strip_view
self.last_copy_bytes = 0
else:
# Not a strip this helper extended: .copy() so the original
# buffer is released rather than kept alive by a view.
self.cached_array = self.cached_array[:, cut:].copy()
self.last_copy_bytes = self.cached_array.nbytes
self._defer_image()
self.total_scroll_width = self.cached_array.shape[1]
self.scroll_position -= cut self.scroll_position -= cut
self.total_distance_scrolled = max(0.0, self.total_distance_scrolled - cut) self.total_distance_scrolled = max(0.0, self.total_distance_scrolled - cut)
@@ -729,9 +875,46 @@ class ScrollHelper:
) )
return cut return cut
def patch_columns(self, x: int, pixels: np.ndarray) -> int:
"""Overwrite the strip's columns from ``x`` with ``pixels``, in place.
What a live Vegas element update is (src/vegas_mode/elements.py): the
strip keeps its width, the scroll keeps its position, and only these
columns change. Call it between frames on the thread that draws them;
every frame copies its slice out of the strip (get_visible_portion),
so no frame already handed on can see half a patch.
Clipped to the strip at both ends. Refused (0) for an array this
helper may not write -- the multi-display follower adopts a read-only
one -- or for pixels of another height. The PIL image is deferred, so
a later read of cached_image shows the patch.
Args:
x: Strip column of the first column of ``pixels``
pixels: uint8 array (height, width, 3)
Returns:
Bytes written.
"""
strip = self.cached_array
if strip is None or not strip.flags.writeable:
return 0
if pixels.ndim != 3 or pixels.shape[0] != strip.shape[0] \
or pixels.shape[2] != strip.shape[2]:
return 0
width = pixels.shape[1]
lo, hi = max(0, int(x)), min(strip.shape[1], int(x) + width)
if hi <= lo:
return 0
strip[:, lo:hi] = pixels[:, lo - int(x):hi - int(x)]
if self.__dict__.get('_cached_image') is not None \
or self.__dict__.get('_image_source') is not None:
self._defer_image()
return (hi - lo) * strip.shape[0] * strip.shape[2]
def remaining_unscrolled(self) -> int: def remaining_unscrolled(self) -> int:
"""Columns of strip still to the right of the viewport.""" """Columns of strip still to the right of the viewport."""
if self.cached_image is None: if not self.has_strip():
return 0 return 0
return max(0, self.total_scroll_width - int(self.scroll_position) return max(0, self.total_scroll_width - int(self.scroll_position)
- self.display_width) - self.display_width)
@@ -776,11 +959,24 @@ class ScrollHelper:
self.clear_cache() self.clear_cache()
return return
# Every frame is cut from cached_array with Image.frombytes('RGB', ...),
# which reads a 4-channel (RGBA) array as garbage and raises on a
# 1-channel (L) one. Transparent pixels go to black, the panel's
# background, rather than to whatever colour hides under the alpha.
if image.mode != 'RGB':
if 'A' in image.mode or 'transparency' in image.info:
rgba = image.convert('RGBA')
image = Image.new('RGB', rgba.size, (0, 0, 0))
image.paste(rgba, (0, 0), rgba)
else:
image = image.convert('RGB')
# Set the cached image # Set the cached image
self.cached_image = image self.cached_image = image
# Convert to numpy array for fast operations (required for get_visible_portion) # Convert to numpy array for fast operations (required for get_visible_portion)
self.cached_array = np.array(image) self.cached_array = np.array(image)
self._forget_strip_buffer()
# Update scroll width # Update scroll width
self.total_scroll_width = image.width self.total_scroll_width = image.width
@@ -802,6 +998,9 @@ class ScrollHelper:
self.scroll_start_time = now self.scroll_start_time = now
self.last_progress_log_time = now self.last_progress_log_time = now
self.last_step_time = now # Initialize step timer for frame-based scrolling self.last_step_time = now # Initialize step timer for frame-based scrolling
# The position just went back to 0; the first update must not advance
# it by however long the helper sat idle before this image arrived.
self.last_update_time = now
self.logger.debug("Set scrolling image: %dx%d, total_scroll_width=%d", self.logger.debug("Set scrolling image: %dx%d, total_scroll_width=%d",
image.width, image.height, self.total_scroll_width) image.width, image.height, self.total_scroll_width)
@@ -1035,6 +1234,7 @@ class ScrollHelper:
""" """
self.cached_image = None self.cached_image = None
self.cached_array = None self.cached_array = None
self._forget_strip_buffer()
self.total_scroll_width = 0 self.total_scroll_width = 0
self.scroll_position = 0.0 self.scroll_position = 0.0
self.total_distance_scrolled = 0.0 self.total_distance_scrolled = 0.0
@@ -1064,5 +1264,8 @@ class ScrollHelper:
'elapsed_time': (time.time() - self.scroll_start_time) 'elapsed_time': (time.time() - self.scroll_start_time)
if self.scroll_start_time if self.scroll_start_time
else None, else None,
'cached_image_size': (self.cached_image.width, self.cached_image.height) if self.cached_image else None # From the array: reading cached_image would build a deferred one.
'cached_image_size': ((self.cached_array.shape[1], self.cached_array.shape[0])
if self.cached_array is not None and self.has_strip()
else None)
} }
+2 -1
View File
@@ -144,7 +144,8 @@ def resolve_font_color(config: Optional[Dict[str, Any]],
if len(matches) > 1: if len(matches) > 1:
configured = [] configured = []
for element in matches: for element in matches:
colour = element_color(config, element, None, mode) # None as the default makes it come back when unconfigured.
colour = element_color(config, element, None, mode) # type: ignore[arg-type]
if colour is not None and colour not in configured: if colour is not None and colour not in configured:
configured.append(colour) configured.append(colour)
if len(configured) == 1: if len(configured) == 1:
+136
View File
@@ -0,0 +1,136 @@
"""The ``sports_card`` delegations every scoreboard's game renderer carries.
After the card helpers moved to ``sports_card`` (3.3.0), each of the eight
scoreboards with a ``game_renderer.py`` -- afl, baseball, basketball,
football, hockey, lacrosse, nrl and soccer -- kept one-line methods that
forward to them with its own ``config`` and ``logger``. Seventeen are
identical in all eight (executable AST, docstrings stripped) or in all but
football, and were copied here from ledmatrix-plugins ``30455671``
(origin/main, 2026-09-29) under their existing names. Football's own
``_format_game_date`` and ``_upcoming_center_mode`` (they follow the
switch-mode settings when it draws the full-screen scorebug) stay in football
and override these.
``_schema_font_size`` and ``_resolve_font_size`` look the same in every copy
but are not moved: they read ``_SCHEMA_PATH``, a module global that is each
plugin's own ``config_schema.json``.
These are the methods ``SportsGameRendererMixin`` (``sports_game_renderer``)
lists among what its host must provide, so a renderer that inherits both no
longer has to write them. Like that mixin this has no ``__init__`` and no
state. It is a separate module rather than more methods there for the reason
``sports_helpers`` gives: a missing module fails at load, where the version
checks see it; a missing method fails mid-render.
WHAT A HOST MUST PROVIDE
------------------------
Derived by walking every ``self.<attr>`` the mixin reads; the host-contract
test in ``test/test_sports_card_wrappers.py`` fails if a read is added
without being listed here.
- ``config`` and ``logger``.
- ``fonts``, read with ``getattr`` -- ``_font_color``.
- ``_FONT_NAME_ALIASES`` and ``_FONT_PIXEL_GRID`` class attributes --
``_crisp_size``, which passes them to ``sports_card.crisp_size`` so a
renderer that declares extra faces keeps them.
Add it as a base of the plugin's renderer, e.g.
``class GameRenderer(SportsCardWrappersMixin, SportsGameRendererMixin)``.
The two define no name in common; a method on the plugin's own class still
wins over either.
"""
import logging
from typing import Any, ClassVar, Dict, Optional, Tuple
from src.common import sports_card as _card
class SportsCardWrappersMixin:
"""The game renderer's ``sports_card`` delegations. See module docstring."""
# The host contract, declared for type checking only: these create no
# attributes, so the host's own values are what the methods read.
config: Dict[str, Any]
logger: logging.Logger
_FONT_NAME_ALIASES: ClassVar[Dict[str, str]]
_FONT_PIXEL_GRID: ClassVar[Dict[str, Any]]
# ---- fonts ---------------------------------------------------------
@classmethod
def _crisp_size(cls, font_file, desired):
"""``sports_card.crisp_size`` with this renderer's font tables."""
return _card.crisp_size(font_file, desired,
cls._FONT_NAME_ALIASES, cls._FONT_PIXEL_GRID)
def _unshare_element_fonts(self, fonts):
"""``sports_card.unshare_element_fonts``."""
return _card.unshare_element_fonts(self.logger, fonts)
def _font_color(self, font, default: Tuple[int, int, int] = (255, 255, 255)):
"""``sports_card.font_color`` for one of ``self.fonts``."""
return _card.font_color(self.config, getattr(self, "fonts", None), font, default)
# ---- colours and favourites ---------------------------------------
@staticmethod
def _coerce_rgb(value, fallback):
"""``sports_card.coerce_rgb``."""
return _card.coerce_rgb(value, fallback)
@staticmethod
def _side_is_favorite(game: Dict[str, Any], side: str, favorites: set) -> bool:
"""``sports_card.side_is_favorite``."""
return _card.side_is_favorite(game, side, favorites)
@staticmethod
def _side_score(game: Dict[str, Any], side: str) -> Optional[int]:
"""``sports_card.side_score``."""
return _card.side_score(game, side)
def _favorite_result(self, game: Dict[str, Any]) -> Optional[str]:
"""``sports_card.favorite_result``."""
return _card.favorite_result(self.config, game)
def _score_color_for(self, game: Dict[str, Any], game_type: str, default=None):
"""``sports_card.score_color_for``."""
return _card.score_color_for(self.config, self.logger, game, game_type, default)
def _recent_score_color(self, game: Dict[str, Any], default):
"""``sports_card.recent_score_color``."""
return _card.recent_score_color(self.config, self.logger, game, default)
def _element_color(self, element: str, default: Tuple[int, int, int] = (255, 255, 255)):
"""``sports_card.element_color``."""
return _card.element_color(self.config, element, default)
# ---- card options, dates and times --------------------------------
def _scroll_card_option(self, key: str, default: Any = None) -> Any:
"""``sports_card.scroll_card_option``."""
return _card.scroll_card_option(self.config, key, default)
def _upcoming_center_mode(self) -> str:
"""``sports_card.upcoming_center_mode``."""
return _card.upcoming_center_mode(self.config)
def _vs_text(self) -> str:
"""``sports_card.vs_text``."""
return _card.vs_text(self.config)
def _format_game_date(self, date_text: str, game: Optional[Dict] = None) -> str:
"""``sports_card.format_game_date``."""
return _card.format_game_date(self.config, self.logger, date_text, game)
def _weekday_for(self, game: Optional[Dict]) -> str:
"""``sports_card.weekday_for``."""
return _card.weekday_for(self.config, self.logger, game)
def _card_tzinfo(self):
"""``sports_card.card_tzinfo``."""
return _card.card_tzinfo(self.config, self.logger)
def _format_game_time(self, time_text: str) -> str:
"""``sports_card.format_game_time``."""
return _card.format_game_time(self.config, time_text)
+780
View File
@@ -0,0 +1,780 @@
"""How the scoreboards draw a score or win celebration.
Five scoreboards -- afl, football, hockey, nrl and soccer -- take over the
panel when a team scores or wins: a backdrop in the scoring team's colours
read off its crest, scenery for the kind of score, confetti, the headline and
the score with the scoring side's digits breathing. The drawing is identical
in all five ``sports.py`` copies (executable AST, docstrings stripped), and
so are the colour helpers it uses; they were copied here from
ledmatrix-plugins ``30455671`` (origin/main, 2026-09-29).
Only the drawing moved. What *arms* a celebration stays in each plugin,
because it differs: which scores count (``_check_for_goal`` /
``_check_for_score``, and nrl matches favourites by team id), the phrase and
the scenery (``_start_celebration``), and when a win fires
(``_check_for_win``). So does ``display()``, which decides whether the
takeover or the scorebug is on screen. A plugin hands this mixin a
celebration dict and it draws it.
The colour helpers are public free functions here (``logo_palette``,
``lift_color``, ``mix_color``, ...); in the plugins they were the same
functions with a leading underscore.
THE CELEBRATION DICT
--------------------
Built by the plugin's ``_start_celebration``. Read here: ``game`` (a
view-model dict; ``<side>_id``, ``<side>_abbr``, ``<side>_logo_path`` and
``<side>_logo_url`` for the crests, ``id`` for the confetti seed),
``scored_side`` (``"away"`` or ``"home"``), ``away_score``, ``home_score``,
``phrase``, ``started_at`` (a ``time.time()`` value) and ``motif``
(``"score"``, ``"kick"``, ``"touchdown"``, ``"net"`` or ``"win"``; anything
else draws the ``"score"`` diagonals). The drawing caches what it derives in
the same dict, under ``_palette``, ``_backdrop``, ``_confetti`` and
``_crests``, so each is worked out once per celebration.
WHAT A HOST MUST PROVIDE
------------------------
Derived by walking every ``self.<attr>`` the mixin reads; the host-contract
test in ``test/test_sports_celebration.py`` fails if a read is added without
being listed here. All five scoreboards' ``SportsLive`` provide them.
- ``display_manager`` -- ``image`` is replaced with the frame, then
``update_display()``; ``clear()`` on ``force_clear``. Its ``matrix``
width and height are used when it has a matrix, else ``display_width`` /
``display_height``.
- ``fonts`` -- ``"time"`` and ``"status"`` for the headline (the first that
fits), ``"score"`` for the score.
- ``logger``.
- ``_load_and_resize_logo(team_id, abbr, logo_path, logo_url)`` -- a crest
as an RGBA image, or ``None``.
- ``_draw_text_with_outline(draw, text, position, font, fill=...)`` -- on
``SportsCoreSharedMixin``.
- ``celebration_duration``, ``celebration_team_colors`` and
``celebration_confetti``, read with ``getattr`` (defaults 8, on, on).
Mix it in ahead of the mode classes, e.g.
``class SportsLive(SportsCelebrationMixin, SportsLiveSharedMixin,
SportsCore)``. It defines nothing any of them define, so the order only
matters for a plugin that keeps its own copy of one of these methods: a
method on the plugin's class always wins over the mixin's.
"""
import colorsys
import logging
import math
import random
import time
from typing import Any, Callable, ClassVar, Dict, List, Optional, Sequence, Tuple
from PIL import Image, ImageDraw
#: A colour as the helpers return it: three 0-255 channels.
Color = Tuple[int, ...]
#: ``deep``, ``glow``, ``headline`` and ``accent``; see ``logo_palette``.
Palette = Dict[str, Color]
#: One confetti flake: column, start height, fall speed, sway phase, size
#: in pixels, colour.
Flake = Tuple[float, float, float, float, int, Color]
# ----------------------------------------------------------------------
# Colour helpers for the score/win celebration
#
# Module level rather than methods: they are pure, which is what makes the
# palette testable without standing up a live manager, and they are shared by
# the takeover's backdrop, confetti and text.
# ----------------------------------------------------------------------
#: The crest is sampled at this resolution. Big enough that a secondary
#: colour survives (a helmet stripe, a trim), small enough that the whole
#: sample is ~1600 pixels of pure-Python work, once per team.
_PALETTE_SAMPLE_PX = 40
#: Above this, a colour carries team identity; below it, it is a grey.
_PALETTE_VIVID_SATURATION = 0.22
#: Ignore pixels this dark -- crest outlines, drop shadows, anti-aliasing.
_PALETTE_MIN_CHANNEL = 24
#: How far apart two bins must be to count as a second, different colour.
_PALETTE_DISTINCT_DISTANCE = 90.0
#: Never bleed a lifted colour below this saturation; past it a hue stops
#: being the team's colour and starts being a pastel.
_PALETTE_MIN_SATURATION = 0.42
#: Lift a headline colour until it is at least this luminous. Chosen so
#: midnight navy reaches a blue that reads at 6px on a panel without
#: becoming a different colour.
_PALETTE_HEADLINE_LUMINANCE = 112.0
#: A crest colour this luminous already reads on a panel, so it is preferred
#: over a darker one that would have to be lifted to get there. Lifting is a
#: compromise -- Green Bay's dark green only reaches legibility as a teal --
#: and most teams whose primary is dark carry a bright second colour that is
#: just as much theirs. This is what picks the Packers' gold over that teal.
_PALETTE_LEGIBLE_LUMINANCE = 90.0
#: ...but only from a colour the crest actually means. The pixels where a
#: bright edge is anti-aliased into a dark fill are luminous too, and there is
#: always a band of them: Kansas City's white-on-red outline leaves a pink at
#: luminance 90 that would otherwise be preferred over the red itself. A blend
#: is a mix, so it is markedly less saturated than either colour it sits
#: between -- that pink is 0.48 where the red is 0.96 and the Packers' gold,
#: which this must keep, is 0.89.
_PALETTE_LEGIBLE_SATURATION = 0.65
#: And it has to be a band of the crest, not a speck of one.
_PALETTE_LEGIBLE_AREA = 0.02
#: Cap the backdrop's luminance so the headline stays legible over it,
#: and the scenery's so it stays behind the headline. Both are luminance and
#: not HSV value on purpose: a silver crest -- the Raiders, or the grey
#: placeholder a failed logo download leaves behind -- has a value of ~0.95,
#: and capping that at 0.34 still yields a light grey card that white text
#: then vanishes into. Scaling the channels down is also hue-exact, which is
#: what lets this be the plain arithmetic that lifting a colour cannot be.
_PALETTE_BACKDROP_LUMINANCE = 34.0
_PALETTE_SCENERY_LUMINANCE = 70.0
def rgb_luminance(color: Sequence[float]) -> float:
"""Rec. 709 relative luminance, 0-255."""
return 0.2126 * color[0] + 0.7152 * color[1] + 0.0722 * color[2]
def rgb_saturation(color: Sequence[float]) -> float:
"""HSV saturation, 0-1."""
high = max(color)
return (high - min(color)) / high if high else 0.0
def color_distance(a: Sequence[float], b: Sequence[float]) -> float:
"""Euclidean distance between two colours in RGB."""
return math.sqrt(sum((x - y) ** 2 for x, y in zip(a, b)))
def mix_color(a: Sequence[float], b: Sequence[float], t: float) -> Color:
"""Blend ``a`` towards ``b``; t=0 is all a, t=1 is all b."""
t = min(max(t, 0.0), 1.0)
return tuple(int(round(a[i] + (b[i] - a[i]) * t)) for i in range(3))
def scale_color(color: Sequence[float], factor: float) -> Color:
"""Scale a colour's brightness, clamped to the panel's range."""
return tuple(min(255, max(0, int(round(c * factor)))) for c in color)
def lift_color(color: Sequence[float], min_luminance: float = _PALETTE_HEADLINE_LUMINANCE,
cap_saturation: float = 0.92) -> Color:
"""Raise a colour's brightness until it reads on a panel, keeping its hue.
Scaling the channels directly is what the obvious version of this does,
and it shifts hue badly on exactly the colours that need lifting: it turns
Baltimore's navy-purple into magenta. Working in HSV and raising only the
value leaves the hue where the team put it.
"""
if rgb_luminance(color) >= min_luminance:
return tuple(int(c) for c in color)
hue, saturation, value = colorsys.rgb_to_hsv(*[c / 255.0 for c in color])
if saturation < 0.12:
# A grey or a silver has no hue to preserve; just make it bright.
lifted = colorsys.hsv_to_rgb(hue, saturation, max(value, 0.85))
return tuple(int(round(c * 255)) for c in lifted)
saturation = min(saturation, cap_saturation)
def _rgb(s: float, v: float) -> Color:
return tuple(int(round(c * 255)) for c in colorsys.hsv_to_rgb(hue, s, v))
out = _rgb(saturation, value)
while value < 1.0 and rgb_luminance(out) < min_luminance:
value = min(1.0, value + 0.05)
out = _rgb(saturation, value)
# Blue carries almost no luminance -- pure blue sits at 18 of 255 -- so a
# navy or a deep purple runs out of value long before it is legible.
# Bleeding saturation out of it is the only way up, and it keeps the hue
# (Baltimore stays purple, just a lighter one) where giving up would
# leave the headline unreadable. Floored so it never washes out to white.
while saturation > _PALETTE_MIN_SATURATION and rgb_luminance(out) < min_luminance:
saturation = max(_PALETTE_MIN_SATURATION, saturation - 0.05)
out = _rgb(saturation, value)
return out
def cap_luminance(color: Sequence[float], max_luminance: float) -> Color:
"""Darken a colour until it is no brighter than ``max_luminance``.
A straight channel scale, which is exactly hue-preserving on the way down
-- unlike lifting, where clamping at 255 is what bends the hue.
"""
luminance = rgb_luminance(color)
if luminance <= max_luminance or luminance <= 0:
return tuple(int(c) for c in color)
return scale_color(color, max_luminance / luminance)
def dim_rgba(image: Image.Image, factor: float) -> Image.Image:
"""Scale an RGBA image's colour channels, leaving its alpha alone.
ImageEnhance.Brightness would scale the alpha band too, which fades the
crest out instead of dimming it and leaves its anti-aliased edge looking
chewed against the backdrop.
"""
red, green, blue, alpha = image.split()
lut = [min(255, int(i * factor)) for i in range(256)]
return Image.merge(
"RGBA", (red.point(lut), green.point(lut), blue.point(lut), alpha)
)
_Buckets = Dict[Tuple[int, int, int], List[int]]
def _palette_buckets(logo: Image.Image) -> Tuple[_Buckets, _Buckets]:
"""Bucket a crest's opaque pixels into coarse colour bins.
Returns ``(vivid, neutral)``; each maps a 3-bit-per-channel key to
``[r_sum, g_sum, b_sum, count]``. Neutral holds the greys, silvers and
whites that carry no identity on their own but are all a monochrome crest
-- the Raiders' silver on black -- has to offer.
"""
sample = logo.convert("RGBA")
sample.thumbnail((_PALETTE_SAMPLE_PX, _PALETTE_SAMPLE_PX), Image.Resampling.BOX)
vivid: _Buckets = {}
neutral: _Buckets = {}
# tobytes() rather than getdata(): same pixels, no per-pixel Python
# object, and getdata() is deprecated from Pillow 14.
raw = sample.tobytes()
for i in range(0, len(raw) - 3, 4):
red, green, blue, alpha = raw[i], raw[i + 1], raw[i + 2], raw[i + 3]
if alpha < 160:
continue
high, low = max(red, green, blue), min(red, green, blue)
if high < _PALETTE_MIN_CHANNEL:
continue
target = vivid if (high - low) / high >= _PALETTE_VIVID_SATURATION else neutral
acc = target.setdefault((red >> 5, green >> 5, blue >> 5), [0, 0, 0, 0])
acc[0] += red
acc[1] += green
acc[2] += blue
acc[3] += 1
return vivid, neutral
def _bucket_mean(acc: List[int]) -> Color:
count = acc[3]
return (acc[0] // count, acc[1] // count, acc[2] // count)
def _bucket_headline_score(acc: List[int]) -> float:
"""How well a colour bin would serve as 6px of text on a panel.
Area alone picks the biggest block of colour, which on a lot of crests is
a dark navy fill -- correct as a backdrop, invisible as text. Weighting
area by saturation and by luminance picks the colour the team is loud in:
Chicago's orange over its navy, Baltimore's gold over its purple.
"""
color = _bucket_mean(acc)
return (
acc[3]
* (0.30 + 0.70 * rgb_saturation(color))
* (0.20 + 0.80 * min(1.0, rgb_luminance(color) / 120.0))
)
def logo_palette(logo: Image.Image) -> Optional[Palette]:
"""Pick a celebration palette out of a team crest, or None.
Two rankings, because a crest's largest colour and its most legible one
are usually not the same and the takeover needs both:
* ``deep`` -- the largest vivid area, darkened into the background wash.
This is what the team reads as at a glance: Chicago navy, Dallas navy,
Baltimore purple.
* ``headline`` -- the vivid area that best survives being shrunk to text,
then lifted until it is legible: Chicago orange, Baltimore gold.
* ``accent`` -- the next vivid colour far enough away from the headline to
be told apart, for confetti. Falls back to the headline.
A crest with no vivid pixels at all falls back to its brightest neutral,
which for the Raiders' silver-on-black is exactly the right answer.
"""
try:
vivid, neutral = _palette_buckets(logo)
except Exception: # noqa: BLE001 - a crest is never worth the takeover
return None
pool = list(vivid.values())
if not pool and neutral:
pool = [
max(
neutral.values(),
key=lambda acc: acc[3]
* (0.2 + 0.8 * min(1.0, rgb_luminance(_bucket_mean(acc)) / 160.0)),
)
]
if not pool:
return None
deep_base = _bucket_mean(max(pool, key=lambda acc: acc[3]))
ranked = sorted(pool, key=_bucket_headline_score, reverse=True)
headline_base = _bucket_mean(ranked[0])
vivid_pixels = sum(acc[3] for acc in pool)
for acc in ranked:
candidate = _bucket_mean(acc)
if (
rgb_luminance(candidate) >= _PALETTE_LEGIBLE_LUMINANCE
and rgb_saturation(candidate) >= _PALETTE_LEGIBLE_SATURATION
and acc[3] >= max(3, vivid_pixels * _PALETTE_LEGIBLE_AREA)
):
headline_base = candidate
break
headline = lift_color(headline_base)
accent = headline
for acc in ranked[1:]:
candidate = _bucket_mean(acc)
if color_distance(candidate, headline_base) > _PALETTE_DISTINCT_DISTANCE:
accent = lift_color(candidate)
break
deep = cap_luminance(deep_base, _PALETTE_BACKDROP_LUMINANCE)
return {
"deep": deep,
# Scenery is the backdrop carried a little way towards the headline:
# tied to the team's colours, and guaranteed to be visible even when
# the backdrop is nearly black.
"glow": cap_luminance(
mix_color(deep, headline, 0.22), _PALETTE_SCENERY_LUMINANCE
),
"headline": headline,
"accent": accent,
}
class SportsCelebrationMixin:
"""Draws a score/win celebration takeover. See the module docstring."""
# The host contract, declared for type checking only: these create no
# attributes, so the host's own values are what the methods read.
display_manager: Any
display_width: int
display_height: int
fonts: Dict[str, Any]
logger: logging.Logger
_load_and_resize_logo: Callable[..., Optional[Image.Image]]
_draw_text_with_outline: Callable[..., None]
def _fit_font(self, draw, text: str, max_width: int, fonts: list):
"""Return the first font whose rendered ``text`` fits ``max_width``,
falling back to the last (smallest) font."""
for font in fonts:
if draw.textlength(text, font=font) <= max_width - 2:
return font
return fonts[-1]
# ------------------------------------------------------------------
# Celebration palette
#
# The takeover is drawn in the scoring team's own colours, taken from the
# pixels of its crest.
#
# ESPN does serve team.color / team.alternateColor, but only inside
# _extract_game_details_common -- a function each scoreboard lineage
# keeps its own copy of -- so reading it there would drag every one of
# them into a celebration change. The crest is already downloaded,
# decoded and sitting in the logo cache by the time a celebration draws,
# so the colours come from it instead: no extra request, no per-league
# colour table to maintain, and it works for any team ESPN can name --
# including the FCS opponents no table would list.
#
# Where a crest's colour differs from the club's published one it
# tends to differ usefully: a published primary is often a near-black
# navy, or an actual #000000, where what the crest carries is the colour
# that reads on an LED panel. Measured across all 32 clubs in
# football-scoreboard.
# ------------------------------------------------------------------
#: Used when the crest yields nothing (no logo on disk yet, or the grey
#: placeholder a failed download leaves) or team colours are switched
#: off -- the navy and amber the celebration wore before it had a palette.
_DEFAULT_CELEBRATION_PALETTE: ClassVar[Palette] = {
"deep": (10, 10, 40),
"glow": (30, 30, 86),
"headline": (255, 208, 56),
"accent": (255, 255, 255),
}
def _celebration_palette(self, celebration: Dict) -> Palette:
"""The scoring team's colours, derived once per celebration."""
cached: Optional[Palette] = celebration.get("_palette")
if cached is not None:
return cached
palette = dict(self._DEFAULT_CELEBRATION_PALETTE)
if getattr(self, "celebration_team_colors", True):
try:
game = celebration["game"]
side = celebration.get("scored_side") or "home"
logo = self._load_and_resize_logo(
game.get("%s_id" % side),
game.get("%s_abbr" % side),
game.get("%s_logo_path" % side),
game.get("%s_logo_url" % side),
)
derived = logo_palette(logo) if logo is not None else None
if derived:
palette = derived
except Exception as e: # noqa: BLE001 - never lose a takeover to a crest
self.logger.debug(f"Celebration palette fell back to the default: {e}")
celebration["_palette"] = palette
return palette
# ------------------------------------------------------------------
# Celebration choreography
#
# Every frame is a finished card. The beats below shift the emphasis --
# an opening colour hit, confetti, a breathing score -- but none of them
# leaves the panel mid-wipe, because on a switch-mode board the core
# drives this plugin at 1 FPS (display_controller reserves its high-FPS
# loop for plugins that scroll or declare needs_high_fps), so any single
# frame may be the only one a viewer ever sees of it.
# ------------------------------------------------------------------
#: Fraction of the celebration spent on the opening colour hit.
_CELEBRATION_IMPACT: ClassVar[float] = 0.11
#: Fraction of it after which the takeover eases back down.
_CELEBRATION_SETTLE: ClassVar[float] = 0.80
#: Seconds per breath of the scoring side's digits. Deliberately a
#: continuous sine rather than an on/off toggle: the 4 Hz flash this
#: replaced was sampled once a second on a switch-mode board, which
#: aliases into a colour that changes at random. A ramp degrades into a
#: slow glow instead, and still reads as a pulse at 125 FPS.
_CELEBRATION_BREATH_SECONDS: ClassVar[float] = 1.7
def _celebration_backdrop(
self,
celebration: Dict,
width: int,
height: int,
palette: Palette,
) -> Image.Image:
"""The static half of the takeover: a team-colour gradient with the
scenery for this kind of score painted into it.
Built once per celebration per panel size and copied per frame, so the
per-pixel work never lands on the render path.
"""
cached: Optional[Tuple[Tuple[int, int], Image.Image]] = celebration.get("_backdrop")
if cached is not None and cached[0] == (width, height):
return cached[1]
# One column, then stretched: filling the panel pixel by pixel would
# be `width` times the work for the same image.
column = Image.new("RGB", (1, max(height, 1)))
pixels: Any = column.load()
for y in range(height):
k = y / max(height - 1, 1)
pixels[0, y] = mix_color(palette["deep"], (0, 0, 0), 0.18 + 0.82 * k)
backdrop = column.resize((width, height)).convert("RGBA")
try:
self._draw_celebration_motif(
ImageDraw.Draw(backdrop),
celebration.get("motif") or "score",
width,
height,
palette,
)
except Exception as e: # noqa: BLE001 - scenery is never worth a blank panel
self.logger.debug(f"Celebration motif skipped: {e}")
celebration["_backdrop"] = ((width, height), backdrop)
return backdrop
def _draw_celebration_motif(
self,
draw,
motif: str,
width: int,
height: int,
palette: Palette,
) -> None:
"""Paint the scenery for one kind of score, dim enough to stay behind
the headline and the score instead of competing with them."""
glow = palette["glow"]
if motif == "kick":
# The uprights a field goal or an extra point went through,
# spread wide enough to frame the score rather than sit beside it.
half = max(8, min(width // 3, height))
mid = width // 2
crossbar = int(height * 0.60)
draw.line([(mid - half, int(height * 0.08)), (mid - half, crossbar)], fill=glow)
draw.line([(mid + half, int(height * 0.08)), (mid + half, crossbar)], fill=glow)
draw.line([(mid - half, crossbar), (mid + half, crossbar)], fill=glow)
draw.line([(mid, crossbar), (mid, height - 1)], fill=glow)
elif motif == "touchdown":
# The goal line, with its hash marks.
line_y = int(height * 0.36)
draw.line([(0, line_y), (width, line_y)], fill=glow)
for x in range(3, width, 9):
draw.line([(x, line_y - 2), (x, line_y + 2)], fill=glow)
elif motif == "net":
# The goal a puck just went into: frame, posts and mesh, sized to
# frame the score the way the uprights do.
half = max(7, min(width // 4, height))
mid = width // 2
top = int(height * 0.34)
draw.rectangle([(mid - half, top), (mid + half, height - 1)], outline=glow)
step = max(3, (half * 2) // 6)
for x in range(mid - half + step, mid + half, step):
draw.line([(x, top + 1), (x, height - 2)], fill=glow)
for y in range(top + step, height - 1, step):
draw.line([(mid - half + 1, y), (mid + half - 1, y)], fill=glow)
elif motif == "win":
# A sunburst behind the winner.
cx, cy = width // 2, height // 2
reach = max(width, height)
for i in range(10):
angle = (math.pi * 2 * i / 10) + math.pi / 20
draw.line(
[
(cx, cy),
(cx + math.cos(angle) * reach, cy + math.sin(angle) * reach),
],
fill=glow,
)
else:
for x in range(-height, width + height, 11):
draw.line([(x, height), (x + height, 0)], fill=glow)
def _celebration_confetti(
self,
celebration: Dict,
width: int,
height: int,
palette: Palette,
) -> List[Flake]:
"""Seed the confetti once per celebration.
Seeded from the game rather than the clock, so the same score always
produces the same fall -- which is what lets a golden screen lock the
effect down instead of having to tolerate it.
"""
cached: Optional[Tuple[Tuple[int, int], List[Flake]]] = celebration.get("_confetti")
if cached is not None and cached[0] == (width, height):
return cached[1]
# Sparse on purpose. At one flake per 170 square pixels a 128x32
# panel carried 24 single-pixel specks over the headline and the
# score, which reads as a dead-pixel problem rather than as confetti.
count = max(6, min(22, (width * height) // 260))
seed = "%s/%s" % (
(celebration.get("game") or {}).get("id", "?"),
celebration.get("phrase", ""),
)
rng = random.Random(seed) # nosec B311 - confetti, not security
# Team colours, plus a pale tint of the headline rather than a flat
# white, so the fall still belongs to the team that scored.
colors = [
palette["headline"],
palette["accent"],
mix_color(palette["headline"], (255, 255, 255), 0.55),
]
flakes = [
(
float(rng.randrange(max(width, 1))), # column
rng.uniform(0.0, float(height)), # start height
rng.uniform(0.40, 1.15), # fall speed
rng.uniform(0.0, math.pi * 2), # sway phase
2 if rng.random() < 0.6 else 1, # size in pixels
colors[rng.randrange(len(colors))],
)
for _ in range(count)
]
celebration["_confetti"] = ((width, height), flakes)
return flakes
def _draw_celebration_confetti(
self,
draw,
celebration: Dict,
width: int,
height: int,
palette: Palette,
elapsed: float,
progress: float,
) -> None:
"""Draw the confetti for this instant, thinning it out as the
celebration eases back towards the scorebug."""
flakes = self._celebration_confetti(celebration, width, height, palette)
fade = 1.0
if progress > self._CELEBRATION_SETTLE:
fade = max(
0.0,
1.0
- (progress - self._CELEBRATION_SETTLE)
/ (1.0 - self._CELEBRATION_SETTLE),
)
if fade <= 0.02:
return
alpha = int(235 * fade)
for column, start, speed, phase, size, color in flakes:
y = (start + speed * elapsed * height * 0.42) % (height + 4) - 2
x = column + math.sin(elapsed * 2.1 + phase) * 2.4
draw.rectangle(
[(int(x), int(y)), (int(x) + size - 1, int(y) + size - 1)],
fill=tuple(color) + (alpha,),
)
def _celebration_crests(
self, celebration: Dict, height: int
) -> Dict[str, Optional[Image.Image]]:
"""The two crests for the takeover, with the side that did not score
dimmed so the scoring team reads at a glance."""
cached: Optional[Tuple[int, Dict[str, Optional[Image.Image]]]] = celebration.get("_crests")
if cached is not None and cached[0] == height:
return cached[1]
game = celebration["game"]
scored = celebration.get("scored_side")
crests: Dict[str, Optional[Image.Image]] = {}
for side in ("away", "home"):
logo = None
try:
logo = self._load_and_resize_logo(
game.get("%s_id" % side),
game.get("%s_abbr" % side),
game.get("%s_logo_path" % side),
game.get("%s_logo_url" % side),
)
except Exception as e: # noqa: BLE001 - a crest is never worth the panel
self.logger.debug(f"Celebration logo load failed: {e}")
if logo is not None and side != scored:
logo = dim_rgba(logo, 0.40)
crests[side] = logo
celebration["_crests"] = (height, crests)
return crests
def _draw_celebration_layout(self, celebration: Dict, force_clear: bool = False) -> None:
"""Render the full-screen goal/win takeover."""
if force_clear:
self.display_manager.clear()
display_width = (
self.display_manager.matrix.width
if hasattr(self.display_manager, "matrix") and self.display_manager.matrix
else self.display_width
)
display_height = (
self.display_manager.matrix.height
if hasattr(self.display_manager, "matrix") and self.display_manager.matrix
else self.display_height
)
elapsed = max(0.0, time.time() - celebration["started_at"])
# getattr throughout the render path: the golden-screen tests build a
# live manager through __new__ and set only what they draw with, and a
# celebration must never be lost to a missing knob.
duration = max(float(getattr(self, "celebration_duration", 8) or 8), 0.5)
progress = min(elapsed / duration, 1.0)
palette = self._celebration_palette(celebration)
main_img = self._celebration_backdrop(
celebration, display_width, display_height, palette
).copy()
# Crests at the edges, bleeding off as the scorebug's do.
crests = self._celebration_crests(celebration, display_height)
center_y = display_height // 2
home_logo, away_logo = crests.get("home"), crests.get("away")
if home_logo is not None:
main_img.paste(
home_logo,
(display_width - home_logo.width + 2, center_y - home_logo.height // 2),
home_logo,
)
if away_logo is not None:
main_img.paste(away_logo, (-2, center_y - away_logo.height // 2), away_logo)
# The opening hit: the team's headline colour washes the panel and
# decays out of it. Held below opaque so the outlined text drawn on
# top still reads in whichever frame happens to catch it.
impact = max(0.0, 1.0 - progress / self._CELEBRATION_IMPACT)
if impact > 0.0:
# Scaled by how colourful the team is. A saturated crest gets the
# full hit; a silver one -- the Raiders, or the grey placeholder a
# failed logo download leaves -- would otherwise wash the whole
# panel out to the same flat grey as its own headline colour.
punch = 0.45 + 0.55 * rgb_saturation(palette["headline"])
alpha = int(140 * punch * (impact ** 1.5))
if alpha > 0:
main_img = Image.alpha_composite(
main_img,
Image.new(
"RGBA",
(display_width, display_height),
tuple(palette["headline"]) + (alpha,),
),
)
overlay = Image.new("RGBA", (display_width, display_height), (0, 0, 0, 0))
draw = ImageDraw.Draw(overlay)
if getattr(self, "celebration_confetti", True):
try:
self._draw_celebration_confetti(
draw,
celebration,
display_width,
display_height,
palette,
elapsed,
progress,
)
except Exception as e: # noqa: BLE001
self.logger.debug(f"Celebration confetti skipped: {e}")
# Headline across the top, shrunk to fit the panel width, struck
# white on the opening hit and settling into the team's colour.
phrase = celebration["phrase"]
phrase_font = self._fit_font(
draw, phrase, display_width, [self.fonts["time"], self.fonts["status"]]
)
phrase_width = draw.textlength(phrase, font=phrase_font)
# Eased in by colour rather than by position. Sliding it down into
# place put the first frame at y=-3 with its top row cut off, and on a
# 1 FPS board that clipped frame can be the only one anyone sees.
self._draw_text_with_outline(
draw,
phrase,
((display_width - phrase_width) // 2, 1),
phrase_font,
fill=mix_color(palette["headline"], (255, 255, 255), impact),
)
# Score centred low, the scoring side's digits breathing in the team's
# headline colour so the change reads at a glance.
away_text = str(celebration["away_score"])
home_text = str(celebration["home_score"])
score_font = self.fonts["score"]
segments = [
(away_text, celebration["scored_side"] == "away"),
("-", False),
(home_text, celebration["scored_side"] == "home"),
]
total_width = sum(draw.textlength(seg, font=score_font) for seg, _ in segments)
breath = 0.72 + 0.28 * (
0.5
+ 0.5 * math.sin(2 * math.pi * elapsed / self._CELEBRATION_BREATH_SECONDS)
)
highlight = scale_color(palette["headline"], breath)
x = (display_width - total_width) // 2
# display_height - 14 was sized for the old fixed 8px score. #338
# scales the score with the panel (16px at 48 and 64 tall), which put
# the bottom of the digits off the panel. Lift it by the measured ink
# (+1 for the outline stroke) only when it would clip, so panels where
# it always fitted render exactly as before.
score_text = "".join(seg for seg, _ in segments)
ink_bottom = draw.textbbox((0, 0), score_text, font=score_font)[3]
y = min(display_height - 14, display_height - ink_bottom - 2)
for seg, is_highlight in segments:
color = highlight if is_highlight else (216, 216, 216)
self._draw_text_with_outline(draw, seg, (int(x), y), score_font, fill=color)
x += draw.textlength(seg, font=score_font)
main_img = Image.alpha_composite(main_img, overlay).convert("RGB")
self.display_manager.image = main_img
self.display_manager.update_display()
+164
View File
@@ -0,0 +1,164 @@
"""Which games a scoreboard shows, for how long, and what its scorebug dates say.
Four ``sports.py`` methods are identical (executable AST, docstrings
stripped, decorators compared) in every scoreboard that carries them, and
were copied here from ledmatrix-plugins ``56c4f15`` (origin/main,
2026-09-30) under their existing names. They split into two mixins because
their carriers differ, and a plugin should not gain an override it did not
have:
``SportsCardOptionsMixin`` -- afl, baseball, basketball, football, hockey,
lacrosse, nrl and soccer (ufc draws no team scorebug):
- ``_card_option`` -- reads one ``scroll_card`` key through
``SportsCoreSharedMixin._card_option``, but never lets the upcoming
scorebug lose both its date and its time (the combination a settings-form
bug saved for a whole cohort of boards);
- ``_recent_date_text`` -- the date line of the full-screen recent scorebug.
``SportsGameRulesMixin`` -- all nine:
- ``_filtered_or_all`` (all but football, which has no such method) -- the
quality and division filters on a board with no favourites, failing open
to every game rather than a blank panel;
- ``_effective_live_duration`` (all but ufc, which has none) -- how long a
live game stays up: ``non_favorite_live_game_duration`` for a
non-favourite when favourites are set, else ``game_display_duration``.
afl, nrl and soccer carry it on ``SportsCore``, the other five on
``SportsLive``; the bodies are the same.
The plugin missing a method gains one it never calls, which changes nothing:
nothing in that plugin, nor in core, calls it.
A new module rather than more methods on ``sports_shared``, for the reason
``sports_helpers`` gives: a missing module fails at load, where the version
checks see it; a missing method fails mid-update.
WHAT A HOST MUST PROVIDE
------------------------
Derived by walking every ``self.<attr>`` the mixins read; the host-contract
test in ``test/test_sports_display_rules.py`` fails if a read is added
without being listed here.
``SportsCardOptionsMixin``:
- ``SportsCoreSharedMixin`` (``src.common.sports_shared``) in the MRO
**after** this mixin: ``_card_option`` calls that mixin's ``_card_option``
and ``_switch_upcoming_center`` by name, and ``_recent_date_text`` its
``_format_game_date``. List this mixin first --
``class SportsCore(SportsCardOptionsMixin, SportsGameRulesMixin,
SportsFetchMixin, SportsCoreSharedMixin, SportsHelpersMixin, ABC)`` --
or ``SportsCoreSharedMixin._card_option`` wins and the rescue is lost.
Through it: ``config`` (the ``scroll_card`` block it reads).
``SportsGameRulesMixin``:
- ``_passes_other_filters(game)`` -- the plugin's own quality/division
filter (``_filtered_or_all``).
- ``_check_ranking_coverage(games)`` -- from ``SportsCoreSharedMixin``.
- ``favorite_teams``, ``game_display_duration`` and
``_is_favorite_game(game)``; ``non_favorite_live_game_duration`` read with
``getattr`` (``_effective_live_duration``).
Neither mixin has an ``__init__`` or state. A method on the plugin's own
class still wins over either.
"""
from typing import Any, Callable, Dict, List, Optional
from src.common.sports_shared import SportsCoreSharedMixin
class SportsCardOptionsMixin:
"""The scorebug's ``scroll_card`` reads. See module docstring."""
# The host contract, declared for type checking only.
_format_game_date: Callable[..., str]
def _card_option(self, key: str, default: Any = None) -> Any:
"""Read one scroll_card key, never blanking the upcoming scorebug.
With the middle set to "date and time" and both of those lines
switched off, the full-screen upcoming scorebug is two logos and
"Next Game" with nothing to say when the game is. Nobody picks that
on purpose -- "vs" and "none" are the settings for a card without the
stack -- yet a whole cohort of boards has it: switch_show_date/_time
shipped while the core's settings form still drew keys missing from
the saved config as unchecked boxes, so the next Save wrote both as
false (fixed in LEDMatrix #597). That one combination therefore reads
as both on. Hiding either line alone, or both under "vs" or "none",
is still honoured.
"""
# The mixin named outright, not super(): tests lift this method onto
# stand-in classes that are not SportsCore subclasses.
base = SportsCoreSharedMixin._card_option
keys = ("switch_show_date", "switch_show_time")
value = base(self, key, default) # type: ignore[arg-type]
if (key in keys and not value
and not any(base(self, k, True) for k in keys) # type: ignore[arg-type]
and SportsCoreSharedMixin._switch_upcoming_center(self) == "date_time"): # type: ignore[arg-type]
return True
return value
def _recent_date_text(self, game: Optional[Dict]) -> str:
"""When a finished game was played, for the full-screen scorebug.
Formatted by switch_date_format, like the upcoming scorebug, so the
two dates on this display agree; its "numeric" default returns the
extractor's "9/23" unchanged. ``switch_recent_show_date`` (default
true) is the off switch.
"""
if not self._card_option("switch_recent_show_date", True):
return ""
return self._format_game_date(str((game or {}).get("game_date") or ""), game)
class SportsGameRulesMixin:
"""Which games are worth showing, and for how long. See module docstring."""
# The host contract, declared for type checking only.
favorite_teams: List[str]
game_display_duration: float
_passes_other_filters: Callable[[Dict], bool]
_check_ranking_coverage: Callable[[List[Dict]], None]
_is_favorite_game: Callable[[Dict], bool]
def _filtered_or_all(self, games: List[Dict]) -> List[Dict]:
"""The games worth watching, or all of them if that leaves none.
With no favourites configured every game selected is a non-favourite
game, so the quality and division settings have to apply here too. They
governed only the top-up slice, which this branch never uses, so a
board with an empty favourites list had both settings silently inert --
it could ask for ranked games only and still get the next N kickoffs.
Fails open as a whole, not just per check. `_passes_other_filters`
allows a game whose data could not be resolved, but a filter working
exactly as asked can still match nothing on a given day, and here there
is no favourite left to carry the mode -- an empty list is a blank
panel rather than a short one.
"""
kept = [g for g in games if self._passes_other_filters(g)]
self._check_ranking_coverage(games)
return kept or games
def _effective_live_duration(self, game) -> float:
"""How long the given live game should stay on screen before rotating.
Non-favorite live games use non_favorite_live_game_duration, but only
when it is set (> 0) AND favorite teams are configured. With no favorites
(or the knob at 0) every live game uses game_display_duration - identical
to the prior single-duration behavior. When show_favorite_teams_only is
on, non-favorite games are never shown, so this naturally never fires."""
non_fav = getattr(self, "non_favorite_live_game_duration", 0) or 0
if (
non_fav > 0
and self.favorite_teams
and game is not None
and not self._is_favorite_game(game)
):
return non_fav
return self.game_display_duration
__all__ = ["SportsCardOptionsMixin", "SportsGameRulesMixin"]
+188
View File
@@ -0,0 +1,188 @@
"""Which requests a scoreboard makes: season fetches, the lookback, live odds.
Four ``SportsCore`` methods are identical (executable AST, docstrings
stripped) in all nine scoreboards' ``sports.py`` -- afl, baseball,
basketball, football, hockey, lacrosse, nrl, soccer and ufc -- and were
copied here from ledmatrix-plugins ``30455671`` (origin/main, 2026-09-29)
under their existing names:
- ``_background_fetches_espn_ranges`` -- whether the core's background
service can fetch an ESPN date range, or the plugin must;
- ``_fetch_season_directly`` -- fetch and cache a season in chunks ESPN
accepts, on the calling thread;
- ``_needs_previous_day`` (with ``_LOOKBACK_CUTOFF_HOUR``) -- whether the
live fetch still has to ask for yesterday;
- ``_wants_live_odds`` (with ``_LIVE_ODDS_LOOKAHEAD``) -- whether a live
game is close enough to the screen to be worth an odds request.
Three other ``SportsCore`` methods are as identical and stay in the plugins,
for the reasons ``sports_shared`` gives: ``_get_timezone`` binds each
plugin's own ``resolve_timezone`` shim, and ``_extract_game_details`` /
``_fetch_data`` are the abstract sport-specific contract. So does
``SportsUpcoming.__init__``: the mixins in ``src/common`` hold no
constructor, so the plugins' constructor signature stays theirs.
A new module rather than more methods on ``sports_shared``, for the reason
``sports_helpers`` gives: a missing module fails at load, where the version
checks see it; a missing method fails mid-update.
WHAT A HOST MUST PROVIDE
------------------------
Derived by walking every ``self.<attr>`` the mixin reads; the host-contract
test in ``test/test_sports_fetch.py`` fails if a read is added without being
listed here.
- ``session``, ``headers``, ``cache_manager`` and ``logger`` --
``_fetch_season_directly``.
- ``_games_lock`` -- ``_wants_live_odds``, which also reads ``live_games``,
``current_game_index`` and ``_rotation_schedule`` with ``getattr``
(only ``SportsLive`` has them).
- ``live_games``, read with ``getattr`` -- ``_needs_previous_day``.
- ``background_service``, read with ``getattr`` --
``_background_fetches_espn_ranges``.
Add it as a base of the plugin's ``SportsCore``, e.g.
``class SportsCore(SportsFetchMixin, SportsCoreSharedMixin,
SportsHelpersMixin, ABC)``. It defines nothing those define; a method or
constant on the plugin's own class still wins over the mixin's.
"""
import logging
import threading
from datetime import datetime, timedelta
from typing import Any, ClassVar, Dict, Optional
from src.common.espn_dates import ESPN_MAX_LIMIT, fetch_espn_scoreboard
class SportsFetchMixin:
"""Season fetch, lookback and live-odds decisions. See module docstring."""
# The host contract, declared for type checking only: these create no
# attributes, so the host's own values are what the methods read.
session: Any
headers: Dict[str, str]
cache_manager: Any
logger: logging.Logger
_games_lock: threading.RLock
#: How many games past the one on screen keep their odds warm. One is
#: enough for the line to be ready when the rotation advances; more just
#: re-creates the whole-slate fetch this replaced.
_LIVE_ODDS_LOOKAHEAD: ClassVar[int] = 1
def _wants_live_odds(self, game: Dict) -> bool:
"""Whether a live game is near enough the front of the rotation to be
worth an odds request.
Odds used to be fetched for *every* live game in the league on every
update. The renderer only ever draws ``current_game``, and a full
rotation of a big slate takes minutes while ``live_odds_update_interval``
is 60s -- so all but one of those requests expired before the game they
belonged to came round.
Measured 2026-09-19 over a full college-football slate: 11,978 odds
requests in 13h on one rig, 54% of all its ESPN traffic, across only
~140 distinct games. The eager loop also cost up to 2s of ``update()``
per live game, because ``_fetch_odds`` waits on its worker thread.
Mirrors the narrowing already applied to the upcoming path and to
``_attach_odds_to_rotated_games``: only games about to be on screen are
asked about. ``get_odds`` still caches per game, so a game re-entering
the window inside its TTL costs a cache lookup, not a request.
The rotation state read here is the previous cycle's -- the new list is
still being built -- which is exactly the question being asked: is this
game at or near the position currently on the panel?
"""
# Read defensively: this predicate lives on SportsCore so it sits
# beside _fetch_odds, but live_games/_rotation_schedule belong to
# SportsLive, which is the only caller.
with self._games_lock:
games = list(getattr(self, "live_games", ()) or ())
index = getattr(self, "current_game_index", 0)
schedule = list(getattr(self, "_rotation_schedule", ()) or ())
if not games:
# Cold start: nothing is on screen yet, so let the games seen on
# this first pass through rather than render a blank line for a
# whole cycle. Bounded -- the next pass has a rotation to narrow by.
return True
order = schedule or [g.get("id") for g in games]
if not order:
return True
start = index if 0 <= index < len(order) else 0
wanted = {
order[(start + offset) % len(order)]
for offset in range(self._LIVE_ODDS_LOOKAHEAD + 1)
}
return game.get("id") in wanted
#: Hour of the Eastern day past which last night's games are assumed over.
#:
#: The live fetch asks ESPN for a two-day window so a game that started
#: yesterday and is still running is not lost. ESPN rejects date *ranges*,
#: so that window is split into one request per day -- doubling every live
#: poll. Measured 2026-09-19: 1,858 requests per rig spent on yesterday's
#: date, which after breakfast holds nothing but final games.
#:
#: No sport on these boards runs six hours past midnight, and one that
#: somehow did is still covered: a game already being tracked keeps its own
#: day in the window regardless of the hour.
_LOOKBACK_CUTOFF_HOUR: ClassVar[int] = 6
def _needs_previous_day(self, now: datetime) -> bool:
"""Whether the previous Eastern day can still hold a live game."""
if now.hour < self._LOOKBACK_CUTOFF_HOUR:
return True
previous = (now - timedelta(days=1)).strftime("%Y%m%d")
for game in (getattr(self, "live_games", None) or []):
start: Any = game.get("start_time_utc") if hasattr(game, "get") else None
try:
if start.astimezone(now.tzinfo).strftime("%Y%m%d") == previous:
return True
except (AttributeError, ValueError, OSError, OverflowError):
continue
return False
def _background_fetches_espn_ranges(self) -> bool:
"""Can the core's background service fetch an ESPN date range?
Cores from before the 2026-09-15 fix send a season range to ESPN as-is,
which now answers 400 for every sport. On those cores the managers fetch
the season themselves with _fetch_season_directly instead.
"""
service = getattr(self, "background_service", None)
return bool(getattr(service, "handles_espn_date_ranges", False))
def _fetch_season_directly(
self,
url: str,
datestring: str,
cache_key: str,
label: str,
ttl: Optional[int] = None,
) -> Optional[Dict]:
"""Fetch a season schedule on this thread, in chunks ESPN accepts, and cache it.
``label`` names the schedule in log lines, e.g. ``"2026 season"``.
"""
try:
data = fetch_espn_scoreboard(
self.session,
url,
params={"dates": datestring, "limit": ESPN_MAX_LIMIT},
headers=self.headers,
timeout=30,
logger=self.logger,
)
except Exception as e:
self.logger.error(f"Failed to fetch {label} schedule: {e}")
return None
if ttl is None:
self.cache_manager.set(cache_key, data)
else:
self.cache_manager.set(cache_key, data, ttl=ttl)
self.logger.info(
f"Fetched {label} schedule: {len(data.get('events', []))} events"
)
return data
+41
View File
@@ -0,0 +1,41 @@
"""Where a scoreboard's bundled font file is, whatever the working directory.
Every scoreboard's ``sports.py`` (nine) and ``game_renderer.py`` (eight)
carries the same module-level ``_resolve_font_path``. It predates
:func:`src.common.font_layout.resolve_asset_path`, and probes the core for
it: the path as given when it exists (relative to the cwd), else the core's
resolver (``FontManager._resolve_asset_path``, which delegates to
``resolve_asset_path``), else the path joined to the install root, else the
path unchanged so the caller's ``ImageFont.truetype`` raises and falls back
as before.
On every core this module ships in, the probe always finds the resolver, and
the install-root join repeats what the resolver already tried. What is left
is two steps, and :func:`resolve_font_path` is exactly those: the cwd first,
then ``resolve_asset_path``. ``test/test_sports_font_path.py`` checks that
against the plugins' own copies, path for path. It is the same rule as
``sports_shared._resolve_font_path``, made public so a plugin can import it.
Why not ``resolve_asset_path`` alone: it never consults the cwd, so a
process started from another checkout would switch to the install root's
fonts. Keeping the cwd first keeps that behaviour exactly.
"""
import os
from src.common.font_layout import resolve_asset_path
def resolve_font_path(path: str) -> str:
"""``path`` if it exists, else :func:`resolve_asset_path` of it.
Absolute paths that exist come back untouched; a relative path is tried
against the cwd, then the install root; a path found nowhere comes back
unchanged, so the caller still raises and falls back.
"""
if os.path.exists(path):
return path
return resolve_asset_path(path)
__all__ = ["resolve_font_path"]
+277
View File
@@ -0,0 +1,277 @@
"""Keep a live scoreboard's scrolling strip current without restarting it.
In scroll mode a scoreboard renders its games into one wide image and
scrolls it past the panel. The strip used to be rebuilt only when a cycle
completed, so a score changed mid-cycle stayed frozen in the pixels until the
marquee finished. Eight scoreboards -- afl, baseball, basketball, football,
hockey, lacrosse, nrl and soccer (ufc has no live strip) -- carry the same
fix in their ``manager.py``: fingerprint the live games, rebuild when the
fingerprint changes (rate-limited, and never for the clock alone), and keep
the marquee's position across the rebuild. Its eight methods and two class
constants are identical (executable AST, docstrings stripped, decorators
compared) in all eight and were copied here from ledmatrix-plugins
``56c4f15`` (origin/main, 2026-09-30) under their existing names:
- ``_live_scroll_managers`` -- the live managers whose games are on the strip;
- ``_refresh_live_scroll_managers`` -- let them refresh before they are
fingerprinted, off the render thread;
- ``_live_scroll_fields``, ``_fingerprint_games`` and
``_live_scroll_fingerprint`` -- what the strip was drawn from;
- ``_live_scroll_needs_rebuild`` (with ``LIVE_SCROLL_REBUILD_MIN_SECONDS``
and ``LIVE_SCROLL_REBUILD_DUTY_DIVISOR``) and ``_note_live_scroll_built``
-- when to rebuild;
- ``_preserving_scroll_position`` -- a context manager that keeps the marquee
where it was across a rebuild.
``LIVE_VOLATILE_FIELDS`` stays in each plugin: afl, nrl and soccer also
exclude ``period_text``, which embeds the clock in those sports.
A separate module from ``sports_plugin_host`` because ufc has no live strip:
it inherits that mixin and not this one, so none of this is in its MRO.
WHAT A HOST MUST PROVIDE
------------------------
Derived by walking every ``self.<attr>`` / ``cls.<attr>`` the mixin reads;
the host-contract test in ``test/test_sports_live_scroll.py`` fails if a read
is added without being listed here.
- ``LIVE_VOLATILE_FIELDS`` -- a class constant: the game-dict keys a rebuild
ignores (the clock, and what the display pipeline adds).
- ``_live_scroll_fingerprints``, ``_live_scroll_rebuilt_at`` and
``_live_scroll_rebuild_cost`` -- empty dicts the host creates in
``__init__``, keyed by scroll key.
- ``logger``.
- ``_dispatch_switch_refresh(manager)`` -- from ``SportsPluginHostMixin``.
- ``_league_registry`` (``{league: {"enabled": bool, "managers": {"live":
manager}}}``) or a ``_get_manager(mode_type)`` accessor, both read with
``getattr`` -- ``_live_scroll_managers``. A host with neither gets no
managers, which leaves the feature inert rather than wrong.
- ``_scroll_manager``, read with ``getattr`` --
``_preserving_scroll_position`` asks it for the mode's scroll helper.
Add it as a base of the plugin class beside ``SportsPluginHostMixin``, before
``BasePlugin``: ``class SoccerScoreboardPlugin(SportsPluginHostMixin,
SportsLiveScrollMixin, BasePlugin)``. The two define no name in common. No
``__init__``; a method on the plugin's own class still wins over the mixin's.
"""
import logging
import time
from contextlib import contextmanager
from typing import Any, Callable, ClassVar, Dict, FrozenSet, Iterator, List
class SportsLiveScrollMixin:
"""Mid-cycle rebuilds of a live scroll strip. See module docstring."""
# The host contract, declared for type checking only: these create no
# attributes, so the host's own values are what the methods read.
logger: logging.Logger
LIVE_VOLATILE_FIELDS: ClassVar[FrozenSet[str]]
_live_scroll_fingerprints: Dict[Any, Any]
_live_scroll_rebuilt_at: Dict[Any, float]
_live_scroll_rebuild_cost: Dict[Any, float]
_dispatch_switch_refresh: Callable[[Any], None]
#: Floor between mid-cycle strip rebuilds, and the duty-cycle cap that can
#: raise it.
#:
#: A rebuild re-renders every card into one wide image, on the render
#: thread, so the marquee is frozen for however long it takes. Measured on a
#: Pi 4: 28ms for one game, 139ms for five, 435ms for fifteen. A fixed 5s
#: floor is fine for one game and wrong for a full slate -- with fifteen
#: live games a pitch lands somewhere every second or so, the fingerprint
#: changes continuously, and 435ms every 5s is nearly a tenth of the time
#: spent not scrolling.
#:
#: So the floor also scales with what the last rebuild actually cost: never
#: spend more than 1/LIVE_SCROLL_REBUILD_DUTY_DIVISOR of wall time
#: rebuilding. Fifteen games self-limits to a rebuild every ~8.7s; one game
#: stays on the 5s floor. No per-sport tuning, and it adapts to slate size
#: and panel width on its own.
LIVE_SCROLL_REBUILD_MIN_SECONDS: ClassVar[float] = 5.0
LIVE_SCROLL_REBUILD_DUTY_DIVISOR: ClassVar[float] = 20.0
def _live_scroll_managers(self, league=None):
"""The live managers whose games are on the strip.
Two shapes across the scoreboard lineage: a _league_registry (baseball,
basketball, hockey, lacrosse, soccer, football) and a _get_manager
accessor on the single-league plugins (afl, nrl). Anything else returns
nothing, which leaves this feature inert rather than wrong.
"""
registry = getattr(self, "_league_registry", None)
if isinstance(registry, dict) and registry:
managers = []
for league_id, entry in registry.items():
if league is not None and league_id != league:
continue
entry = entry or {}
if not entry.get("enabled", False):
continue
manager = (entry.get("managers") or {}).get("live")
if manager is not None:
managers.append(manager)
return managers
getter = getattr(self, "_get_manager", None)
if callable(getter):
try:
# pylint: disable=not-callable
# The lineages that lack _get_manager infer this as None, so a
# static checker calls it uncallable. callable() above is the
# runtime guard; the branch is simply dead in those plugins.
manager = getter("live")
except (AttributeError, KeyError, TypeError, ValueError, OSError):
return []
return [manager] if manager is not None else []
return []
def _refresh_live_scroll_managers(self, league=None) -> None:
"""Let the live managers refresh before their games are fingerprinted.
Switch mode stays current because _try_manager_display() calls
_ensure_manager_updated() on every pass. Scroll mode had no equivalent:
its only refresh sat inside the block gated by the rebuild decision, and
that decision is computed from the data the refresh would replace. So
once the first strip was built nothing could change it, and the score on
the marquee stayed frozen until the process restarted.
The refresh runs off the render thread -- see _dispatch_switch_refresh().
This is called on every scroll frame, and a due manager.update() is a
network round trip: run inline, it froze the marquee for the length of
the ESPN request. The refreshed games land a few frames later, and the
fingerprint check that follows this call picks them up on the next frame
after they do. Dispatches for a manager are rate-limited, so the frames
where nothing is due cost a dict lookup and a clock read.
Deliberately NOT gated on mode_type == "live". A recent/upcoming strip
never rebuilds from the fingerprint (_live_scroll_needs_rebuild returns
early for those), so refreshing here looks like wasted work -- but with
live_priority the plugin only switches TO live mode once it knows live
games exist, and it learns that from these same managers. Refreshing
only while live mode is on screen would rebuild the same circularity one
level up, and a game that went live would wait for the background
plugin update -- an hour, on a rig that sets update_interval: 3600.
"""
for manager in self._live_scroll_managers(league) or []:
try:
self._dispatch_switch_refresh(manager)
except (AttributeError, KeyError, TypeError, ValueError, OSError,
RuntimeError) as exc:
# Narrow on purpose: the update itself runs on another thread,
# and _ensure_manager_updated() swallows whatever it raises, so
# anything arriving here is a lookup error or a thread that
# could not be started, not a fetch failure.
self.logger.debug("Live scroll refresh skipped: %s", exc)
@classmethod
def _live_scroll_fields(cls, game) -> tuple:
"""One game as sorted ``(key, value)`` strings, minus the volatile keys."""
try:
items = list(game.items())
except AttributeError:
return (("<not-a-dict>", str(game)),)
return tuple(sorted((str(k), str(v)) for k, v in items
if k not in cls.LIVE_VOLATILE_FIELDS))
@classmethod
def _fingerprint_games(cls, games) -> tuple:
"""Order-independent fingerprint of a list of games."""
return tuple(sorted(cls._live_scroll_fields(g) for g in (games or [])))
def _live_scroll_fingerprint(self, league=None) -> tuple:
"""Fingerprint of every live game the strip's managers hold now."""
games: List[Any] = []
for manager in self._live_scroll_managers(league):
games.extend(getattr(manager, "live_games", None) or [])
return self._fingerprint_games(games)
def _live_scroll_needs_rebuild(self, scroll_key, mode_type, league=None) -> bool:
"""True when the live card would draw differently than the strip does.
_scroll_prepared is cleared only when the cycle *completes*, so a score
scored mid-cycle stayed frozen in the rendered strip until the marquee
finished -- minutes, for a long game list. Restarting the display forces
a rebuild, which is the workaround users find.
"""
if mode_type != "live":
return False
known = self._live_scroll_fingerprints.get(scroll_key)
if known is None:
return False # nothing built yet; normal path
if self._live_scroll_fingerprint(league) == known:
return False
last = self._live_scroll_rebuilt_at.get(scroll_key, 0.0)
cost = self._live_scroll_rebuild_cost.get(scroll_key, 0.0)
floor = max(self.LIVE_SCROLL_REBUILD_MIN_SECONDS,
cost * self.LIVE_SCROLL_REBUILD_DUTY_DIVISOR)
if time.time() - last < floor:
return False # deferred, not dropped
return True
def _note_live_scroll_built(self, scroll_key, mode_type, fingerprint=None,
league=None) -> None:
"""Record what the strip was built from.
Takes a fingerprint captured from the *managers* immediately before the
render, not one computed from the games handed to the renderer. Those
two are not comparable: _collect_games_for_scroll() decorates each game
with extra keys ("league", "status"), so a fingerprint taken from its
output can never equal one taken from the managers -- every check past
the rate limiter would rebuild, defeating the clock exclusion entirely.
That is not hypothetical; it is what the first version of this did, and
an end-to-end simulation caught it rebuilding on a bare clock tick.
Capturing before the render also closes the race a plain re-read would
open: a background update landing mid-render would otherwise be recorded
as though the strip already contained it.
"""
if mode_type != "live":
return
self._live_scroll_fingerprints[scroll_key] = (
fingerprint if fingerprint is not None
else self._live_scroll_fingerprint(league))
self._live_scroll_rebuilt_at[scroll_key] = time.time()
@contextmanager
def _preserving_scroll_position(self, mode_type, active, scroll_key=None) -> Iterator[None]:
"""Keep the marquee where it is across a mid-cycle rebuild.
ScrollHelper.set_scrolling_image() resets two counters and both matter:
scroll_position (without it the marquee snaps back to the start, which
looks worse than the stale score being fixed) and total_distance_scrolled
(without it the cycle restarts, so a game that keeps scoring could stop
the strip ever completing). Restored clamped to the new strip, since a
score gaining a digit changes its card's width by a few pixels.
A no-op unless `active` -- a first build should start at zero.
"""
helper = None
if active and getattr(self, "_scroll_manager", None):
try:
helper = self._scroll_manager.get_scroll_display(mode_type).scroll_helper # type: ignore[attr-defined]
except Exception: # pragma: no cover - defensive
helper = None
position = getattr(helper, "scroll_position", None) if helper else None
distance = getattr(helper, "total_distance_scrolled", None) if helper else None
started = time.time()
try:
yield
finally:
# What this render cost, so the next floor can scale with it. Keyed by
# scroll_key, which is what _live_scroll_needs_rebuild() reads --
# they are only the same string in some of these plugins, and keying
# by mode_type made the duty cap silently inert in the rest.
self._live_scroll_rebuild_cost[scroll_key or mode_type] = time.time() - started
if helper is not None and position is not None:
width = max(getattr(helper, "total_scroll_width", 0) - 1, 0)
helper.scroll_position = min(position, width)
if distance is not None:
helper.total_distance_scrolled = distance
helper.scroll_complete = False
self.logger.info(
"[Scroll] Live card changed; rebuilt the %s strip in place "
"at position %d", mode_type, int(helper.scroll_position))
__all__ = ["SportsLiveScrollMixin"]
+270
View File
@@ -0,0 +1,270 @@
"""The scoreboard plugin class's helpers every ``manager.py`` copies.
Each scoreboard's ``manager.py`` holds its ``BasePlugin`` subclass (the
"host": ``SoccerScoreboardPlugin``, ``UFCScoreboardPlugin``, ...). Ten of its
methods, and the class constant one of them reads, are identical
(executable AST, docstrings stripped, decorators compared) in all nine
scoreboards -- afl, baseball, basketball, football, hockey, lacrosse, nrl,
soccer and ufc -- and were copied here from ledmatrix-plugins ``56c4f15``
(origin/main, 2026-09-30) under their existing names:
- ``_dispatch_switch_refresh`` (with ``_SWITCH_REFRESH_MIN_GAP_SECONDS``) --
run a manager's refresh on a daemon thread so ``display()`` never blocks
on the network;
- ``get_vegas_priority_weight``, ``_favorite_team_is_live``,
``_favorite_scan_targets``, ``_favorite_scan_games`` and
``_game_involves`` -- how many Vegas slots the plugin asks for, and
whether a configured favourite is playing live;
- ``get_vegas_content_type`` -- ``'multi'``: a scoreboard is a list of games;
- ``_dynamic_feature_enabled``, ``_get_total_games_for_manager`` and
``_build_manager_key`` -- small dynamic-duration helpers.
This is stage 4 of the consolidation (docs/SPORTS_UNIFICATION.md): the
families that needed no reconciling. The rest of ``manager.py`` has drifted
and is reconciled one family per release before it moves.
A new module rather than more methods on an existing mixin, for the reason
``sports_helpers`` gives: a missing module fails at load, where the version
checks see it; a missing method fails mid-frame.
WHAT A HOST MUST PROVIDE
------------------------
Derived by walking every ``self.<attr>`` the mixin reads; the host-contract
test in ``test/test_sports_plugin_host.py`` fails if a read is added without
being listed here.
- ``_ensure_manager_updated(manager)`` -- ``_dispatch_switch_refresh`` runs
it on the thread it starts. It must swallow its own errors: nothing joins
the thread.
- ``global_config``, ``has_live_priority()``, ``has_live_content()`` and
``supports_dynamic_duration()`` -- all on ``BasePlugin``; the scoreboards
override the last three.
- ``is_enabled`` -- set by each scoreboard's ``__init__`` (``BasePlugin``
calls its flag ``enabled``).
- ``_switch_refresh_threads`` and ``_switch_refresh_at``, read with
``getattr`` -- ``_dispatch_switch_refresh`` creates both on first use, so
a host need not.
- The live managers it scans for favourites are found through ``vars(self)``
(``_favorite_scan_targets``): any attribute, or value of a dict
attribute, with ``favorite_teams`` (or ``favorite_fighters``) and
``live_games`` (or ``live_matches``, or an ``active_celebration`` dict
holding a ``game``).
Add it as a base of the plugin class, **before** ``BasePlugin``, e.g.
``class SoccerScoreboardPlugin(SportsPluginHostMixin, BasePlugin)``:
``get_vegas_priority_weight`` and ``get_vegas_content_type`` override
``BasePlugin``'s defaults. A method on the plugin's own class still wins over
the mixin's. The mixin has no ``__init__`` and creates no class attributes
beyond its one constant.
"""
import logging
import threading
import time
from typing import Any, Callable, ClassVar, Dict, Iterator, Optional
class SportsPluginHostMixin:
"""The scoreboard plugin class's identical helpers. See module docstring."""
# The host contract, declared for type checking only: these create no
# attributes, so the host's own values are what the methods read.
logger: logging.Logger
is_enabled: bool
global_config: Dict[str, Any]
_ensure_manager_updated: Callable[[Any], Any]
has_live_priority: Callable[[], bool]
has_live_content: Callable[[], bool]
supports_dynamic_duration: Callable[[], bool]
# Created on first use by _dispatch_switch_refresh, per instance.
_switch_refresh_threads: Dict[int, threading.Thread]
_switch_refresh_at: Dict[int, float]
#: Floor between two draw-time refresh dispatches for one manager. The
#: manager's own update() still decides whether anything is fetched; this
#: only stops display() starting a thread on every frame just to be told
#: the interval has not elapsed.
_SWITCH_REFRESH_MIN_GAP_SECONDS: ClassVar[float] = 5.0
def _dispatch_switch_refresh(self, manager) -> None:
"""Run _ensure_manager_updated(manager) on a daemon thread.
Called from display(), so it must not block: when an update is due,
manager.update() fetches rankings and the schedule over the network,
and doing that inline stalled the frame for the length of the round
trip. The refreshed games land in the manager a few frames later --
still within the manager's own interval, which is the freshness the
switch path was missing.
At most one refresh per manager runs at a time, and dispatches for the
same manager are at least _SWITCH_REFRESH_MIN_GAP_SECONDS apart. Only
the render thread touches the two bookkeeping dicts, so they need no
lock; manager.update() stamps last_update before it fetches, so a
concurrent background plugin.update() for the same manager returns
early rather than fetching twice.
"""
threads: Optional[Dict[int, threading.Thread]] = getattr(self, "_switch_refresh_threads", None)
if threads is None:
threads = self._switch_refresh_threads = {}
stamps: Optional[Dict[int, float]] = getattr(self, "_switch_refresh_at", None)
if stamps is None:
stamps = self._switch_refresh_at = {}
key = id(manager)
running = threads.get(key)
if running is not None and running.is_alive():
return
now = time.monotonic()
last = stamps.get(key)
if last is not None and now - last < self._SWITCH_REFRESH_MIN_GAP_SECONDS:
return
stamps[key] = now
thread = threading.Thread(
target=self._ensure_manager_updated,
args=(manager,),
daemon=True,
name="SwitchRefresh-%s" % type(manager).__name__,
)
threads[key] = thread
thread.start()
# ---- Vegas weighting: is a favourite playing? -----------------------
#
# With display.vegas_scroll.live_in_ticker set, the marquee keeps running
# through a live game and plugins can claim more than one slot per cycle.
# The core already gives any plugin with live content `live_weight`; this
# exists for the one thing the core cannot work out for itself, which is
# *whose* game is live. See PLUGIN_API_REFERENCE, "Vegas scroll hooks",
# and ADVANCED_FEATURES, "Live content in the ticker".
def get_vegas_priority_weight(self):
"""Slots per Vegas cycle: more when a favorite team is playing.
Returns None when nothing is live, which leaves the decision to the
core rather than asserting a weight of 1 -- the core may have its own
reason to boost this plugin later.
"""
try:
if not (self.has_live_priority() and self.has_live_content()):
return None
vegas = (self.global_config or {}).get('display', {}).get(
'vegas_scroll', {})
if self._favorite_team_is_live():
return vegas.get('favorite_live_weight', 5)
return vegas.get('live_weight', 3)
except Exception:
# Never let a weighting question break the rotation; the core
# treats an exception as weight 1 anyway, and None says the same
# thing more cheaply.
return None
def _favorite_team_is_live(self):
"""Whether any live game or fight involves a configured favorite.
The sports plugins do not share one data shape, so this enumerates the
real ones rather than assuming. An earlier version looked only for an
attribute holding `live_games` alongside `favorite_teams`, which was
true of five plugins and quietly false for four others -- they simply
never reported a favorite, and no test noticed because the tests used
the assumed shape rather than each plugin's own.
Handled:
* managers held directly on the plugin *and* inside a dict such as
``self._managers`` (nrl, afl)
* ``live_games`` (most) and ``live_matches`` (cricket)
* ``favorite_teams`` (most) and ``favorite_fighters`` (ufc)
* identifiers ``home_abbr``/``away_abbr``, ``home_id``/``away_id``,
``fighter1_name``/``fighter2_name``, and cricket's nested
``teams: [{name, abbr, short_name}]``
* ``active_celebration["game"]``, a snapshot the live manager keeps
precisely because the game leaves ``live_games`` while the
celebration is still on screen
"""
for holder in self._favorite_scan_targets():
favorites = (getattr(holder, 'favorite_teams', None)
or getattr(holder, 'favorite_fighters', None))
if not favorites:
continue
wanted = {str(f).strip().lower() for f in favorites if f}
if not wanted:
continue
for game in self._favorite_scan_games(holder):
if self._game_involves(game, wanted):
return True
return False
def _favorite_scan_targets(self) -> Iterator[Any]:
"""Objects that might carry live content: attributes, and dict values.
nrl and afl keep their per-league managers in a ``self._managers``
dict, so walking attribute values alone finds the dict and stops.
"""
for value in list(vars(self).values()):
yield value
if isinstance(value, dict):
for nested in list(value.values()):
yield nested
@staticmethod
def _favorite_scan_games(holder) -> Iterator[Dict[str, Any]]:
"""Every game/fight on a holder that a favorite could be playing in."""
for attr in ('live_games', 'live_matches'):
for game in (getattr(holder, attr, None) or []):
if isinstance(game, dict):
yield game
celebration = getattr(holder, 'active_celebration', None)
if isinstance(celebration, dict) and isinstance(celebration.get('game'), dict):
yield celebration['game']
@staticmethod
def _game_involves(game, wanted) -> bool:
"""Whether a game/fight involves one of the wanted names."""
for field in ('home_abbr', 'away_abbr', 'home_id', 'away_id',
'fighter1_name', 'fighter2_name'):
value = game.get(field)
if value is not None and str(value).strip().lower() in wanted:
return True
# Cricket nests its sides and matches on any of three names, by
# substring -- "india" should match "India Women". Mirrors that
# plugin's own _match_has_team rather than inventing a second rule.
for team in (game.get('teams') or []):
if not isinstance(team, dict):
continue
hay = " ".join(str(team.get(k) or '') for k in
('name', 'abbr', 'short_name')).lower()
if any(name in hay for name in wanted):
return True
return False
def get_vegas_content_type(self) -> str:
"""Plugin provides multiple scrollable items (games)."""
return 'multi'
# ---- dynamic duration ------------------------------------------------
def _dynamic_feature_enabled(self) -> bool:
"""Dynamic duration applies: the plugin is enabled and supports it."""
if not self.is_enabled:
return False
return self.supports_dynamic_duration()
@staticmethod
def _get_total_games_for_manager(manager) -> int:
"""How many games a manager holds, from the first list it carries."""
if manager is None:
return 0
for attr in ("live_games", "games_list", "recent_games", "upcoming_games"):
value = getattr(manager, attr, None)
if isinstance(value, list):
return len(value)
return 0
@staticmethod
def _build_manager_key(mode_name: str, manager) -> str:
"""``"<mode>:<manager class>"``, the key progress is tracked under."""
manager_name = manager.__class__.__name__ if manager else "None"
return f"{mode_name}:{manager_name}"
__all__ = ["SportsPluginHostMixin"]
+184 -2
View File
@@ -42,17 +42,19 @@ Usage::
from __future__ import annotations from __future__ import annotations
import functools
import logging import logging
import time import time
from typing import Any, Dict, List, Optional from typing import Any, Callable, Dict, List, Optional, Tuple
from PIL import Image from PIL import Image
from src.common import scroll_config from src.common import scroll_config, sports_vegas
from src.common.scroll_helper import ScrollHelper from src.common.scroll_helper import ScrollHelper
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
#: Defaults every copy agreed on. A subclass overrides #: Defaults every copy agreed on. A subclass overrides
#: :meth:`SportsScrollDisplay.scroll_settings_defaults` to change them — #: :meth:`SportsScrollDisplay.scroll_settings_defaults` to change them —
#: the soccer lineage uses a 24px gap and min/max duration keys instead. #: the soccer lineage uses a 24px gap and min/max duration keys instead.
@@ -413,6 +415,160 @@ class SportsScrollDisplay:
"""Whether content is prepared and ready to scroll.""" """Whether content is prepared and ready to scroll."""
return bool(self.scroll_helper.cached_image) return bool(self.scroll_helper.cached_image)
# ------------------------------------------------------------------
# Live Vegas cards
# ------------------------------------------------------------------
#
# One live element per game (src/plugin_system/vegas_elements.py): the
# ticker swaps a card in place when its game changes. A sport opts in by
# implementing make_vegas_renderer(); everything else is here.
def make_vegas_renderer(self, card_width: int,
rankings_cache: Optional[Dict[str, int]] = None) -> Any:
"""The renderer this sport draws one game card with, at ``card_width``.
**Override point.** Return the object whose ``render_game_card(game,
game_type)`` draws one card exactly ``card_width`` wide at the display's
height -- the one prepare_scroll_content already builds -- without the
black padding prepare_scroll_content adds around each card (the ticker
adds its own). Raising NotImplementedError, the default, keeps the
plugin on its ordinary Vegas content.
"""
raise NotImplementedError(
f"{type(self).__name__} has no live Vegas cards (make_vegas_renderer)")
def _determine_game_type(self, game: Dict[str, Any]) -> str:
"""The card a game is drawn as: 'live', 'recent' or 'upcoming'.
From the game's state; a sport whose scroll display decides it
differently (most define their own) overrides this.
"""
return {'in': 'live', 'post': 'recent'}.get(sports_vegas._state(game), 'upcoming')
def render_vegas_card(self, renderer: Any, game: Dict[str, Any]) -> Image.Image:
"""Draw one game's card. Override only if the renderer is called differently."""
card: Image.Image = renderer.render_game_card(game, self._determine_game_type(game))
return card
def vegas_separator(self, league: str) -> Optional[Image.Image]:
"""The league separator shown before a league's cards, if there is an icon."""
icon = self._separator_icons.get(league)
if icon is None:
return None
gap = self._vegas_settings(league).get("gap_between_games", 48)
pad = max(4, int(gap) // 2)
image = Image.new('RGB', (icon.width + pad * 2, self.display_height), (0, 0, 0))
mask = icon if icon.mode == 'RGBA' else None
image.paste(icon, (pad, (self.display_height - icon.height) // 2), mask)
return image
def _vegas_memo(self) -> Dict[Any, Any]:
"""Per-size, per-config memo for the live path; emptied when either changes."""
stamp = (self.display_width, self.display_height, id(self.config))
memo: Optional[Tuple[Any, Dict[Any, Any]]] = getattr(self, '_vegas_memo_store', None)
if memo is None or memo[0] != stamp:
memo = (stamp, {})
self._vegas_memo_store = memo
store: Dict[Any, Any] = memo[1]
return store
def _vegas_settings(self, league: Optional[str]) -> Dict[str, Any]:
"""A league's scroll settings, looked up once per size and config.
The live path asks after every update; a sport's settings lookup can
be expensive (sizing the default card width builds probe renderers).
"""
memo = self._vegas_memo()
key = ('settings', league)
if key not in memo:
memo[key] = dict(self._get_scroll_settings(league))
settings: Dict[str, Any] = memo[key]
return settings
def _vegas_renderer(self, card_width: int,
rankings_cache: Optional[Dict[str, int]]) -> Any:
"""The sport's renderer for one card width, built once rather than per slate.
Building one loads fonts and, for the default card width, probes the
layout; the scroll path pays that on every prepare, which the live
path would repeat on every update.
"""
memo = self._vegas_memo()
key = ('renderer', card_width)
if key not in memo:
memo[key] = self.make_vegas_renderer(card_width, rankings_cache)
renderer = memo[key]
if hasattr(renderer, 'set_rankings_cache'):
# Every time, empty included: the renderer is reused across
# slates, and ranks cleared since must not stay drawn.
renderer.set_rankings_cache(rankings_cache or {})
return renderer
def build_vegas_elements(
self,
games: List[Dict[str, Any]],
leagues: List[str],
rankings_cache: Optional[Dict[str, int]] = None,
fingerprint: Optional[Callable[[Dict[str, Any]], Any]] = None,
now: Optional[float] = None,
) -> Optional[List[Any]]:
"""The slate as live Vegas elements: one card per game, separators between leagues.
Only cards whose fingerprint changed are drawn; the rest come from the
cache. ``fingerprint(game)`` should return what the card draws (the
plugin's own signature fields, the clock included for live games); by
default the whole game dict is used, which redraws on any change. The
teams' ranks from ``rankings_cache`` count too: the renderer draws
them from there, not from the game.
Raises NotImplementedError when the sport has no make_vegas_renderer.
"""
from src.plugin_system.vegas_elements import VegasElement
games = sports_vegas.dedupe_games(games)
if not games:
return None
# Settings follow each game's own league, not the slate's first one:
# a card's width must not change because another league has no games
# today (the ticker refuses a redraw of another width).
first = self._vegas_settings(leagues[0] if leagues else None)
cards = getattr(self, '_vegas_cards', None)
if cards is None:
cards = self._vegas_cards = sports_vegas.VegasCardCache()
odds = getattr(self, '_vegas_odds', None)
if odds is None:
odds = self._vegas_odds = sports_vegas.StickyOdds()
fingerprint = fingerprint or sports_vegas.game_fingerprint
elements: List[Any] = []
keys: List[str] = []
current_league = None
separators = 0
for game in games:
league = game.get("league")
settings = self._vegas_settings(league) if league else first
card_width = int(settings.get("game_card_width", self.display_width))
if settings.get("show_league_separators", True) and league != current_league:
separator = self.vegas_separator(league) if league else None
if separator is not None:
elements.append(VegasElement(
key=f"sep:{separators}:{league}", image=separator, live=False))
separators += 1
current_league = league
key = sports_vegas.game_key(game)
drawn = odds.apply(key, game, now)
ranks = (rankings_cache.get(str(drawn.get("home_abbr"))),
rankings_cache.get(str(drawn.get("away_abbr")))) if rankings_cache else None
renderer = self._vegas_renderer(card_width, rankings_cache)
elements.append(cards.element(
key, (fingerprint(drawn), ranks, card_width, self.display_height),
functools.partial(self.render_vegas_card, renderer, drawn)))
keys.append(key)
cards.retain(keys)
odds.retain(keys)
return elements
def get_current_game_count(self) -> int: def get_current_game_count(self) -> int:
return len(self._current_games) return len(self._current_games)
@@ -525,6 +681,32 @@ class SportsScrollDisplayManager:
scroll_display.clear() scroll_display.clear()
self._current_game_type = "" self._current_game_type = ""
def get_vegas_elements_for(
self,
game_type: str,
games: List[Dict[str, Any]],
leagues: List[str],
rankings_cache: Optional[Dict[str, int]] = None,
fingerprint: Optional[Callable[[Dict[str, Any]], Any]] = None,
) -> Optional[List[Any]]:
"""Live Vegas cards for a slate, built on the ``game_type`` display.
None when the sport has no live cards (it does not implement
make_vegas_renderer) or building them failed, so the plugin's
get_vegas_elements() can return it and the ticker falls back to the
plugin's ordinary Vegas content.
"""
scroll_display = self.get_scroll_display(game_type)
try:
return scroll_display.build_vegas_elements(
games, leagues, rankings_cache, fingerprint)
except NotImplementedError:
return None
except Exception:
# Built straight from feed data, like prepare_scroll_content.
self.logger.exception("Error building live Vegas cards")
return None
def get_all_vegas_content_items(self) -> List[Image.Image]: def get_all_vegas_content_items(self) -> List[Image.Image]:
"""Every display's Vegas items, for splicing into the marquee.""" """Every display's Vegas items, for splicing into the marquee."""
items: List[Image.Image] = [] items: List[Image.Image] = []
+49
View File
@@ -1348,6 +1348,55 @@ class SportsLiveSharedMixin:
or candidate < current): or candidate < current):
self._next_scheduled_start_ts = candidate self._next_scheduled_start_ts = candidate
#: How long a game that finished live is still reported by
#: finished_games_snapshot(): long enough for the recent-games list, which
#: refreshes about hourly, to take it over well before most slates would.
FINISHED_GAME_TTL = 900.0
def _record_finished_game(self, details: Dict) -> None:
"""Remember a game that was live and has just gone final (or looks over).
A finished game leaves ``live_games`` at the next poll, and the recent
list that will show it refreshes about hourly, so in between nothing
holds the game's final score -- and a live Vegas card for it would keep
its last live score. Call this wherever a poll drops a game as final
or over. Only a game this manager had as live is taken; one already
held takes the newer details (a game dropped by an "is it over"
heuristic, then marked final by the feed) but keeps its expiry, so a
feed that lists finals all day cannot keep one here all day.
"""
game_id = details.get("id") if isinstance(details, dict) else None
if not game_id:
return
finished = self.__dict__.setdefault("_finished_games", {})
held = finished.get(game_id)
if held is not None:
finished[game_id] = (held[0], dict(details))
return
if not any(g.get("id") == game_id for g in getattr(self, "live_games", ()) or ()):
return
finished[game_id] = (time.monotonic(), dict(details))
def finished_games_snapshot(self) -> List[Dict]:
"""Games that went final here within FINISHED_GAME_TTL, newest data first.
Copies, safe to decorate. The caller dedupes them against its other
lists (src/common/sports_vegas.dedupe_games keeps the liveliest copy,
and a final beats nothing but a live one).
"""
finished = self.__dict__.get("_finished_games")
if not finished:
return []
now = time.monotonic()
# A copy first: a manager finishing its update in the background (off
# the plugin's lock) may record a game while the ticker reads these.
held = list(finished.items())
for game_id, (seen, _game) in held:
if now - seen > self.FINISHED_GAME_TTL:
finished.pop(game_id, None)
return [dict(game) for _id, (seen, game) in held
if now - seen <= self.FINISHED_GAME_TTL]
def _note_live_fetch(self, found_live: bool) -> None: def _note_live_fetch(self, found_live: bool) -> None:
"""Record whether a look for live games found any.""" """Record whether a look for live games found any."""
if found_live: if found_live:
+262
View File
@@ -0,0 +1,262 @@
"""Timezone resolution for the scoreboard plugins.
Game start times arrive from ESPN in UTC and have to be converted to the
user's local zone before they are drawn. This module owns the "which zone?"
decision so every part of a scoreboard (its scorebug, its scroll-mode game
card and its plugin manager) agrees.
The scoreboards each carried a copy of this module as ``<sport>_timezone.py``.
The copies were identical apart from two per-plugin values, which are
keyword-only arguments here: ``plugin_label``, the name the final warning
tells the user to open, and ``writeback_fixed_in`` (see below).
Resolution order, first valid wins:
1. ``timezone`` in the plugin's own config (explicit per-plugin override)
2. The LEDMatrix global timezone via ``plugin_manager.config_manager``
3. The LEDMatrix global timezone via ``cache_manager.config_manager``
4. The host system's zone (``TZ``, ``/etc/timezone``, ``/etc/localtime``)
5. UTC
Steps 2 and 3 matter because the core does not consistently hang
``config_manager`` off both objects -- reading only one of them is what made
a scoreboard fall through to UTC while the clock plugin (which checks
``plugin_manager`` first) showed the right time on the same device. Step 4 is
the backstop for cores that expose no ``config_manager`` at all: a Pi with its
system clock set correctly should never end up rendering UTC.
Two traps this module exists to avoid, both of which render every start time
in UTC on a correctly-configured device:
* **The stale ``"UTC"`` artifact.** Some plugins once wrote
``"timezone": "UTC"`` into the *saved* config whenever resolution failed, and
that write-back stuck -- thereafter shadowing the real global timezone. Such
a plugin passes ``writeback_fixed_in`` (the release that fixed it), and step 1
then treats a bare ``"UTC"`` as suspect: it is honored only when nothing
downstream disagrees. ``Etc/UTC`` is the unambiguous spelling for "I really
do want UTC"; the bug never produced it, so it is always honored. A plugin
that never had the bug leaves ``writeback_fixed_in`` as ``None``, and its
plugin-level ``"UTC"`` is honored verbatim.
* **``get_timezone()``'s own default.** The core's
``ConfigManager.get_timezone()`` is ``self.config.get('timezone', 'UTC')``, so
it hands back ``"UTC"`` for a config that simply has no ``timezone`` key.
Steps 2 and 3 read the raw config dict instead, so an absent key falls
through to the system zone rather than latching onto that default.
"""
import logging
import os
from typing import Any, Dict, Iterable, Optional, Tuple
import pytz
logger = logging.getLogger(__name__)
def _from_config_manager(config_manager: Any, log: logging.Logger) -> Optional[str]:
"""Pull the global timezone out of a core ConfigManager, if it has one.
Reads the raw config dict in preference to ``get_timezone()``. The core's
``ConfigManager.get_timezone()`` is ``self.config.get('timezone', 'UTC')`` --
it substitutes its own ``"UTC"`` when the key is absent, which is
indistinguishable from the user deliberately choosing UTC. Taking that at
face value would mask a missing global setting and stop resolution ever
reaching the host system zone. So: if the raw config is readable and has no
``timezone`` key, report "nothing here" and let the caller fall through.
``get_timezone()`` is only consulted for cores that expose no raw config.
"""
if config_manager is None:
return None
raw_readable = False
for loader_name in ("get_config", "load_config"):
loader = getattr(config_manager, loader_name, None)
if not callable(loader):
continue
try:
main_config = loader()
except Exception:
log.debug("config_manager.%s() failed", loader_name, exc_info=True)
continue
if isinstance(main_config, dict):
raw_readable = True
name: Optional[str] = main_config.get("timezone")
if name:
return name
if raw_readable:
return None
getter = getattr(config_manager, "get_timezone", None)
if callable(getter):
try:
name = getter()
if name:
return name
except Exception:
log.debug("config_manager.get_timezone() failed", exc_info=True)
return None
def system_timezone_name() -> Optional[str]:
"""Best-effort IANA name for the host's configured timezone."""
name = os.environ.get("TZ")
if name:
return name
# Debian / Raspberry Pi OS record the zone name here.
try:
with open("/etc/timezone", "r", encoding="utf-8") as handle:
name = handle.read().strip()
if name:
return name
except OSError:
pass
# Otherwise /etc/localtime is a symlink into the zoneinfo tree.
try:
path = os.path.realpath("/etc/localtime")
marker = "zoneinfo" + os.sep
if marker in path:
return path.split(marker, 1)[1]
except OSError:
pass
return None
def _validated(name: object, source: str, log: logging.Logger) -> Optional[str]:
"""Return a usable IANA name, or None if blank/absent/not a real zone."""
if not isinstance(name, str):
return None
name = name.strip()
if not name:
return None
try:
pytz.timezone(name)
except pytz.UnknownTimeZoneError:
log.warning("Ignoring invalid timezone %r from %s", name, source)
return None
except Exception:
# Not an unknown-zone error, so something else went wrong inside pytz.
# Log it loudly rather than silently reclassifying it as "invalid" --
# but still don't propagate: this runs in the render path, and a
# mislabelled zone beats taking the whole display down.
log.warning(
"Unexpected error validating timezone %r from %s; ignoring it",
name, source, exc_info=True,
)
return None
return name
def resolve_timezone_name(
config: Optional[Dict[str, Any]] = None,
plugin_manager: Any = None,
cache_manager: Any = None,
log: Optional[logging.Logger] = None,
*,
plugin_label: str,
writeback_fixed_in: Optional[str] = None,
) -> str:
"""Return the IANA timezone name to render game times in.
Never raises and never returns an empty string; falls back to ``"UTC"``
only when every source is missing or invalid.
``plugin_label`` names the plugin in the warning logged when nothing
resolves (e.g. ``"hockey scoreboard"``). ``writeback_fixed_in`` is the
plugin release that stopped writing ``"UTC"`` back into the saved config,
for a plugin that ever did; ``None`` (the default) otherwise.
"""
log = log or logger
def downstream():
"""Yield (source, name) for everything except the plugin's own config.
Lazy: ``SportsCore._get_timezone()`` runs this once per game and the
answer is almost always already in the plugin config, so evaluating on
demand keeps the common case from calling into both config managers and
stat-ing the host timezone files every time.
"""
yield (
"plugin_manager.config_manager",
_from_config_manager(getattr(plugin_manager, "config_manager", None), log),
)
yield (
"cache_manager.config_manager",
_from_config_manager(getattr(cache_manager, "config_manager", None), log),
)
yield "system timezone", system_timezone_name()
def first_valid(
sources: Iterable[Tuple[str, Any]],
) -> Tuple[Optional[str], Optional[str]]:
for source, name in sources:
name = _validated(name, source, log)
if name:
return source, name
return None, None
plugin_value = _validated((config or {}).get("timezone"), "plugin config", log)
if writeback_fixed_in is not None and plugin_value and plugin_value.lower() == "utc":
# Before writeback_fixed_in this plugin wrote "timezone": "UTC" into
# the saved config whenever it failed to resolve a global timezone, and
# that write-back persisted. A bare "UTC" is therefore far more likely
# to be that artifact than a deliberate choice -- it only ever appeared
# on failure. Honor it only when nothing downstream disagrees; a user
# who genuinely wants UTC writes the unambiguous "Etc/UTC", which the
# bug never produced and which falls through to the normal path below.
source, downstream_name = first_valid(downstream())
if downstream_name and downstream_name.lower() not in ("utc", "etc/utc"):
log.warning(
"Ignoring the plugin-level timezone 'UTC': it is almost "
"certainly left over from the write-back bug fixed in %s, and "
"%s says %s. Using %s. If you really do want UTC here, set "
"this plugin's timezone to 'Etc/UTC' instead.",
writeback_fixed_in, source, downstream_name, downstream_name,
)
return downstream_name
log.debug("Plugin-level timezone 'UTC' agrees with %s; using UTC", source or "no other source")
return "UTC"
if plugin_value:
log.debug("Resolved timezone %s from plugin config", plugin_value)
return plugin_value
source, name = first_valid(downstream())
if name:
log.debug("Resolved timezone %s from %s", name, source)
return name
log.warning(
"Could not determine a timezone from the plugin config, the LEDMatrix "
"config or the system; game times will be shown in UTC. Set a timezone "
"in the %s's Advanced Settings to override.",
plugin_label,
)
return "UTC"
def resolve_timezone(
config: Optional[Dict[str, Any]] = None,
plugin_manager: Any = None,
cache_manager: Any = None,
log: Optional[logging.Logger] = None,
*,
plugin_label: str,
writeback_fixed_in: Optional[str] = None,
):
"""``resolve_timezone_name`` as a ready-to-use tzinfo object."""
return pytz.timezone(
resolve_timezone_name(
config=config,
plugin_manager=plugin_manager,
cache_manager=cache_manager,
log=log,
plugin_label=plugin_label,
writeback_fixed_in=writeback_fixed_in,
)
)
+250
View File
@@ -0,0 +1,250 @@
"""Live Vegas cards for the sports scoreboards.
A scoreboard hands the Vegas ticker one card per game. As live elements
(src/plugin_system/vegas_elements.py) those cards change on the panel while
they scroll: a goal redraws its game's card and the ticker swaps it in place.
This module is what every scoreboard needs for that and would otherwise write
nine times:
- :func:`game_key` -- a stable key per game, so the ticker can tell which card
a redraw belongs to however the slate is re-sorted.
- :class:`VegasCardCache` -- draws a card only when what it shows changed
(its fingerprint), so an unchanged slate costs a dictionary lookup per game
and a changed one only the cards that changed.
- :class:`StickyOdds` -- live odds are fetched only for games near the front
of the rotation, so a card's odds come and go between polls; this keeps the
last odds for a while instead of redrawing the card without them.
- :func:`dedupe_games` -- a game present in two managers' lists (live and
recent, around the final whistle) appears once, its liveliest copy.
- :func:`finished_games` / :func:`with_finished_games` -- a game that has just
gone final keeps its card, now showing FINAL, where its live card was,
until the recent list (refreshed about hourly) takes it over.
- :func:`game_fingerprint` -- what a card is redrawn on by default: the whole
game dict, frozen hashable.
SportsScrollDisplay.build_vegas_elements (src/common/sports_scroll.py) puts
them together; a plugin adopts it by implementing make_vegas_renderer().
"""
from __future__ import annotations
import time
from collections import OrderedDict
from typing import Any, Callable, Dict, Hashable, Iterable, List, Optional, Tuple
from PIL import Image
#: Which copy of a duplicated game wins: the liveliest.
_STATE_PRIORITY = {'in': 3, 'post': 2, 'pre': 1}
def _state(game: Dict[str, Any]) -> str:
status = game.get('status')
state = status.get('state') if isinstance(status, dict) else status
if isinstance(state, str):
return state
if game.get('is_live'):
return 'in'
if game.get('is_final'):
return 'post'
return 'pre'
def _freeze(value: Any) -> Any:
"""A hashable, order-stable copy of feed data."""
if isinstance(value, dict):
return tuple(sorted((str(k), _freeze(v)) for k, v in value.items()))
if isinstance(value, (list, tuple)):
return tuple(_freeze(v) for v in value)
if isinstance(value, (str, int, float, bool)) or value is None:
return value
return repr(value)
def game_fingerprint(game: Dict[str, Any]) -> Hashable:
"""Everything in a game dict, hashable: a card drawn from it changes only if this does.
The default card version. Nothing a card could draw is left out, so no
field is ever frozen on the panel; the cost is a redraw when a field the
card does not draw changes too, which feed data rarely does between polls.
"""
frozen: Hashable = _freeze(game)
return frozen
def game_key(game: Dict[str, Any]) -> str:
"""A key that names this game and nothing else, across polls.
``game:<league>:<id>`` from the feed's own id. A game without one falls
back to its teams and start time, which is stable for the life of a game.
"""
league = game.get('league') or 'game'
game_id = game.get('id') or game.get('game_id')
if game_id not in (None, ''):
return f"game:{league}:{game_id}"
away = game.get('away_abbr') or game.get('away_team') or '?'
home = game.get('home_abbr') or game.get('home_team') or '?'
start = game.get('start_time_utc') or game.get('start_time') or ''
return f"game:{league}:{away}@{home}:{start}"
def dedupe_games(games: Iterable[Dict[str, Any]],
key_fn: Callable[[Dict[str, Any]], str] = game_key) -> List[Dict[str, Any]]:
"""Each game once, in first-seen order, keeping its liveliest copy.
Around a final whistle a game can be in the live list (last poll) and the
recent list (next poll) at once; two cards with one key would be refused
by the ticker, and showing the game twice is wrong anyway.
"""
chosen: "OrderedDict[str, Dict[str, Any]]" = OrderedDict()
for game in games:
key = key_fn(game)
current = chosen.get(key)
if current is None or _STATE_PRIORITY.get(_state(game), 0) > \
_STATE_PRIORITY.get(_state(current), 0):
chosen[key] = game
return list(chosen.values())
def finished_games(
live_managers: Iterable[Tuple[str, Any]]) -> List[Dict[str, Any]]:
"""Games that just left these live managers' lists, final ones as recent games.
``live_managers`` pairs each league with its live manager (None is
skipped). Each manager reports what SportsLiveSharedMixin recorded
(finished_games_snapshot, copies), with its league. A final game is
drawn as a recent card. One a poll only judged over -- a tied end of
regulation looks like that too -- keeps its last live state, so its card
never says FINAL early; if play resumes the live list has it again, and
dedupe_games keeps that copy.
"""
finished: List[Dict[str, Any]] = []
for league, manager in live_managers:
snapshot = getattr(manager, 'finished_games_snapshot', None)
if not callable(snapshot):
continue
for game in snapshot():
game['league'] = league
if game.get('is_final'):
status = game.get('status')
status = dict(status) if isinstance(status, dict) else {}
status['state'] = 'post'
game.update(status=status, is_live=False)
finished.append(game)
return finished
def with_finished_games(
games: List[Dict[str, Any]], leagues: List[str],
finished: List[Dict[str, Any]],
) -> Tuple[List[Dict[str, Any]], List[str]]:
"""The slate with games that just went final where their live cards were.
A slate lists each league's games together, live ones first. Each
finished game goes after its league's live games, ahead of the rest; a
league with no games left in the slate is added at the end. A finished
game the slate also has (the recent list caught up) is left for
dedupe_games, which keeps one copy.
"""
if not finished:
return list(games), list(leagues)
pending: "OrderedDict[Any, List[Dict[str, Any]]]" = OrderedDict()
for game in finished:
pending.setdefault(game.get('league'), []).append(game)
merged: List[Dict[str, Any]] = []
for index, game in enumerate(games):
league = game.get('league')
if league in pending and _state(game) != 'in':
merged.extend(pending.pop(league))
merged.append(game)
following = games[index + 1] if index + 1 < len(games) else None
if league in pending and (following is None or following.get('league') != league):
merged.extend(pending.pop(league)) # the league's games were all live
leagues = list(leagues)
for league, rest in pending.items():
merged.extend(rest)
if league not in leagues:
leagues.append(league)
return merged, leagues
class VegasCardCache:
"""Cards drawn once per fingerprint, kept for as long as their game is.
``element(key, fingerprint, render)`` returns a VegasElement whose image is
``render()``'s -- called only when the fingerprint differs from the one the
cached card was drawn for. The fingerprint is also the element's version,
so the ticker skips unchanged cards without comparing pixels.
Bounded: keys not passed to :meth:`retain` after a slate are dropped, and
at most ``max_entries`` are ever held (oldest first).
"""
def __init__(self, max_entries: int = 96) -> None:
self.max_entries = max(1, int(max_entries))
self._cards: "OrderedDict[str, Tuple[Hashable, Image.Image]]" = OrderedDict()
self.renders = 0
def element(self, key: str, fingerprint: Hashable,
render: Callable[[], Image.Image], live: bool = True) -> Any:
from src.plugin_system.vegas_elements import VegasElement
cached = self._cards.get(key)
if cached is not None and cached[0] == fingerprint:
self._cards.move_to_end(key)
image = cached[1]
else:
image = render()
self.renders += 1
self._cards[key] = (fingerprint, image)
self._cards.move_to_end(key)
while len(self._cards) > self.max_entries:
self._cards.popitem(last=False)
return VegasElement(key=key, image=image, version=fingerprint, live=live)
def retain(self, keys: Iterable[str]) -> None:
"""Forget every card whose key is not in ``keys``."""
keep = set(keys)
for key in [k for k in self._cards if k not in keep]:
self._cards.pop(key, None)
def clear(self) -> None:
self._cards.clear()
def __len__(self) -> int:
return len(self._cards)
class StickyOdds:
"""Keep a game's last odds on its card while a live poll leaves them out.
Live odds are fetched only for games near the front of the rotation
(src/common/sports_fetch.py), so the same game's dict has odds on one poll
and none on the next. Drawn as-is that redraws the card every poll with
the odds flickering in and out. ``apply`` returns the game with its last
non-empty odds put back, for up to ``ttl_s`` seconds after they were seen.
"""
def __init__(self, ttl_s: float = 600.0) -> None:
self.ttl_s = float(ttl_s)
self._seen: Dict[str, Tuple[float, Any]] = {}
def apply(self, key: str, game: Dict[str, Any],
now: Optional[float] = None) -> Dict[str, Any]:
now = time.monotonic() if now is None else now
odds = game.get('odds')
if odds:
self._seen[key] = (now, odds)
return game
seen = self._seen.get(key)
if seen is None or now - seen[0] > self.ttl_s:
self._seen.pop(key, None)
return game
refilled = dict(game)
refilled['odds'] = seen[1]
return refilled
def retain(self, keys: Iterable[str]) -> None:
keep = set(keys)
for key in [k for k in self._seen if k not in keep]:
self._seen.pop(key, None)

Some files were not shown because too many files have changed in this diff Show More