Compare commits

..
Author SHA1 Message Date
ChuckandClaude Opus 5.5 e39dcb18b4 feat(scroll): report a panel that cannot reach its refresh cap, and suggest one it can hold
Scroll speeds are solved against display.hardware.limit_refresh_rate_hz,
which is only a ceiling. A panel that cannot reach it still moves whole
pixels per frame, but every scroll runs slow by the shortfall and the
"smooth" ladder is the cap's, not the panel's. A user rig (Pi 4, 2x128x64,
adafruit-hat-pwm, pwm_bits 9, gpio_slowdown 5) measured 107.6-113.1 Hz under
a 120 Hz cap: 60 px/s ran at 55, and nothing said why.

- scroll_config: refresh_shortfall() (more than 3% under the planned rate),
  holdable_cap() (a multiple of 10, 5% under the measurement, since the
  measurement is the fast end of an uncapped panel's drift), and
  describe_refresh_shortfall().
- FrameTimingRecorder.plan_refresh(): once the measured period has held for
  three trusted windows, a shortfall is logged once as a warning naming the
  cap to use. DisplayManager calls it only for a real panel, not the
  emulator or the fallback canvas. The stats file records
  planned_refresh_hz (additive).
- GET /api/v3/config/refresh-rate, plus a hint under the Display tab's
  Limit Refresh Rate field with a button that fills in the suggested cap.
- _panel_refresh_hz (behind the Vegas slider's advice) ignores a measurement
  written under a different cap, so a changed cap stops being advised from
  the old rate before the display restarts.

Verified on ledpi with a temporary 200 Hz cap: the warning logged about a
minute after the restart ("about 132 Hz ... Set Limit Refresh Rate to
120 Hz"), the endpoint returned the same shortfall, and the Display tab
showed the hint; its button filled in 120. ledpi was restored afterwards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 17:11:09 -04:00
17 changed files with 407 additions and 544 deletions
+19 -11
View File
@@ -19,6 +19,25 @@ accepts both, but the store flags the old spelling as deprecated
## Unreleased
### Scroll speed: a panel slower than its refresh cap is reported
- Scroll speeds are solved against `limit_refresh_rate_hz`, so a panel that
cannot reach its cap ran every scroll slow by the shortfall, with no sign
why (one Pi 4 on a 120 Hz cap refreshed at ~110 Hz: 60 px/s ran at 55).
Once the display has measured the real rate over three windows of
scrolling, a panel more than 3% short of the cap is logged once, as a
warning from `src.common.frame_timing` that names a cap it can hold (a
multiple of 10, 5% under the measurement). The Display tab shows the same
under Limit Refresh Rate, with a button that fills it in, from the new
`GET /api/v3/config/refresh-rate`. Not checked in the emulator or on the
fallback canvas.
- The frame-stats file records `planned_refresh_hz` (additive), and the
scroll-speed advice behind the Vegas slider ignores a measurement written
under a different cap. Until now, after the cap changed, the slider kept
advising from the old rate until the display restarted.
- New in `src.common.scroll_config`: `refresh_shortfall()`, `holdable_cap()`
and `describe_refresh_shortfall()`.
### Fixed
- The web preview and `/api/v3/display/current` no longer stay black for a
@@ -1417,17 +1436,6 @@ read any of them:
### Fixes
- Updating a plugin from the store no longer deletes the files it wrote
beside itself. A monorepo update replaces the plugin directory with the
fresh download and deletes the old copy, so calendar's Google OAuth files
(`token.pickle`, `credentials.json`) were lost on every update and the
calendar stopped until they were restored by hand. Before the old copy is
removed, the update now copies over anything the plugin's `.gitignore`
excludes plus known secret/state files (`*.pickle`, `token.json`,
`credentials.json`, `config_secrets.json`, `.pkce_code_verifier`); files the
new release ships are never overwritten, and byte code is not carried. A
plugin updated with `git pull` no longer sweeps an untracked token into the
auto-stash, which is never popped (`src/plugin_system/plugin_local_files.py`).
- Quieter routine logging. Every rotation logged each mode twice
("Switching to mode", then "Processing mode"), and a mode with nothing to
show added "display() returned False" and "No content to display". Those
+25
View File
@@ -77,6 +77,31 @@ The Vegas **Scroll Speed** slider in the web UI shows the same thing live: a
line under it says what your speed will run as on this panel, and links to the
nearest smooth speeds.
### A panel that cannot reach its cap
Speeds are solved against `limit_refresh_rate_hz`, the configured cap, but a
cap is only a ceiling: a long chain, a high `pwm_bits` or a big
`gpio_slowdown` can leave the panel below it. One Pi 4 driving 2×128×64 on
`adafruit-hat-pwm` with `pwm_bits 9` and `gpio_slowdown 5` measured
107.6–113.1 Hz under a 120 Hz cap. Frames still move whole pixels, but
every scroll runs that much slower than configured (60 px/s ran at 55 px/s),
and the smooth speeds are the cap's rather than the panel's.
The display measures the real rate from its own frames. About a minute
into scrolling, a panel more than 3% short of its cap is logged once:
```
WARNING - src.common.frame_timing - The panel refreshes at about 113 Hz, below
the 120 Hz that scroll speeds are planned for ... Set Limit Refresh Rate to
100 Hz (web UI, Display tab), which this panel can hold, and restart.
```
The Display tab says the same under **Limit Refresh Rate**, with a button
that fills in the suggested cap (`GET /api/v3/config/refresh-rate`). The
suggestion is a multiple of 10 at least 5% under the measurement, because
an uncapped panel drifts and the measurement is the fast end of it. A cap the
panel holds also stops the drift.
### How a slow speed stays crisp
`SwapOnVSync(canvas, framerate_fraction)` holds each frame for N panel
+41
View File
@@ -135,6 +135,8 @@ import time
import traceback
from typing import Any, Callable, Dict, List, Optional, Tuple, TypedDict
from src.common import scroll_config
logger = logging.getLogger(__name__)
#: Bumped when a field changes meaning, so a reader can refuse stale files.
@@ -179,6 +181,12 @@ MAX_REFRESH_DROP = 0.2
#: trusted -- about a second of scrolling.
MIN_FRAMES_FOR_REFRESH = 90
#: Trusted windows, counting the one that adopted the period, before a panel
#: slower than its cap is reported. The estimate can still fall (the refresh
#: rate rise) by up to MAX_REFRESH_DROP per window early on; the warning
#: should not name a rate one more window would have corrected.
REFRESH_CHECK_WINDOWS = 3
FLUSH_INTERVAL = 10.0
#: A scroll's last frame older than this is a stall worth a stack dump.
@@ -441,6 +449,12 @@ class FrameTimingRecorder:
1.0 / refresh_hz if refresh_hz and refresh_hz > 0 else None)
# The first estimate, until a second window agrees with it.
self._refresh_candidate: Optional[float] = None
# The rate scroll speeds are solved against; see plan_refresh().
self.planned_refresh_hz: Optional[float] = None
# Trusted windows seen since the period was adopted, until the
# shortfall check has run.
self._refresh_windows = 0
self._shortfall_checked = True
self.totals: Dict[str, Any] = {
"static_frames": 0,
"scroll_frames": 0,
@@ -639,7 +653,12 @@ class FrameTimingRecorder:
self._refresh_candidate = estimate
elif current * (1.0 - MAX_REFRESH_DROP) <= estimate < current:
self.refresh_period = estimate
if self.refresh_period is not None:
self._refresh_windows += 1
period = self.refresh_period
if (period and not self._shortfall_checked
and self._refresh_windows >= REFRESH_CHECK_WINDOWS):
self._check_refresh_shortfall(1.0 / period)
histograms = self.histograms
for frame in batch:
@@ -688,6 +707,25 @@ class FrameTimingRecorder:
elif missed <= -1:
totals["early_frames"] += 1
def plan_refresh(self, hz: Optional[float]) -> None:
"""Say what rate scroll speeds are solved against, before frames arrive.
``DisplayManager.refresh_hz``: the configured cap. Once the measured
rate has held for :data:`REFRESH_CHECK_WINDOWS` windows, a panel that
falls short of it is logged once, with a cap it can hold (see
:func:`src.common.scroll_config.refresh_shortfall`). The display
manager calls this only for a real panel.
"""
self.planned_refresh_hz = hz
self._shortfall_checked = not hz
def _check_refresh_shortfall(self, measured_hz: float) -> None:
"""Log, once, a panel that cannot reach the rate speeds assume."""
self._shortfall_checked = True
shortfall = scroll_config.refresh_shortfall(measured_hz, self.planned_refresh_hz)
if shortfall:
logger.warning(scroll_config.describe_refresh_shortfall(shortfall))
def snapshot(self) -> Dict[str, Any]:
"""The JSON document: cumulative since this process started."""
if not self._binding_checked:
@@ -704,6 +742,9 @@ class FrameTimingRecorder:
"bucket_ms": BUCKET_MS,
"freeze_seconds": FREEZE_SECONDS,
"measured_refresh_hz": round(1.0 / period, 2) if period else None,
# Additive: what scroll speeds were solved against, so a reader can
# tell a stale file (written under another cap) from this one.
"planned_refresh_hz": self.planned_refresh_hz,
"binding_releases_gil": self._binding_gil,
"info": info,
"totals": copy.deepcopy(self.totals),
+63
View File
@@ -538,3 +538,66 @@ def speed_advice(
"smooth": smooth,
"alternatives": [as_dict(c) for c in alternatives],
}
#: A measured refresh this far below the rate speeds are planned for means
#: the panel cannot reach its cap. Smaller gaps are the cap's own slack and
#: the estimate's: one rig measured 99.95 Hz under a 100 Hz cap.
REFRESH_SHORTFALL = 0.03
#: How far under the measured rate a suggested cap sits. The measurement is
#: the fast end of the panel's refreshes (frame_timing takes the 10th
#: percentile of intervals), and an uncapped panel drifts: one read
#: 107.6-113.1 Hz over 15 seconds. A cap inside that band would not hold.
CAP_HEADROOM = 0.05
def holdable_cap(measured_hz: Any) -> Optional[int]:
"""A refresh cap the panel can hold: a multiple of 10, 5% under what it measured.
A multiple of 10 because its whole-pixel speeds are round numbers (a
100 Hz cap gives 50 and 100 px/s). None without a usable measurement, or
when the panel is too slow for any cap of 10 Hz or more.
"""
hz = _coerce(measured_hz)
if hz is None:
return None
cap = int(hz * (1.0 - CAP_HEADROOM) // 10) * 10
return cap if cap >= 10 else None
def refresh_shortfall(measured_hz: Any, planned_hz: Any) -> Optional[Dict[str, Any]]:
"""When the panel refreshes measurably slower than speeds are planned for.
``planned_hz`` is what :func:`configure` solves against -- the
``limit_refresh_rate_hz`` cap, or :data:`DEFAULT_REFRESH_HZ` when it is 0.
A panel that cannot reach it still moves whole pixels per frame, but every
speed runs slow by the shortfall and the ladder of smooth speeds is the
cap's, not the panel's. None when there is no measurement, or the panel
reaches the cap (or beats it, as some do by a few Hz).
"""
measured, planned = _coerce(measured_hz), _coerce(planned_hz)
if measured is None or planned is None:
return None
if measured >= planned * (1.0 - REFRESH_SHORTFALL):
return None
return {
"measured_hz": round(measured, 1),
"planned_hz": round(planned, 1),
"suggested_cap_hz": holdable_cap(measured),
"slow_percent": round((1.0 - measured / planned) * 100),
}
def describe_refresh_shortfall(shortfall: Dict[str, Any]) -> str:
"""One log line for :func:`refresh_shortfall`'s answer."""
text = (
f"The panel refreshes at about {shortfall['measured_hz']:.0f} Hz, below "
f"the {shortfall['planned_hz']:.0f} Hz that scroll speeds are planned "
f"for (display.hardware.limit_refresh_rate_hz), so every scroll runs "
f"about {shortfall['slow_percent']}% slower than configured and the "
f"smooth speeds are worked out for a rate this panel never reaches.")
if shortfall.get("suggested_cap_hz"):
text += (f" Set Limit Refresh Rate to {shortfall['suggested_cap_hz']} Hz "
f"(web UI, Display tab), which this panel can hold, and restart.")
return text
+8 -1
View File
@@ -393,6 +393,11 @@ class DisplayManager:
self._setup_matrix()
logger.info("Matrix setup completed in %.3f seconds", time.time() - start_time)
# Only a real panel's swaps wait on its refresh: the emulator and the
# fallback canvas pace themselves, so "slower than the cap" would be
# noise there.
if self.matrix is not None and os.environ.get('EMULATOR', 'false') != 'true':
self.frame_timing.plan_refresh(self.refresh_hz)
self._setup_scan_order_compensation()
font_time = time.time()
@@ -1501,7 +1506,9 @@ class DisplayManager:
fractional-pixel motion. See src/common/scroll_config.py.
Note this is the configured *cap*, not necessarily what the panel
achieves -- scripts/scroll_speeds.py --measure reports the real rate.
achieves -- scripts/scroll_speeds.py --measure reports the real rate,
and the frame-timing recorder logs a warning, with a cap the panel can
hold, once it has measured a panel that falls short of this.
"""
hardware = (self.config.get('display') or {}).get('hardware') or {}
try:
-204
View File
@@ -1,204 +0,0 @@
"""
Files a plugin writes beside itself at runtime, which an update must keep.
A store update replaces a plugin's directory with a fresh download and then
deletes the old copy. Anything the plugin created there -- OAuth tokens, a
client-secrets file, a PKCE verifier, cached state -- is in no release, so the
fresh download does not contain it and deleting the old copy destroys it. On
2026-10-04 updating calendar 1.2.9 -> 1.2.12 that way deleted its
``token.pickle`` and ``credentials.json``, and the calendar stopped until they
were restored from a backup.
What counts as "the plugin's own local file" is the union of:
* :data:`KNOWN_STATE_PATTERNS` -- secret and state files plugins are known to
write, kept even when a plugin forgot to gitignore them; and
* whatever the plugin's own ``.gitignore`` (old copy or new) excludes. A file
the author ignores is by definition not part of a release.
A file the new release ships is never overwritten: tracked content wins. Byte
code (``__pycache__``, ``*.pyc``) and ``.git`` are never carried, since they
belong to the old code rather than to the user.
"""
from __future__ import annotations
import fnmatch
import os
import re
import shutil
from pathlib import Path
from typing import Iterable, List, Optional, Pattern, Tuple
__all__ = [
'KNOWN_STATE_PATTERNS',
'carry_over_local_files',
'is_known_state_file',
'local_files_to_keep',
]
# Basename globs. Kept even when the plugin's .gitignore does not list them.
KNOWN_STATE_PATTERNS: Tuple[str, ...] = (
'token.pickle',
'*.pickle',
'token.json',
'credentials.json',
'config_secrets.json',
'.pkce_code_verifier',
)
_NEVER_CARRY_DIRS = frozenset({'.git', '__pycache__'})
_NEVER_CARRY_SUFFIXES = ('.pyc', '.pyo')
def is_known_state_file(rel_path: str) -> bool:
"""True when ``rel_path``'s basename is a known secret/state file."""
name = rel_path.replace('\\', '/').rsplit('/', 1)[-1]
return any(fnmatch.fnmatchcase(name, p) for p in KNOWN_STATE_PATTERNS)
class _GitIgnore:
"""The subset of gitignore semantics plugin .gitignore files use.
Supports comments, ``!`` negation (last match wins), a trailing ``/`` for
directory-only patterns, anchoring by a leading or embedded ``/``, ``*``,
``?``, ``[...]`` and ``**``. As in git, a file under an ignored directory
is ignored regardless of later negations.
"""
def __init__(self, lines: Iterable[str]):
self._rules: List[Tuple[Pattern[str], bool, bool]] = []
for raw in lines:
line = raw.rstrip('\n').rstrip()
if not line or line.startswith('#'):
continue
negate = line.startswith('!')
if negate:
line = line[1:]
elif line.startswith('\\'):
line = line[1:]
dir_only = line.endswith('/')
line = line.rstrip('/')
if not line:
continue
anchored = '/' in line
line = line.lstrip('/')
body = self._translate(line)
regex = body if anchored else r'(?:.*/)?' + body
self._rules.append((re.compile(r'\A' + regex + r'\Z'), negate, dir_only))
@staticmethod
def _translate(pattern: str) -> str:
out, i, n = [], 0, len(pattern)
while i < n:
if pattern.startswith('**/', i):
out.append(r'(?:.*/)?')
i += 3
elif pattern.startswith('/**', i) and i + 3 == n:
out.append(r'/.*')
i += 3
elif pattern.startswith('**', i):
out.append(r'.*')
i += 2
elif pattern[i] == '*':
out.append(r'[^/]*')
i += 1
elif pattern[i] == '?':
out.append(r'[^/]')
i += 1
elif pattern[i] == '[':
end = pattern.find(']', i + 1)
if end == -1:
out.append(re.escape('['))
i += 1
else:
cls = pattern[i + 1:end]
if cls.startswith('!'):
cls = '^' + cls[1:]
out.append('[' + cls.replace('\\', '\\\\') + ']')
i = end + 1
else:
out.append(re.escape(pattern[i]))
i += 1
return ''.join(out)
def _decide(self, rel: str, is_dir: bool) -> Optional[bool]:
verdict = None
for regex, negate, dir_only in self._rules:
if dir_only and not is_dir:
continue
if regex.match(rel):
verdict = not negate
return verdict
def ignores(self, rel_path: str) -> bool:
if not self._rules:
return False
parts = rel_path.replace('\\', '/').split('/')
for depth in range(1, len(parts)):
if self._decide('/'.join(parts[:depth]), True):
return True
return bool(self._decide('/'.join(parts), False))
def _read_gitignore(plugin_dir: Path) -> List[str]:
try:
return (plugin_dir / '.gitignore').read_text(
encoding='utf-8', errors='replace').splitlines()
except OSError:
return []
def local_files_to_keep(old_dir: Path, new_dir: Path) -> List[str]:
"""Relative paths (``/``-separated) in ``old_dir`` to copy into ``new_dir``.
Regular files only; symlinks and anything the new release already ships
are skipped.
"""
old_dir, new_dir = Path(old_dir), Path(new_dir)
ignore = _GitIgnore(_read_gitignore(old_dir) + _read_gitignore(new_dir))
keep: List[str] = []
for root, dirs, files in os.walk(old_dir):
dirs[:] = sorted(d for d in dirs if d not in _NEVER_CARRY_DIRS
and not os.path.islink(os.path.join(root, d)))
rel_root = os.path.relpath(root, old_dir)
for name in sorted(files):
if name.endswith(_NEVER_CARRY_SUFFIXES):
continue
full = os.path.join(root, name)
if os.path.islink(full) or not os.path.isfile(full):
continue
rel = name if rel_root == '.' else f"{rel_root}/{name}".replace('\\', '/')
if not (is_known_state_file(rel) or ignore.ignores(rel)):
continue
if os.path.lexists(new_dir / rel):
continue
keep.append(rel)
return keep
def carry_over_local_files(
old_dir: Path, new_dir: Path
) -> Tuple[List[str], List[Tuple[str, str]]]:
"""Copy the plugin's local files from ``old_dir`` into ``new_dir``.
Copies rather than moves, so ``old_dir`` stays a complete copy until the
caller deletes it. Returns ``(copied, failed)`` where ``failed`` pairs a
relative path with the error; the caller should keep ``old_dir`` when
anything failed.
"""
copied: List[str] = []
failed: List[Tuple[str, str]] = []
try:
candidates = local_files_to_keep(old_dir, new_dir)
except OSError as e:
return copied, [('.', str(e))]
for rel in candidates:
dest = Path(new_dir) / rel
try:
dest.parent.mkdir(parents=True, exist_ok=True)
shutil.copy2(Path(old_dir) / rel, dest)
copied.append(rel)
except OSError as e:
failed.append((rel, str(e)))
return copied, failed
+4 -33
View File
@@ -22,7 +22,6 @@ from src.plugin_system.plugin_loader import (
contained_plugin_dir, requirements_to_install,
)
from src.plugin_system.plugin_dirs import BACKUP_MARKER
from src.plugin_system.plugin_local_files import carry_over_local_files
from src.plugin_system.repo_urls import (
USER_AGENT, github_api_headers, github_owner_repo, normalize_repo_url,
)
@@ -93,9 +92,7 @@ class _InstallMixin:
raise
if installed:
self._discard_backup(
plugin_id, backup_path, "install",
new_path=self._existing_install(plugin_id) or plugin_path)
self._discard_backup(plugin_id, backup_path, "install")
return True
self._restore_backup(plugin_id, plugin_path, backup_path, "Install")
@@ -136,33 +133,8 @@ class _InstallMixin:
return f"could not set aside {plugin_path}: {e}"
return None
def _discard_backup(
self, plugin_id: str, backup_path: Path, action: str,
new_path: Optional[Path] = None,
) -> None:
"""Remove the set-aside copy after a successful (re)install.
With ``new_path`` (where the new copy landed), first carries the
plugin's own runtime files -- OAuth tokens, client secrets, anything
its .gitignore excludes -- from the old copy into the new one: no
release contains them, so deleting the old copy would destroy them.
See src/plugin_system/plugin_local_files.py. If any could not be
copied the old copy is kept, so nothing is lost.
"""
if new_path is not None and new_path.is_dir():
copied, failed = carry_over_local_files(backup_path, new_path)
if copied:
self.logger.info(
"Kept %d local file(s) of %s across the %s: %s",
len(copied), plugin_id, action, ", ".join(copied))
if failed:
self.logger.error(
"Could not carry %s's local files into the new copy (%s); "
"the previous copy is kept at %s -- copy them back by hand",
plugin_id,
"; ".join(f"{rel}: {err}" for rel, err in failed),
backup_path)
return
def _discard_backup(self, plugin_id: str, backup_path: Path, action: str) -> None:
"""Remove the set-aside copy after a successful (re)install."""
if not self._safe_remove_directory(backup_path):
self.logger.warning(
"%s of %s succeeded but the previous copy at %s could not be "
@@ -570,8 +542,7 @@ class _InstallMixin:
raise
temp_dir = None # Prevent cleanup since we moved it
if backup_path is not None:
self._discard_backup(
plugin_id, backup_path, "install", new_path=final_path)
self._discard_backup(plugin_id, backup_path, "install")
# Install dependencies
self._install_dependencies(final_path)
+5 -24
View File
@@ -10,9 +10,6 @@ import subprocess # nosec B404 - list-form argv only, no shell # nosemgrep
from pathlib import Path
from typing import Dict, Optional, Tuple
from src.plugin_system.plugin_dirs import BACKUP_MARKER
from src.plugin_system.plugin_local_files import (
KNOWN_STATE_PATTERNS, is_known_state_file,
)
from src.plugin_system.repo_urls import same_repo
@@ -305,11 +302,7 @@ class _UpdateMixin:
installed = False
if installed:
# install_plugin may land the new copy under the manifest id
# rather than the old directory name.
self._discard_backup(
plugin_id, backup_path, "update",
new_path=self._existing_install(plugin_id) or plugin_path)
self._discard_backup(plugin_id, backup_path, "update")
return True
# Bad network, registry error...: the user keeps a working plugin.
@@ -516,12 +509,8 @@ class _UpdateMixin:
for line in untracked_result.stdout.strip().split('\n'):
if line.startswith('??'):
# Untracked file
file_path = line[3:].strip().strip('"')
# Tokens and secrets stay out of the
# stash (see below), so they alone are
# not a reason to stash.
if not is_known_state_file(file_path):
untracked_files.append(file_path)
file_path = line[3:].strip()
untracked_files.append(file_path)
# Check for tracked file changes
status_result = subprocess.run(
@@ -548,17 +537,9 @@ class _UpdateMixin:
if has_changes:
self.logger.info(f"Stashing local changes in {plugin_id} before update")
try:
# Use -u to include untracked files in stash --
# except the plugin's tokens and secrets, which a
# repo may have forgotten to gitignore. The stash
# is never popped, so a stashed token.pickle would
# vanish from the plugin and break it.
stash_cmd = (
['git', '-C', str(plugin_path), 'stash', 'push', '-u',
'-m', f'LEDMatrix auto-stash before update {plugin_id}', '--', '.']
+ [f':(exclude,glob)**/{p}' for p in KNOWN_STATE_PATTERNS])
# Use -u to include untracked files in stash
stash_result = subprocess.run(
stash_cmd,
['git', '-C', str(plugin_path), 'stash', 'push', '-u', '-m', f'LEDMatrix auto-stash before update {plugin_id}'],
capture_output=True,
text=True,
timeout=30,
+9
View File
@@ -175,6 +175,15 @@
"POST"
]
],
[
"/api/v3/config/refresh-rate",
"api_v3.get_refresh_rate",
[
"GET",
"HEAD",
"OPTIONS"
]
],
[
"/api/v3/config/schedule",
"api_v3.get_schedule_config",
+51
View File
@@ -807,3 +807,54 @@ def test_a_process_with_the_gc_monitor_exits_cleanly():
assert proc.returncode == 0, proc.stderr
assert "Exception ignored" not in proc.stderr
assert "installed at exit: False" in proc.stdout
SLOW = 1 / 110.0 # a panel that cannot reach a 120 Hz cap
def _windows(recorder, n, interval, start=0.0):
for i in range(n):
_feed(recorder, [interval] * 200, start=start + 50.0 * i)
_aggregate(recorder)
def _shortfall_warnings(caplog):
return [r for r in caplog.records
if r.name == "src.common.frame_timing" and "Limit Refresh Rate" in r.getMessage()]
def test_a_panel_slower_than_its_cap_is_reported_once(tmp_path, caplog):
r = _recorder(tmp_path)
r.plan_refresh(120.0)
caplog.set_level("WARNING")
_windows(r, 3, SLOW) # adopted on the 2nd window, checked on the 4th
assert _shortfall_warnings(caplog) == []
_windows(r, 3, SLOW, start=1000.0)
warnings = _shortfall_warnings(caplog)
assert len(warnings) == 1
assert "about 110 Hz" in warnings[0].getMessage()
assert "to 100 Hz" in warnings[0].getMessage()
def test_a_panel_that_reaches_its_cap_is_not_reported(tmp_path, caplog):
r = _recorder(tmp_path)
r.plan_refresh(100.0)
caplog.set_level("WARNING")
_windows(r, 6, PERIOD)
assert _shortfall_warnings(caplog) == []
def test_without_a_planned_rate_nothing_is_checked(tmp_path, caplog):
# The emulator and the fallback canvas: DisplayManager never calls
# plan_refresh(), since their frames are not paced by a panel.
r = _recorder(tmp_path)
caplog.set_level("WARNING")
_windows(r, 6, SLOW)
assert _shortfall_warnings(caplog) == []
def test_the_snapshot_records_the_planned_rate(tmp_path):
r = _recorder(tmp_path)
assert r.snapshot()["planned_refresh_hz"] is None
r.plan_refresh(120.0)
assert r.snapshot()["planned_refresh_hz"] == 120.0
+6 -14
View File
@@ -21,18 +21,6 @@ SPORTS_MODES = ['nfl_live', 'nfl_recent', 'nfl_upcoming',
'ncaa_fb_live', 'ncaa_fb_recent', 'ncaa_fb_upcoming']
def _last_write(cache_manager, key):
"""The last ``cache_manager.set(key, ...)`` call.
Not simply the last ``set`` call: the controller's font-usage publisher
thread writes ``font_usage_snapshot`` to the same cache manager whenever
it wakes, so on a slow runner it can land after the write under test.
"""
writes = [c for c in cache_manager.set.call_args_list if c.args and c.args[0] == key]
assert writes, f"nothing was written to {key!r}"
return writes[-1]
def _sports_plugin(has_live_content=False):
plugin = MagicMock(spec=['display', 'has_live_content', 'has_live_priority',
'get_live_modes'])
@@ -99,7 +87,8 @@ class TestANamedLiveModeIsShown:
def test_the_named_mode_survives_a_restart(self, football):
football._activate_on_demand({'plugin_id': 'football-scoreboard',
'mode': 'ncaa_fb_live'})
saved = _last_write(football.cache_manager, 'display_on_demand_config')
saved = football.cache_manager.set.call_args_list[-1]
assert saved.args[0] == 'display_on_demand_config'
config = saved.args[1]
assert config['named_mode'] == 'ncaa_fb_live'
@@ -131,7 +120,10 @@ class TestARestoreWithNothingToResume:
def test_it_is_reported_as_an_error(self, restored):
assert restored.on_demand_status == 'error'
assert restored.on_demand_last_error == 'restore-failed'
published = _last_write(restored.cache_manager, 'display_on_demand_state')
# The last on-demand state write, not the last write of any key: the
# font-usage publisher thread writes its own key at its own pace.
published = [c for c in restored.cache_manager.set.call_args_list
if c.args and c.args[0] == 'display_on_demand_state'][-1]
assert published.args[1]['status'] == 'error'
assert published.args[1]['error'] == 'restore-failed'
+44
View File
@@ -21,6 +21,7 @@ from src.common.scroll_config import ( # noqa: E402
refresh_hz_from_config,
resolve,
)
from src.common import scroll_config # noqa: E402
class FakeHelper:
@@ -504,3 +505,46 @@ class TestSpeedAdvice:
got = solve_crisp(50, 125.74)
assert got.steppiness == "smooth"
assert got.pixels_per_frame == 1
class TestRefreshShortfall:
"""A panel that cannot reach its cap runs every scroll slow."""
def test_the_ledmatrix_rig_is_told_to_cap_at_100(self):
# Pi 4, 2x128x64 on adafruit-hat-pwm under a 120 Hz cap: measured
# 107.6-113.1 Hz, and frame_timing reports the fast end.
s = scroll_config.refresh_shortfall(113.1, 120)
assert s == {"measured_hz": 113.1, "planned_hz": 120.0,
"suggested_cap_hz": 100, "slow_percent": 6}
def test_a_panel_that_holds_its_cap_is_fine(self):
assert scroll_config.refresh_shortfall(99.95, 100) is None
assert scroll_config.refresh_shortfall(97.5, 100) is None
def test_a_panel_that_beats_its_cap_is_fine(self):
assert scroll_config.refresh_shortfall(125.7, 120) is None
def test_nothing_measured_says_nothing(self):
assert scroll_config.refresh_shortfall(None, 120) is None
assert scroll_config.refresh_shortfall(0, 120) is None
assert scroll_config.refresh_shortfall("fast", 120) is None
def test_the_suggestion_leaves_headroom_under_the_measurement(self):
assert scroll_config.holdable_cap(113.1) == 100
assert scroll_config.holdable_cap(95.0) == 90
# 5% under 105 is 99.75: 100 would sit inside the panel's drift.
assert scroll_config.holdable_cap(105.0) == 90
assert scroll_config.holdable_cap(9.0) is None
assert scroll_config.holdable_cap(None) is None
def test_the_log_line_names_the_cap_to_use(self):
text = scroll_config.describe_refresh_shortfall(
scroll_config.refresh_shortfall(113.1, 120))
assert "about 113 Hz" in text and "120 Hz" in text
assert "6% slower" in text
assert "Set Limit Refresh Rate to 100 Hz" in text
def test_no_suggestion_for_a_panel_too_slow_for_any_cap(self):
text = scroll_config.describe_refresh_shortfall(
scroll_config.refresh_shortfall(9.0, 100))
assert "Set Limit Refresh Rate" not in text
-244
View File
@@ -1,244 +0,0 @@
"""A plugin update must keep the files the plugin wrote beside itself.
Field incident, 2026-10-04: updating calendar 1.2.9 -> 1.2.12 from the web UI
replaced plugin-repos/calendar/ with the fresh download and deleted the old
copy -- and with it token.pickle and credentials.json, the plugin's Google
OAuth files. No release contains them (the repo gitignores them), so the hot
reload logged "Credentials file not found" and the calendar stayed broken
until the files were restored by hand.
Both update routes are covered: a monorepo plugin (registry ``plugin_path``),
which is reinstalled into a fresh directory, and a plugin installed from its
own git repository, which is updated with ``git pull`` after an auto-stash.
"""
import json
import shutil
import subprocess
import pytest
from src.plugin_system.plugin_local_files import (
is_known_state_file, local_files_to_keep,
)
from src.plugin_system.store_manager import PluginStoreManager
PLUGIN_ID = "calendar"
def _manifest(version):
return {"id": PLUGIN_ID, "name": "Calendar", "class_name": "CalendarPlugin",
"display_modes": ["calendar"], "version": version}
def _write_release(target, version):
"""What a download of ``version`` puts on disk."""
target.mkdir(parents=True, exist_ok=True)
(target / "manifest.json").write_text(json.dumps(_manifest(version)))
(target / "manager.py").write_text(f"VERSION = {version!r}\n")
(target / ".gitignore").write_text("credentials.json\ntoken.pickle\ncache/\n")
def _drop_local_files(plugin_dir):
"""What the plugin writes at runtime: OAuth files plus cached state."""
(plugin_dir / "token.pickle").write_bytes(b"\x80\x04oauth-token")
(plugin_dir / "credentials.json").write_text('{"installed": {}}')
(plugin_dir / "cache").mkdir()
(plugin_dir / "cache" / "events.json").write_text("[]")
def _assert_local_files_kept(plugin_dir):
assert (plugin_dir / "token.pickle").read_bytes() == b"\x80\x04oauth-token"
assert (plugin_dir / "credentials.json").read_text() == '{"installed": {}}'
assert (plugin_dir / "cache" / "events.json").read_text() == "[]"
def _leftover_backups(plugins_dir):
return [p.name for p in plugins_dir.iterdir() if "standalone-backup" in p.name]
@pytest.fixture
def store(tmp_path, monkeypatch):
mgr = PluginStoreManager(
plugins_dir=str(tmp_path / "plugin-repos"),
uninstalled_registry_path=str(tmp_path / "uninstalled.json"))
mgr.plugins_dir.mkdir(parents=True, exist_ok=True)
monkeypatch.setattr(mgr, "_install_dependencies", lambda *a, **k: True)
monkeypatch.setattr(mgr, "fetch_registry", lambda *a, **k: {"plugins": []})
return mgr
class TestMonorepoUpdate:
@pytest.fixture
def installed(self, store, monkeypatch):
registry_entry = {
"id": PLUGIN_ID, "repo": "https://github.com/ChuckBuilds/ledmatrix-plugins",
"plugin_path": "plugins/calendar", "branch": "main",
"latest_version": "1.2.9",
}
monkeypatch.setattr(store, "get_plugin_info", lambda *a, **k: registry_entry)
release = {"version": "1.2.9"}
def fake_monorepo_download(download_url, plugin_subpath, target):
assert plugin_subpath == "plugins/calendar"
_write_release(target, release["version"])
return True
monkeypatch.setattr(store, "_install_from_monorepo", fake_monorepo_download)
assert store.install_plugin(PLUGIN_ID) is True
def publish(version):
registry_entry["latest_version"] = release["version"] = version
return store, store.plugins_dir / PLUGIN_ID, publish
def test_update_keeps_token_and_gitignored_files(self, installed):
store, plugin_dir, publish = installed
_drop_local_files(plugin_dir)
publish("1.2.12")
assert store.update_plugin(PLUGIN_ID) is True
assert json.loads((plugin_dir / "manifest.json").read_text())["version"] == "1.2.12"
_assert_local_files_kept(plugin_dir)
assert _leftover_backups(store.plugins_dir) == []
def test_token_is_kept_even_when_the_release_does_not_gitignore_it(self, installed):
store, plugin_dir, publish = installed
(plugin_dir / ".gitignore").unlink()
(plugin_dir / "token.pickle").write_bytes(b"tok")
(plugin_dir / "config_secrets.json").write_text("{}")
publish("1.2.12")
assert store.update_plugin(PLUGIN_ID) is True
assert (plugin_dir / "token.pickle").read_bytes() == b"tok"
assert (plugin_dir / "config_secrets.json").read_text() == "{}"
def test_release_content_wins_and_old_code_is_not_carried(self, installed):
store, plugin_dir, publish = installed
# A file the old copy had that the new release dropped, byte code, and
# an old copy of a file the new release also ships.
(plugin_dir / "removed_module.py").write_text("OLD = True\n")
(plugin_dir / "__pycache__").mkdir()
(plugin_dir / "__pycache__" / "manager.cpython-313.pyc").write_bytes(b"pyc")
publish("1.2.12")
assert store.update_plugin(PLUGIN_ID) is True
assert not (plugin_dir / "removed_module.py").exists()
assert not (plugin_dir / "__pycache__").exists()
assert "1.2.12" in (plugin_dir / "manager.py").read_text()
def test_reinstall_over_an_existing_copy_keeps_them_too(self, installed):
store, plugin_dir, publish = installed
_drop_local_files(plugin_dir)
assert store.install_plugin(PLUGIN_ID) is True
_assert_local_files_kept(plugin_dir)
assert _leftover_backups(store.plugins_dir) == []
class TestInstallFromUrlReplace:
def test_replacing_an_installed_copy_keeps_the_token(self, store, monkeypatch):
plugin_dir = store.plugins_dir / PLUGIN_ID
_write_release(plugin_dir, "1.0.0")
_drop_local_files(plugin_dir)
def fake_clone(repo_url, target, branches):
_write_release(target, "2.0.0")
return "main"
monkeypatch.setattr(store, "_install_via_git", fake_clone)
result = store.install_from_url(
"https://github.com/example/ledmatrix-calendar", plugin_id=PLUGIN_ID)
assert result["success"] is True
assert json.loads((plugin_dir / "manifest.json").read_text())["version"] == "2.0.0"
_assert_local_files_kept(plugin_dir)
def _git(*args, cwd):
subprocess.run(["git", "-c", "user.email=t@example.com", "-c", "user.name=t",
"-c", "core.autocrlf=false", *args],
cwd=cwd, check=True, capture_output=True)
@pytest.mark.skipif(shutil.which("git") is None, reason="git not installed")
class TestGitRepoUpdate:
@pytest.fixture
def cloned(self, store, tmp_path, monkeypatch):
monkeypatch.setattr(store, "get_plugin_info", lambda *a, **k: None)
upstream = tmp_path / "upstream"
_write_release(upstream, "1.0.0")
# This repo does NOT gitignore the token: an untracked, non-ignored
# file is exactly what `git stash push -u` used to sweep away.
(upstream / ".gitignore").write_text("cache/\n")
_git("init", "-q", "-b", "main", cwd=upstream)
_git("add", ".", cwd=upstream)
_git("commit", "-qm", "1.0.0", cwd=upstream)
plugin_dir = store.plugins_dir / PLUGIN_ID
_git("clone", "-q", str(upstream), str(plugin_dir), cwd=tmp_path)
def publish(version):
(upstream / "manifest.json").write_text(json.dumps(_manifest(version)))
_git("commit", "-qam", version, cwd=upstream)
return store, plugin_dir, publish
def test_pull_update_keeps_untracked_token(self, cloned):
store, plugin_dir, publish = cloned
_drop_local_files(plugin_dir)
# An unrelated untracked file, so the update really does stash.
(plugin_dir / "notes.txt").write_text("scratch")
publish("1.1.0")
assert store.update_plugin(PLUGIN_ID) is True
assert json.loads((plugin_dir / "manifest.json").read_text())["version"] == "1.1.0"
_assert_local_files_kept(plugin_dir)
def test_token_alone_does_not_trigger_a_stash(self, cloned):
store, plugin_dir, publish = cloned
(plugin_dir / "token.pickle").write_bytes(b"tok")
publish("1.1.0")
assert store.update_plugin(PLUGIN_ID) is True
assert (plugin_dir / "token.pickle").read_bytes() == b"tok"
stashes = subprocess.run(["git", "-C", str(plugin_dir), "stash", "list"],
capture_output=True, text=True, check=True)
assert stashes.stdout.strip() == ""
class TestWhatIsKept:
@pytest.mark.parametrize("path,expected", [
("token.pickle", True),
("data/session.pickle", True),
("credentials.json", True),
("token.json", True),
("config_secrets.json", True),
(".pkce_code_verifier", True),
("manager.py", False),
("config.json", False),
])
def test_known_state_files(self, path, expected):
assert is_known_state_file(path) is expected
def test_gitignore_rules(self, tmp_path):
old, new = tmp_path / "old", tmp_path / "new"
new.mkdir()
for rel in ["a.log", "logs/x.txt", "sub/deep/b.log", "keep.log",
"anchored.txt", "sub/anchored.txt", "assets/x/y_backup/z.png",
"manager.py", "shipped.log"]:
(old / rel).parent.mkdir(parents=True, exist_ok=True)
(old / rel).write_text("x")
(new / "shipped.log").write_text("new")
(old / ".gitignore").write_text(
"# comment\n*.log\n!keep.log\nlogs/\n/anchored.txt\n"
"assets/**/*_backup/\n")
assert local_files_to_keep(old, new) == [
"a.log", "anchored.txt", "assets/x/y_backup/z.png",
"logs/x.txt", "sub/deep/b.log",
]
@@ -0,0 +1,63 @@
"""GET /api/v3/config/refresh-rate: the cap, the measured rate, a cap to hold."""
import json
from unittest.mock import MagicMock
import pytest
from flask import Flask
from web_interface.blueprints.api_v3 import api_v3
@pytest.fixture
def client(monkeypatch, tmp_path):
stats = tmp_path / "stats.json"
monkeypatch.setattr("src.common.frame_timing.default_stats_path", lambda: str(stats))
manager = MagicMock()
manager.load_config.return_value = {
"display": {"hardware": {"limit_refresh_rate_hz": 120}}}
monkeypatch.setattr(api_v3, "config_manager", manager, raising=False)
app = Flask(__name__)
app.register_blueprint(api_v3, url_prefix="/api/v3")
c = app.test_client()
c.stats_path = stats
return c
def _get(client):
body = client.get("/api/v3/config/refresh-rate").get_json()
assert body["status"] == "success"
return body["data"]
def test_nothing_measured_yet(client):
data = _get(client)
assert data == {"planned_hz": 120.0, "measured_hz": None, "shortfall": None}
def test_a_panel_short_of_its_cap_gets_a_cap_it_can_hold(client):
client.stats_path.write_text(json.dumps(
{"measured_refresh_hz": 110.4, "planned_refresh_hz": 120.0}))
data = _get(client)
assert data["measured_hz"] == 110.4
assert data["shortfall"]["suggested_cap_hz"] == 100
assert data["shortfall"]["slow_percent"] == 8
def test_a_panel_at_its_cap_has_no_shortfall(client):
client.stats_path.write_text(json.dumps(
{"measured_refresh_hz": 121.3, "planned_refresh_hz": 120.0}))
assert _get(client)["shortfall"] is None
def test_a_file_written_under_another_cap_is_stale(client):
# The cap was changed to 120 but the display still runs under 100 Hz.
client.stats_path.write_text(json.dumps(
{"measured_refresh_hz": 99.9, "planned_refresh_hz": 100.0}))
data = _get(client)
assert data["measured_hz"] is None
assert data["shortfall"] is None
def test_a_file_from_a_display_too_old_to_record_its_cap_still_counts(client):
client.stats_path.write_text(json.dumps({"measured_refresh_hz": 110.4}))
assert _get(client)["shortfall"]["suggested_cap_hz"] == 100
@@ -1131,17 +1131,9 @@ class TestPixletEditorHostDefaultsButDoesNotOverride:
class FakeProcess:
pid = 424242
real_popen = mod.subprocess.Popen
def fake_popen(cmd, *args, env=None, **kwargs):
# Only the editor launch is faked. Patching subprocess.Popen
# patches it for the whole request, and the captive-portal
# before_request hook runs `systemctl is-active hostapd` through
# subprocess.run whenever its 30s cache has expired -- which
# needs a real process (run() uses it as a context manager).
if str(script) not in cmd:
return real_popen(cmd, *args, env=env, **kwargs)
captured['env'] = env
if env is not None:
captured['env'] = env
return FakeProcess()
with patch.object(mod, '_validate_starlark_app_path',
+31 -3
View File
@@ -158,11 +158,16 @@ def _panel_refresh_hz(config):
cap = scroll_config.refresh_hz_from_config(config)
try:
with open(frame_timing.default_stats_path(), encoding='utf-8') as fh:
measured = float(json.load(fh).get('measured_refresh_hz') or 0)
stats = json.load(fh)
measured = float(stats.get('measured_refresh_hz') or 0)
planned = float(stats.get('planned_refresh_hz') or 0)
except (OSError, ValueError, TypeError, AttributeError):
measured = planned = 0.0
# Reject a stale file from a previous hardware config: one written under
# another cap (the display has not restarted since it changed), or, from
# a display too old to record its cap, a measurement far off this one.
if planned and abs(planned - cap) > 0.5:
measured = 0.0
# Reject a stale file from a previous hardware config: a measurement far
# off the cap says the config changed since it was written.
if measured > 0 and 0.5 * cap <= measured <= 1.5 * cap:
return measured, 'measured'
return cap, 'configured'
@@ -189,6 +194,29 @@ def get_scroll_speed_advice():
return jsonify({'status': 'success', 'data': advice})
@api_v3.route('/config/refresh-rate', methods=['GET'])
def get_refresh_rate():
"""The refresh cap, what the panel measured, and a cap it can hold.
Backs the hint under the Display tab's Limit Refresh Rate field. Scroll
speeds are solved against the cap, so a panel that cannot reach it runs
every scroll slow; ``shortfall`` (None when the panel keeps up, or nothing
has been measured yet) says by how much and suggests a cap.
"""
from src.common import scroll_config
if not api_v3.config_manager:
return jsonify({'status': 'error', 'message': 'Config manager not initialized'}), 500
config = api_v3.config_manager.load_config()
planned = scroll_config.refresh_hz_from_config(config)
hz, source = _panel_refresh_hz(config)
measured = hz if source == 'measured' else None
return jsonify({'status': 'success', 'data': {
'planned_hz': planned,
'measured_hz': round(measured, 1) if measured else None,
'shortfall': scroll_config.refresh_shortfall(measured, planned),
}})
@api_v3.route('/config/schedule', methods=['GET'])
def get_schedule_config():
"""Get current schedule configuration"""
@@ -318,6 +318,7 @@
min="0"
max="1000"
class="form-control">
<p id="limit_refresh_rate_hz_hint" class="mt-1 text-xs text-amber-700" aria-live="polite"></p>
</div>
</div>
@@ -915,6 +916,41 @@ document.getElementById('brightness').addEventListener('input', function() {
});
}
// Say so when the panel cannot reach its refresh cap. Scroll speeds are
// worked out against the cap, so every scroll then runs slow, and the
// display has measured a cap the panel can hold.
(function refreshRateHint() {
const hint = document.getElementById('limit_refresh_rate_hz_hint');
const input = document.getElementById('limit_refresh_rate_hz');
if (!hint || !input) return;
fetch('/api/v3/config/refresh-rate')
.then(function(r) { return r.json(); })
.then(function(body) {
const s = body.status === 'success' && body.data.shortfall;
hint.textContent = '';
if (!s) return;
hint.appendChild(document.createTextNode(
'This panel refreshes at about ' + Math.round(s.measured_hz) +
' Hz, below this ' + Math.round(s.planned_hz) + ' Hz cap, so scrolls run about ' +
s.slow_percent + '% slower than set.' + (s.suggested_cap_hz ? ' ' : '')));
if (!s.suggested_cap_hz) return;
const btn = document.createElement('button');
btn.type = 'button';
btn.className = 'underline font-medium';
btn.textContent = 'Use ' + s.suggested_cap_hz + ' Hz';
btn.addEventListener('click', function() {
input.value = s.suggested_cap_hz;
input.dispatchEvent(new Event('input', {bubbles: true}));
input.dispatchEvent(new Event('change', {bubbles: true}));
hint.textContent = 'Save, then restart the display, to apply ' +
s.suggested_cap_hz + ' Hz.';
});
hint.appendChild(btn);
hint.appendChild(document.createTextNode(', a cap it can hold.'));
})
.catch(function() { hint.textContent = ''; });
})();
// Declared before first use: let is not hoisted usably.
let scrollHintTimer = null;
let scrollHintSeq = 0;