Compare commits

...
Author SHA1 Message Date
Chuck 782731052d fix(store): serialize concurrent updates per plugin, check cleanup results
CodeRabbit review on #405 flagged two things in
_reinstall_with_rollback, both verified against current code:

- Real race: the web UI runs Flask with threaded=True and there's a
  single update route, so two overlapping requests for the same
  plugin_id (double-click, two tabs) can interleave. The loser could
  rename the winner's in-progress install aside mid-download, deleting
  its own rollback safety net — worse than the bug this function
  exists to fix. Added a lazy per-plugin_id lock dict (mirrors the
  plugin_manager per-plugin lock pattern) held for the whole function.
- _safe_remove_directory's return value was ignored at both call
  sites. Stale-aside cleanup failure now aborts cleanly instead of
  falling through to a rename that would fail anyway with a less
  useful error; post-success backup-removal failure now logs instead
  of failing silently (still returns True — the update itself
  succeeded, and the next update self-heals the leftover aside).

Left the third nitpick (test_stale_aside_from_previous_crash_is_cleared)
addressed by asserting the stale dir is actually gone and that
install_plugin was reached, rather than just the end-to-end result.

Added a concurrency regression test asserting install_plugin never
runs for the same plugin_id while another call is in flight.
2026-07-13 09:11:21 -04:00
ChuckandClaude Fable 5 641296990d fix(store): plugin updates keep the old install until the new one succeeds
Both reinstall paths in update_plugin — the monorepo-migration remote
switch AND the routine archive update every store user hits — deleted
the installed plugin directory BEFORE downloading its replacement. A
mid-update failure (bad network, registry error) permanently destroyed
the plugin. Seen in the field: a Pi with broken DNS lost 12 plugins in
one update pass during the monorepo migration.

New _reinstall_with_rollback: rename the old install aside (using the
'.standalone-backup-' name pattern plugin discovery already excludes),
run install_plugin, remove the aside on success — restore it on ANY
failure, clearing partial-download debris first. A stale aside from a
previous crash is cleared before starting.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqzC1nzTWL4kaqgMaQZFam
2026-07-12 17:00:05 -04:00
05e7c43b27 fix(plugins): replace dependency marker files with a real satisfaction check (#390)
* fix(plugins): replace dependency marker files with a real satisfaction check

The .dependencies_installed hash-marker system only tracked "was this exact
requirements.txt hashed before" — not whether the packages it names are
actually present. That made it fragile (a wiped venv, a manually removed
package, or a lost/corrupted marker forces a needless full pip reinstall or,
worse, a false skip) and produced dead weight for the ~10 plugins whose
requirements.txt is comment-only (they still paid a pip subprocess on first
boot before a marker existed).

Replace it with requirements_are_satisfied() in plugin_loader.py, which
checks each real requirement line against importlib.metadata directly, so
install_dependencies() only shells out to pip when something is actually
missing or version-mismatched. Drops the marker file entirely: removed all
marker read/write sites in plugin_loader.py and store_manager.py, the
now-pointless marker-cleanup step in the git-update path, the unused legacy
marker implementation in plugin_manager.py, and the already-stale
clear_dependency_markers.sh script.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KEZK1P1Q1fu5pcuVrkrCFZ

* fix(security): close path-injection gap in dependency-satisfaction checks

CodeQL flagged 2 new high-severity "uncontrolled data used in path
expression" alerts at the open() calls inside this PR's new
requirements_has_real_deps()/requirements_are_satisfied() -- both are
reachable from paths that were never run through the basename+trusted-base
sanitiser this codebase already uses elsewhere:

- PluginLoader.install_dependencies() only applied that sanitiser when its
  optional plugins_dir argument was actually passed; the "no plugins_dir"
  branch trusted plugin_dir_real directly. Made plugins_dir required (not
  Optional) so that branch can't exist, and added an explicit guard in
  load_plugin() so install_deps=True without a plugins_dir fails loudly
  instead of silently. Production's only real caller (PluginManager) always
  passes plugins_dir already; the harness/dev-server/render-plugin callers
  all use install_deps=False and are unaffected.

- StoreManager._install_dependencies() never sanitised plugin_path at all,
  and its call sites ultimately derive that path from a plugin's own
  manifest.json "id" field (install_plugin_from_url) -- a malicious plugin
  could otherwise point requirements_file outside plugins_dir. Applied the
  same os.path.basename()-based containment pattern PluginLoader already
  uses (and that CodeQL recognises as a real sanitiser).

Added test_install_dependencies_requires_plugins_dir and
test_install_dependencies_rejects_path_outside_plugins_dir to lock in the
actual security property, not just quiet the scanner. Verified: all 20
tests in test_plugin_loader.py pass, plus the PR's existing test plan
(test_plugin_system.py, test_store_manager_caches.py: 53 passed) and the
full CI plugin-safety suite (test_harness.py, test_visual_rendering.py,
test_plugin_matrix.py: 52 passed, 2 pre-existing skips) all still pass.

* fix(security): replace basename-only sanitiser with a trusted-enumeration check

The previous commit's os.path.basename() + os.path.join() pattern (which a
pre-existing code comment claimed CodeQL recognises as a sanitiser) did not
actually clear the alert -- the next CodeQL run still flagged the same 2
sink lines, plus a new one at the os.path.join() call itself. Taking a
substring of tainted data apparently isn't treated as a barrier by this
query, whatever the comment assumed.

Replaced it with find_trusted_subdir(): enumerate the trusted plugins_dir
via os.scandir() and only use a name that scandir itself produced, matched
by equality against the caller's requested name. The path is then built
from that enumerated entry, not from the caller's string -- a value
sourced from iterating a trusted, non-tainted directory carries no taint
regardless of what it happens to equal, which is a stronger and more
conventional allowlist-style barrier than string-stripping. Applied
identically in both PluginLoader.install_dependencies() and
StoreManager._install_dependencies(), sharing one implementation.

Re-verified: all 65 tests across test_plugin_loader.py (20, including the
2 new security regression tests), test_store_manager_caches.py (35),
test_plugin_system.py (10) pass, plus the full CI plugin-safety suite
(test_harness.py/test_visual_rendering.py/test_plugin_matrix.py: 52
passed, 2 pre-existing skips).

* fix(security): redact URL credentials from pip subprocess output before logging

CodeQL flagged 3 clear-text-logging-of-secrets alerts in
install_requirements_file() (src/common/permission_utils.py:353,360,371).
Pre-existing on main, unrelated to this PR's own diff, but now visible
since the path-injection alerts that previously took priority in the
annotation list are fixed.

The underlying risk is real: pip can echo a private index URL's embedded
basic-auth credentials (from a requirements.txt --index-url line or
PIP_INDEX_URL) back verbatim in its own stderr/stdout on failure, and this
function both logs that output directly and returns it to callers --
store_manager.py's _install_dependencies() logs result.stderr from this
same function too.

Added _redact_url_credentials(), applied immediately after each of the two
subprocess.run() calls (mutating result.stderr/stdout in place) rather
than patching each log call site individually. This closes the leak at
the source: every downstream use -- the three flagged log lines, the
"note" string embedded in the returned stdout, and store_manager.py's own
logging of the returned result -- gets the redacted text for free.

Verified the fixed-phrase "denied" check (`"a password is required" in
result.stderr`) is unaffected, since URL syntax and those phrases don't
overlap -- covered explicitly by
test_does_not_touch_denied_check_phrases. Added
test/test_permission_utils.py (6 tests) covering the redaction helper
directly and both subprocess.run() call sites (the sudo-wrapper branch,
which this repo's scripts/fix_perms/safe_pip_install.sh makes live, and
the no-wrapper fallback branch). All pass.

* fix(security): stop interpolating req_file/pip-output into log calls

The previous commit's redaction (mutating result.stderr/stdout right after
each subprocess.run()) didn't clear CodeQL's clear-text-logging alerts --
same lesson as the path-injection fix earlier in this PR: a static
analyzer can't tell "this value was already sanitised two lines up" from
"this is still the raw tainted value" just by looking at a single log
call in isolation, so it conservatively keeps flagging it regardless of
what the redaction function actually does.

Removed all dynamic interpolation (req_file, result.stderr) from the 3
flagged logger.warning() calls entirely, replacing them with fixed
messages plus (for the one that had it) result.returncode, which is a
plain int with no possible taint. The full redacted detail is still
available where it actually matters -- in the returned
CompletedProcess.stderr/stdout and the "note" text -- just not duplicated
into a log line a scanner has to reason about in isolation.

Re-verified: all 6 test_permission_utils.py tests still pass (they assert
on the returned result, not log call arguments), plus the full
test_plugin_loader.py/test_store_manager_caches.py/test_plugin_system.py
suite (71 passed, 1 pre-existing deselect, 4 subtests).

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-11 08:55:50 -04:00
ChuckandGitHub 2ffc57cf40 fix(plugin-harness): add no-op process_deferred_updates to test double (#391)
The safety harness's VisualTestDisplayManager (base of
BoundsCheckingDisplayManager) doesn't implement process_deferred_updates(),
which 5 first-party ledmatrix-plugins call unconditionally between
set_scrolling_state() and their scroll-position update: news, odds-ticker,
ledmatrix-leaderboard, stock-news, and ledmatrix-stocks. Any of them fails
the harness with AttributeError the moment it's touched (surfaced when
ledmatrix-plugins#177 had to add a local hasattr guard in ledmatrix-stocks
just to pass CI). Add the method as a no-op, mirroring the existing
"no-op for testing" pattern already used for set_scrolling_state, so these
plugins render under the harness without every touching PR needing its own
guard.
2026-07-11 08:55:03 -04:00
ChuckandGitHub aab0e9ade0 fix(plugin-manager): fix TypeError breaking every plugin's scheduled update (#392)
run_scheduled_updates()'s resource-monitor branch wrapped the update call
in a closure stored as a *class* attribute on a dynamically-built type
(type('obj', (object,), {'update': monitored_update})()). The descriptor
protocol turns a function found via class-attribute lookup into a bound
method on instance access, silently prepending the synthetic instance as
an implicit first argument -- but monitored_update() takes none, so every
call raised "monitored_update() takes 0 positional arguments but 1 was
given", was caught by run_scheduled_updates' try/except, and recorded as
an update failure.

self.resource_monitor is None by default and was dormant until PR #388
("activate dormant plugin health/metrics subsystem") wired it up in both
display_controller.py and web_interface/app.py -- meaning this bug went
live in every real deployment as of that merge (2026-07-09) despite the
buggy line itself dating back to 2025-12-27. In practice this means no
plugin's update() has succeeded since upgrading past #388: circuit
breakers cycle through half-open -> immediate failure -> reopened every
health-check interval forever, and all plugin data (scores, odds, prices,
etc.) goes stale from whatever was last fetched before the upgrade.
Confirmed live on a running instance: odds-ticker (and stock-news,
ledmatrix-stocks, baseball-scoreboard, ledmatrix-leaderboard, of-the-day)
failing this exact way every 5-minute circuit-breaker retry.

Fixed by using types.SimpleNamespace(update=monitored_update) instead of
a dynamic class: SimpleNamespace stores attributes on the instance
itself, so attribute lookup returns the plain function unchanged --
never routed through the class-attribute descriptor protocol that
injects an implicit self.

Added test_run_scheduled_updates_calls_update_with_resource_monitor to
test/test_plugin_system.py using a real PluginResourceMonitor (not a
mock of it), so the test exercises the actual descriptor-binding
behavior that caused this. Verified the test fails with the exact
reported error against the pre-fix code and passes against the fix.
2026-07-11 08:54:30 -04:00
12 changed files with 714 additions and 244 deletions
+3
View File
@@ -43,6 +43,9 @@ websocket-client>=1.8.0,<2.0.0
# JSON Schema validation # JSON Schema validation
jsonschema>=4.20.0,<5.0.0 jsonschema>=4.20.0,<5.0.0
# Requirement specifier parsing (plugin dependency satisfaction checks)
packaging>=23.0,<27.0
# Testing dependencies # Testing dependencies
pytest>=9.0.3,<10.0.0 pytest>=9.0.3,<10.0.0
pytest-cov>=4.1.0,<5.0.0 pytest-cov>=4.1.0,<5.0.0
-29
View File
@@ -1,29 +0,0 @@
#!/bin/bash
# Clear all plugin dependency markers to force fresh dependency check
# Useful after updating plugins or troubleshooting dependency issues
echo "Clearing plugin dependency markers..."
# Check both possible cache locations
CACHE_DIRS=(
"/var/cache/ledmatrix"
"$HOME/.cache/ledmatrix"
)
for CACHE_DIR in "${CACHE_DIRS[@]}"; do
if [ -d "$CACHE_DIR" ]; then
echo "Checking $CACHE_DIR..."
marker_count=$(find "$CACHE_DIR" -name "plugin_*_deps_installed" 2>/dev/null | wc -l)
if [ "$marker_count" -gt 0 ]; then
echo "Found $marker_count dependency marker(s) in $CACHE_DIR"
find "$CACHE_DIR" -name "plugin_*_deps_installed" -delete
echo "Cleared $marker_count marker(s)"
else
echo "No dependency markers found in $CACHE_DIR"
fi
fi
done
echo "Done! Dependency markers cleared."
echo "Next startup will check and install dependencies as needed."
+43 -8
View File
@@ -8,6 +8,7 @@ files that need to be accessible by both root service and web user.
import os import os
import logging import logging
import re
import shutil as _shutil import shutil as _shutil
import subprocess import subprocess
import sys import sys
@@ -16,6 +17,25 @@ from typing import Optional
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
# Matches the credentials portion of a "scheme://user:pass@host" URL, so pip's
# own error output can be logged/displayed without echoing back a private
# index URL's embedded basic-auth secret verbatim (e.g. from a
# requirements.txt --index-url line or the PIP_INDEX_URL env var).
_URL_CREDENTIALS_RE = re.compile(r'://[^/\s@:]+:[^/\s@]+@')
def _redact_url_credentials(text: Optional[str]) -> str:
"""Replace embedded user:pass@ URL credentials in text with a placeholder.
Safe to call on any subprocess output destined for logs: it only ever
shortens/replaces the credential substring, never changes the presence
or absence of the specific fixed phrases callers check for
(e.g. "a password is required"), so it can't affect control flow.
"""
if not text:
return text or ""
return _URL_CREDENTIALS_RE.sub('://***:***@', text)
# System directories that should never have their permissions modified # System directories that should never have their permissions modified
# These directories have special system-level permissions that must be preserved # These directories have special system-level permissions that must be preserved
PROTECTED_SYSTEM_DIRECTORIES = { # nosec B108 - these are checked to PREVENT permission changes, not to use as temp paths PROTECTED_SYSTEM_DIRECTORIES = { # nosec B108 - these are checked to PREVENT permission changes, not to use as temp paths
@@ -338,6 +358,13 @@ def install_requirements_file(req_file: Path, timeout: int = 300) -> subprocess.
["sudo", "-n", bash_path, str(wrapper), str(req_file)], ["sudo", "-n", bash_path, str(wrapper), str(req_file)],
capture_output=True, text=True, timeout=timeout, cwd=str(project_root) capture_output=True, text=True, timeout=timeout, cwd=str(project_root)
) )
# Redact immediately: pip can echo a private index URL's embedded
# basic-auth credentials back in its own error/progress output
# (e.g. from a requirements.txt --index-url line). Doesn't affect
# the fixed-phrase "denied" check below -- those phrases never
# overlap with URL syntax.
result.stderr = _redact_url_credentials(result.stderr)
result.stdout = _redact_url_credentials(result.stdout)
if result.returncode == 0: if result.returncode == 0:
return result return result
# Distinguish "sudo rejected this exact command line" (worth # Distinguish "sudo rejected this exact command line" (worth
@@ -348,16 +375,24 @@ def install_requirements_file(req_file: Path, timeout: int = 300) -> subprocess.
for phrase in ("a password is required", "is not allowed to run", "no tty present") for phrase in ("a password is required", "is not allowed to run", "no tty present")
) )
if not denied: if not denied:
# Deliberately don't interpolate req_file or the pip output here:
# this log line is scanner-visible, and a static analyzer can't
# tell "already redacted above" from "still raw" just by looking
# at this call in isolation. The full (redacted) text is still
# available to callers via the returned CompletedProcess.
logger.warning( logger.warning(
"Root pip install failed (rc=%s) for %s: %s", "Root pip install failed (rc=%s); see the returned "
result.returncode, req_file, result.stderr.strip()[:500], "CompletedProcess.stderr for details.",
result.returncode,
) )
return result return result
# Same reasoning as above: no req_file / pip-output interpolation in
# this log line, only in the returned note/CompletedProcess.
logger.warning( logger.warning(
"Root pip install wrapper denied via sudo for %s; falling back to " "Root pip install wrapper denied via sudo for all candidates; "
"user-level install: %s", "falling back to user-level install. See the returned "
req_file, result.stderr.strip()[:500] if result else "no bash candidates found", "CompletedProcess.stderr for details."
) )
note = ( note = (
f"[Root install unavailable ({(result.stderr.strip() if result else 'sudo denied') or 'sudo denied'}); " f"[Root install unavailable ({(result.stderr.strip() if result else 'sudo denied') or 'sudo denied'}); "
@@ -367,8 +402,7 @@ def install_requirements_file(req_file: Path, timeout: int = 300) -> subprocess.
) )
else: else:
logger.warning( logger.warning(
"safe_pip_install.sh not found; falling back to user-level install for %s", "safe_pip_install.sh not found; falling back to user-level install."
req_file,
) )
note = ( note = (
"[safe_pip_install.sh not found; installed for the current process's " "[safe_pip_install.sh not found; installed for the current process's "
@@ -386,6 +420,7 @@ def install_requirements_file(req_file: Path, timeout: int = 300) -> subprocess.
[sys.executable, "-m", "pip", "install", "--break-system-packages", "--ignore-installed", "-r", str(req_file)], [sys.executable, "-m", "pip", "install", "--break-system-packages", "--ignore-installed", "-r", str(req_file)],
capture_output=True, text=True, timeout=timeout, cwd=str(project_root) capture_output=True, text=True, timeout=timeout, cwd=str(project_root)
) )
result.stdout = note + (result.stdout or "") result.stderr = _redact_url_credentials(result.stderr)
result.stdout = note + _redact_url_credentials(result.stdout)
return result return result
+139 -67
View File
@@ -5,10 +5,10 @@ Handles plugin module imports, dependency installation, and class instantiation.
Extracted from PluginManager to improve separation of concerns. Extracted from PluginManager to improve separation of concerns.
""" """
import hashlib
import json
import importlib import importlib
import importlib.metadata
import importlib.util import importlib.util
import json
import os import os
import sys import sys
import subprocess import subprocess
@@ -17,12 +17,101 @@ from pathlib import Path
from typing import Dict, Any, Optional, Tuple, Type from typing import Dict, Any, Optional, Tuple, Type
import logging import logging
from packaging.requirements import InvalidRequirement, Requirement
from src.exceptions import PluginError from src.exceptions import PluginError
from src.logging_config import get_logger from src.logging_config import get_logger
from src.common.permission_utils import (
ensure_file_permissions,
get_plugin_file_mode def requirements_has_real_deps(requirements_file: str) -> bool:
) """
Check whether a requirements.txt actually specifies anything to install.
Plugins that ship all their dependencies with LEDMatrix core often keep a
requirements.txt where every line is commented out, for documentation
purposes only. Running pip against such a file still pays the full
subprocess/resolver cost for zero effect, so callers should skip the
install step entirely when this returns False.
"""
try:
with open(requirements_file, 'r', encoding='utf-8') as fh:
for line in fh:
line = line.strip()
if line and not line.startswith('#'):
return True
except OSError:
# Let the caller's own file handling report the error.
return True
return False
def requirements_are_satisfied(requirements_file: str) -> bool:
"""
Check whether every real requirement line in requirements.txt is already
satisfied by packages installed in the current interpreter.
This replaces marker-file tracking with a direct fact check, so it's
immune to stale/missing/corrupted markers: it looks at what's actually
importable right now rather than trusting a hash comparison from a
previous run. Anything ambiguous (pip options, unparseable lines,
extras, unresolvable versions) conservatively returns False so the
caller falls through to running pip — this check only ever saves work,
never masks a real install.
"""
try:
with open(requirements_file, 'r', encoding='utf-8') as fh:
lines = fh.readlines()
except OSError:
return False
for raw_line in lines:
line = raw_line.strip()
if not line or line.startswith('#'):
continue
if line.startswith('-'):
return False # pip option (-r, --index-url, ...), can't verify
try:
req = Requirement(line)
except InvalidRequirement:
return False
if req.extras:
return False # verifying extras' sub-dependencies isn't worth it here
if req.marker is not None and not req.marker.evaluate():
continue # not applicable on this platform/interpreter
try:
installed_version = importlib.metadata.version(req.name)
except importlib.metadata.PackageNotFoundError:
return False
if req.specifier and not req.specifier.contains(installed_version, prereleases=True):
return False
return True
def find_trusted_subdir(trusted_dir: str, name: str) -> Optional[str]:
"""Return `name` if it names an actual subdirectory of trusted_dir, else None.
Used as a containment check for a directory name derived from untrusted
input (a manifest-declared plugin id, an externally-supplied plugin
path): the returned value always comes from enumerating trusted_dir
itself via os.scandir(), so a caller that builds a path by joining
trusted_dir with this return value is joining against a name the
filesystem produced under a trusted root -- not the caller's original
string, which could otherwise smuggle a traversal sequence through.
"""
try:
with os.scandir(trusted_dir) as entries:
for entry in entries:
if entry.name == name and entry.is_dir():
return entry.name
except OSError:
pass
return None
class PluginLoader: class PluginLoader:
@@ -139,7 +228,7 @@ class PluginLoader:
self, self,
plugin_dir: Path, plugin_dir: Path,
plugin_id: str, plugin_id: str,
plugins_dir: Optional[Path] = None, plugins_dir: Path,
timeout: int = 300 timeout: int = 300
) -> bool: ) -> bool:
""" """
@@ -148,7 +237,12 @@ class PluginLoader:
Args: Args:
plugin_dir: Plugin directory path plugin_dir: Plugin directory path
plugin_id: Plugin identifier plugin_id: Plugin identifier
plugins_dir: Trusted base plugins directory for path containment check plugins_dir: Trusted base plugins directory for path containment check.
Required (not optional) so every caller reconstructs the plugin
path through the sanitiser below rather than trusting plugin_dir
directly -- CodeQL's path-injection query (and a malicious
manifest/plugin_id in practice) can't tell a legitimate
plugin_dir from one crafted to traverse outside plugins_dir.
timeout: Installation timeout in seconds timeout: Installation timeout in seconds
Returns: Returns:
@@ -160,59 +254,42 @@ class PluginLoader:
# Resolve to a canonical absolute path (normalises .. and symlinks) # Resolve to a canonical absolute path (normalises .. and symlinks)
plugin_dir_real = os.path.realpath(str(plugin_dir)) plugin_dir_real = os.path.realpath(str(plugin_dir))
plugins_dir_real = os.path.realpath(str(plugins_dir))
requested_name = os.path.basename(plugin_dir_real)
if plugins_dir is not None: # Match the requested directory against an entry actually enumerated
# Reconstruct the plugin path from a trusted base + a sanitised # from the trusted plugins_dir, and build the path from that entry --
# directory name. os.path.basename() is CodeQL's recognised # not from requested_name. A name that came out of os.scandir() on a
# py/path-injection sanitiser: it strips all directory components # trusted root carries no taint regardless of what the caller asked
# so the result cannot contain traversal sequences. Joining it # for, so this is a real containment guarantee (an allowlist check
# with the resolved, trusted plugins_dir produces a path that # against a trusted source), not a string-sanitisation of untrusted
# CodeQL considers untainted. # input that a static analyzer has to trust blindly.
plugins_dir_real = os.path.realpath(str(plugins_dir)) matched_name = find_trusted_subdir(plugins_dir_real, requested_name)
safe_dir_name = os.path.basename(plugin_dir_real) if matched_name is None:
if not safe_dir_name: self.logger.error(
self.logger.error("Could not determine plugin directory name for %s", plugin_id) "Plugin directory for %s not found inside plugins dir", plugin_id
return False )
safe_plugin_dir = os.path.join(plugins_dir_real, safe_dir_name) return False
if not os.path.isdir(safe_plugin_dir):
self.logger.error(
"Plugin directory for %s not found inside plugins dir", plugin_id
)
return False
else:
safe_plugin_dir = plugin_dir_real
if not os.path.isdir(safe_plugin_dir):
self.logger.error("Plugin directory does not exist: %s", plugin_dir)
return False
safe_plugin_dir = os.path.join(plugins_dir_real, matched_name)
requirements_file = os.path.join(safe_plugin_dir, "requirements.txt") requirements_file = os.path.join(safe_plugin_dir, "requirements.txt")
marker_file = os.path.join(safe_plugin_dir, ".dependencies_installed")
if not os.path.isfile(requirements_file): if not os.path.isfile(requirements_file):
return True # No dependencies needed return True # No dependencies needed
try: if not requirements_has_real_deps(requirements_file):
with open(requirements_file, 'rb') as fh: self.logger.debug(
current_hash = hashlib.sha256(fh.read()).hexdigest() "requirements.txt for %s has no real dependencies (comments/blank only), skipping pip",
except OSError as e: plugin_id
self.logger.error("Failed to read requirements.txt for %s: %s", plugin_id, e) )
return False return True
# Skip if requirements.txt hasn't changed since last install if requirements_are_satisfied(requirements_file):
if os.path.isfile(marker_file): self.logger.debug(
try: "Dependencies for %s already satisfied in current environment, skipping pip",
with open(marker_file, 'r', encoding='utf-8') as fh: plugin_id
stored_hash = fh.read().strip() )
except OSError as e: return True
self.logger.warning(
"Could not read dependency marker for %s (%s), will reinstall dependencies",
plugin_id, e
)
else:
if stored_hash == current_hash:
self.logger.debug("Dependencies already installed for %s (requirements unchanged)", plugin_id)
return True
self.logger.info("Requirements changed for %s, reinstalling dependencies", plugin_id)
try: try:
self.logger.info("Installing dependencies for plugin %s...", plugin_id) self.logger.info("Installing dependencies for plugin %s...", plugin_id)
@@ -225,12 +302,6 @@ class PluginLoader:
) )
if result.returncode == 0: if result.returncode == 0:
try:
with open(marker_file, 'w', encoding='utf-8') as fh:
fh.write(current_hash)
ensure_file_permissions(Path(marker_file), get_plugin_file_mode())
except OSError as marker_err:
self.logger.debug("Could not write dependency marker for %s: %s", plugin_id, marker_err)
self.logger.info("Dependencies installed successfully for %s", plugin_id) self.logger.info("Dependencies installed successfully for %s", plugin_id)
return True return True
else: else:
@@ -242,8 +313,7 @@ class PluginLoader:
# the system copy instead of trying to replace it — matching the # the system copy instead of trying to replace it — matching the
# retry already used by install_dependencies_apt.py / safe_pip_install.sh. # retry already used by install_dependencies_apt.py / safe_pip_install.sh.
# Without this retry, the plugin would silently keep running against # Without this retry, the plugin would silently keep running against
# whatever version the system happened to ship, even though the # whatever version the system happened to ship.
# marker below claims the requirement is satisfied.
if "uninstall-no-record-file" in stderr: if "uninstall-no-record-file" in stderr:
self.logger.warning( self.logger.warning(
"Dependencies for %s conflict with a system-managed package " "Dependencies for %s conflict with a system-managed package "
@@ -280,12 +350,6 @@ class PluginLoader:
"system-managed version satisfies the requirement", "system-managed version satisfies the requirement",
plugin_id plugin_id
) )
try:
with open(marker_file, 'w', encoding='utf-8') as fh:
fh.write(current_hash)
ensure_file_permissions(Path(marker_file), get_plugin_file_mode())
except OSError as marker_err:
self.logger.debug("Could not write dependency marker for %s: %s", plugin_id, marker_err)
return True return True
self.logger.warning( self.logger.warning(
"Dependency installation returned non-zero exit code for %s: %s", "Dependency installation returned non-zero exit code for %s: %s",
@@ -653,6 +717,14 @@ class PluginLoader:
""" """
# Install dependencies if needed # Install dependencies if needed
if install_deps: if install_deps:
if plugins_dir is None:
raise PluginError(
f"plugins_dir is required to install dependencies for plugin {plugin_id} "
"(needed for path containment; pass install_deps=False if the caller "
"doesn't have a trusted plugins directory to supply)",
plugin_id=plugin_id,
context={'plugin_dir': str(plugin_dir)},
)
if not self.install_dependencies(plugin_dir, plugin_id, plugins_dir=plugins_dir): if not self.install_dependencies(plugin_dir, plugin_id, plugins_dir=plugins_dir):
raise PluginError( raise PluginError(
f"Dependency installation failed for plugin {plugin_id} in {plugin_dir}", f"Dependency installation failed for plugin {plugin_id} in {plugin_dir}",
+12 -86
View File
@@ -9,9 +9,9 @@ API Version: 1.0.0
import json import json
import sys import sys
import subprocess
import time import time
import threading import threading
import types
from pathlib import Path from pathlib import Path
from typing import Dict, List, Optional, Any from typing import Dict, List, Optional, Any
import logging import logging
@@ -177,90 +177,6 @@ class PluginManager:
return plugin_ids return plugin_ids
def _get_dependency_marker_path(self, plugin_id: str) -> Path:
"""Get path to dependency installation marker file."""
plugin_dir = self.plugins_dir / plugin_id
if not plugin_dir.exists():
# Try with ledmatrix- prefix
plugin_dir = self.plugins_dir / f"ledmatrix-{plugin_id}"
return plugin_dir / ".dependencies_installed"
def _check_dependencies_installed(self, plugin_id: str) -> bool:
"""Check if dependencies are already installed for a plugin."""
marker_path = self._get_dependency_marker_path(plugin_id)
return marker_path.exists()
def _mark_dependencies_installed(self, plugin_id: str) -> None:
"""Mark dependencies as installed for a plugin."""
marker_path = self._get_dependency_marker_path(plugin_id)
try:
marker_path.touch()
# Set proper file permissions after creating marker
from src.common.permission_utils import (
ensure_file_permissions,
get_plugin_file_mode
)
ensure_file_permissions(marker_path, get_plugin_file_mode())
except (OSError, PermissionError) as e:
self.logger.warning("Could not create dependency marker for %s: %s", plugin_id, e)
def _remove_dependency_marker(self, plugin_id: str) -> None:
"""Remove dependency installation marker."""
marker_path = self._get_dependency_marker_path(plugin_id)
try:
if marker_path.exists():
marker_path.unlink()
except (OSError, PermissionError) as e:
self.logger.warning("Could not remove dependency marker for %s: %s", plugin_id, e)
def _install_plugin_dependencies(self, requirements_file: Path) -> bool:
"""
Install plugin dependencies from requirements.txt.
Args:
requirements_file: Path to requirements.txt
Returns:
True if installation succeeded or not needed, False on error
"""
try:
self.logger.info("Installing dependencies from %s", requirements_file)
result = subprocess.run(
[sys.executable, "-m", "pip", "install", "--break-system-packages", "--no-cache-dir", "-r", str(requirements_file)],
capture_output=True,
text=True,
timeout=300,
check=False
)
if result.returncode == 0:
self.logger.info("Dependencies installed successfully")
return True
else:
self.logger.warning("Dependency installation returned non-zero exit code: %s", result.stderr)
return False
except subprocess.TimeoutExpired:
self.logger.error("Dependency installation timed out")
return False
except FileNotFoundError as e:
self.logger.warning("Command not found: %s. Skipping dependency installation", e)
return True
except (BrokenPipeError, OSError) as e:
# Handle broken pipe errors (errno 32) which can occur during pip downloads
# Often caused by network interruptions or output buffer issues
if isinstance(e, OSError) and e.errno == 32:
self.logger.error(
"Broken pipe error during dependency installation. "
"This usually indicates a network interruption or pip output buffer issue. "
"Try installing again or check your network connection."
)
else:
self.logger.error("OS error during dependency installation: %s", e)
return False
except Exception as e:
self.logger.error("Unexpected error installing dependencies: %s", e, exc_info=True)
return True
def load_plugin(self, plugin_id: str) -> bool: def load_plugin(self, plugin_id: str) -> bool:
""" """
Load a plugin by ID. Load a plugin by ID.
@@ -828,8 +744,18 @@ class PluginManager:
# If resource monitor exists, wrap the call # If resource monitor exists, wrap the call
def monitored_update(): def monitored_update():
self.resource_monitor.monitor_call(plugin_id, plugin_instance.update) self.resource_monitor.monitor_call(plugin_id, plugin_instance.update)
# SimpleNamespace stores `update` as an *instance*
# attribute, so attribute lookup returns the plain
# function object as-is. A dynamically-built class
# (`type(..., {'update': monitored_update})`) instead
# stores it as a *class* attribute, which the
# descriptor protocol turns into a bound method on
# access -- silently prepending the instance as an
# implicit first argument to a function that takes
# none, raising "monitored_update() takes 0
# positional arguments but 1 was given" on every call.
success = self.plugin_executor.execute_update( success = self.plugin_executor.execute_update(
type('obj', (object,), {'update': monitored_update})(), types.SimpleNamespace(update=monitored_update),
plugin_id plugin_id
) )
else: else:
+126 -34
View File
@@ -5,7 +5,6 @@ Handles plugin discovery, installation, updates, and uninstallation
from both the official registry and custom GitHub repositories. from both the official registry and custom GitHub repositories.
""" """
import hashlib
import os import os
import re import re
import json import json
@@ -26,6 +25,9 @@ import logging
from urllib.parse import urlparse from urllib.parse import urlparse
from src.common.permission_utils import sudo_remove_directory, install_requirements_file from src.common.permission_utils import sudo_remove_directory, install_requirements_file
from src.plugin_system.plugin_loader import (
requirements_has_real_deps, requirements_are_satisfied, find_trusted_subdir
)
try: try:
from jsonschema import Draft7Validator, ValidationError from jsonschema import Draft7Validator, ValidationError
@@ -140,9 +142,28 @@ class PluginStoreManager:
# then get the result from the warm cache (double-checked locking). # then get the result from the warm cache (double-checked locking).
self._registry_fetch_lock = threading.Lock() self._registry_fetch_lock = threading.Lock()
# Per-plugin locks for _reinstall_with_rollback: the web UI runs
# Flask with threaded=True, so two overlapping requests for the
# same plugin_id (double-click, two browser tabs) would otherwise
# both rename the same directory aside — one succeeds, and the
# loser can end up renaming the winner's in-progress install aside
# mid-download, stealing its own rollback safety net. Keyed by
# plugin_id so unrelated plugins still update concurrently.
self._reinstall_locks: Dict[str, threading.Lock] = {}
self._reinstall_locks_guard = threading.Lock()
# Ensure plugins directory exists # Ensure plugins directory exists
self.plugins_dir.mkdir(exist_ok=True) self.plugins_dir.mkdir(exist_ok=True)
def _get_reinstall_lock(self, plugin_id: str) -> threading.Lock:
"""Lazily create (or fetch) the per-plugin reinstall lock."""
with self._reinstall_locks_guard:
lock = self._reinstall_locks.get(plugin_id)
if lock is None:
lock = threading.Lock()
self._reinstall_locks[plugin_id] = lock
return lock
def _record_cache_backoff(self, cache_dict: Dict, cache_key: str, def _record_cache_backoff(self, cache_dict: Dict, cache_key: str,
cache_timeout: int, payload: Any) -> None: cache_timeout: int, payload: Any) -> None:
"""Bump a cache entry's timestamp so subsequent lookups hit the """Bump a cache entry's timestamp so subsequent lookups hit the
@@ -1907,12 +1928,38 @@ class PluginStoreManager:
Returns: Returns:
True if successful or no requirements file True if successful or no requirements file
""" """
requirements_file = plugin_path / "requirements.txt" # Reconstruct the plugin path from the trusted self.plugins_dir base +
# an entry actually enumerated from it, rather than trusting
# plugin_path directly -- callers ultimately derive it from a
# plugin-supplied manifest "id" field (see install_plugin_from_url),
# so without this a malicious manifest could point requirements_file
# outside plugins_dir. find_trusted_subdir()'s return value always
# comes from os.scandir() on the trusted root, so building the path
# from it (not from the caller's string) is a real containment
# guarantee, matching the pattern in PluginLoader.install_dependencies().
plugin_dir_real = os.path.realpath(str(plugin_path))
plugins_dir_real = os.path.realpath(str(self.plugins_dir))
requested_name = os.path.basename(plugin_dir_real)
matched_name = find_trusted_subdir(plugins_dir_real, requested_name)
if matched_name is None:
self.logger.error("Plugin directory not found inside plugins dir for dependency install")
return False
safe_plugin_path = Path(os.path.join(plugins_dir_real, matched_name))
requirements_file = safe_plugin_path / "requirements.txt"
if not requirements_file.exists(): if not requirements_file.exists():
self.logger.debug(f"No requirements.txt found in {plugin_path.name}") self.logger.debug(f"No requirements.txt found in {plugin_path.name}")
return True return True
if not requirements_has_real_deps(str(requirements_file)):
self.logger.debug(f"requirements.txt for {plugin_path.name} has no real dependencies, skipping pip")
return True
if requirements_are_satisfied(str(requirements_file)):
self.logger.debug(f"Dependencies for {plugin_path.name} already satisfied, skipping pip")
return True
try: try:
self.logger.info(f"Installing dependencies for {plugin_path.name}") self.logger.info(f"Installing dependencies for {plugin_path.name}")
# Routed through the shared root-visible installer (same one the # Routed through the shared root-visible installer (same one the
@@ -1929,12 +1976,6 @@ class PluginStoreManager:
) )
return False return False
self.logger.info(f"Dependencies installed successfully for {plugin_path.name}") self.logger.info(f"Dependencies installed successfully for {plugin_path.name}")
# Write hash marker so plugin_loader skips redundant pip run on next startup
try:
current_hash = hashlib.sha256(requirements_file.read_bytes()).hexdigest()
(plugin_path / ".dependencies_installed").write_text(current_hash, encoding='utf-8')
except OSError as marker_err:
self.logger.debug("Could not write dependency marker for %s: %s", plugin_path.name, marker_err)
return True return True
except subprocess.TimeoutExpired: except subprocess.TimeoutExpired:
@@ -2241,6 +2282,74 @@ class PluginStoreManager:
self.logger.error(f"Error uninstalling plugin {plugin_id}: {e}") self.logger.error(f"Error uninstalling plugin {plugin_id}: {e}")
return False return False
def _reinstall_with_rollback(self, plugin_id: str, plugin_path: Path) -> bool:
"""Replace an installed plugin with a fresh install, atomically.
The old install is renamed aside (not deleted) until the new install
succeeds, then removed; on ANY install failure the old directory is
restored. This is the difference between a failed update and a
destroyed plugin: the previous delete-then-install flow permanently
removed plugins whenever the download failed mid-update (seen in the
field during the monorepo migration on a Pi with broken DNS — every
old-remote plugin was deleted and none could be re-downloaded).
The aside name embeds '.standalone-backup-' so plugin discovery
(plugin_manager._scan_directory_for_plugins) ignores it even though
it still contains a manifest.json.
Held for the whole operation under a per-plugin_id lock: two
overlapping requests for the same plugin (double-click, two
browser tabs — the web UI runs Flask with threaded=True) must not
interleave their renames, or the second could steal the first's
rollback safety net mid-install. Other plugin_ids are unaffected.
"""
with self._get_reinstall_lock(plugin_id):
backup_path = plugin_path.with_name(
f"{plugin_path.name}.standalone-backup-migrating")
# A stale aside from a previous crash would block the rename
if backup_path.exists():
if not self._safe_remove_directory(backup_path):
self.logger.error(
f"Could not clear stale backup for {plugin_id} at "
f"{backup_path}; leaving old install in place")
return False
try:
plugin_path.rename(backup_path)
except OSError as e:
self.logger.error(
f"Could not set aside old plugin directory for {plugin_id}: {e}")
return False
try:
installed = self.install_plugin(plugin_id)
except Exception as e:
self.logger.error(f"Reinstall of {plugin_id} raised: {e}")
installed = False
if installed:
if not self._safe_remove_directory(backup_path):
self.logger.warning(
f"Update of {plugin_id} succeeded but the old backup "
f"at {backup_path} could not be removed; it will be "
f"cleared on the next update")
return True
# Install failed (bad network, registry error...) — put the old
# version back so the user still has a working plugin.
self.logger.error(
f"Reinstall of {plugin_id} failed; restoring previous version")
try:
if plugin_path.exists():
# partial download debris from the failed install
self._safe_remove_directory(plugin_path)
backup_path.rename(plugin_path)
self.logger.info(f"Restored previous install of {plugin_id}")
except OSError as e:
self.logger.error(
f"CRITICAL: could not restore {plugin_id} from {backup_path}: {e}. "
f"The previous install is preserved there — rename it back manually.")
return False
def update_plugin(self, plugin_id: str) -> bool: def update_plugin(self, plugin_id: str) -> bool:
""" """
Update a plugin to the latest commit on its upstream branch. Update a plugin to the latest commit on its upstream branch.
@@ -2303,10 +2412,7 @@ class PluginStoreManager:
f"Plugin {resolved_id} git remote ({local_remote}) differs from registry ({registry_repo}). " f"Plugin {resolved_id} git remote ({local_remote}) differs from registry ({registry_repo}). "
f"Reinstalling from registry to migrate to new source." f"Reinstalling from registry to migrate to new source."
) )
if not self._safe_remove_directory(plugin_path): return self._reinstall_with_rollback(resolved_id, plugin_path)
self.logger.error(f"Failed to remove old plugin directory for {resolved_id}")
return False
return self.install_plugin(resolved_id)
# Check if already up to date # Check if already up to date
if remote_sha and local_sha and remote_sha.startswith(local_sha): if remote_sha and local_sha and remote_sha.startswith(local_sha):
@@ -2432,19 +2538,6 @@ class PluginStoreManager:
file_path = line[3:].strip() file_path = line[3:].strip()
untracked_files.append(file_path) untracked_files.append(file_path)
# Remove marker files that are safe to delete (they'll be regenerated)
safe_to_remove = ['.dependencies_installed']
removed_files = []
for file_name in safe_to_remove:
file_path = plugin_path / file_name
if file_path.exists() and file_name in untracked_files:
try:
file_path.unlink()
removed_files.append(file_name)
self.logger.info(f"Removed marker file {file_name} from {plugin_id} before update")
except Exception as e:
self.logger.warning(f"Could not remove {file_name} from {plugin_id}: {e}")
# Check for tracked file changes # Check for tracked file changes
status_result = subprocess.run( status_result = subprocess.run(
['git', '-C', str(plugin_path), 'status', '--porcelain', '--untracked-files=no'], ['git', '-C', str(plugin_path), 'status', '--porcelain', '--untracked-files=no'],
@@ -2455,10 +2548,9 @@ class PluginStoreManager:
) )
has_changes = bool(status_result.stdout.strip()) has_changes = bool(status_result.stdout.strip())
# If there are remaining untracked files (not safe to remove), stash them # If there are untracked files, stash them
remaining_untracked = [f for f in untracked_files if f not in removed_files] if untracked_files:
if remaining_untracked: self.logger.info(f"Found {len(untracked_files)} untracked files in {plugin_id}, will stash them")
self.logger.info(f"Found {len(remaining_untracked)} untracked files in {plugin_id}, will stash them")
has_changes = True has_changes = True
except subprocess.TimeoutExpired: except subprocess.TimeoutExpired:
# If status check times out, assume there might be changes and proceed # If status check times out, assume there might be changes and proceed
@@ -2624,11 +2716,11 @@ class PluginStoreManager:
# Plugin is not a git repo but is in registry and has a newer version - reinstall # Plugin is not a git repo but is in registry and has a newer version - reinstall
self.logger.info(f"Plugin {plugin_id} not installed via git; re-installing latest archive (registry id: {registry_id})") self.logger.info(f"Plugin {plugin_id} not installed via git; re-installing latest archive (registry id: {registry_id})")
# Remove directory and reinstall fresh # Reinstall with the old version kept aside until the new
if not self._safe_remove_directory(plugin_path): # download succeeds — this is the path every routine store
self.logger.error(f"Failed to remove old plugin directory for {plugin_id}") # update takes, and a mid-update network failure must not
return False # destroy the user's plugin.
return self.install_plugin(registry_id) return self._reinstall_with_rollback(registry_id, plugin_path)
except Exception as e: except Exception as e:
import traceback import traceback
@@ -454,6 +454,18 @@ class VisualTestDisplayManager:
"""Check if display is currently scrolling.""" """Check if display is currently scrolling."""
return self._scrolling_state['is_scrolling'] return self._scrolling_state['is_scrolling']
def process_deferred_updates(self):
"""Process any deferred updates (no-op for testing).
Several ticker-style plugins (news, odds-ticker, leaderboard,
stock-news, stocks) call this unconditionally between
set_scrolling_state() and their scroll-position update, mirroring the
real display_manager's deferred-update queue. This double has no such
queue, so there is nothing to process the no-op just lets those
plugins render under the harness instead of raising AttributeError.
"""
pass
# ------------------------------------------------------------------ # ------------------------------------------------------------------
# Utility methods # Utility methods
# ------------------------------------------------------------------ # ------------------------------------------------------------------
+10
View File
@@ -172,6 +172,16 @@ class TestVisualDisplayManager:
vdm.set_scrolling_state(False) vdm.set_scrolling_state(False)
assert vdm.is_currently_scrolling() is False assert vdm.is_currently_scrolling() is False
def test_process_deferred_updates_is_noop(self):
# Ticker-style plugins (news, odds-ticker, leaderboard, stock-news,
# stocks) call this unconditionally alongside set_scrolling_state();
# it must exist and be harmless so those plugins render under the
# harness instead of raising AttributeError.
vdm = VisualTestDisplayManager(width=128, height=32)
vdm.set_scrolling_state(True)
vdm.process_deferred_updates() # should not raise
assert vdm.is_currently_scrolling() is True
def test_format_date_with_ordinal(self): def test_format_date_with_ordinal(self):
from datetime import datetime from datetime import datetime
vdm = VisualTestDisplayManager(width=128, height=32) vdm = VisualTestDisplayManager(width=128, height=32)
+83
View File
@@ -0,0 +1,83 @@
"""
Tests for src.common.permission_utils's URL-credential redaction.
Covers the fix for a CodeQL clear-text-logging-of-secrets alert:
install_requirements_file() must never let a private index URL's embedded
user:pass@ credentials reach logs or its returned CompletedProcess, since
pip can echo that URL back verbatim in its own stderr/stdout on failure.
"""
from pathlib import Path
from unittest.mock import MagicMock, patch
from src.common.permission_utils import _redact_url_credentials, install_requirements_file
class TestRedactUrlCredentials:
def test_redacts_embedded_basic_auth(self):
text = "Could not fetch URL https://alice:s3cr3t@pypi.example.com/simple/: 403"
redacted = _redact_url_credentials(text)
assert "s3cr3t" not in redacted
assert "alice" not in redacted
assert "https://***:***@pypi.example.com/simple/" in redacted
def test_leaves_credential_free_text_unchanged(self):
text = "ERROR: Could not find a version that satisfies the requirement foo==1.0"
assert _redact_url_credentials(text) == text
def test_handles_none_and_empty(self):
assert _redact_url_credentials(None) == ""
assert _redact_url_credentials("") == ""
def test_does_not_touch_denied_check_phrases(self):
"""The fixed phrases install_requirements_file greps for must survive
redaction untouched -- they don't overlap with URL syntax, but this
pins that assumption so a regex change can't silently break it."""
text = "sudo: a password is required"
assert _redact_url_credentials(text) == text
class TestInstallRequirementsFileRedaction:
@patch('src.common.permission_utils.subprocess.run')
def test_wrapper_path_redacts_stderr_and_stdout(self, mock_run, tmp_path):
"""safe_pip_install.sh exists in this repo, so install_requirements_file
takes the sudo-wrapper branch; a failing result must come back
with any embedded index-URL credentials already redacted."""
req_file = tmp_path / "requirements.txt"
req_file.write_text("requests\n")
mock_run.return_value = MagicMock(
returncode=1,
stdout="Looking in indexes: https://bob:hunter2@pypi.internal/simple\n",
stderr="ERROR https://bob:hunter2@pypi.internal/simple/foo: 401",
)
result = install_requirements_file(req_file, timeout=5)
assert "hunter2" not in result.stdout
assert "hunter2" not in result.stderr
assert "https://***:***@pypi.internal" in result.stdout
assert "https://***:***@pypi.internal" in result.stderr
@patch('src.common.permission_utils.subprocess.run')
@patch('src.common.permission_utils.Path.exists', return_value=False)
def test_no_wrapper_fallback_path_redacts_stderr_and_stdout(self, mock_exists, mock_run, tmp_path):
"""No safe_pip_install.sh wrapper -> falls straight to the
sys.executable pip fallback (the second subprocess.run call site);
its result must come back redacted too, independent of the wrapper
branch's own redaction above."""
req_file = tmp_path / "requirements.txt"
req_file.write_text("requests\n")
mock_run.return_value = MagicMock(
returncode=1,
stdout="Looking in indexes: https://carol:swordfish@pypi.internal/simple\n",
stderr="ERROR https://carol:swordfish@pypi.internal/simple/foo: 401",
)
result = install_requirements_file(req_file, timeout=5)
assert "swordfish" not in result.stdout
assert "swordfish" not in result.stderr
assert "https://***:***@pypi.internal" in result.stdout
assert "https://***:***@pypi.internal" in result.stderr
+55 -10
View File
@@ -193,7 +193,7 @@ class TestPluginLoader:
mock_subprocess.return_value = MagicMock(returncode=0) mock_subprocess.return_value = MagicMock(returncode=0)
result = plugin_loader.install_dependencies(plugin_dir, "test_plugin") result = plugin_loader.install_dependencies(plugin_dir, "test_plugin", plugins_dir=tmp_plugins_dir)
assert result is True assert result is True
mock_subprocess.assert_called_once() mock_subprocess.assert_called_once()
@@ -204,7 +204,7 @@ class TestPluginLoader:
plugin_dir = tmp_plugins_dir / "test_plugin" plugin_dir = tmp_plugins_dir / "test_plugin"
plugin_dir.mkdir() plugin_dir.mkdir()
result = plugin_loader.install_dependencies(plugin_dir, "test_plugin") result = plugin_loader.install_dependencies(plugin_dir, "test_plugin", plugins_dir=tmp_plugins_dir)
assert result is True assert result is True
mock_subprocess.assert_not_called() mock_subprocess.assert_not_called()
@@ -219,17 +219,20 @@ class TestPluginLoader:
mock_subprocess.return_value = MagicMock(returncode=1) mock_subprocess.return_value = MagicMock(returncode=1)
result = plugin_loader.install_dependencies(plugin_dir, "test_plugin") result = plugin_loader.install_dependencies(plugin_dir, "test_plugin", plugins_dir=tmp_plugins_dir)
assert result is False assert result is False
@patch('src.plugin_system.plugin_loader.requirements_are_satisfied', return_value=False)
@patch('subprocess.run') @patch('subprocess.run')
def test_install_dependencies_retries_with_ignore_installed_on_apt_conflict( def test_install_dependencies_retries_with_ignore_installed_on_apt_conflict(
self, mock_subprocess, plugin_loader, tmp_plugins_dir self, mock_subprocess, mock_satisfied, plugin_loader, tmp_plugins_dir
): ):
"""An apt-managed package with no pip RECORD file triggers a retry with """An apt-managed package with no pip RECORD file triggers a retry with
--ignore-installed rather than silently assuming the old version satisfies --ignore-installed rather than silently assuming the old version satisfies
the requirement.""" the requirement. requirements_are_satisfied() is mocked False here because
this scenario is exactly the case where the installed (apt) version does
NOT satisfy the pin that's why pip attempts a reinstall in the first place."""
plugin_dir = tmp_plugins_dir / "test_plugin" plugin_dir = tmp_plugins_dir / "test_plugin"
plugin_dir.mkdir() plugin_dir.mkdir()
requirements_file = plugin_dir / "requirements.txt" requirements_file = plugin_dir / "requirements.txt"
@@ -242,16 +245,17 @@ class TestPluginLoader:
retry_attempt = MagicMock(returncode=0, stderr="") retry_attempt = MagicMock(returncode=0, stderr="")
mock_subprocess.side_effect = [first_attempt, retry_attempt] mock_subprocess.side_effect = [first_attempt, retry_attempt]
result = plugin_loader.install_dependencies(plugin_dir, "test_plugin") result = plugin_loader.install_dependencies(plugin_dir, "test_plugin", plugins_dir=tmp_plugins_dir)
assert result is True assert result is True
assert mock_subprocess.call_count == 2 assert mock_subprocess.call_count == 2
retry_cmd = mock_subprocess.call_args_list[1][0][0] retry_cmd = mock_subprocess.call_args_list[1][0][0]
assert "--ignore-installed" in retry_cmd assert "--ignore-installed" in retry_cmd
@patch('src.plugin_system.plugin_loader.requirements_are_satisfied', return_value=False)
@patch('subprocess.run') @patch('subprocess.run')
def test_install_dependencies_apt_conflict_retry_also_fails( def test_install_dependencies_apt_conflict_retry_also_fails(
self, mock_subprocess, plugin_loader, tmp_plugins_dir self, mock_subprocess, mock_satisfied, plugin_loader, tmp_plugins_dir
): ):
"""Still tolerates the failure (returns True) if the --ignore-installed """Still tolerates the failure (returns True) if the --ignore-installed
retry itself fails, matching the prior soft-fallback behavior.""" retry itself fails, matching the prior soft-fallback behavior."""
@@ -267,14 +271,15 @@ class TestPluginLoader:
retry_attempt = MagicMock(returncode=1, stderr="some other pip error") retry_attempt = MagicMock(returncode=1, stderr="some other pip error")
mock_subprocess.side_effect = [first_attempt, retry_attempt] mock_subprocess.side_effect = [first_attempt, retry_attempt]
result = plugin_loader.install_dependencies(plugin_dir, "test_plugin") result = plugin_loader.install_dependencies(plugin_dir, "test_plugin", plugins_dir=tmp_plugins_dir)
assert result is True assert result is True
assert mock_subprocess.call_count == 2 assert mock_subprocess.call_count == 2
@patch('src.plugin_system.plugin_loader.requirements_are_satisfied', return_value=False)
@patch('subprocess.run') @patch('subprocess.run')
def test_install_dependencies_apt_conflict_retry_times_out( def test_install_dependencies_apt_conflict_retry_times_out(
self, mock_subprocess, plugin_loader, tmp_plugins_dir self, mock_subprocess, mock_satisfied, plugin_loader, tmp_plugins_dir
): ):
"""A retry timeout must be tolerated the same way as a retry failure """A retry timeout must be tolerated the same way as a retry failure
(return True), not propagate to the outer TimeoutExpired handler and (return True), not propagate to the outer TimeoutExpired handler and
@@ -293,7 +298,47 @@ class TestPluginLoader:
subprocess.TimeoutExpired(cmd="pip", timeout=300), subprocess.TimeoutExpired(cmd="pip", timeout=300),
] ]
result = plugin_loader.install_dependencies(plugin_dir, "test_plugin") result = plugin_loader.install_dependencies(plugin_dir, "test_plugin", plugins_dir=tmp_plugins_dir)
assert result is True assert result is True
assert mock_subprocess.call_count == 2 assert mock_subprocess.call_count == 2
@patch('subprocess.run')
def test_install_dependencies_already_satisfied_skips_pip(self, mock_subprocess, plugin_loader, tmp_plugins_dir):
"""A requirement already satisfied in the current environment shouldn't invoke pip."""
plugin_dir = tmp_plugins_dir / "test_plugin"
plugin_dir.mkdir()
requirements_file = plugin_dir / "requirements.txt"
requirements_file.write_text("pytest>=1.0\n")
result = plugin_loader.install_dependencies(plugin_dir, "test_plugin", plugins_dir=tmp_plugins_dir)
assert result is True
mock_subprocess.assert_not_called()
def test_install_dependencies_requires_plugins_dir(self, plugin_loader, tmp_plugins_dir):
"""plugins_dir is a required argument, not an optional trust-me flag --
calling without it must fail loudly (TypeError) rather than silently
falling back to trusting plugin_dir unchecked."""
plugin_dir = tmp_plugins_dir / "test_plugin"
plugin_dir.mkdir()
with pytest.raises(TypeError):
plugin_loader.install_dependencies(plugin_dir, "test_plugin")
@patch('subprocess.run')
def test_install_dependencies_rejects_path_outside_plugins_dir(
self, mock_subprocess, plugin_loader, tmp_path, tmp_plugins_dir
):
"""A plugin_dir that doesn't actually live inside plugins_dir (e.g. a
manifest-derived id crafted to traverse elsewhere) must be rejected
rather than read from -- this is the path-injection containment
check CodeQL flagged as missing."""
outside_dir = tmp_path / "outside"
outside_dir.mkdir()
(outside_dir / "requirements.txt").write_text("requests>=2.0\n")
result = plugin_loader.install_dependencies(outside_dir, "evil_plugin", plugins_dir=tmp_plugins_dir)
assert result is False
mock_subprocess.assert_not_called()
+52 -2
View File
@@ -3,6 +3,7 @@ from unittest.mock import MagicMock, patch
from pathlib import Path from pathlib import Path
from src.plugin_system.plugin_manager import PluginManager from src.plugin_system.plugin_manager import PluginManager
from src.plugin_system.plugin_state import PluginState from src.plugin_system.plugin_state import PluginState
from src.plugin_system.resource_monitor import PluginResourceMonitor
class TestPluginManager: class TestPluginManager:
"""Test PluginManager functionality.""" """Test PluginManager functionality."""
@@ -78,14 +79,63 @@ class TestPluginManager:
assert result is False assert result is False
assert pm.state_manager.get_state("non_existent_plugin") == PluginState.ERROR assert pm.state_manager.get_state("non_existent_plugin") == PluginState.ERROR
def test_run_scheduled_updates_calls_update_with_resource_monitor(
self, mock_config_manager, mock_display_manager, mock_cache_manager
):
"""Regression test: run_scheduled_updates() must actually call a
plugin's update() when self.resource_monitor is set (as it is in
every real deployment -- display_controller.py and web_interface/
app.py both assign a real PluginResourceMonitor after construction).
Previously, the resource_monitor branch wrapped the call in a
function stored as a *class* attribute on a dynamically-built type
(`type('obj', (object,), {'update': monitored_update})()`), which
the descriptor protocol turns into a bound method on access --
silently passing the synthetic instance as an implicit first
argument to monitored_update(), which takes none. Every plugin's
scheduled update failed with "monitored_update() takes 0 positional
arguments but 1 was given" and was silently swallowed into a
circuit-breaker retry loop that never succeeded, so plugin data
(scores, odds, etc.) never refreshed.
"""
with patch('src.plugin_system.plugin_manager.ensure_directory_permissions'):
pm = PluginManager(
plugins_dir="plugins",
config_manager=mock_config_manager,
display_manager=mock_display_manager,
cache_manager=mock_cache_manager
)
plugin_instance = MagicMock()
plugin_instance.enabled = True
plugin_instance.update = MagicMock()
pm.plugins["test_plugin"] = plugin_instance
pm.plugin_manifests["test_plugin"] = {"update_interval": 10}
pm.state_manager.set_state("test_plugin", PluginState.ENABLED)
# Plain MagicMock, not the mock_cache_manager fixture: this test
# is about run_scheduled_updates() actually invoking update()
# through the resource-monitor wrapper, not about
# PluginResourceMonitor's own cache-backed metrics persistence
# (which calls cache_manager.get(..., memory_ttl=...) --
# a kwarg the fixture's mock_get() doesn't accept).
pm.resource_monitor = PluginResourceMonitor(MagicMock())
pm.run_scheduled_updates(current_time=time.time())
plugin_instance.update.assert_called_once()
assert "test_plugin" in pm.plugin_last_update
assert pm.state_manager.get_state("test_plugin") == PluginState.ENABLED
class TestPluginLoader: class TestPluginLoader:
"""Test PluginLoader functionality.""" """Test PluginLoader functionality."""
def test_dependency_check(self): def test_dependency_check(self):
"""Test dependency checking logic.""" """Test dependency checking logic."""
# This would test _check_dependencies_installed and _install_plugin_dependencies # Covered by test_plugin_loader.py's install_dependencies tests,
# which requires mocking subprocess calls and file operations # which exercise requirements_has_real_deps/requirements_are_satisfied
# and the pip subprocess fallback.
class TestPluginExecutor: class TestPluginExecutor:
+171
View File
@@ -0,0 +1,171 @@
"""Tests for atomic plugin updates (store_manager._reinstall_with_rollback).
Regression for a field data-loss incident: update_plugin's reinstall paths
(monorepo migration AND routine archive updates) deleted the installed
plugin BEFORE downloading its replacement a mid-update network failure
permanently destroyed the plugin. Seen live: a Pi with broken DNS lost 12
plugins from one update pass.
"""
import json
import os
import sys
import threading
import time
from pathlib import Path
from unittest.mock import patch
import pytest
sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
from src.plugin_system.store_manager import PluginStoreManager # noqa: E402
PLUGIN_ID = "rollback-test-plugin"
@pytest.fixture
def store(tmp_path):
mgr = PluginStoreManager(plugins_dir=str(tmp_path))
plugin_dir = tmp_path / PLUGIN_ID
plugin_dir.mkdir()
(plugin_dir / "manifest.json").write_text(json.dumps(
{"id": PLUGIN_ID, "name": "Rollback Test", "version": "1.0.0"}))
(plugin_dir / "manager.py").write_text("# old version marker\n")
return mgr, plugin_dir
class TestReinstallWithRollback:
def test_failed_install_restores_old_version(self, store):
"""The whole point: a failed download must leave the old install."""
mgr, plugin_dir = store
with patch.object(mgr, "install_plugin", return_value=False):
ok = mgr._reinstall_with_rollback(PLUGIN_ID, plugin_dir)
assert ok is False
assert plugin_dir.exists()
assert "old version marker" in (plugin_dir / "manager.py").read_text()
# no aside debris left behind
leftovers = [p for p in plugin_dir.parent.iterdir()
if "standalone-backup" in p.name]
assert leftovers == []
def test_install_exception_restores_old_version(self, store):
mgr, plugin_dir = store
with patch.object(mgr, "install_plugin",
side_effect=RuntimeError("network down")):
ok = mgr._reinstall_with_rollback(PLUGIN_ID, plugin_dir)
assert ok is False
assert plugin_dir.exists()
assert "old version marker" in (plugin_dir / "manager.py").read_text()
def test_successful_install_removes_aside(self, store):
mgr, plugin_dir = store
def fake_install(plugin_id):
new_dir = plugin_dir # same path, new content
new_dir.mkdir(exist_ok=True)
(new_dir / "manager.py").write_text("# new version\n")
(new_dir / "manifest.json").write_text(json.dumps(
{"id": PLUGIN_ID, "name": "Rollback Test", "version": "2.0.0"}))
return True
with patch.object(mgr, "install_plugin", side_effect=fake_install):
ok = mgr._reinstall_with_rollback(PLUGIN_ID, plugin_dir)
assert ok is True
assert "new version" in (plugin_dir / "manager.py").read_text()
leftovers = [p for p in plugin_dir.parent.iterdir()
if "standalone-backup" in p.name]
assert leftovers == []
def test_partial_download_debris_is_replaced_by_old_version(self, store):
"""A failed install that left a partial directory must still roll back."""
mgr, plugin_dir = store
def fake_partial_install(plugin_id):
plugin_dir.mkdir(exist_ok=True)
(plugin_dir / "half-downloaded.tmp").write_text("junk")
return False
with patch.object(mgr, "install_plugin", side_effect=fake_partial_install):
ok = mgr._reinstall_with_rollback(PLUGIN_ID, plugin_dir)
assert ok is False
assert "old version marker" in (plugin_dir / "manager.py").read_text()
assert not (plugin_dir / "half-downloaded.tmp").exists()
def test_stale_aside_from_previous_crash_is_cleared(self, store):
mgr, plugin_dir = store
stale = plugin_dir.parent / f"{PLUGIN_ID}.standalone-backup-migrating"
stale.mkdir()
(stale / "old.txt").write_text("stale")
with patch.object(mgr, "install_plugin", return_value=False) as mock_install:
ok = mgr._reinstall_with_rollback(PLUGIN_ID, plugin_dir)
# The reinstall itself still fails (mocked) and the old install is
# restored, but the stale aside must not have survived — otherwise
# it would have blocked this run's own rename (or a future one).
assert not stale.exists()
mock_install.assert_called_once_with(PLUGIN_ID)
assert ok is False
assert plugin_dir.exists()
assert "old version marker" in (plugin_dir / "manager.py").read_text()
def test_concurrent_updates_for_same_plugin_are_serialized(self, store):
"""Two overlapping requests for the same plugin_id (double-click,
two browser tabs the web UI runs Flask with threaded=True) must
not interleave: the loser must wait for the winner to finish
rather than renaming the winner's in-progress install aside and
stealing its rollback safety net."""
mgr, plugin_dir = store
active = 0
max_active = 0
guard = threading.Lock()
def fake_install(plugin_id):
nonlocal active, max_active
with guard:
active += 1
max_active = max(max_active, active)
time.sleep(0.05)
plugin_dir.mkdir(exist_ok=True)
(plugin_dir / "manager.py").write_text("# new version\n")
(plugin_dir / "manifest.json").write_text(json.dumps(
{"id": PLUGIN_ID, "name": "Rollback Test", "version": "2.0.0"}))
with guard:
active -= 1
return True
results = []
def worker():
results.append(mgr._reinstall_with_rollback(PLUGIN_ID, plugin_dir))
with patch.object(mgr, "install_plugin", side_effect=fake_install):
threads = [threading.Thread(target=worker) for _ in range(2)]
for t in threads:
t.start()
for t in threads:
t.join(timeout=5)
assert max_active == 1, "install_plugin ran concurrently for the same plugin_id"
assert results == [True, True]
assert plugin_dir.exists()
assert "new version" in (plugin_dir / "manager.py").read_text()
leftovers = [p for p in plugin_dir.parent.iterdir()
if "standalone-backup" in p.name]
assert leftovers == []
def test_aside_name_is_invisible_to_discovery(self, store, tmp_path):
"""The aside still contains a manifest.json — discovery must skip it
(relies on the existing '.standalone-backup-' exclusion)."""
mgr, plugin_dir = store
from src.plugin_system.plugin_manager import PluginManager
aside = plugin_dir.parent / f"{PLUGIN_ID}.standalone-backup-migrating"
plugin_dir.rename(aside)
pm = PluginManager(plugins_dir=str(tmp_path), config_manager=None,
display_manager=None, cache_manager=None)
found = pm._scan_directory_for_plugins(Path(tmp_path))
assert PLUGIN_ID not in found
if __name__ == "__main__":
sys.exit(pytest.main([__file__, "-v"]))