rick/LEDMatrix
1
0
Fork 0
mirror of https://github.com/ChuckBuilds/LEDMatrix.git synced 2026-10-05 23:05:10 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(plugins): store and plugin-manager bugs; tidy src/plugin_system (#635)

Browse Source 
* fix(store): don't read a ZIP-installed plugin's remote from the LEDMatrix repo

update_plugin looked up remote.origin.url with `git -C <plugin> config
--local` for plugins that are not git checkouts. Under plugin-repos/ git
walks up to the enclosing LEDMatrix repository, so the lookup returned
LEDMatrix's own URL and a plugin missing from the registry was
"reinstalled" from the LEDMatrix repo. Only ask git when the plugin
directory has its own .git, the test _get_local_git_info already uses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(schema): report each missing required field once, by name

validate_config_against_schema ran its own required-fields loop after
Draft7Validator.iter_errors, which already yields one `required` error
per missing field, so every missing top-level field was listed twice.
The validator's copy also printed the schema's whole `required` list
("Missing required property '['api_key', 'city']'") instead of the field.
Drop the loop and take the field name from the error itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(store): stop mangling repository URLs that contain ".git"

install_from_url and fetch_registry_from_url cleaned URLs with
`rstrip('/').replace('.git', '')`, which removes ".git" anywhere:
https://github.com/user/my.github.io became .../myhub.io, so installing
or browsing that repository asked GitHub for one that does not exist.

Add src/plugin_system/repo_urls.py with one anchored normalize_repo_url(),
same_repo() for comparisons, github_owner_repo() and github_api_headers(),
and use them for the five copies of the owner/repo parsing and GitHub
headers in the store and for saved repositories. GitHub URLs are now
recognised by urlparse().hostname everywhere: _get_latest_commit_info
used a substring test, and _install_from_monorepo_api parsed any host.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(store): install a repository whose only branch is not main/master

_install_via_git returned None both when every clone failed and when the
last-resort clone of the repository's default branch succeeded.
_install_plugin_impl papered over it with `and not plugin_path.exists()`;
install_from_url did not, so a repository whose only branch is e.g.
`develop` was cloned, then treated as a failure, then "downloaded" from
main/master archives that do not exist.

After a default-branch clone, return the branch the clone checked out
(read from .git/HEAD), so None means failure and nothing else, and give
both callers the same `branch_used is None` fallback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(plugins): judge the memory limit on each call's own growth

monitor_call stores `metrics.memory_mb = max(previous, growth)`, and
_check_limits compared that high-water mark with max_memory_mb. It never
decreases, so once one update() grew the process past the limit every
later call raised ResourceLimitExceeded and the circuit breaker kept
reopening. Pass the call's own RSS growth to _check_limits; keep the
high-water mark for reporting and document what it measures.

Remove ResourceMetrics.update_average_execution_time: nothing called it,
and it overwrote the running total with the average.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(plugins): reload_plugin re-reads the manifest from the discovered directory

reload_plugin read `plugins_dir / plugin_id / "manifest.json"`, ignoring
the discovery map and the plugin_dirs rules. For a plugin whose
directory name differs from its manifest id the path did not exist, the
re-read was skipped without a word, and the reload kept the stale
manifest. Resolve the directory with find_plugin_directory, as
load_plugin does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(plugins): drop the always-null last_display from plugin state info

PluginStateManager reported `last_display` from `_last_display`, which
nothing ever wrote, so it was null for every plugin. Recording it in
PluginExecutor.execute_display would not help: get_state_info's only
reader is the web process, whose PluginManager never calls display().
Remove the field, its dict and get_last_display() (no caller in core,
the web UI or the plugin monorepo).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(store): share the rollback and requirements helpers, drop dead code

- install_plugin and _reinstall_with_rollback set aside, discard and
  restore the old copy through _set_aside/_discard_backup/_restore_backup
  instead of two copies of the same blocks.
- The loader and the store run the same pre-pip checks through
  contained_plugin_dir() and requirements_to_install() in plugin_loader.
  They still invoke pip differently (sys.executable -m pip vs. the sudo
  wrapper). `except (BrokenPipeError, OSError)` + `isinstance(e, OSError)`
  becomes `except OSError` checking errno.EPIPE.
- load_module never returns None, so load_plugin's check is gone and the
  docstring says what it raises.
- Remove the always-true JSONSCHEMA_AVAILABLE, the inline re-imports of
  re and permission_utils, the fake status_result object nobody reads,
  hasattr(git_error, 'cmd'), a redundant "merge conflict" test and
  `import traceback` (exc_info=True does it).
- Correct comments: install_from_url names the directory for the
  caller's id when given (not always the manifest id), _get_local_git_info
  saves one git subprocess (not four), _enrich calls two helpers,
  search_plugins documents all its arguments, _find_plugin_path states
  its behaviour instead of a TODO, and history narration is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(plugins): tidy base_plugin, correct plugin_manager/state comments

- base_plugin: drop the unused `import logging`; get_display_duration
  runs the instance value and the config value through one
  _positive_seconds() helper instead of two copies of the coercion; the
  'static'/'none'/fallback branches of get_vegas_display_mode, which all
  returned FIXED_SEGMENT, are one; fix the mis-indented validate_config
  example; say that get_supported_vegas_modes/get_vegas_segment_width
  are not consulted by core (kept, plugins override them).
- schema_manager: import expand_style_elements normally rather than
  swallowing an ImportError of a core module.
- plugin_manager: the plugins directory is the configured one
  (plugin-repos/ by default), not plugins/; get_config() returns the live
  dict, not a copy, so the interval cache comments say what it saves.
- state_manager: config_version and the file version are not used to
  detect corruption; say what they are.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(plugins): stop writing data/plugin_operations.json

PluginOperationQueue wrote its finished-operation history to
data/plugin_operations.json after every operation, and read it back only
into its own in-memory list, which only get_operation_history() exposes
-- and nothing calls that. The operation-history endpoint reads
OperationHistory (data/operation_history.json). No code in src/,
web_interface/, scripts/ or test/ reads the file.

Drop the history_file/lazy_load parameters and the load/save code; the
bounded in-memory history stays. web_interface/app.py and the
integration test stop passing the removed arguments. An existing
data/plugin_operations.json is left in place (data/* is gitignored).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(changelog): plugin-system

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Chuck
2026-09-24 17:32:02 -04:00
committed by GitHub
co-authored by Claude Opus 5.5
parent 3967a6cffc
commit b11bcfa204
24 changed files with 958 additions and 756 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+4 -4
View File
@@ -81,10 +81,10 @@ assets/stocks/crypto_icons/
# Plugin operation state written at runtime. # Plugin operation state written at runtime.
# #
# web_interface/app.py writes data/plugin_operations.json, data/plugin_state.json # web_interface/app.py writes data/plugin_state.json and data/operation_history.json
# and data/operation_history.json as the web interface runs, into a directory that # (older releases also data/plugin_operations.json) as the web interface runs, into
# ships tracked (data/.gitkeep) and was otherwise unignored. So every rig that ever # a directory that ships tracked (data/.gitkeep). Unignored, every rig that ever
# opened the web UI -- and every test run that constructs the app -- left three # opened the web UI -- and every test run that constructs the app -- would leave
# untracked files behind and a permanently dirty `git status`. Same reasoning as # untracked files behind and a permanently dirty `git status`. Same reasoning as
# the logo rule above: a checkout that is always dirty is a checkout nobody reads. # the logo rule above: a checkout that is always dirty is a checkout nobody reads.
data/* data/*
CHANGELOG.md
+9
View File
@@ -33,6 +33,15 @@ accepts both, but the store flags the old spelling as deprecated
- `src/common/README.md` covers every module. - `src/common/README.md` covers every module.
- Stale setup, service and troubleshooting claims are corrected. - Stale setup, service and troubleshooting claims are corrected.
- Plugin store and plugin manager fixes:
- Updating a plugin that was installed from a ZIP no longer tries to reinstall it from the LEDMatrix repository's own URL.
- Repository URLs with `.git` in the middle are no longer mangled. The URL helpers now live in `src/plugin_system/repo_urls.py`.
- Installing from a URL works when the repository's only branch isn't `main` or `master`.
- A missing required config field is reported once, by name.
- A plugin that went over `max_memory_mb` once is no longer refused on every call after that.
- `reload_plugin` reads the manifest from the plugin's discovered directory.
- Removed: `last_display` from plugin state info and `get_last_display()` (nothing recorded them); `PluginOperationQueue`'s `history_file` and `lazy_load` arguments; and `data/plugin_operations.json`, which nothing read.
- The web service (`ledmatrix-web`) logs through `src.logging_config` like the - The web service (`ledmatrix-web`) logs through `src.logging_config` like the
display service, so `journalctl -p err -u ledmatrix-web` works. Successful display service, so `journalctl -p err -u ledmatrix-web` works. Successful
GET/HEAD/OPTIONS requests (the UI's polling) are logged at DEBUG instead of GET/HEAD/OPTIONS requests (the UI's polling) are logged at DEBUG instead of
src/plugin_system/base_plugin.py
+47 -100
View File
@@ -11,7 +11,6 @@ Stability: Stable - maintains backward compatibility
from abc import ABC, abstractmethod from abc import ABC, abstractmethod
from enum import Enum from enum import Enum
from typing import Dict, Any, Optional, List from typing import Dict, Any, Optional, List
import logging
import os import os
import sys import sys
from src.logging_config import get_logger from src.logging_config import get_logger
@@ -511,104 +510,53 @@ class BasePlugin(ABC):
""" """
Get the display duration for this plugin instance. Get the display duration for this plugin instance.
Automatically detects duration from: Uses, in order, the first positive number among:
1. self.display_duration instance variable (if exists) 1. ``self.display_duration`` (a common pattern in scoreboard plugins)
2. self.config.get("display_duration", 15.0) (fallback) 2. ``self.config["display_duration"]``
3. 15.0
Can be overridden by plugins to provide dynamic durations based Numeric strings count as numbers. Can be overridden by plugins to
on content (e.g., longer duration for more complex displays). provide dynamic durations based on content (e.g., longer duration for
more complex displays).
Returns: Returns:
Duration in seconds to display this plugin's content Duration in seconds to display this plugin's content
""" """
# Check for instance variable first (common pattern in scoreboard plugins)
if hasattr(self, 'display_duration'):
try: try:
duration = getattr(self, 'display_duration') duration = getattr(self, 'display_duration', None)
# Handle None case
if duration is None:
pass # Fall through to config
# Try to convert to float if it's a number or numeric string.
# bool is excluded: it's an int subclass, and True would
# otherwise read as a 1-second duration.
elif isinstance(duration, (int, float)) and not isinstance(duration, bool):
if duration > 0:
return float(duration)
else:
self.logger.debug(
"display_duration instance variable is non-positive (%s), using config fallback",
duration
)
# Try converting string representations of numbers
elif isinstance(duration, str):
try:
duration_float = float(duration)
if duration_float > 0:
return duration_float
else:
self.logger.debug(
"display_duration string value is non-positive (%s), using config fallback",
duration
)
except (ValueError, TypeError):
self.logger.warning(
"display_duration instance variable has invalid string value '%s', using config fallback",
duration
)
else:
self.logger.warning(
"display_duration instance variable has unexpected type %s (value: %s), using config fallback",
type(duration).__name__, duration
)
except (TypeError, ValueError, AttributeError) as e: except (TypeError, ValueError, AttributeError) as e:
# A plugin may define display_duration as a property that raises.
self.logger.warning( self.logger.warning(
"Error reading display_duration instance variable: %s, using config fallback", "Error reading display_duration instance variable: %s, using config fallback", e)
e duration = None
) if duration is not None:
seconds = self._positive_seconds(duration, "display_duration instance variable")
if seconds is not None:
return seconds
# Fall back to config seconds = self._positive_seconds(
config_duration = self.config.get("display_duration", 15.0) self.config.get("display_duration", 15.0), "config display_duration")
return seconds if seconds is not None else 15.0
def _positive_seconds(self, value: Any, source: str) -> Optional[float]:
"""``value`` as a positive float, or None (with a log line) if it is not one.
bool is rejected although it is an int subclass: True would otherwise
read as a 1-second duration.
"""
if isinstance(value, bool) or not isinstance(value, (int, float, str)):
self.logger.warning("%s has unexpected type %s (value: %s), ignoring it",
source, type(value).__name__, value)
return None
try: try:
# Ensure config value is also a valid float (bool excluded — an seconds = float(value)
# int subclass that would otherwise read True as 1 second)
if isinstance(config_duration, (int, float)) and not isinstance(config_duration, bool):
if config_duration > 0:
return float(config_duration)
else:
self.logger.debug(
"Config display_duration is non-positive (%s), using default 15.0",
config_duration
)
return 15.0
elif isinstance(config_duration, str):
try:
duration_float = float(config_duration)
if duration_float > 0:
return duration_float
else:
self.logger.debug(
"Config display_duration string is non-positive (%s), using default 15.0",
config_duration
)
return 15.0
except ValueError: except ValueError:
self.logger.warning( self.logger.warning("%s has invalid value %r, ignoring it", source, value)
"Config display_duration has invalid string value '%s', using default 15.0", return None
config_duration if seconds > 0:
) return seconds
return 15.0 self.logger.debug("%s is non-positive (%s), ignoring it", source, value)
else: return None
self.logger.warning(
"Config display_duration has unexpected type %s (value: %s), using default 15.0",
type(config_duration).__name__, config_duration
)
except (ValueError, TypeError) as e:
self.logger.warning(
"Error processing config display_duration: %s, using default 15.0",
e
)
return 15.0
# --------------------------------------------------------------------- # ---------------------------------------------------------------------
# Dynamic duration support hooks # Dynamic duration support hooks
@@ -926,25 +874,20 @@ class BasePlugin(ABC):
config_mode, self.plugin_id config_mode, self.plugin_id
) )
# Fall back to mapping legacy content_type # Fall back to mapping legacy content_type. 'none' (excluded from
content_type = self.get_vegas_content_type() # Vegas) also maps to FIXED_SEGMENT: exclusion is decided by checking
if content_type == 'multi': # get_vegas_content_type() separately.
if self.get_vegas_content_type() == 'multi':
return VegasDisplayMode.SCROLL return VegasDisplayMode.SCROLL
elif content_type == 'static':
return VegasDisplayMode.FIXED_SEGMENT
elif content_type == 'none':
# 'none' means excluded - return FIXED_SEGMENT as default
# The exclusion is handled by checking get_vegas_content_type() separately
return VegasDisplayMode.FIXED_SEGMENT
return VegasDisplayMode.FIXED_SEGMENT return VegasDisplayMode.FIXED_SEGMENT
def get_supported_vegas_modes(self) -> List[VegasDisplayMode]: def get_supported_vegas_modes(self) -> List[VegasDisplayMode]:
""" """
Return list of Vegas display modes this plugin supports. Return list of Vegas display modes this plugin supports.
Used by the web UI to show available mode options for user configuration. Not currently consulted by core: neither Vegas mode nor the web UI
Override to customize which modes are available for this plugin. calls it. It is kept, and plugins override it, as the declared set of
modes a future mode picker would offer.
By default: By default:
- 'multi' content type plugins support SCROLL and FIXED_SEGMENT - 'multi' content type plugins support SCROLL and FIXED_SEGMENT
@@ -972,6 +915,10 @@ class BasePlugin(ABC):
""" """
Get the preferred width for this plugin in Vegas FIXED_SEGMENT mode. Get the preferred width for this plugin in Vegas FIXED_SEGMENT mode.
Not currently consulted by core: Vegas mode sizes a card from the
``vegas_width_pct`` / ``vegas_scroll.render_width_pct`` settings
(see get_vegas_render_width()). Kept because plugins override it.
Returns the number of panels this plugin should occupy when displayed Returns the number of panels this plugin should occupy when displayed
as a fixed segment. The actual pixel width is calculated as: as a fixed segment. The actual pixel width is calculated as:
width = panels * single_panel_width width = panels * single_panel_width
src/plugin_system/operation_queue.py
+6 -75
View File
@@ -9,8 +9,6 @@ import threading
import queue import queue
from typing import Dict, Optional, List, Callable, Any from typing import Dict, Optional, List, Callable, Any
from datetime import datetime from datetime import datetime
from pathlib import Path
import json
from src.plugin_system.operation_types import ( from src.plugin_system.operation_types import (
PluginOperation, OperationType, OperationStatus PluginOperation, OperationType, OperationStatus
@@ -28,28 +26,22 @@ class PluginOperationQueue:
- Prevents concurrent operations on same plugin - Prevents concurrent operations on same plugin
- Operation status tracking - Operation status tracking
- Operation cancellation - Operation cancellation
- Operation history - In-memory history of finished operations
The history is not persisted. The web UI's operation history comes from
OperationHistory (operation_history.py), which has its own file; a copy
written here was never read back by anything.
""" """
def __init__( def __init__(self, max_history: int = 100):
self,
history_file: Optional[str] = None,
max_history: int = 100,
lazy_load: bool = False
):
""" """
Initialize operation queue. Initialize operation queue.
Args: Args:
history_file: Optional path to file for persisting operation history
max_history: Maximum number of operations to keep in history max_history: Maximum number of operations to keep in history
lazy_load: If True, defer loading history file until first access
""" """
self.logger = get_logger(__name__) self.logger = get_logger(__name__)
self.history_file = Path(history_file) if history_file else None
self.max_history = max_history self.max_history = max_history
self._lazy_load = lazy_load
self._history_loaded = False
# Operation tracking # Operation tracking
self._operations: Dict[str, PluginOperation] = {} self._operations: Dict[str, PluginOperation] = {}
@@ -62,20 +54,8 @@ class PluginOperationQueue:
self._worker_thread: Optional[threading.Thread] = None self._worker_thread: Optional[threading.Thread] = None
self._stop_event = threading.Event() self._stop_event = threading.Event()
# Load history from file if it exists (unless lazy loading)
if not self._lazy_load and self.history_file and self.history_file.exists():
self._load_history()
self._history_loaded = True
# Start worker thread
self._start_worker() self._start_worker()
def _ensure_loaded(self) -> None:
"""Ensure history is loaded (for lazy loading)."""
if not self._history_loaded and self.history_file and self.history_file.exists():
self._load_history()
self._history_loaded = True
def enqueue_operation( def enqueue_operation(
self, self,
operation_type: OperationType, operation_type: OperationType,
@@ -139,7 +119,6 @@ class PluginOperationQueue:
Returns: Returns:
PluginOperation if found, None otherwise PluginOperation if found, None otherwise
""" """
self._ensure_loaded()
with self._lock: with self._lock:
return self._operations.get(operation_id) return self._operations.get(operation_id)
@@ -184,7 +163,6 @@ class PluginOperationQueue:
Returns: Returns:
List of operations, sorted by creation time (newest first) List of operations, sorted by creation time (newest first)
""" """
self._ensure_loaded()
with self._lock: with self._lock:
# Sort by creation time (newest first) # Sort by creation time (newest first)
history = sorted( history = sorted(
@@ -314,12 +292,8 @@ class PluginOperationQueue:
if self._active_operations[operation.plugin_id].operation_id == operation.operation_id: if self._active_operations[operation.plugin_id].operation_id == operation.operation_id:
del self._active_operations[operation.plugin_id] del self._active_operations[operation.plugin_id]
# Add to history
self._add_to_history(operation) self._add_to_history(operation)
# Save history to file
self._save_history()
def _add_to_history(self, operation: PluginOperation) -> None: def _add_to_history(self, operation: PluginOperation) -> None:
"""Add operation to history, maintaining max_history limit.""" """Add operation to history, maintaining max_history limit."""
self._operation_history.append(operation) self._operation_history.append(operation)
@@ -330,46 +304,6 @@ class PluginOperationQueue:
self._operation_history.sort(key=lambda op: op.created_at) self._operation_history.sort(key=lambda op: op.created_at)
self._operation_history = self._operation_history[-self.max_history:] self._operation_history = self._operation_history[-self.max_history:]
def _save_history(self) -> None:
"""Save operation history to file."""
if not self.history_file:
return
try:
with self._lock:
# Convert operations to dicts
history_data = [op.to_dict() for op in self._operation_history]
# Ensure directory exists
self.history_file.parent.mkdir(parents=True, exist_ok=True)
# Write to file
with open(self.history_file, 'w') as f:
json.dump(history_data, f, indent=2)
except Exception as e:
self.logger.warning(f"Error saving operation history: {e}")
def _load_history(self) -> None:
"""Load operation history from file."""
if not self.history_file or not self.history_file.exists():
return
try:
with open(self.history_file, 'r') as f:
history_data = json.load(f)
with self._lock:
self._operation_history = [
PluginOperation.from_dict(op_data)
for op_data in history_data
]
self.logger.info(f"Loaded {len(self._operation_history)} operations from history")
except Exception as e:
self.logger.warning(f"Error loading operation history: {e}")
def shutdown(self) -> None: def shutdown(self) -> None:
"""Shutdown the operation queue and worker thread.""" """Shutdown the operation queue and worker thread."""
self.logger.info("Shutting down plugin operation queue") self.logger.info("Shutting down plugin operation queue")
@@ -378,6 +312,3 @@ class PluginOperationQueue:
if self._worker_thread and self._worker_thread.is_alive(): if self._worker_thread and self._worker_thread.is_alive():
self._worker_thread.join(timeout=5.0) self._worker_thread.join(timeout=5.0)
# Save history one last time
self._save_history()
src/plugin_system/plugin_loader.py
+58 -42
View File
@@ -5,6 +5,7 @@ Handles plugin module imports, dependency installation, and class instantiation.
Extracted from PluginManager to improve separation of concerns. Extracted from PluginManager to improve separation of concerns.
""" """
import errno
import importlib import importlib
import importlib.metadata import importlib.metadata
import importlib.util import importlib.util
@@ -184,6 +185,46 @@ def find_trusted_subdir(trusted_dir: str, name: str) -> Optional[str]:
return None return None
def contained_plugin_dir(plugin_dir: Path, plugins_dir: Path) -> Optional[str]:
"""``plugin_dir`` rebuilt from an entry enumerated under ``plugins_dir``.
Returns None when ``plugin_dir`` is not a subdirectory of ``plugins_dir``.
Callers derive ``plugin_dir`` from a manifest-declared id, so the path is
rebuilt from :func:`find_trusted_subdir`'s answer rather than trusted: a
name that came out of ``os.scandir()`` on the trusted root carries no
taint, which is a real containment guarantee (and one CodeQL's
path-injection query can follow), not a string sanitiser.
"""
plugin_dir_real = os.path.realpath(str(plugin_dir))
plugins_dir_real = os.path.realpath(str(plugins_dir))
matched_name = find_trusted_subdir(plugins_dir_real, os.path.basename(plugin_dir_real))
if matched_name is None:
return None
return os.path.join(plugins_dir_real, matched_name)
def requirements_to_install(plugin_dir: str, logger: logging.Logger,
label: str) -> Optional[str]:
"""The plugin's requirements.txt if pip has work to do, else None.
None when there is no requirements.txt, when it lists nothing (plugins
whose dependencies ship with core often keep an all-comments file), or
when every requirement is already installed. Shared by the loader and the
store so both skip pip for the same reasons; they differ only in how they
run it.
"""
requirements_file = os.path.join(plugin_dir, "requirements.txt")
if not os.path.isfile(requirements_file):
return None
if not requirements_has_real_deps(requirements_file):
logger.debug("requirements.txt for %s has no real dependencies, skipping pip", label)
return None
if requirements_are_satisfied(requirements_file):
logger.debug("Dependencies for %s already satisfied, skipping pip", label)
return None
return requirements_file
class PluginLoader: class PluginLoader:
"""Handles plugin module loading and class instantiation.""" """Handles plugin module loading and class instantiation."""
@@ -273,43 +314,15 @@ class PluginLoader:
if not plugin_id: if not plugin_id:
return False return False
# Resolve to a canonical absolute path (normalises .. and symlinks) safe_plugin_dir = contained_plugin_dir(plugin_dir, plugins_dir)
plugin_dir_real = os.path.realpath(str(plugin_dir)) if safe_plugin_dir is None:
plugins_dir_real = os.path.realpath(str(plugins_dir))
requested_name = os.path.basename(plugin_dir_real)
# Match the requested directory against an entry actually enumerated
# from the trusted plugins_dir, and build the path from that entry --
# not from requested_name. A name that came out of os.scandir() on a
# trusted root carries no taint regardless of what the caller asked
# for, so this is a real containment guarantee (an allowlist check
# against a trusted source), not a string-sanitisation of untrusted
# input that a static analyzer has to trust blindly.
matched_name = find_trusted_subdir(plugins_dir_real, requested_name)
if matched_name is None:
self.logger.error( self.logger.error(
"Plugin directory for %s not found inside plugins dir", plugin_id "Plugin directory for %s not found inside plugins dir", plugin_id
) )
return False return False
safe_plugin_dir = os.path.join(plugins_dir_real, matched_name) requirements_file = requirements_to_install(safe_plugin_dir, self.logger, plugin_id)
requirements_file = os.path.join(safe_plugin_dir, "requirements.txt") if requirements_file is None:
if not os.path.isfile(requirements_file):
return True # No dependencies needed
if not requirements_has_real_deps(requirements_file):
self.logger.debug(
"requirements.txt for %s has no real dependencies (comments/blank only), skipping pip",
plugin_id
)
return True
if requirements_are_satisfied(requirements_file):
self.logger.debug(
"Dependencies for %s already satisfied in current environment, skipping pip",
plugin_id
)
return True return True
try: try:
@@ -348,8 +361,8 @@ class PluginLoader:
# below). # below).
try: try:
# sys.executable is this process's own interpreter (not # sys.executable is this process's own interpreter (not
# attacker-influenced), and requirements_file is a path # attacker-influenced), and requirements_file is rebuilt
# built internally by find_plugin_directory, never raw # by contained_plugin_dir() from a trusted listing, never raw
# external input. # external input.
retry_result = subprocess.run( # nosec B603 - no shell invoked (list-form argv) # nosemgrep retry_result = subprocess.run( # nosec B603 - no shell invoked (list-form argv) # nosemgrep
[sys.executable, "-m", "pip", "install", "--break-system-packages", [sys.executable, "-m", "pip", "install", "--break-system-packages",
@@ -384,10 +397,10 @@ class PluginLoader:
except FileNotFoundError: except FileNotFoundError:
self.logger.warning("pip not found. Skipping dependency installation for %s", plugin_id) self.logger.warning("pip not found. Skipping dependency installation for %s", plugin_id)
return True return True
except (BrokenPipeError, OSError) as e: except OSError as e:
# Handle broken pipe errors (errno 32) which can occur during pip downloads # A broken pipe (EPIPE) happens when pip's output pipe closes
# Often caused by network interruptions or output buffer issues # mid-download, usually a network interruption.
if isinstance(e, OSError) and e.errno == 32: if e.errno == errno.EPIPE:
self.logger.error( self.logger.error(
"Broken pipe error during dependency installation for %s. " "Broken pipe error during dependency installation for %s. "
"This usually indicates a network interruption or pip output buffer issue. " "This usually indicates a network interruption or pip output buffer issue. "
@@ -528,7 +541,7 @@ class PluginLoader:
plugin_id: str, plugin_id: str,
plugin_dir: Path, plugin_dir: Path,
entry_point: str entry_point: str
) -> Optional[Any]: ) -> Any:
""" """
Load a plugin module from file. Load a plugin module from file.
@@ -547,7 +560,12 @@ class PluginLoader:
entry_point: Entry point filename (e.g., 'manager.py') entry_point: Entry point filename (e.g., 'manager.py')
Returns: Returns:
Loaded module or None on error The loaded module
Raises:
PluginError: If the plugin id, directory or entry point is
invalid. Whatever the module raises while executing
propagates unchanged.
""" """
plugin_id = os.path.basename(plugin_id or '') plugin_id = os.path.basename(plugin_id or '')
if not plugin_id: if not plugin_id:
@@ -782,8 +800,6 @@ class PluginLoader:
# Load module # Load module
entry_point = manifest.get('entry_point', 'manager.py') entry_point = manifest.get('entry_point', 'manager.py')
module = self.load_module(plugin_id, plugin_dir, entry_point) module = self.load_module(plugin_id, plugin_dir, entry_point)
if module is None:
raise PluginError(f"Failed to load module for plugin {plugin_id}", plugin_id=plugin_id)
# Get plugin class # Get plugin class
class_name = manifest.get('class_name') class_name = manifest.get('class_name')
src/plugin_system/plugin_manager.py
+27 -22
View File
@@ -2,7 +2,8 @@
Plugin Manager Plugin Manager
Manages plugin discovery, loading, and lifecycle for the LEDMatrix system. Manages plugin discovery, loading, and lifecycle for the LEDMatrix system.
Handles dynamic plugin loading from the plugins/ directory. Loads plugins from the configured plugins directory
(``plugin_system.plugins_directory``, ``plugin-repos/`` by default).
API Version: 1.0.0 API Version: 1.0.0
""" """
@@ -40,7 +41,7 @@ class PluginManager:
Manages plugin discovery, loading, and lifecycle. Manages plugin discovery, loading, and lifecycle.
The PluginManager is responsible for: The PluginManager is responsible for:
- Discovering plugins in the plugins/ directory - Discovering plugins in the configured plugins directory
- Loading plugin modules and instantiating plugin classes - Loading plugin modules and instantiating plugin classes
- Managing plugin lifecycle (load, unload, reload) - Managing plugin lifecycle (load, unload, reload)
- Providing access to loaded plugins - Providing access to loaded plugins
@@ -99,10 +100,9 @@ class PluginManager:
self.plugin_directories: Dict[str, Path] = {} self.plugin_directories: Dict[str, Path] = {}
self.plugin_last_update: Dict[str, float] = {} self.plugin_last_update: Dict[str, float] = {}
# Cached data-fetch intervals per plugin_id. # Cached static data-fetch intervals per plugin_id, so the render
# _get_plugin_update_interval falls back to config_manager.get_config() # loop's scheduling tick does not repeat the manifest/config lookup
# (a full dict copy) when the manifest lacks an interval — caching avoids # for every plugin. Cleared on load/unload.
# that copy on every 30-fps tick. Cleared on load/unload.
self._update_interval_cache: Dict[str, Optional[float]] = {} self._update_interval_cache: Dict[str, Optional[float]] = {}
# Health tracking (optional, set by display_controller if available) # Health tracking (optional, set by display_controller if available)
@@ -110,14 +110,12 @@ class PluginManager:
self.resource_monitor = None self.resource_monitor = None
# --- Asynchronous plugin updates ------------------------------- # --- Asynchronous plugin updates -------------------------------
# update() used to run inline in the render loop (execute_update's # Run inline in the render loop, one slow plugin HTTP fetch in
# internal thread.join(timeout=30) blocked it), so one slow plugin # update() freezes scrolling for the whole fetch. Scheduling happens
# HTTP fetch froze scrolling for the whole fetch. Scheduling still # on the render thread (run_scheduled_updates); execution happens on
# happens on the render thread (run_scheduled_updates), but # this single background worker. Per-plugin locks keep a plugin's
# execution moves to this single background worker. Per-plugin # update() and display() mutually exclusive, including across the
# locks keep a plugin's update() and display() mutually exclusive — # post-timeout window.
# today's implicit guarantee, now explicit (and, unlike today,
# also held across the post-timeout window).
# Kill switch: plugin_system.synchronous_updates: true restores the # Kill switch: plugin_system.synchronous_updates: true restores the
# inline path. # inline path.
self._update_queue: "queue.Queue[Optional[Tuple[str, float]]]" = queue.Queue() self._update_queue: "queue.Queue[Optional[Tuple[str, float]]]" = queue.Queue()
@@ -661,9 +659,15 @@ class PluginManager:
if not self.unload_plugin(plugin_id): if not self.unload_plugin(plugin_id):
return False return False
# Re-discover to get updated manifest # Re-read the manifest so an edit to it takes effect, from the
manifest_path = self.plugins_dir / plugin_id / "manifest.json" # directory discovery found the plugin in: a directory's name need not
if manifest_path.exists(): # be the id its manifest declares.
with self._discovery_lock:
directories = dict(self.plugin_directories)
plugin_dir = self.plugin_loader.find_plugin_directory(
plugin_id, self.plugins_dir, directories)
manifest_path = plugin_dir / "manifest.json" if plugin_dir is not None else None
if manifest_path is not None and manifest_path.exists():
try: try:
with open(manifest_path, 'r', encoding='utf-8') as f: with open(manifest_path, 'r', encoding='utf-8') as f:
manifest = json.load(f) manifest = json.load(f)
@@ -881,11 +885,12 @@ class PluginManager:
updating, since a scheduler that propagates a plugin bug stops every updating, since a scheduler that propagates a plugin bug stops every
other plugin too. other plugin too.
The static result is cached per plugin_id after the first lookup to The static result is cached per plugin_id after the first lookup, so
avoid calling config_manager.get_config() — which returns a full dict the manifest/config resolution is not repeated on every scheduling
copy — on every tick of the 30-fps display loop. The cache is tick of the display loop. A change to ``update_interval`` in
invalidated when a plugin is loaded or unloaded. The dynamic hook is config.json therefore takes effect when the plugin is next loaded or
deliberately *not* cached: caching it would defeat its only purpose. unloaded, which clears the cache. The dynamic hook is deliberately
*not* cached: caching it would defeat its only purpose.
""" """
dynamic = self._dynamic_update_interval(plugin_id, plugin_instance) dynamic = self._dynamic_update_interval(plugin_id, plugin_instance)
if dynamic is not None: if dynamic is not None:
src/plugin_system/plugin_state.py
-7
View File
@@ -41,7 +41,6 @@ class PluginStateManager:
self._state_transition_counts: Dict[str, int] = {} self._state_transition_counts: Dict[str, int] = {}
self._error_info: Dict[str, Dict[str, Any]] = {} self._error_info: Dict[str, Dict[str, Any]] = {}
self._last_update: Dict[str, datetime] = {} self._last_update: Dict[str, datetime] = {}
self._last_display: Dict[str, datetime] = {}
def _record_transition(self, plugin_id: str) -> None: def _record_transition(self, plugin_id: str) -> None:
"""Count a state transition. Callers must already hold ``_lock``.""" """Count a state transition. Callers must already hold ``_lock``."""
@@ -182,10 +181,6 @@ class PluginStateManager:
"""Get timestamp of last update() call.""" """Get timestamp of last update() call."""
return self._last_update.get(plugin_id) return self._last_update.get(plugin_id)
def get_last_display(self, plugin_id: str) -> Optional[datetime]:
"""Get timestamp of last display() call."""
return self._last_display.get(plugin_id)
def get_state_info(self, plugin_id: str) -> Dict[str, Any]: def get_state_info(self, plugin_id: str) -> Dict[str, Any]:
""" """
Get comprehensive state information for a plugin. Get comprehensive state information for a plugin.
@@ -212,7 +207,6 @@ class PluginStateManager:
'is_error': self.is_error(plugin_id), 'is_error': self.is_error(plugin_id),
'can_execute': self.can_execute(plugin_id), 'can_execute': self.can_execute(plugin_id),
'last_update': self.get_last_update(plugin_id), 'last_update': self.get_last_update(plugin_id),
'last_display': self.get_last_display(plugin_id),
'error_info': self.get_error_info(plugin_id), 'error_info': self.get_error_info(plugin_id),
'state_history_count': self._state_transition_counts.get(plugin_id, 0) 'state_history_count': self._state_transition_counts.get(plugin_id, 0)
} }
@@ -231,5 +225,4 @@ class PluginStateManager:
self._state_transition_counts.pop(plugin_id, None) self._state_transition_counts.pop(plugin_id, None)
self._error_info.pop(plugin_id, None) self._error_info.pop(plugin_id, None)
self._last_update.pop(plugin_id, None) self._last_update.pop(plugin_id, None)
self._last_display.pop(plugin_id, None)
src/plugin_system/repo_urls.py
+70
View File
@@ -0,0 +1,70 @@
"""
Repository URL helpers shared by the plugin store and saved repositories.
One definition of "the same repository URL", of how a GitHub URL maps to
``owner/repo``, and of the headers sent to the GitHub API.
"""
from typing import Dict, Optional, Tuple
from urllib.parse import urlparse
#: Hosts whose URLs name a GitHub repository. Matched against
#: ``urlparse(url).hostname``, never by substring: a substring test accepts
#: ``https://github.com.example.org/...`` as GitHub.
GITHUB_HOSTS = frozenset({'github.com', 'www.github.com'})
#: Sent on every request the store makes to GitHub.
USER_AGENT = 'LEDMatrix-Plugin-Manager/1.0'
def normalize_repo_url(url: str) -> str:
"""``url`` without surrounding whitespace, trailing slashes or a trailing ``.git``.
Only a *trailing* ``.git`` is removed. The unanchored
``url.replace('.git', '')`` this replaces turned
``https://github.com/user/my.github.io`` into ``.../myhub.io``.
Case is preserved; compare with :func:`same_repo`.
"""
url = url.strip().rstrip('/')
if url.endswith('.git'):
url = url[:-4]
return url
def same_repo(url_a: str, url_b: str) -> bool:
"""Whether two URLs name the same repository.
GitHub owner and repository names are case-insensitive, so
``ChuckBuilds/LEDMatrix-Plugins`` and ``chuckbuilds/ledmatrix-plugins``
are one repository.
"""
return normalize_repo_url(url_a).lower() == normalize_repo_url(url_b).lower()
def github_owner_repo(url: str) -> Optional[Tuple[str, str]]:
"""``(owner, repo)`` for a github.com repository URL, else None.
The first two path segments, so a URL that points inside the repository
(``.../owner/repo/tree/main/plugins/x``) still names ``owner/repo``.
"""
parsed = urlparse(normalize_repo_url(url))
if parsed.hostname not in GITHUB_HOSTS:
return None
parts = [part for part in parsed.path.split('/') if part]
if len(parts) < 2:
return None
return parts[0], normalize_repo_url(parts[1])
def github_api_headers(token: Optional[str] = None) -> Dict[str, str]:
"""Headers for a GitHub REST API request, authenticated when ``token`` is set.
An authenticated request gets 5000 requests an hour instead of 60.
"""
headers = {
'Accept': 'application/vnd.github.v3+json',
'User-Agent': USER_AGENT,
}
if token:
headers['Authorization'] = f'token {token}'
return headers
src/plugin_system/resource_monitor.py
+27 -16
View File
@@ -33,7 +33,14 @@ class ResourceLimits:
@dataclass @dataclass
class ResourceMetrics: class ResourceMetrics:
"""Resource usage metrics for a plugin.""" """Resource usage metrics for a plugin.
``memory_mb`` is the largest growth in this *process's* resident memory
seen across a single monitored call -- a high-water mark, not current
usage, and not the plugin's own footprint (another thread allocating
during the call counts too). ``cpu_percent`` is the whole process's CPU
use since the previous sample.
"""
memory_mb: float = 0.0 memory_mb: float = 0.0
cpu_percent: float = 0.0 cpu_percent: float = 0.0
execution_time: float = 0.0 execution_time: float = 0.0
@@ -43,11 +50,6 @@ class ResourceMetrics:
min_execution_time: float = float('inf') min_execution_time: float = float('inf')
last_update_time: float = field(default_factory=time.time) last_update_time: float = field(default_factory=time.time)
def update_average_execution_time(self):
"""Update average execution time."""
if self.call_count > 0:
self.total_execution_time = self.total_execution_time / self.call_count
#: How often a plugin's metrics are written to the cache, in seconds. #: How often a plugin's metrics are written to the cache, in seconds.
#: #:
@@ -287,6 +289,7 @@ class PluginResourceMonitor:
# Calculate execution time # Calculate execution time
execution_time = time.time() - start_time execution_time = time.time() - start_time
memory_growth_mb = 0.0
# Update metrics # Update metrics
with self._lock: with self._lock:
@@ -302,17 +305,17 @@ class PluginResourceMonitor:
# Update memory and CPU if monitoring enabled # Update memory and CPU if monitoring enabled
if self.enable_monitoring: if self.enable_monitoring:
end_memory = self._get_process_memory_mb() memory_growth_mb = self._get_process_memory_mb() - start_memory
metrics.memory_mb = max(metrics.memory_mb, end_memory - start_memory) metrics.memory_mb = max(metrics.memory_mb, memory_growth_mb)
# CPU is harder to measure per-call, so we track it separately # CPU is harder to measure per-call, so we track it separately
metrics.cpu_percent = self._get_process_cpu_percent() metrics.cpu_percent = self._get_process_cpu_percent()
# Persist metrics, at most once per interval per plugin. # Persist metrics, at most once per interval per plugin.
self._persist_metrics(plugin_id, metrics) self._persist_metrics(plugin_id, metrics)
# Check limits
if limits: if limits:
self._check_limits(plugin_id, metrics, limits, execution_time) self._check_limits(plugin_id, metrics, limits, execution_time,
memory_growth_mb)
return result return result
@@ -327,8 +330,16 @@ class PluginResourceMonitor:
raise raise
def _check_limits(self, plugin_id: str, metrics: ResourceMetrics, def _check_limits(self, plugin_id: str, metrics: ResourceMetrics,
limits: ResourceLimits, execution_time: float) -> None: limits: ResourceLimits, execution_time: float,
"""Check if plugin has exceeded resource limits.""" memory_growth_mb: float) -> None:
"""Raise ResourceLimitExceeded if this call went over a limit.
Execution time and memory growth are this call's own; CPU is the
latest process sample. Judging memory by the stored high-water mark
(``metrics.memory_mb``) instead would fail every call after the first
expensive one, so the health tracker's circuit breaker would reopen
on every recovery probe and the plugin would never update again.
"""
warnings = [] warnings = []
errors = [] errors = []
@@ -343,13 +354,13 @@ class PluginResourceMonitor:
) )
# Check memory # Check memory
if limits.max_memory_mb and metrics.memory_mb > limits.max_memory_mb: if limits.max_memory_mb and memory_growth_mb > limits.max_memory_mb:
errors.append( errors.append(
f"Memory usage {metrics.memory_mb:.2f}MB exceeds limit {limits.max_memory_mb:.2f}MB" f"Memory growth {memory_growth_mb:.2f}MB exceeds limit {limits.max_memory_mb:.2f}MB"
) )
elif limits.max_memory_mb and metrics.memory_mb > limits.max_memory_mb * limits.warning_threshold: elif limits.max_memory_mb and memory_growth_mb > limits.max_memory_mb * limits.warning_threshold:
warnings.append( warnings.append(
f"Memory usage {metrics.memory_mb:.2f}MB approaching limit {limits.max_memory_mb:.2f}MB" f"Memory growth {memory_growth_mb:.2f}MB approaching limit {limits.max_memory_mb:.2f}MB"
) )
# Check CPU # Check CPU
src/plugin_system/saved_repositories.py
+5 -15
View File
@@ -10,6 +10,8 @@ import os
from pathlib import Path from pathlib import Path
from typing import List, Dict, Optional from typing import List, Dict, Optional
from src.plugin_system.repo_urls import normalize_repo_url
class SavedRepositoriesManager: class SavedRepositoriesManager:
"""Manages saved GitHub repository URLs.""" """Manages saved GitHub repository URLs."""
@@ -71,18 +73,6 @@ class SavedRepositoriesManager:
pass pass
return False return False
@staticmethod
def _clean_url(repo_url: str) -> str:
"""Normalize a repo URL: strip whitespace, trailing slashes, and a
trailing ``.git`` suffix ONLY. (The old ``.replace('.git', '')``
was an unanchored substring replace that mangled URLs merely
containing ``.git``, e.g. ``https://github.com/user/my.github.io``.)
"""
repo_url = repo_url.strip().rstrip('/')
if repo_url.endswith('.git'):
repo_url = repo_url[:-4]
return repo_url
def get_all(self) -> List[Dict[str, str]]: def get_all(self) -> List[Dict[str, str]]:
"""Get all saved repositories.""" """Get all saved repositories."""
return self.repositories.copy() return self.repositories.copy()
@@ -98,7 +88,7 @@ class SavedRepositoriesManager:
Returns: Returns:
True if added successfully True if added successfully
""" """
repo_url = self._clean_url(repo_url) repo_url = normalize_repo_url(repo_url)
# Check if already exists # Check if already exists
for repo in self.repositories: for repo in self.repositories:
@@ -138,7 +128,7 @@ class SavedRepositoriesManager:
Returns: Returns:
True if removed successfully True if removed successfully
""" """
repo_url = self._clean_url(repo_url) repo_url = normalize_repo_url(repo_url)
previous = self.repositories previous = self.repositories
remaining = [r for r in previous if r.get('url') != repo_url] remaining = [r for r in previous if r.get('url') != repo_url]
@@ -156,7 +146,7 @@ class SavedRepositoriesManager:
def has(self, repo_url: str) -> bool: def has(self, repo_url: str) -> bool:
"""Check if a repository is already saved.""" """Check if a repository is already saved."""
repo_url = self._clean_url(repo_url) repo_url = normalize_repo_url(repo_url)
return any(r.get('url') == repo_url for r in self.repositories) return any(r.get('url') == repo_url for r in self.repositories)
def get_registry_repositories(self) -> List[Dict[str, str]]: def get_registry_repositories(self) -> List[Dict[str, str]]:
src/plugin_system/schema_manager.py
+13 -16
View File
@@ -14,6 +14,7 @@ import jsonschema
from jsonschema import Draft7Validator, ValidationError from jsonschema import Draft7Validator, ValidationError
from src.core_config_keys import CORE_CONFIG_KEYS from src.core_config_keys import CORE_CONFIG_KEYS
from src.element_style import expand_style_elements
def _renders_as_object(prop: Dict[str, Any]) -> bool: def _renders_as_object(prop: Dict[str, Any]) -> bool:
@@ -452,11 +453,7 @@ class SchemaManager:
# full per-element style blocks (font/size/color + layout # full per-element style blocks (font/size/color + layout
# offsets) the web-UI config form renders. No-op for schemas # offsets) the web-UI config form renders. No-op for schemas
# without the declaration; never raises. # without the declaration; never raises.
try:
from src.element_style import expand_style_elements
schema = expand_style_elements(schema) schema = expand_style_elements(schema)
except ImportError:
pass
# Cache the schema # Cache the schema
self._schema_cache[plugin_id] = schema self._schema_cache[plugin_id] = schema
@@ -643,19 +640,11 @@ class SchemaManager:
[name for name in CORE_PLUGIN_PROPERTIES if name not in declared] [name for name in CORE_PLUGIN_PROPERTIES if name not in declared]
) )
# Create validator with enhanced schema # iter_errors reports every violation, including one ``required``
# error per missing field at every depth.
validator = Draft7Validator(enhanced_schema) validator = Draft7Validator(enhanced_schema)
# Collect all validation errors
for error in validator.iter_errors(config): for error in validator.iter_errors(config):
error_msg = self._format_validation_error(error, plugin_id) errors.append(self._format_validation_error(error, plugin_id))
errors.append(error_msg)
# Check required fields
required_fields = enhanced_schema.get('required', [])
for field in required_fields:
if field not in config:
errors.append(f"Missing required field: '{field}'")
if errors: if errors:
return False, errors return False, errors
@@ -687,7 +676,15 @@ class SchemaManager:
field_path = f"'{path}'" if path else "root" field_path = f"'{path}'" if path else "root"
if error.validator == 'required': if error.validator == 'required':
missing = error.validator_value # validator_value is the schema's whole ``required`` list; the
# error itself is about one field, which jsonschema names only in
# its message ("'api_key' is a required property").
missing = next(
(name for name in error.validator_value
if error.message.startswith(f"{name!r} ")),
None)
if missing is None:
return f"Field {field_path}: {error.message}"
return f"Field {field_path}: Missing required property '{missing}'" return f"Field {field_path}: Missing required property '{missing}'"
elif error.validator == 'type': elif error.validator == 'type':
expected = error.validator_value expected = error.validator_value
src/plugin_system/state_manager.py
+5 -2
View File
@@ -34,7 +34,9 @@ class PluginState:
version: Optional[str] = None version: Optional[str] = None
installed_at: Optional[datetime] = None installed_at: Optional[datetime] = None
last_updated: Optional[datetime] = None last_updated: Optional[datetime] = None
config_version: int = 1 # For detecting state corruption # Bumped on every update_plugin_state(). Nothing reads it; it stays so
# plugin_state.json keeps the shape older releases load with cls(**data).
config_version: int = 1
metadata: Dict[str, Any] = None metadata: Dict[str, Any] = None
def __post_init__(self): def __post_init__(self):
@@ -100,6 +102,8 @@ class PluginStateManager:
# State storage # State storage
self._states: Dict[str, PluginState] = {} self._states: Dict[str, PluginState] = {}
# The file's top-level "version", written back as read. Nothing
# checks it yet; it is there for a future format change to branch on.
self._state_version = 1 self._state_version = 1
# Threading # Threading
@@ -193,7 +197,6 @@ class PluginStateManager:
current_state.metadata = {} current_state.metadata = {}
current_state.metadata.update(updates['metadata']) current_state.metadata.update(updates['metadata'])
# Increment config version
current_state.config_version += 1 current_state.config_version += 1
# Store updated state # Store updated state
src/plugin_system/store_manager.py
+214 -333
View File
@@ -5,6 +5,7 @@ Handles plugin discovery, installation, updates, and uninstallation
from both the official registry and custom GitHub repositories. from both the official registry and custom GitHub repositories.
""" """
import errno
import os import os
import re import re
import json import json
@@ -22,21 +23,19 @@ from pathlib import Path
from typing import List, Dict, Optional, Any, Tuple, Set from typing import List, Dict, Optional, Any, Tuple, Set
import logging import logging
from urllib.parse import urlparse from jsonschema import Draft7Validator, ValidationError
from src.common.permission_utils import sudo_remove_directory, install_requirements_file from src.common.permission_utils import (
from src.plugin_system.plugin_loader import ( ensure_directory_permissions, get_plugin_dir_mode, install_requirements_file,
requirements_has_real_deps, requirements_are_satisfied, find_trusted_subdir sudo_remove_directory,
) )
from src.plugin_system.plugin_loader import contained_plugin_dir, requirements_to_install
from src.plugin_system.plugin_dirs import ( from src.plugin_system.plugin_dirs import (
BACKUP_MARKER, PluginDirectoryIndex, resolve_plugin_dir, store_search_dirs, BACKUP_MARKER, PluginDirectoryIndex, resolve_plugin_dir, store_search_dirs,
) )
from src.plugin_system.repo_urls import (
try: USER_AGENT, github_api_headers, github_owner_repo, normalize_repo_url, same_repo,
from jsonschema import Draft7Validator, ValidationError )
JSONSCHEMA_AVAILABLE = True
except ImportError:
JSONSCHEMA_AVAILABLE = False
class PluginStoreManager: class PluginStoreManager:
@@ -93,11 +92,11 @@ class PluginStoreManager:
self.registry_cache_timeout = 900 self.registry_cache_timeout = 900
self.commit_info_cache = {} # Cache for latest commit info: {key: (timestamp, data)} self.commit_info_cache = {} # Cache for latest commit info: {key: (timestamp, data)}
# 30 minutes for commit/manifest caches. Plugin Store users browse # 30 minutes for commit/manifest caches. Plugin Store users browse
# the catalog via /plugins/store/list which fetches commit info and # the catalog via /plugins/store/list, which fetches commit info per
# manifest data per plugin. 5-min TTLs meant every fresh browse on # plugin; with a 5-minute TTL nearly every browse on a Pi4 paid for
# a Pi4 paid for ~3 HTTP requests x N plugins (30-60s serial). 30 # an HTTP request per plugin again. 30 minutes keeps the cache warm
# minutes keeps the cache warm across a realistic session while # across a realistic session while still picking up upstream updates
# still picking up upstream updates within a reasonable window. # within a reasonable window.
self.commit_cache_timeout = 1800 self.commit_cache_timeout = 1800
self.manifest_cache = {} # Cache for GitHub manifest fetches: {key: (timestamp, data)} self.manifest_cache = {} # Cache for GitHub manifest fetches: {key: (timestamp, data)}
self.manifest_cache_timeout = 1800 self.manifest_cache_timeout = 1800
@@ -127,9 +126,9 @@ class PluginStoreManager:
# where ``signature`` is a tuple of (head_mtime, resolved_ref_mtime, # where ``signature`` is a tuple of (head_mtime, resolved_ref_mtime,
# head_contents) so a fast-forward update to the current branch # head_contents) so a fast-forward update to the current branch
# (which touches .git/refs/heads/<branch> but NOT .git/HEAD) still # (which touches .git/refs/heads/<branch> but NOT .git/HEAD) still
# invalidates the cache. Before this cache, every # invalidates the cache. Without it every /plugins/installed request
# /plugins/installed request fired 4 git subprocesses per plugin, # runs a git subprocess per plugin, which adds up on a Pi4 with a
# which pegged the CPU on a Pi4 with a dozen plugins. The cached # dozen plugins. The cached
# ``data`` dict is the same shape returned by ``_get_local_git_info`` # ``data`` dict is the same shape returned by ``_get_local_git_info``
# itself (sha / short_sha / branch / optional remote_url, date_iso, # itself (sha / short_sha / branch / optional remote_url, date_iso,
# date) — all string-keyed strings. # date) — all string-keyed strings.
@@ -368,13 +367,7 @@ class PluginStoreManager:
# Validate token by making a lightweight API call to /user endpoint # Validate token by making a lightweight API call to /user endpoint
try: try:
api_url = "https://api.github.com/user" api_url = "https://api.github.com/user"
headers = { response = requests.get(api_url, headers=github_api_headers(token), timeout=5)
'Accept': 'application/vnd.github.v3+json',
'User-Agent': 'LEDMatrix-Plugin-Manager/1.0',
'Authorization': f'token {token}'
}
response = requests.get(api_url, headers=headers, timeout=5)
if response.status_code == 200: if response.status_code == 200:
# Token is valid # Token is valid
@@ -502,9 +495,6 @@ class PluginStoreManager:
Returns: Returns:
List of validation error messages (empty if valid or schema unavailable) List of validation error messages (empty if valid or schema unavailable)
""" """
if not JSONSCHEMA_AVAILABLE:
return []
try: try:
# Load manifest schema # Load manifest schema
schema_path = Path(__file__).parent.parent.parent / "schema" / "manifest_schema.json" schema_path = Path(__file__).parent.parent.parent / "schema" / "manifest_schema.json"
@@ -535,48 +525,46 @@ class PluginStoreManager:
self.logger.debug(f"Error validating manifest schema for {plugin_id}: {e}") self.logger.debug(f"Error validating manifest schema for {plugin_id}: {e}")
return [] return []
def _get_github_repo_info(self, repo_url: str) -> Dict[str, Any]: _EMPTY_REPO_INFO: Dict[str, Any] = {
"""Fetch GitHub repository information (stars, etc.)""" 'stars': 0,
# Extract owner/repo from URL 'forks': 0,
try: 'open_issues': 0,
# Handle different URL formats 'updated_at_iso': '',
_parsed_url = urlparse(repo_url) 'last_commit_iso': '',
if _parsed_url.hostname in ('github.com', 'www.github.com'): 'last_commit_date': '',
parts = repo_url.strip('/').split('/') 'language': '',
if len(parts) >= 2: 'license': '',
owner = parts[-2] 'default_branch': 'main',
repo = parts[-1] }
if repo.endswith('.git'):
repo = repo[:-4]
def _get_github_repo_info(self, repo_url: str) -> Dict[str, Any]:
"""GitHub metadata for a repository (stars, default branch, last push).
Returns zeroed defaults (``_EMPTY_REPO_INFO``) for a non-GitHub URL or
when GitHub cannot be asked and nothing is cached.
"""
try:
owner_repo = github_owner_repo(repo_url)
if owner_repo is None:
return dict(self._EMPTY_REPO_INFO)
owner, repo = owner_repo
cache_key = f"{owner}/{repo}" cache_key = f"{owner}/{repo}"
# Check cache first
if cache_key in self.github_cache: if cache_key in self.github_cache:
cached_time, cached_data = self.github_cache[cache_key] cached_time, cached_data = self.github_cache[cache_key]
if time.time() - cached_time < self.cache_timeout: if time.time() - cached_time < self.cache_timeout:
return cached_data return cached_data
# Fetch from GitHub API
api_url = f"https://api.github.com/repos/{owner}/{repo}" api_url = f"https://api.github.com/repos/{owner}/{repo}"
headers = {
'Accept': 'application/vnd.github.v3+json',
'User-Agent': 'LEDMatrix-Plugin-Manager/1.0'
}
# Add authentication if token is available
if self.github_token:
headers['Authorization'] = f'token {self.github_token}'
try: try:
response = requests.get(api_url, headers=headers, timeout=10) response = requests.get(
api_url, headers=github_api_headers(self.github_token), timeout=10)
except requests.RequestException as req_err: except requests.RequestException as req_err:
# Network error: prefer a stale cache hit over an # Network error: prefer a stale cache hit over an empty
# empty default so the UI keeps working on a flaky # default so the UI keeps working on a flaky Pi WiFi link.
# Pi WiFi link. Bump the cached entry's timestamp # Bump the cached entry's timestamp into a short backoff
# into a short backoff window so subsequent # window so subsequent requests serve the stale payload
# requests serve the stale payload cheaply instead # cheaply instead of re-hitting the network on every request.
# of re-hitting the network on every request.
if cache_key in self.github_cache: if cache_key in self.github_cache:
_, stale = self.github_cache[cache_key] _, stale = self.github_cache[cache_key]
self._record_cache_backoff(self.github_cache, cache_key, self.cache_timeout, stale) self._record_cache_backoff(self.github_cache, cache_key, self.cache_timeout, stale)
@@ -601,18 +589,13 @@ class PluginStoreManager:
'license': data.get('license', {}).get('name', '') if data.get('license') else '', 'license': data.get('license', {}).get('name', '') if data.get('license') else '',
'default_branch': data.get('default_branch', 'main') 'default_branch': data.get('default_branch', 'main')
} }
# Cache the result
self.github_cache[cache_key] = (time.time(), repo_info) self.github_cache[cache_key] = (time.time(), repo_info)
return repo_info return repo_info
elif response.status_code == 403:
# Rate limit or authentication issue. If we have a if response.status_code == 403:
# previously-cached value, serve it rather than # Rate limit or authentication issue. A stale star count is
# returning empty defaults — a stale star count is # better than a reset to zero, and the backoff bump stops the
# better than a reset to zero. Apply the same # store hammering the API while rate-limited.
# failure-backoff bump as the network-error path
# so we don't hammer the API with repeat requests
# while rate-limited.
if cache_key in self.github_cache: if cache_key in self.github_cache:
_, stale = self.github_cache[cache_key] _, stale = self.github_cache[cache_key]
self._record_cache_backoff(self.github_cache, cache_key, self.cache_timeout, stale) self._record_cache_backoff(self.github_cache, cache_key, self.cache_timeout, stale)
@@ -638,31 +621,11 @@ class PluginStoreManager:
self._record_cache_backoff(self.github_cache, cache_key, self.cache_timeout, stale) self._record_cache_backoff(self.github_cache, cache_key, self.cache_timeout, stale)
return stale return stale
return { return dict(self._EMPTY_REPO_INFO)
'stars': 0,
'forks': 0,
'open_issues': 0,
'updated_at_iso': '',
'last_commit_iso': '',
'last_commit_date': '',
'language': '',
'license': '',
'default_branch': 'main'
}
except Exception as e: except Exception as e:
self.logger.error(f"Error fetching GitHub repo info for {repo_url}: {e}") self.logger.error(f"Error fetching GitHub repo info for {repo_url}: {e}")
return { return dict(self._EMPTY_REPO_INFO)
'stars': 0,
'forks': 0,
'open_issues': 0,
'updated_at_iso': '',
'last_commit_iso': '',
'last_commit_date': '',
'language': '',
'license': '',
'default_branch': 'main'
}
def _http_get_with_retries(self, url: str, *, timeout: int = 10, stream: bool = False, headers: Dict[str, str] = None, max_retries: int = 3, backoff_sec: float = 0.75): def _http_get_with_retries(self, url: str, *, timeout: int = 10, stream: bool = False, headers: Dict[str, str] = None, max_retries: int = 3, backoff_sec: float = 0.75):
""" """
@@ -697,27 +660,17 @@ class PluginStoreManager:
Registry dict with plugins list, or None if not found/invalid Registry dict with plugins list, or None if not found/invalid
""" """
try: try:
# Clean up URL repo_url = normalize_repo_url(repo_url)
repo_url = repo_url.rstrip('/').replace('.git', '')
# Try to find plugins.json in common locations # plugins.json or registry.json at the root of main, then master.
# First try root directory
registry_urls = [] registry_urls = []
owner_repo = github_owner_repo(repo_url)
# Extract owner/repo from URL if owner_repo is not None:
_parsed_repo_url = urlparse(repo_url) owner, repo = owner_repo
if _parsed_repo_url.hostname in ('github.com', 'www.github.com'):
parts = repo_url.split('/')
if len(parts) >= 2:
owner = parts[-2]
repo = parts[-1]
# Try common branch names
for branch in ['main', 'master']: for branch in ['main', 'master']:
registry_urls.append(f"https://raw.githubusercontent.com/{owner}/{repo}/{branch}/plugins.json") registry_urls.append(f"https://raw.githubusercontent.com/{owner}/{repo}/{branch}/plugins.json")
registry_urls.append(f"https://raw.githubusercontent.com/{owner}/{repo}/{branch}/registry.json") registry_urls.append(f"https://raw.githubusercontent.com/{owner}/{repo}/{branch}/registry.json")
# Try each URL
for url in registry_urls: for url in registry_urls:
try: try:
response = self._http_get_with_retries(url, timeout=10) response = self._http_get_with_retries(url, timeout=10)
@@ -815,15 +768,20 @@ class PluginStoreManager:
""" """
Search for plugins in the registry with enhanced metadata. Search for plugins in the registry with enhanced metadata.
GitHub is now treated as the source of truth for live metadata like GitHub supplies live metadata such as stars and last commit
stars and last commit timestamps. The registry provides descriptive timestamps; the registry supplies descriptive information (name,
information (name, description, repo URL, etc.). description, repo URL, etc.).
Args: Args:
query: Search query string (searches name, description, id) query: Search query string (searches name, description, id, author)
category: Filter by category (e.g., 'sports', 'weather', 'time') category: Filter by category (e.g., 'sports', 'weather', 'time')
tags: Filter by tags (matches any tag in list) tags: Filter by tags (matches any tag in list)
fetch_commit_info: If True (default), fetch commit metadata from GitHub. fetch_commit_info: If True (default), fetch commit metadata from GitHub.
include_saved_repos: If True (default), also search the
registry-style repositories the user saved.
saved_repositories_manager: The SavedRepositoriesManager holding
those repositories; without it only the official registry is
searched.
Returns: Returns:
List of matching plugin metadata enriched with GitHub information List of matching plugin metadata enriched with GitHub information
@@ -877,11 +835,11 @@ class PluginStoreManager:
def _enrich(plugin: Dict) -> Dict: def _enrich(plugin: Dict) -> Dict:
"""Enrich a single plugin with GitHub metadata. """Enrich a single plugin with GitHub metadata.
Called concurrently from a ThreadPoolExecutor. Each underlying Called concurrently from a ThreadPoolExecutor. Both HTTP helpers
HTTP helper (``_get_github_repo_info`` / ``_get_latest_commit_info`` (``_get_github_repo_info`` / ``_get_latest_commit_info``) are
/ ``_fetch_manifest_from_github``) is thread-safe — they use thread-safe -- they use ``requests`` and write their own cache
``requests`` and write their own cache keys on Python dicts, keys on Python dicts, which is atomic under the GIL for
which is atomic under the GIL for single-key assignments. single-key assignments.
""" """
enhanced_plugin = plugin.copy() enhanced_plugin = plugin.copy()
repo_url = plugin.get('repo', '') repo_url = plugin.get('repo', '')
@@ -912,24 +870,21 @@ class PluginStoreManager:
# The registry's plugins.json already carries ``description`` # The registry's plugins.json already carries ``description``
# (it is generated from each plugin's manifest by # (it is generated from each plugin's manifest by
# ``update_registry.py``), and ``last_updated`` is filled in # ``update_registry.py``), and ``last_updated`` is filled in
# from the commit info above. An earlier implementation # from the commit info above. Fetching manifest.json per
# fetched manifest.json per plugin anyway, which meant one # plugin costs one extra HTTPS round trip per result; on a Pi4
# extra HTTPS round trip per result; on a Pi4 with a flaky # with a flaky WiFi link the tail retries of that one call
# WiFi link the tail retries of that one extra call
# (_http_get_with_retries does 3 attempts with exponential # (_http_get_with_retries does 3 attempts with exponential
# backoff) dominated wall time even after parallelization. # backoff) dominate wall time even with the thread pool.
return enhanced_plugin return enhanced_plugin
# Fan out the per-plugin GitHub enrichment. The previous # Fan out the per-plugin GitHub enrichment. Serially, a Pi4 with ~15
# implementation did this serially, which on a Pi4 with ~15 plugins # plugins and a cold cache makes 30+ HTTP requests in strict sequence
# and a fresh cache meant 30+ HTTP requests in strict sequence (the # (the "connecting to display" hang users reported). With a thread
# "connecting to display" hang reported by users). With a thread
# pool, latency is dominated by the slowest request rather than # pool, latency is dominated by the slowest request rather than
# their sum. Workers capped at 10 to stay well under the # their sum. Workers capped at 10 to stay well under the
# unauthenticated GitHub rate limit burst and avoid overwhelming a # unauthenticated GitHub rate limit burst and avoid overwhelming a
# Pi's WiFi link. For a small number of plugins the pool is # Pi's WiFi link.
# essentially free.
if not filtered: if not filtered:
return [] return []
@@ -959,21 +914,11 @@ class PluginStoreManager:
Manifest data or None if not found Manifest data or None if not found
""" """
try: try:
# Convert repo URL to raw content URL owner_repo = github_owner_repo(repo_url)
# https://github.com/user/repo -> https://raw.githubusercontent.com/user/repo/branch/manifest.json if owner_repo is None:
_parsed_manifest_url = urlparse(repo_url) return None
if _parsed_manifest_url.hostname in ('github.com', 'www.github.com'): owner, repo = owner_repo
# Handle different URL formats
repo_url = repo_url.rstrip('/')
if repo_url.endswith('.git'):
repo_url = repo_url[:-4]
parts = repo_url.split('/')
if len(parts) >= 2:
owner = parts[-2]
repo = parts[-1]
# Check cache first
cache_key = f"{owner}/{repo}:{branch}:{manifest_path}" cache_key = f"{owner}/{repo}:{branch}:{manifest_path}"
if not force_refresh and cache_key in self.manifest_cache: if not force_refresh and cache_key in self.manifest_cache:
cached_time, cached_data = self.manifest_cache[cache_key] cached_time, cached_data = self.manifest_cache[cache_key]
@@ -981,15 +926,12 @@ class PluginStoreManager:
return cached_data return cached_data
raw_url = f"https://raw.githubusercontent.com/{owner}/{repo}/{branch}/{manifest_path}" raw_url = f"https://raw.githubusercontent.com/{owner}/{repo}/{branch}/{manifest_path}"
response = self._http_get_with_retries(raw_url, timeout=10) response = self._http_get_with_retries(raw_url, timeout=10)
if response.status_code == 200: if response.status_code == 200:
result = response.json() result = response.json()
self.manifest_cache[cache_key] = (time.time(), result) self.manifest_cache[cache_key] = (time.time(), result)
return result return result
elif response.status_code == 404: if response.status_code == 404 and branch != "main":
# Try main branch instead
if branch != "main":
raw_url = f"https://raw.githubusercontent.com/{owner}/{repo}/main/{manifest_path}" raw_url = f"https://raw.githubusercontent.com/{owner}/{repo}/main/{manifest_path}"
response = self._http_get_with_retries(raw_url, timeout=10) response = self._http_get_with_retries(raw_url, timeout=10)
if response.status_code == 200: if response.status_code == 200:
@@ -997,7 +939,8 @@ class PluginStoreManager:
self.manifest_cache[cache_key] = (time.time(), result) self.manifest_cache[cache_key] = (time.time(), result)
return result return result
# Cache negative result # Cache the miss too, so a plugin without a manifest at this path
# is not re-fetched on every browse.
self.manifest_cache[cache_key] = (time.time(), None) self.manifest_cache[cache_key] = (time.time(), None)
except Exception as e: except Exception as e:
self.logger.debug(f"Could not fetch manifest from GitHub for {repo_url}: {e}") self.logger.debug(f"Could not fetch manifest from GitHub for {repo_url}: {e}")
@@ -1007,21 +950,11 @@ class PluginStoreManager:
def _get_latest_commit_info(self, repo_url: str, branch: str = "main", force_refresh: bool = False) -> Optional[Dict[str, Any]]: def _get_latest_commit_info(self, repo_url: str, branch: str = "main", force_refresh: bool = False) -> Optional[Dict[str, Any]]:
"""Return metadata about the latest commit on the given branch.""" """Return metadata about the latest commit on the given branch."""
try: try:
if 'github.com' not in repo_url: owner_repo = github_owner_repo(repo_url)
if owner_repo is None:
return None return None
owner, repo = owner_repo
repo_url = repo_url.rstrip('/')
if repo_url.endswith('.git'):
repo_url = repo_url[:-4]
parts = repo_url.split('/')
if len(parts) < 2:
return None
owner = parts[-2]
repo = parts[-1]
# Check cache first
cache_key = f"{owner}/{repo}:{branch}" cache_key = f"{owner}/{repo}:{branch}"
if not force_refresh and cache_key in self.commit_info_cache: if not force_refresh and cache_key in self.commit_info_cache:
cached_time, cached_data = self.commit_info_cache[cache_key] cached_time, cached_data = self.commit_info_cache[cache_key]
@@ -1029,14 +962,7 @@ class PluginStoreManager:
return cached_data return cached_data
branches_to_try = self._distinct_sequence([branch, 'main', 'master']) branches_to_try = self._distinct_sequence([branch, 'main', 'master'])
headers = github_api_headers(self.github_token)
headers = {
'Accept': 'application/vnd.github.v3+json',
'User-Agent': 'LEDMatrix-Plugin-Manager/1.0'
}
if self.github_token:
headers['Authorization'] = f'token {self.github_token}'
last_error = None last_error = None
for branch_name in branches_to_try: for branch_name in branches_to_try:
@@ -1254,46 +1180,57 @@ class PluginStoreManager:
backup_path = plugin_path.with_name( backup_path = plugin_path.with_name(
f"{plugin_path.name}{BACKUP_MARKER}preinstall") f"{plugin_path.name}{BACKUP_MARKER}preinstall")
if backup_path.exists() and not self._safe_remove_directory(backup_path): problem = self._set_aside(plugin_path, backup_path)
# Can't stage a safety net. Better to attempt the install than if problem:
# to refuse outright, which is what callers got before this # Can't stage a safety net. Attempting the install anyway is
# existed. # what callers got before the net existed; refusing would be
# a new failure mode for a direct install.
self.logger.warning( self.logger.warning(
"Could not clear stale pre-install backup for %s at %s; " "Installing %s without a rollback net: %s", plugin_id, problem)
"installing without a rollback net", plugin_id, backup_path)
return self._install_plugin_impl(plugin_id, branch)
try:
plugin_path.rename(backup_path)
except OSError as e:
self.logger.warning(
"Could not set aside existing install of %s (%s); "
"installing without a rollback net", plugin_id, e)
return self._install_plugin_impl(plugin_id, branch) return self._install_plugin_impl(plugin_id, branch)
try: try:
installed = self._install_plugin_impl(plugin_id, branch) installed = self._install_plugin_impl(plugin_id, branch)
except Exception: except Exception:
self._restore_preinstall_backup(plugin_id, plugin_path, backup_path) self._restore_backup(plugin_id, plugin_path, backup_path, "Install")
raise raise
if installed: if installed:
if not self._safe_remove_directory(backup_path): self._discard_backup(plugin_id, backup_path, "install")
self.logger.warning(
"Install of %s succeeded but the previous copy at %s "
"could not be removed; it will be cleared on the next "
"install", plugin_id, backup_path)
return True return True
self._restore_preinstall_backup(plugin_id, plugin_path, backup_path) self._restore_backup(plugin_id, plugin_path, backup_path, "Install")
return False return False
def _restore_preinstall_backup( def _set_aside(self, plugin_path: Path, backup_path: Path) -> Optional[str]:
self, plugin_id: str, plugin_path: Path, backup_path: Path """Rename an installed plugin to ``backup_path`` so a failed
(re)install can put it back.
A stale backup left by a crash is cleared first, since it would block
the rename. Returns None on success, otherwise why it could not.
"""
if backup_path.exists() and not self._safe_remove_directory(backup_path):
return f"could not clear stale backup at {backup_path}"
try:
plugin_path.rename(backup_path)
except OSError as e:
return f"could not set aside {plugin_path}: {e}"
return None
def _discard_backup(self, plugin_id: str, backup_path: Path, action: str) -> None:
"""Remove the set-aside copy after a successful (re)install."""
if not self._safe_remove_directory(backup_path):
self.logger.warning(
"%s of %s succeeded but the previous copy at %s could not be "
"removed; it will be cleared on the next %s",
action.capitalize(), plugin_id, backup_path, action)
def _restore_backup(
self, plugin_id: str, plugin_path: Path, backup_path: Path, action: str
) -> None: ) -> None:
"""Put the previous install back after a failed (re)install.""" """Put the set-aside copy back after a failed (re)install."""
self.logger.error( self.logger.error(
"Install of %s failed; restoring the previous version", plugin_id) "%s of %s failed; restoring the previous version", action, plugin_id)
try: try:
if plugin_path.exists(): if plugin_path.exists():
# Partial download debris from the failed install. # Partial download debris from the failed install.
@@ -1375,8 +1312,7 @@ class PluginStoreManager:
return False return False
else: else:
branch_used = self._install_via_git(repo_url, plugin_path, branch_candidates) branch_used = self._install_via_git(repo_url, plugin_path, branch_candidates)
if branch_used is None and not plugin_path.exists(): if branch_used is None:
# Git failed entirely; fall back to zip download
self.logger.info("Git not available or clone failed, attempting archive download...") self.logger.info("Git not available or clone failed, attempting archive download...")
for candidate in branch_candidates: for candidate in branch_candidates:
download_url = f"{repo_url}/archive/refs/heads/{candidate}.zip" download_url = f"{repo_url}/archive/refs/heads/{candidate}.zip"
@@ -1384,7 +1320,7 @@ class PluginStoreManager:
branch_used = candidate branch_used = candidate
break break
if branch_used is None and not plugin_path.exists(): if branch_used is None:
self.logger.error(f"Failed to install plugin {plugin_id} via git or archive download") self.logger.error(f"Failed to install plugin {plugin_id} via git or archive download")
return False return False
@@ -1523,8 +1459,7 @@ class PluginStoreManager:
branch_info = f" (branch: {branch})" if branch else "" branch_info = f" (branch: {branch})" if branch else ""
self.logger.info(f"Installing plugin from custom URL: {repo_url}{branch_info}" + (f" (subpath: {plugin_path})" if plugin_path else "")) self.logger.info(f"Installing plugin from custom URL: {repo_url}{branch_info}" + (f" (subpath: {plugin_path})" if plugin_path else ""))
# Clean up URL (remove .git suffix if present) repo_url = normalize_repo_url(repo_url)
repo_url = repo_url.rstrip('/').replace('.git', '')
temp_dir = None temp_dir = None
try: try:
@@ -1549,13 +1484,11 @@ class PluginStoreManager:
'error': f'Failed to download or extract plugin from monorepo subdirectory: {plugin_path}' 'error': f'Failed to download or extract plugin from monorepo subdirectory: {plugin_path}'
} }
else: else:
# Try git clone for direct plugin repos
branch_used = self._install_via_git(repo_url, temp_dir, branch_candidates) branch_used = self._install_via_git(repo_url, temp_dir, branch_candidates)
if branch_used: if branch_used is not None:
self.logger.info(f"Cloned via git (branch: {branch_used})") self.logger.info(f"Cloned via git (branch: {branch_used})")
else: else:
# Git failed; try downloading as zip self.logger.info("Git not available or clone failed, attempting archive download...")
branch_used = None
for candidate in branch_candidates: for candidate in branch_candidates:
download_url = f"{repo_url}/archive/refs/heads/{candidate}.zip" download_url = f"{repo_url}/archive/refs/heads/{candidate}.zip"
if self._install_via_download(download_url, temp_dir): if self._install_via_download(download_url, temp_dir):
@@ -1634,8 +1567,10 @@ class PluginStoreManager:
json.dump(manifest, f, indent=2) json.dump(manifest, f, indent=2)
self.logger.info(f"Added missing entry_point field to {plugin_id} manifest (defaulted to manager.py)") self.logger.info(f"Added missing entry_point field to {plugin_id} manifest (defaulted to manager.py)")
# Move to plugins directory - use manifest ID as source of truth # The directory is named for the caller's plugin_id when one was
# This ensures directory name always matches manifest ID # given (update_plugin passes the installed id), else for the
# manifest's id -- so it can differ from the manifest id, which
# discovery tolerates by reading the manifest.
final_path = self.plugins_dir / plugin_id final_path = self.plugins_dir / plugin_id
if final_path.exists(): if final_path.exists():
self.logger.warning(f"Plugin {plugin_id} already exists, removing existing copy") self.logger.warning(f"Plugin {plugin_id} already exists, removing existing copy")
@@ -1648,8 +1583,6 @@ class PluginStoreManager:
shutil.move(str(temp_dir), str(final_path)) shutil.move(str(temp_dir), str(final_path))
temp_dir = None # Prevent cleanup since we moved it temp_dir = None # Prevent cleanup since we moved it
# Note: plugin_id here is already from manifest (line 749), so directory name matches manifest ID
# Install dependencies # Install dependencies
self._install_dependencies(final_path) self._install_dependencies(final_path)
@@ -1701,7 +1634,6 @@ class PluginStoreManager:
Class name if found, None otherwise Class name if found, None otherwise
""" """
try: try:
import re
with open(manager_file, 'r', encoding='utf-8') as f: with open(manager_file, 'r', encoding='utf-8') as f:
content = f.read() content = f.read()
@@ -1723,7 +1655,18 @@ class PluginStoreManager:
return None return None
def _install_via_git(self, repo_url: str, target_path: Path, branches: Optional[List[str]] = None) -> Optional[str]: def _install_via_git(self, repo_url: str, target_path: Path, branches: Optional[List[str]] = None) -> Optional[str]:
"""Clone a repository into ``target_path``. Returns the branch name on success.""" """Clone a repository into ``target_path``.
Tries each of ``branches`` (default ``main``, ``master``), then the
repository's own default branch, so a repository whose only branch
is e.g. ``develop`` still installs.
Returns:
The branch that was cloned, or None when every clone failed and
``target_path`` has been removed. After a default-branch clone
this is the branch the clone checked out (``'HEAD'`` if the
remote's HEAD is detached), never None.
"""
branches_to_try = self._distinct_sequence(branches or []) branches_to_try = self._distinct_sequence(branches or [])
if not branches_to_try: if not branches_to_try:
branches_to_try = ['main', 'master'] branches_to_try = ['main', 'master']
@@ -1758,7 +1701,7 @@ class PluginStoreManager:
timeout=60 timeout=60
) )
self.logger.debug(f"Successfully cloned {repo_url} (git default branch) to {target_path}") self.logger.debug(f"Successfully cloned {repo_url} (git default branch) to {target_path}")
return None # Unknown branch name, git default used return self._checked_out_branch(target_path)
except (subprocess.CalledProcessError, subprocess.TimeoutExpired, FileNotFoundError) as e: except (subprocess.CalledProcessError, subprocess.TimeoutExpired, FileNotFoundError) as e:
last_error = e last_error = e
if target_path.exists(): if target_path.exists():
@@ -1767,6 +1710,19 @@ class PluginStoreManager:
self.logger.error(f"Git clone failed for all attempted branches: {last_error}") self.logger.error(f"Git clone failed for all attempted branches: {last_error}")
return None return None
@staticmethod
def _checked_out_branch(checkout: Path) -> str:
"""The branch a fresh clone has checked out, read from ``.git/HEAD``.
``'HEAD'`` when HEAD is detached or unreadable.
"""
try:
head = (checkout / '.git' / 'HEAD').read_text(encoding='utf-8').strip()
except OSError:
return 'HEAD'
prefix = 'ref: refs/heads/'
return head[len(prefix):] if head.startswith(prefix) else 'HEAD'
def _install_from_monorepo(self, download_url: str, plugin_subpath: str, target_path: Path) -> bool: def _install_from_monorepo(self, download_url: str, plugin_subpath: str, target_path: Path) -> bool:
""" """
Install a plugin from a monorepo by downloading only the target subdirectory. Install a plugin from a monorepo by downloading only the target subdirectory.
@@ -1815,14 +1771,6 @@ class PluginStoreManager:
pass pass
return None, None return None, None
@staticmethod
def _normalize_repo_url(url: str) -> str:
"""Normalize a GitHub repo URL for comparison (strip trailing / and .git)."""
url = url.rstrip('/')
if url.endswith('.git'):
url = url[:-4]
return url.lower()
def _install_from_monorepo_api(self, repo_url: str, branch: str, plugin_subpath: str, target_path: Path) -> bool: def _install_from_monorepo_api(self, repo_url: str, branch: str, plugin_subpath: str, target_path: Path) -> bool:
""" """
Install a plugin subdirectory using the GitHub Git Trees API. Install a plugin subdirectory using the GitHub Git Trees API.
@@ -1841,25 +1789,15 @@ class PluginStoreManager:
True if successful, False to trigger ZIP fallback True if successful, False to trigger ZIP fallback
""" """
try: try:
# Parse owner/repo from URL owner_repo = github_owner_repo(repo_url)
clean_url = repo_url.rstrip('/') if owner_repo is None:
if clean_url.endswith('.git'):
clean_url = clean_url[:-4]
parts = clean_url.split('/')
if len(parts) < 2:
return False return False
owner, repo = parts[-2], parts[-1] owner, repo = owner_repo
# Step 1: Get the recursive tree listing (1 API call) # Step 1: Get the recursive tree listing (1 API call)
api_url = f"https://api.github.com/repos/{owner}/{repo}/git/trees/{branch}?recursive=true" api_url = f"https://api.github.com/repos/{owner}/{repo}/git/trees/{branch}?recursive=true"
headers = { tree_response = self._http_get_with_retries(
'Accept': 'application/vnd.github.v3+json', api_url, timeout=15, headers=github_api_headers(self.github_token))
'User-Agent': 'LEDMatrix-Plugin-Manager/1.0'
}
if self.github_token:
headers['Authorization'] = f'token {self.github_token}'
tree_response = self._http_get_with_retries(api_url, timeout=15, headers=headers)
if tree_response.status_code != 200: if tree_response.status_code != 200:
self.logger.debug(f"Trees API returned {tree_response.status_code} for {owner}/{repo}") self.logger.debug(f"Trees API returned {tree_response.status_code} for {owner}/{repo}")
return False return False
@@ -1892,10 +1830,6 @@ class PluginStoreManager:
self.logger.info(f"Downloading {len(file_entries)} files for {plugin_subpath} via API") self.logger.info(f"Downloading {len(file_entries)} files for {plugin_subpath} via API")
# Step 3: Create target directory and download each file # Step 3: Create target directory and download each file
from src.common.permission_utils import (
ensure_directory_permissions,
get_plugin_dir_mode
)
ensure_directory_permissions(target_path.parent, get_plugin_dir_mode()) ensure_directory_permissions(target_path.parent, get_plugin_dir_mode())
target_path.mkdir(parents=True, exist_ok=True) target_path.mkdir(parents=True, exist_ok=True)
@@ -1991,10 +1925,6 @@ class PluginStoreManager:
source_plugin_dir = temp_extract / root_dir / plugin_subpath source_plugin_dir = temp_extract / root_dir / plugin_subpath
from src.common.permission_utils import (
ensure_directory_permissions,
get_plugin_dir_mode
)
ensure_directory_permissions(target_path.parent, get_plugin_dir_mode()) ensure_directory_permissions(target_path.parent, get_plugin_dir_mode())
# Ensure target doesn't exist to prevent shutil.move nesting # Ensure target doesn't exist to prevent shutil.move nesting
if target_path.exists(): if target_path.exists():
@@ -2028,7 +1958,7 @@ class PluginStoreManager:
try: try:
self.logger.info(f"Downloading from: {download_url}") self.logger.info(f"Downloading from: {download_url}")
# Allow redirects (GitHub archive URLs redirect to codeload.github.com) # Allow redirects (GitHub archive URLs redirect to codeload.github.com)
response = self._http_get_with_retries(download_url, timeout=60, stream=True, headers={'User-Agent': 'LEDMatrix-Plugin-Manager/1.0'}) response = self._http_get_with_retries(download_url, timeout=60, stream=True, headers={'User-Agent': USER_AGENT})
response.raise_for_status() response.raise_for_status()
# Download to temporary file # Download to temporary file
@@ -2065,10 +1995,6 @@ class PluginStoreManager:
# Move contents from root_dir to target # Move contents from root_dir to target
source_dir = temp_extract / root_dir source_dir = temp_extract / root_dir
if source_dir.exists(): if source_dir.exists():
from src.common.permission_utils import (
ensure_directory_permissions,
get_plugin_dir_mode
)
ensure_directory_permissions(target_path.parent, get_plugin_dir_mode()) ensure_directory_permissions(target_path.parent, get_plugin_dir_mode())
shutil.move(str(source_dir), str(target_path)) shutil.move(str(source_dir), str(target_path))
else: else:
@@ -2094,42 +2020,23 @@ class PluginStoreManager:
""" """
Install Python dependencies from requirements.txt. Install Python dependencies from requirements.txt.
``plugin_path`` is ultimately derived from a plugin-supplied manifest
``id``, so it is only used after contained_plugin_dir() has rebuilt it
from a listing of ``self.plugins_dir``.
Args: Args:
plugin_path: Path to plugin directory plugin_path: Path to plugin directory
Returns: Returns:
True if successful or no requirements file True if successful or no requirements file
""" """
# Reconstruct the plugin path from the trusted self.plugins_dir base + safe_plugin_dir = contained_plugin_dir(plugin_path, self.plugins_dir)
# an entry actually enumerated from it, rather than trusting if safe_plugin_dir is None:
# plugin_path directly -- callers ultimately derive it from a
# plugin-supplied manifest "id" field (see install_plugin_from_url),
# so without this a malicious manifest could point requirements_file
# outside plugins_dir. find_trusted_subdir()'s return value always
# comes from os.scandir() on the trusted root, so building the path
# from it (not from the caller's string) is a real containment
# guarantee, matching the pattern in PluginLoader.install_dependencies().
plugin_dir_real = os.path.realpath(str(plugin_path))
plugins_dir_real = os.path.realpath(str(self.plugins_dir))
requested_name = os.path.basename(plugin_dir_real)
matched_name = find_trusted_subdir(plugins_dir_real, requested_name)
if matched_name is None:
self.logger.error("Plugin directory not found inside plugins dir for dependency install") self.logger.error("Plugin directory not found inside plugins dir for dependency install")
return False return False
safe_plugin_path = Path(os.path.join(plugins_dir_real, matched_name))
requirements_file = safe_plugin_path / "requirements.txt" requirements_file = requirements_to_install(safe_plugin_dir, self.logger, plugin_path.name)
if requirements_file is None:
if not requirements_file.exists():
self.logger.debug(f"No requirements.txt found in {plugin_path.name}")
return True
if not requirements_has_real_deps(str(requirements_file)):
self.logger.debug(f"requirements.txt for {plugin_path.name} has no real dependencies, skipping pip")
return True
if requirements_are_satisfied(str(requirements_file)):
self.logger.debug(f"Dependencies for {plugin_path.name} already satisfied, skipping pip")
return True return True
try: try:
@@ -2141,7 +2048,7 @@ class PluginStoreManager:
# ledmatrix.service, so pip reports success while the package # ledmatrix.service, so pip reports success while the package
# stays invisible to the running plugin (e.g. missing `astral` # stays invisible to the running plugin (e.g. missing `astral`
# for the weather plugin even though "install" succeeded). # for the weather plugin even though "install" succeeded).
result = install_requirements_file(requirements_file, timeout=300) result = install_requirements_file(Path(requirements_file), timeout=300)
if result.returncode != 0: if result.returncode != 0:
self.logger.error( self.logger.error(
f"Error installing dependencies for {plugin_path.name}: {result.stderr}" f"Error installing dependencies for {plugin_path.name}: {result.stderr}"
@@ -2153,10 +2060,10 @@ class PluginStoreManager:
except subprocess.TimeoutExpired: except subprocess.TimeoutExpired:
self.logger.error("Dependency installation timed out") self.logger.error("Dependency installation timed out")
return False return False
except (BrokenPipeError, OSError) as e: except OSError as e:
# Handle broken pipe errors (errno 32) which can occur during pip downloads # A broken pipe (EPIPE) happens when pip's output pipe closes
# Often caused by network interruptions or output buffer issues # mid-download, usually a network interruption.
if isinstance(e, OSError) and e.errno == 32: if e.errno == errno.EPIPE:
self.logger.error( self.logger.error(
f"Broken pipe error during dependency installation for {plugin_path.name}. " f"Broken pipe error during dependency installation for {plugin_path.name}. "
f"This usually indicates a network interruption or pip output buffer issue. " f"This usually indicates a network interruption or pip output buffer issue. "
@@ -2166,7 +2073,6 @@ class PluginStoreManager:
self.logger.error(f"OS error during dependency installation: {e}") self.logger.error(f"OS error during dependency installation: {e}")
return False return False
except Exception as e: except Exception as e:
# Catch any other unexpected errors
self.logger.error(f"Unexpected error installing dependencies for {plugin_path.name}: {e}", exc_info=True) self.logger.error(f"Unexpected error installing dependencies for {plugin_path.name}: {e}", exc_info=True)
return False return False
@@ -2241,9 +2147,9 @@ class PluginStoreManager:
Results are cached keyed on a signature that includes HEAD Results are cached keyed on a signature that includes HEAD
contents plus the mtime of HEAD AND the resolved ref (or contents plus the mtime of HEAD AND the resolved ref (or
packed-refs). Repeated calls skip the four ``git`` subprocesses packed-refs). Repeated calls skip the ``git log`` subprocess when
when nothing has changed, and a ``git pull`` that fast-forwards nothing has changed, and a ``git pull`` that fast-forwards the
the branch correctly invalidates the cache. branch correctly invalidates the cache.
""" """
git_dir = plugin_path / '.git' git_dir = plugin_path / '.git'
if not git_dir.exists(): if not git_dir.exists():
@@ -2412,12 +2318,11 @@ class PluginStoreManager:
No ``ledmatrix-`` prefix and no case folding here, unlike the loader: No ``ledmatrix-`` prefix and no case folding here, unlike the loader:
a store operation may delete what this returns, so it only accepts a a store operation may delete what this returns, so it only accepts a
directory that names the id exactly or declares it. Note that this directory that names the id exactly or declares it. So a registry id
leaves registry ids like `stocks` unresolved when the installed such as `stocks` does not resolve to an installed `ledmatrix-stocks/`
plugin is `ledmatrix-stocks/` declaring `ledmatrix-stocks` (the declaring `ledmatrix-stocks` (the monorepo's leaderboard, music,
monorepo's leaderboard, music, stocks and weather); passing stocks and weather); callers pass the installed id, and
``prefix=True`` would resolve them, but update_plugin()'s reinstall update_plugin() maps it back to the registry id itself.
path has not been checked against that yet.
Args: Args:
plugin_id: Plugin identifier plugin_id: Plugin identifier
@@ -2545,11 +2450,9 @@ class PluginStoreManager:
The old install is renamed aside (not deleted) until the new install The old install is renamed aside (not deleted) until the new install
succeeds, then removed; on ANY install failure the old directory is succeeds, then removed; on ANY install failure the old directory is
restored. This is the difference between a failed update and a restored. Deleting first turns a failed download into a destroyed
destroyed plugin: the previous delete-then-install flow permanently plugin: during the monorepo migration a Pi with broken DNS lost every
removed plugins whenever the download failed mid-update (seen in the old-remote plugin that way, with none able to be re-downloaded.
field during the monorepo migration on a Pi with broken DNS — every
old-remote plugin was deleted and none could be re-downloaded).
The aside name embeds BACKUP_MARKER ('.standalone-backup-') so every The aside name embeds BACKUP_MARKER ('.standalone-backup-') so every
plugin directory lookup (src/plugin_system/plugin_dirs.py) ignores it plugin directory lookup (src/plugin_system/plugin_dirs.py) ignores it
@@ -2564,18 +2467,11 @@ class PluginStoreManager:
with self._get_reinstall_lock(plugin_id): with self._get_reinstall_lock(plugin_id):
backup_path = plugin_path.with_name( backup_path = plugin_path.with_name(
f"{plugin_path.name}{BACKUP_MARKER}migrating") f"{plugin_path.name}{BACKUP_MARKER}migrating")
# A stale aside from a previous crash would block the rename problem = self._set_aside(plugin_path, backup_path)
if backup_path.exists(): if problem:
if not self._safe_remove_directory(backup_path):
self.logger.error( self.logger.error(
f"Could not clear stale backup for {plugin_id} at " "Not updating %s: %s; the installed version is left in place",
f"{backup_path}; leaving old install in place") plugin_id, problem)
return False
try:
plugin_path.rename(backup_path)
except OSError as e:
self.logger.error(
f"Could not set aside old plugin directory for {plugin_id}: {e}")
return False return False
try: try:
@@ -2585,27 +2481,11 @@ class PluginStoreManager:
installed = False installed = False
if installed: if installed:
if not self._safe_remove_directory(backup_path): self._discard_backup(plugin_id, backup_path, "update")
self.logger.warning(
f"Update of {plugin_id} succeeded but the old backup "
f"at {backup_path} could not be removed; it will be "
f"cleared on the next update")
return True return True
# Install failed (bad network, registry error...) — put the old # Bad network, registry error...: the user keeps a working plugin.
# version back so the user still has a working plugin. self._restore_backup(plugin_id, plugin_path, backup_path, "Reinstall")
self.logger.error(
f"Reinstall of {plugin_id} failed; restoring previous version")
try:
if plugin_path.exists():
# partial download debris from the failed install
self._safe_remove_directory(plugin_path)
backup_path.rename(plugin_path)
self.logger.info(f"Restored previous install of {plugin_id}")
except OSError as e:
self.logger.error(
f"CRITICAL: could not restore {plugin_id} from {backup_path}: {e}. "
f"The previous install is preserved there — rename it back manually.")
return False return False
def update_plugin(self, plugin_id: str) -> bool: def update_plugin(self, plugin_id: str) -> bool:
@@ -2665,7 +2545,7 @@ class PluginStoreManager:
# while the registry now points to the monorepo. Detect this and reinstall. # while the registry now points to the monorepo. Detect this and reinstall.
registry_repo = plugin_info_remote.get('repo', '') registry_repo = plugin_info_remote.get('repo', '')
local_remote = git_info.get('remote_url', '') local_remote = git_info.get('remote_url', '')
if local_remote and registry_repo and self._normalize_repo_url(local_remote) != self._normalize_repo_url(registry_repo): if local_remote and registry_repo and not same_repo(local_remote, registry_repo):
self.logger.info( self.logger.info(
f"Plugin {resolved_id} git remote ({local_remote}) differs from registry ({registry_repo}). " f"Plugin {resolved_id} git remote ({local_remote}) differs from registry ({registry_repo}). "
f"Reinstalling from registry to migrate to new source." f"Reinstalling from registry to migrate to new source."
@@ -2814,7 +2694,6 @@ class PluginStoreManager:
# If status check times out, assume there might be changes and proceed # If status check times out, assume there might be changes and proceed
self.logger.warning(f"Git status check timed out for {plugin_id}, proceeding with update") self.logger.warning(f"Git status check timed out for {plugin_id}, proceeding with update")
has_changes = True has_changes = True
status_result = type('obj', (object,), {'stdout': '', 'stderr': 'Status check timed out'})()
stash_info = "" stash_info = ""
# Whether the pull can be undone without destroying work. # Whether the pull can be undone without destroying work.
@@ -2898,7 +2777,7 @@ class PluginStoreManager:
except subprocess.CalledProcessError as git_error: except subprocess.CalledProcessError as git_error:
error_output = git_error.stderr or git_error.stdout or "Unknown error" error_output = git_error.stderr or git_error.stdout or "Unknown error"
cmd_str = ' '.join(git_error.cmd) if hasattr(git_error, 'cmd') else 'unknown' cmd_str = ' '.join(git_error.cmd)
self.logger.error(f"Git update failed for {plugin_id}") self.logger.error(f"Git update failed for {plugin_id}")
self.logger.error(f"Command: {cmd_str}") self.logger.error(f"Command: {cmd_str}")
self.logger.error(f"Return code: {git_error.returncode}") self.logger.error(f"Return code: {git_error.returncode}")
@@ -2914,7 +2793,7 @@ class PluginStoreManager:
self.logger.error(f"Authentication failed for {plugin_id}. Check git credentials or repository permissions.") self.logger.error(f"Authentication failed for {plugin_id}. Check git credentials or repository permissions.")
elif "not found" in error_lower or "does not exist" in error_lower: elif "not found" in error_lower or "does not exist" in error_lower:
self.logger.error(f"Remote branch or repository not found for {plugin_id}. Check repository URL and branch name.") self.logger.error(f"Remote branch or repository not found for {plugin_id}. Check repository URL and branch name.")
elif "merge conflict" in error_lower or "conflict" in error_lower: elif "conflict" in error_lower:
self.logger.error(f"Merge conflict detected for {plugin_id}. Resolve conflicts manually or reinstall plugin.") self.logger.error(f"Merge conflict detected for {plugin_id}. Resolve conflicts manually or reinstall plugin.")
return False return False
@@ -2922,12 +2801,15 @@ class PluginStoreManager:
self.logger.warning(f"Git update timed out for {plugin_id}") self.logger.warning(f"Git update timed out for {plugin_id}")
return False return False
# Not a git repository - try to get repo URL from git config if it exists # A plugin with its own .git that _get_local_git_info could not
# (in case .git directory was removed but remote URL is still in config) # read (e.g. no commits yet) may still name a remote to reinstall
# from. Without its own .git, `git -C <plugin>` walks up and finds
# the enclosing LEDMatrix checkout when plugins live in
# plugin-repos/ -- `--local` does not prevent that -- and the
# "plugin's" remote would be LEDMatrix itself.
repo_url = None repo_url = None
if (plugin_path / '.git').exists():
try: try:
# Use --local to avoid inheriting the parent LEDMatrix repo's git config
# when the plugin directory lives inside the main repo (e.g. plugin-repos/).
remote_url_result = subprocess.run( remote_url_result = subprocess.run(
['git', '-C', str(plugin_path), 'config', '--local', '--get', 'remote.origin.url'], ['git', '-C', str(plugin_path), 'config', '--local', '--get', 'remote.origin.url'],
capture_output=True, capture_output=True,
@@ -2936,9 +2818,10 @@ class PluginStoreManager:
check=False check=False
) )
if remote_url_result.returncode == 0: if remote_url_result.returncode == 0:
repo_url = remote_url_result.stdout.strip() repo_url = remote_url_result.stdout.strip() or None
if repo_url:
self.logger.info(f"Found git remote URL for {plugin_id}: {repo_url}") self.logger.info(f"Found git remote URL for {plugin_id}: {repo_url}")
except Exception as e: except (OSError, subprocess.SubprocessError) as e:
self.logger.debug(f"Could not get git remote URL: {e}") self.logger.debug(f"Could not get git remote URL: {e}")
# Try registry-based update # Try registry-based update
@@ -3027,9 +2910,7 @@ class PluginStoreManager:
return self._reinstall_with_rollback(registry_id, plugin_path) return self._reinstall_with_rollback(registry_id, plugin_path)
except Exception as e: except Exception as e:
import traceback self.logger.error(f"Error updating plugin {plugin_id}: {e}", exc_info=True)
self.logger.error(f"Error updating plugin {plugin_id}: {e}")
self.logger.debug(traceback.format_exc())
return False return False
def list_installed_plugins(self) -> List[str]: def list_installed_plugins(self) -> List[str]:
test/test_config_validation_edge_cases.py
+45
View File
@@ -299,3 +299,48 @@ class TestDefaultMerging:
assert merged["enabled"] is False assert merged["enabled"] is False
assert merged["display_duration"] == 60 assert merged["display_duration"] == 60
class TestMissingRequiredFields:
"""One message per missing field, naming that field.
A manual ``required`` loop used to run after Draft7Validator, which already
reports ``required``, so every missing top-level field was listed twice --
and the validator's copy printed the schema's whole ``required`` list as
if it were the field name.
"""
SCHEMA = {
"type": "object",
"properties": {
"api_key": {"type": "string"},
"city": {"type": "string"},
"units": {"type": "string"},
},
"required": ["api_key", "city", "units"],
}
def test_each_missing_field_is_reported_once_by_name(self):
ok, errors = SchemaManager().validate_config_against_schema(
{"units": "metric"}, self.SCHEMA, "test-plugin")
assert not ok
assert errors == [
"Field root: Missing required property 'api_key'",
"Field root: Missing required property 'city'",
]
def test_nested_missing_field_names_the_field_and_its_parent(self):
schema = {
"type": "object",
"properties": {"nfl": {
"type": "object",
"properties": {"api_key": {"type": "string"}},
"required": ["api_key"],
}},
}
ok, errors = SchemaManager().validate_config_against_schema(
{"nfl": {}}, schema, "test-plugin")
assert not ok
assert errors == ["Field 'nfl': Missing required property 'api_key'"]
test/test_metrics_cache_unknown_fields.py
+6 -2
View File
@@ -116,10 +116,14 @@ def test_values_of_the_wrong_type_fall_back_to_usable_defaults(bad):
assert isinstance(getattr(metrics, field_name), (int, float)), \ assert isinstance(getattr(metrics, field_name), (int, float)), \
f"{field_name} came back as {getattr(metrics, field_name)!r}" f"{field_name} came back as {getattr(metrics, field_name)!r}"
# The real proof: arithmetic on the loaded metrics must not explode. # The real proof: the arithmetic monitor_call and get_metrics_summary do
# on the loaded metrics must not explode.
metrics.call_count += 1 metrics.call_count += 1
metrics.total_execution_time += 0.5 metrics.total_execution_time += 0.5
metrics.update_average_execution_time() metrics.max_execution_time = max(metrics.max_execution_time, 0.5)
metrics.min_execution_time = min(metrics.min_execution_time, 0.5)
metrics.memory_mb = max(metrics.memory_mb, 1.0)
assert metrics.total_execution_time / metrics.call_count >= 0
def test_a_numeric_string_is_accepted_rather_than_discarded(): def test_a_numeric_string_is_accepted_rather_than_discarded():
test/test_plugin_manager_reload.py
+36
View File
@@ -0,0 +1,36 @@
"""reload_plugin re-reads the manifest from the plugin's actual directory.
It read ``plugins_dir / plugin_id / manifest.json``, but a plugin directory's
name need not be the id its manifest declares -- discovery maps ids to
directories for exactly that reason. For such a plugin the path did not exist,
the re-read was skipped silently, and the reload kept the stale manifest.
"""
import json
import pytest
from src.plugin_system.plugin_manager import PluginManager
@pytest.fixture
def pm_with_renamed_dir(tmp_path):
plugins_dir = tmp_path / "plugins"
plugin_dir = plugins_dir / "stock-ticker-v2"
plugin_dir.mkdir(parents=True)
manifest_path = plugin_dir / "manifest.json"
manifest_path.write_text(json.dumps({"id": "stocks", "version": "1.0.0"}))
pm = PluginManager(plugins_dir=str(plugins_dir))
assert pm.discover_plugins() == ["stocks"]
return pm, manifest_path
def test_reload_picks_up_an_edited_manifest(pm_with_renamed_dir, monkeypatch):
pm, manifest_path = pm_with_renamed_dir
manifest_path.write_text(json.dumps({"id": "stocks", "version": "2.0.0"}))
loaded = []
monkeypatch.setattr(pm, "load_plugin", lambda pid: loaded.append(pid) or True)
assert pm.reload_plugin("stocks") is True
assert pm.plugin_manifests["stocks"]["version"] == "2.0.0"
assert loaded == ["stocks"]
test/test_plugin_state_transition_count.py
+17
View File
@@ -115,5 +115,22 @@ def test_get_state_info_is_a_consistent_snapshot():
assert not inconsistent, f"observed a torn snapshot: {inconsistent[:1]}" assert not inconsistent, f"observed a torn snapshot: {inconsistent[:1]}"
def test_state_info_reports_only_what_something_records():
"""No field that is always null.
``last_display`` was reported here, but nothing ever recorded a display()
call, so it was null for every plugin. Its only reader is the web process,
whose PluginManager never calls display(), so recording it in the display
process could not have filled it either.
"""
manager = PluginStateManager()
manager.set_state("clock", PluginState.ENABLED)
manager.record_update("clock")
info = manager.get_state_info("clock")
assert "last_display" not in info
assert info["last_update"] is not None
if __name__ == "__main__": if __name__ == "__main__":
sys.exit(pytest.main([__file__, "-v"])) sys.exit(pytest.main([__file__, "-v"]))
test/test_repo_urls.py
+95
View File
@@ -0,0 +1,95 @@
"""Repository URL handling shared by the plugin store and saved repositories.
The store cleaned URLs with ``url.rstrip('/').replace('.git', '')`` in two
places, which removes ``.git`` anywhere in the URL:
``https://github.com/user/my.github.io`` became ``.../myhub.io``, so installing
or browsing such a repository asked GitHub for one that does not exist.
"""
from unittest.mock import MagicMock
import pytest
from src.plugin_system.repo_urls import (
github_api_headers, github_owner_repo, normalize_repo_url, same_repo,
)
from src.plugin_system.store_manager import PluginStoreManager
PAGES_REPO = "https://github.com/user/my.github.io"
class TestNormalizeRepoUrl:
@pytest.mark.parametrize("raw, expected", [
(PAGES_REPO, PAGES_REPO),
(PAGES_REPO + ".git", PAGES_REPO),
("https://github.com/user/repo.git/", "https://github.com/user/repo"),
(" https://github.com/user/repo/ ", "https://github.com/user/repo"),
])
def test_only_a_trailing_dot_git_is_removed(self, raw, expected):
assert normalize_repo_url(raw) == expected
def test_same_repo_ignores_case_and_suffix(self):
assert same_repo("https://github.com/Owner/Repo.git",
"https://github.com/owner/repo/")
assert not same_repo("https://github.com/owner/repo",
"https://github.com/owner/other")
class TestGithubOwnerRepo:
@pytest.mark.parametrize("url, expected", [
(PAGES_REPO + ".git", ("user", "my.github.io")),
("https://www.github.com/owner/repo", ("owner", "repo")),
("https://github.com/owner/repo/tree/main/plugins/x", ("owner", "repo")),
])
def test_github_urls(self, url, expected):
assert github_owner_repo(url) == expected
@pytest.mark.parametrize("url", [
"https://github.com.example.org/owner/repo",
"https://gitlab.com/owner/repo",
"https://github.com/owner",
"github.com/owner/repo",
])
def test_anything_else_is_not_a_github_repo(self, url):
assert github_owner_repo(url) is None
def test_headers_carry_the_token_only_when_given(self):
assert "Authorization" not in github_api_headers(None)
assert github_api_headers("abc")["Authorization"] == "token abc"
@pytest.fixture
def store(tmp_path):
return PluginStoreManager(
plugins_dir=str(tmp_path / "plugins"),
uninstalled_registry_path=str(tmp_path / "uninstalled.json"))
def test_install_from_url_keeps_an_interior_dot_git(store, monkeypatch):
cloned_from = []
monkeypatch.setattr(store, "_install_via_git",
lambda url, *a, **k: cloned_from.append(url))
downloaded = []
monkeypatch.setattr(store, "_install_via_download",
lambda url, *a, **k: downloaded.append(url) or False)
result = store.install_from_url(PAGES_REPO + ".git")
assert result["success"] is False
assert cloned_from == [PAGES_REPO]
assert all(url.startswith(PAGES_REPO + "/archive/") for url in downloaded)
def test_fetch_registry_from_url_asks_for_the_named_repository(store, monkeypatch):
requested = []
def fake_get(url, **kwargs):
requested.append(url)
return MagicMock(status_code=404)
monkeypatch.setattr(store, "_http_get_with_retries", fake_get)
assert store.fetch_registry_from_url(PAGES_REPO) is None
assert requested
assert all(url.startswith("https://raw.githubusercontent.com/user/my.github.io/")
for url in requested)
test/test_resource_monitor.py
+21
View File
@@ -93,6 +93,27 @@ class TestResourceLimits:
with pytest.raises(ResourceLimitExceeded): with pytest.raises(ResourceLimitExceeded):
mon.monitor_call("p", lambda: time.sleep(0.02)) mon.monitor_call("p", lambda: time.sleep(0.02))
def test_memory_limit_judges_each_call_on_its_own_growth(self):
"""One expensive call must not fail every call after it.
The check used to compare the stored high-water mark, which never
decreases, so after one call grew memory past the limit every later
call raised too and the plugin never updated again.
"""
mon = PluginResourceMonitor(_cache(), enable_monitoring=False)
mon.enable_monitoring = True # measure without needing psutil
readings = iter([100.0, 200.0, # first call grows RSS by 100 MB
200.0, 201.0]) # second call grows it by 1 MB
mon._get_process_memory_mb = lambda: next(readings)
mon._get_process_cpu_percent = lambda: 0.0
mon.set_limits("p", ResourceLimits(max_memory_mb=50))
with pytest.raises(ResourceLimitExceeded):
mon.monitor_call("p", lambda: None)
assert mon.monitor_call("p", lambda: "ok") == "ok"
# The high-water mark is still reported.
assert mon.get_metrics("p").memory_mb == 100.0
def test_reset_metrics_clears_counts(self): def test_reset_metrics_clears_counts(self):
cache = _cache() cache = _cache()
mon = PluginResourceMonitor(cache, enable_monitoring=False) mon = PluginResourceMonitor(cache, enable_monitoring=False)
test/test_saved_repositories.py
+1 -1
View File
@@ -5,7 +5,7 @@ SavedRepositoriesManager contract.
Covers: the three accepted on-disk load shapes (bare list, wrapped Covers: the three accepted on-disk load shapes (bare list, wrapped
{"repositories": [...]}, anything else -> []) and that saves always write {"repositories": [...]}, anything else -> []) and that saves always write
the bare-list form; add/remove/has round trips through a fresh manager; the bare-list form; add/remove/has round trips through a fresh manager;
URL normalization post-fix (_clean_url strips only a TRAILING '.git' after URL normalization post-fix (normalize_repo_url strips only a TRAILING '.git' after
trailing slashes — the old unanchored .replace('.git', '') mangled URLs trailing slashes — the old unanchored .replace('.git', '') mangled URLs
like my.github.io); name derivation and registry-vs-single type like my.github.io); name derivation and registry-vs-single type
classification (the ledmatrix-plugins check is lowercased, the classification (the ledmatrix-plugins check is lowercased, the
test/test_store_install_default_branch.py
+77
View File
@@ -0,0 +1,77 @@
"""A repository whose only branch is neither main nor master still installs.
_install_via_git tries the candidate branches, then the repository's default
branch -- but it returned None both for "every clone failed" and for "the
default-branch clone succeeded". install_from_url took the None as failure,
fell through to the archive download of main/master (which does not exist),
and reported "Failed to clone or download repository" for a repository it had
just cloned.
"""
import json
import shutil
import subprocess
import pytest
from src.plugin_system.store_manager import PluginStoreManager
pytestmark = pytest.mark.skipif(shutil.which("git") is None, reason="git not installed")
MANIFEST = {
"id": "develop-only", "name": "Develop Only", "class_name": "P",
"display_modes": ["develop_only"], "version": "1.0.0",
}
def _git(*args, cwd):
subprocess.run(
["git", "-c", "user.name=t", "-c", "user.email=t@example.invalid", *args],
cwd=cwd, check=True, capture_output=True)
@pytest.fixture
def develop_only_repo(tmp_path):
repo = tmp_path / "upstream"
repo.mkdir()
_git("init", "-q", "-b", "develop", cwd=repo)
(repo / "manifest.json").write_text(json.dumps(MANIFEST))
(repo / "manager.py").write_text("class P: pass\n")
_git("add", ".", cwd=repo)
_git("commit", "-q", "-m", "init", cwd=repo)
return repo.as_uri()
@pytest.fixture
def store(tmp_path, monkeypatch):
mgr = PluginStoreManager(
plugins_dir=str(tmp_path / "plugins"),
uninstalled_registry_path=str(tmp_path / "uninstalled.json"))
monkeypatch.setattr(mgr, "_install_dependencies", lambda *a, **k: True)
downloads = []
monkeypatch.setattr(mgr, "_install_via_download",
lambda url, *a, **k: downloads.append(url) or False)
mgr.downloads = downloads
return mgr
def test_a_default_branch_clone_reports_its_branch(store, develop_only_repo, tmp_path):
target = tmp_path / "clone"
assert store._install_via_git(develop_only_repo, target, ["main", "master"]) == "develop"
assert (target / "manifest.json").exists()
def test_a_failed_clone_reports_none(store, tmp_path):
missing = (tmp_path / "no-such-repo").as_uri()
target = tmp_path / "clone"
assert store._install_via_git(missing, target, ["main"]) is None
assert not target.exists()
def test_install_from_url_installs_a_develop_only_repository(store, develop_only_repo):
result = store.install_from_url(develop_only_repo)
assert result == {"success": True, "plugin_id": "develop-only",
"name": "Develop Only", "branch": "develop"}
assert (store.plugins_dir / "develop-only" / "manifest.json").exists()
assert store.downloads == []
test/test_store_update_non_git_remote.py
+62
View File
@@ -0,0 +1,62 @@
"""update_plugin must not borrow the enclosing LEDMatrix checkout's remote.
Plugins live in ``plugin-repos/`` inside the LEDMatrix git checkout. For a
plugin installed from a ZIP (no ``.git`` of its own), ``git -C <plugin>``
walks up to the LEDMatrix repository, and ``git config --local --get
remote.origin.url`` answers with LEDMatrix's own URL. update_plugin then tried
to "reinstall" the plugin from the LEDMatrix repository.
"""
import json
import shutil
import subprocess
import pytest
from src.plugin_system.store_manager import PluginStoreManager
pytestmark = pytest.mark.skipif(shutil.which("git") is None, reason="git not installed")
PLUGIN_ID = "zip-installed"
PARENT_REMOTE = "https://github.com/example/LEDMatrix"
def _git(*args, cwd):
subprocess.run(["git", *args], cwd=cwd, check=True, capture_output=True)
@pytest.fixture
def store_inside_checkout(tmp_path):
checkout = tmp_path / "LEDMatrix"
checkout.mkdir()
_git("init", "-q", cwd=checkout)
_git("remote", "add", "origin", PARENT_REMOTE, cwd=checkout)
plugins_dir = checkout / "plugin-repos"
plugin_dir = plugins_dir / PLUGIN_ID
plugin_dir.mkdir(parents=True)
(plugin_dir / "manifest.json").write_text(json.dumps(
{"id": PLUGIN_ID, "name": "Zip", "version": "1.0.0"}))
store = PluginStoreManager(
plugins_dir=str(plugins_dir),
uninstalled_registry_path=str(tmp_path / "uninstalled.json"))
return store, plugin_dir
def test_a_plugin_without_its_own_git_has_no_remote(store_inside_checkout, monkeypatch):
store, plugin_dir = store_inside_checkout
# The premise: git itself does report the parent's remote here.
parent_view = subprocess.run(
["git", "-C", str(plugin_dir), "config", "--local", "--get", "remote.origin.url"],
capture_output=True, text=True)
assert parent_view.stdout.strip() == PARENT_REMOTE
monkeypatch.setattr(store, "fetch_registry", lambda *a, **k: {"plugins": []})
monkeypatch.setattr(store, "get_plugin_info", lambda *a, **k: None)
install_calls = []
monkeypatch.setattr(store, "install_from_url",
lambda *a, **k: install_calls.append((a, k)) or {"success": True})
assert store.update_plugin(PLUGIN_ID) is False
assert install_calls == []
test/web_interface/integration/test_plugin_operations.py
+1 -4
View File
@@ -21,10 +21,7 @@ class TestPluginOperationsIntegration(unittest.TestCase):
self.temp_dir = Path(tempfile.mkdtemp()) self.temp_dir = Path(tempfile.mkdtemp())
# Initialize components # Initialize components
self.operation_queue = PluginOperationQueue( self.operation_queue = PluginOperationQueue(max_history=100)
history_file=str(self.temp_dir / "operations.json"),
max_history=100
)
self.state_manager = PluginStateManager( self.state_manager = PluginStateManager(
state_file=str(self.temp_dir / "state.json"), state_file=str(self.temp_dir / "state.json"),
web_interface/app.py
+1 -6
View File
@@ -144,12 +144,7 @@ schema_manager = SchemaManager(
) )
# Initialize operation queue for plugin operations # Initialize operation queue for plugin operations
# Use lazy_load=True to defer file loading until first use (improves startup time) operation_queue = PluginOperationQueue(max_history=500)
operation_queue = PluginOperationQueue(
history_file=str(project_root / "data" / "plugin_operations.json"),
max_history=500,
lazy_load=True
)
# Initialize plugin state manager # Initialize plugin state manager
# Use lazy_load=True to defer file loading until first use (improves startup time) # Use lazy_load=True to defer file loading until first use (improves startup time)