mirror of
https://github.com/ChuckBuilds/LEDMatrix.git
synced 2026-10-04 06:15:09 +00:00
fix(plugins): store and plugin-manager bugs; tidy src/plugin_system (#635)
* fix(store): don't read a ZIP-installed plugin's remote from the LEDMatrix repo update_plugin looked up remote.origin.url with `git -C <plugin> config --local` for plugins that are not git checkouts. Under plugin-repos/ git walks up to the enclosing LEDMatrix repository, so the lookup returned LEDMatrix's own URL and a plugin missing from the registry was "reinstalled" from the LEDMatrix repo. Only ask git when the plugin directory has its own .git, the test _get_local_git_info already uses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(schema): report each missing required field once, by name validate_config_against_schema ran its own required-fields loop after Draft7Validator.iter_errors, which already yields one `required` error per missing field, so every missing top-level field was listed twice. The validator's copy also printed the schema's whole `required` list ("Missing required property '['api_key', 'city']'") instead of the field. Drop the loop and take the field name from the error itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(store): stop mangling repository URLs that contain ".git" install_from_url and fetch_registry_from_url cleaned URLs with `rstrip('/').replace('.git', '')`, which removes ".git" anywhere: https://github.com/user/my.github.io became .../myhub.io, so installing or browsing that repository asked GitHub for one that does not exist. Add src/plugin_system/repo_urls.py with one anchored normalize_repo_url(), same_repo() for comparisons, github_owner_repo() and github_api_headers(), and use them for the five copies of the owner/repo parsing and GitHub headers in the store and for saved repositories. GitHub URLs are now recognised by urlparse().hostname everywhere: _get_latest_commit_info used a substring test, and _install_from_monorepo_api parsed any host. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(store): install a repository whose only branch is not main/master _install_via_git returned None both when every clone failed and when the last-resort clone of the repository's default branch succeeded. _install_plugin_impl papered over it with `and not plugin_path.exists()`; install_from_url did not, so a repository whose only branch is e.g. `develop` was cloned, then treated as a failure, then "downloaded" from main/master archives that do not exist. After a default-branch clone, return the branch the clone checked out (read from .git/HEAD), so None means failure and nothing else, and give both callers the same `branch_used is None` fallback. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(plugins): judge the memory limit on each call's own growth monitor_call stores `metrics.memory_mb = max(previous, growth)`, and _check_limits compared that high-water mark with max_memory_mb. It never decreases, so once one update() grew the process past the limit every later call raised ResourceLimitExceeded and the circuit breaker kept reopening. Pass the call's own RSS growth to _check_limits; keep the high-water mark for reporting and document what it measures. Remove ResourceMetrics.update_average_execution_time: nothing called it, and it overwrote the running total with the average. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(plugins): reload_plugin re-reads the manifest from the discovered directory reload_plugin read `plugins_dir / plugin_id / "manifest.json"`, ignoring the discovery map and the plugin_dirs rules. For a plugin whose directory name differs from its manifest id the path did not exist, the re-read was skipped without a word, and the reload kept the stale manifest. Resolve the directory with find_plugin_directory, as load_plugin does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(plugins): drop the always-null last_display from plugin state info PluginStateManager reported `last_display` from `_last_display`, which nothing ever wrote, so it was null for every plugin. Recording it in PluginExecutor.execute_display would not help: get_state_info's only reader is the web process, whose PluginManager never calls display(). Remove the field, its dict and get_last_display() (no caller in core, the web UI or the plugin monorepo). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(store): share the rollback and requirements helpers, drop dead code - install_plugin and _reinstall_with_rollback set aside, discard and restore the old copy through _set_aside/_discard_backup/_restore_backup instead of two copies of the same blocks. - The loader and the store run the same pre-pip checks through contained_plugin_dir() and requirements_to_install() in plugin_loader. They still invoke pip differently (sys.executable -m pip vs. the sudo wrapper). `except (BrokenPipeError, OSError)` + `isinstance(e, OSError)` becomes `except OSError` checking errno.EPIPE. - load_module never returns None, so load_plugin's check is gone and the docstring says what it raises. - Remove the always-true JSONSCHEMA_AVAILABLE, the inline re-imports of re and permission_utils, the fake status_result object nobody reads, hasattr(git_error, 'cmd'), a redundant "merge conflict" test and `import traceback` (exc_info=True does it). - Correct comments: install_from_url names the directory for the caller's id when given (not always the manifest id), _get_local_git_info saves one git subprocess (not four), _enrich calls two helpers, search_plugins documents all its arguments, _find_plugin_path states its behaviour instead of a TODO, and history narration is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(plugins): tidy base_plugin, correct plugin_manager/state comments - base_plugin: drop the unused `import logging`; get_display_duration runs the instance value and the config value through one _positive_seconds() helper instead of two copies of the coercion; the 'static'/'none'/fallback branches of get_vegas_display_mode, which all returned FIXED_SEGMENT, are one; fix the mis-indented validate_config example; say that get_supported_vegas_modes/get_vegas_segment_width are not consulted by core (kept, plugins override them). - schema_manager: import expand_style_elements normally rather than swallowing an ImportError of a core module. - plugin_manager: the plugins directory is the configured one (plugin-repos/ by default), not plugins/; get_config() returns the live dict, not a copy, so the interval cache comments say what it saves. - state_manager: config_version and the file version are not used to detect corruption; say what they are. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(plugins): stop writing data/plugin_operations.json PluginOperationQueue wrote its finished-operation history to data/plugin_operations.json after every operation, and read it back only into its own in-memory list, which only get_operation_history() exposes -- and nothing calls that. The operation-history endpoint reads OperationHistory (data/operation_history.json). No code in src/, web_interface/, scripts/ or test/ reads the file. Drop the history_file/lazy_load parameters and the load/save code; the bounded in-memory history stays. web_interface/app.py and the integration test stop passing the removed arguments. An existing data/plugin_operations.json is left in place (data/* is gitignored). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(changelog): plugin-system Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Chuck
+4
-4
@@ -81,10 +81,10 @@ assets/stocks/crypto_icons/
|
||||
|
||||
# Plugin operation state written at runtime.
|
||||
#
|
||||
# web_interface/app.py writes data/plugin_operations.json, data/plugin_state.json
|
||||
# and data/operation_history.json as the web interface runs, into a directory that
|
||||
# ships tracked (data/.gitkeep) and was otherwise unignored. So every rig that ever
|
||||
# opened the web UI -- and every test run that constructs the app -- left three
|
||||
# web_interface/app.py writes data/plugin_state.json and data/operation_history.json
|
||||
# (older releases also data/plugin_operations.json) as the web interface runs, into
|
||||
# a directory that ships tracked (data/.gitkeep). Unignored, every rig that ever
|
||||
# opened the web UI -- and every test run that constructs the app -- would leave
|
||||
# untracked files behind and a permanently dirty `git status`. Same reasoning as
|
||||
# the logo rule above: a checkout that is always dirty is a checkout nobody reads.
|
||||
data/*
|
||||
|
||||
@@ -33,6 +33,15 @@ accepts both, but the store flags the old spelling as deprecated
|
||||
- `src/common/README.md` covers every module.
|
||||
- Stale setup, service and troubleshooting claims are corrected.
|
||||
|
||||
- Plugin store and plugin manager fixes:
|
||||
- Updating a plugin that was installed from a ZIP no longer tries to reinstall it from the LEDMatrix repository's own URL.
|
||||
- Repository URLs with `.git` in the middle are no longer mangled. The URL helpers now live in `src/plugin_system/repo_urls.py`.
|
||||
- Installing from a URL works when the repository's only branch isn't `main` or `master`.
|
||||
- A missing required config field is reported once, by name.
|
||||
- A plugin that went over `max_memory_mb` once is no longer refused on every call after that.
|
||||
- `reload_plugin` reads the manifest from the plugin's discovered directory.
|
||||
- Removed: `last_display` from plugin state info and `get_last_display()` (nothing recorded them); `PluginOperationQueue`'s `history_file` and `lazy_load` arguments; and `data/plugin_operations.json`, which nothing read.
|
||||
|
||||
- The web service (`ledmatrix-web`) logs through `src.logging_config` like the
|
||||
display service, so `journalctl -p err -u ledmatrix-web` works. Successful
|
||||
GET/HEAD/OPTIONS requests (the UI's polling) are logged at DEBUG instead of
|
||||
|
||||
@@ -11,7 +11,6 @@ Stability: Stable - maintains backward compatibility
|
||||
from abc import ABC, abstractmethod
|
||||
from enum import Enum
|
||||
from typing import Dict, Any, Optional, List
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
from src.logging_config import get_logger
|
||||
@@ -511,104 +510,53 @@ class BasePlugin(ABC):
|
||||
"""
|
||||
Get the display duration for this plugin instance.
|
||||
|
||||
Automatically detects duration from:
|
||||
1. self.display_duration instance variable (if exists)
|
||||
2. self.config.get("display_duration", 15.0) (fallback)
|
||||
Uses, in order, the first positive number among:
|
||||
1. ``self.display_duration`` (a common pattern in scoreboard plugins)
|
||||
2. ``self.config["display_duration"]``
|
||||
3. 15.0
|
||||
|
||||
Can be overridden by plugins to provide dynamic durations based
|
||||
on content (e.g., longer duration for more complex displays).
|
||||
Numeric strings count as numbers. Can be overridden by plugins to
|
||||
provide dynamic durations based on content (e.g., longer duration for
|
||||
more complex displays).
|
||||
|
||||
Returns:
|
||||
Duration in seconds to display this plugin's content
|
||||
"""
|
||||
# Check for instance variable first (common pattern in scoreboard plugins)
|
||||
if hasattr(self, 'display_duration'):
|
||||
try:
|
||||
duration = getattr(self, 'display_duration')
|
||||
# Handle None case
|
||||
if duration is None:
|
||||
pass # Fall through to config
|
||||
# Try to convert to float if it's a number or numeric string.
|
||||
# bool is excluded: it's an int subclass, and True would
|
||||
# otherwise read as a 1-second duration.
|
||||
elif isinstance(duration, (int, float)) and not isinstance(duration, bool):
|
||||
if duration > 0:
|
||||
return float(duration)
|
||||
else:
|
||||
self.logger.debug(
|
||||
"display_duration instance variable is non-positive (%s), using config fallback",
|
||||
duration
|
||||
)
|
||||
# Try converting string representations of numbers
|
||||
elif isinstance(duration, str):
|
||||
try:
|
||||
duration_float = float(duration)
|
||||
if duration_float > 0:
|
||||
return duration_float
|
||||
else:
|
||||
self.logger.debug(
|
||||
"display_duration string value is non-positive (%s), using config fallback",
|
||||
duration
|
||||
)
|
||||
except (ValueError, TypeError):
|
||||
self.logger.warning(
|
||||
"display_duration instance variable has invalid string value '%s', using config fallback",
|
||||
duration
|
||||
)
|
||||
else:
|
||||
self.logger.warning(
|
||||
"display_duration instance variable has unexpected type %s (value: %s), using config fallback",
|
||||
type(duration).__name__, duration
|
||||
)
|
||||
except (TypeError, ValueError, AttributeError) as e:
|
||||
self.logger.warning(
|
||||
"Error reading display_duration instance variable: %s, using config fallback",
|
||||
e
|
||||
)
|
||||
|
||||
# Fall back to config
|
||||
config_duration = self.config.get("display_duration", 15.0)
|
||||
try:
|
||||
# Ensure config value is also a valid float (bool excluded — an
|
||||
# int subclass that would otherwise read True as 1 second)
|
||||
if isinstance(config_duration, (int, float)) and not isinstance(config_duration, bool):
|
||||
if config_duration > 0:
|
||||
return float(config_duration)
|
||||
else:
|
||||
self.logger.debug(
|
||||
"Config display_duration is non-positive (%s), using default 15.0",
|
||||
config_duration
|
||||
)
|
||||
return 15.0
|
||||
elif isinstance(config_duration, str):
|
||||
try:
|
||||
duration_float = float(config_duration)
|
||||
if duration_float > 0:
|
||||
return duration_float
|
||||
else:
|
||||
self.logger.debug(
|
||||
"Config display_duration string is non-positive (%s), using default 15.0",
|
||||
config_duration
|
||||
)
|
||||
return 15.0
|
||||
except ValueError:
|
||||
self.logger.warning(
|
||||
"Config display_duration has invalid string value '%s', using default 15.0",
|
||||
config_duration
|
||||
)
|
||||
return 15.0
|
||||
else:
|
||||
self.logger.warning(
|
||||
"Config display_duration has unexpected type %s (value: %s), using default 15.0",
|
||||
type(config_duration).__name__, config_duration
|
||||
)
|
||||
except (ValueError, TypeError) as e:
|
||||
duration = getattr(self, 'display_duration', None)
|
||||
except (TypeError, ValueError, AttributeError) as e:
|
||||
# A plugin may define display_duration as a property that raises.
|
||||
self.logger.warning(
|
||||
"Error processing config display_duration: %s, using default 15.0",
|
||||
e
|
||||
)
|
||||
"Error reading display_duration instance variable: %s, using config fallback", e)
|
||||
duration = None
|
||||
if duration is not None:
|
||||
seconds = self._positive_seconds(duration, "display_duration instance variable")
|
||||
if seconds is not None:
|
||||
return seconds
|
||||
|
||||
return 15.0
|
||||
seconds = self._positive_seconds(
|
||||
self.config.get("display_duration", 15.0), "config display_duration")
|
||||
return seconds if seconds is not None else 15.0
|
||||
|
||||
def _positive_seconds(self, value: Any, source: str) -> Optional[float]:
|
||||
"""``value`` as a positive float, or None (with a log line) if it is not one.
|
||||
|
||||
bool is rejected although it is an int subclass: True would otherwise
|
||||
read as a 1-second duration.
|
||||
"""
|
||||
if isinstance(value, bool) or not isinstance(value, (int, float, str)):
|
||||
self.logger.warning("%s has unexpected type %s (value: %s), ignoring it",
|
||||
source, type(value).__name__, value)
|
||||
return None
|
||||
try:
|
||||
seconds = float(value)
|
||||
except ValueError:
|
||||
self.logger.warning("%s has invalid value %r, ignoring it", source, value)
|
||||
return None
|
||||
if seconds > 0:
|
||||
return seconds
|
||||
self.logger.debug("%s is non-positive (%s), ignoring it", source, value)
|
||||
return None
|
||||
|
||||
# ---------------------------------------------------------------------
|
||||
# Dynamic duration support hooks
|
||||
@@ -926,25 +874,20 @@ class BasePlugin(ABC):
|
||||
config_mode, self.plugin_id
|
||||
)
|
||||
|
||||
# Fall back to mapping legacy content_type
|
||||
content_type = self.get_vegas_content_type()
|
||||
if content_type == 'multi':
|
||||
# Fall back to mapping legacy content_type. 'none' (excluded from
|
||||
# Vegas) also maps to FIXED_SEGMENT: exclusion is decided by checking
|
||||
# get_vegas_content_type() separately.
|
||||
if self.get_vegas_content_type() == 'multi':
|
||||
return VegasDisplayMode.SCROLL
|
||||
elif content_type == 'static':
|
||||
return VegasDisplayMode.FIXED_SEGMENT
|
||||
elif content_type == 'none':
|
||||
# 'none' means excluded - return FIXED_SEGMENT as default
|
||||
# The exclusion is handled by checking get_vegas_content_type() separately
|
||||
return VegasDisplayMode.FIXED_SEGMENT
|
||||
|
||||
return VegasDisplayMode.FIXED_SEGMENT
|
||||
|
||||
def get_supported_vegas_modes(self) -> List[VegasDisplayMode]:
|
||||
"""
|
||||
Return list of Vegas display modes this plugin supports.
|
||||
|
||||
Used by the web UI to show available mode options for user configuration.
|
||||
Override to customize which modes are available for this plugin.
|
||||
Not currently consulted by core: neither Vegas mode nor the web UI
|
||||
calls it. It is kept, and plugins override it, as the declared set of
|
||||
modes a future mode picker would offer.
|
||||
|
||||
By default:
|
||||
- 'multi' content type plugins support SCROLL and FIXED_SEGMENT
|
||||
@@ -972,6 +915,10 @@ class BasePlugin(ABC):
|
||||
"""
|
||||
Get the preferred width for this plugin in Vegas FIXED_SEGMENT mode.
|
||||
|
||||
Not currently consulted by core: Vegas mode sizes a card from the
|
||||
``vegas_width_pct`` / ``vegas_scroll.render_width_pct`` settings
|
||||
(see get_vegas_render_width()). Kept because plugins override it.
|
||||
|
||||
Returns the number of panels this plugin should occupy when displayed
|
||||
as a fixed segment. The actual pixel width is calculated as:
|
||||
width = panels * single_panel_width
|
||||
@@ -1025,7 +972,7 @@ class BasePlugin(ABC):
|
||||
required_fields = ['api_key', 'city']
|
||||
for field in required_fields:
|
||||
if field not in self.config:
|
||||
self.logger.error("Missing required field: %s", field)
|
||||
self.logger.error("Missing required field: %s", field)
|
||||
return False
|
||||
return True
|
||||
"""
|
||||
|
||||
@@ -9,8 +9,6 @@ import threading
|
||||
import queue
|
||||
from typing import Dict, Optional, List, Callable, Any
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
import json
|
||||
|
||||
from src.plugin_system.operation_types import (
|
||||
PluginOperation, OperationType, OperationStatus
|
||||
@@ -28,28 +26,22 @@ class PluginOperationQueue:
|
||||
- Prevents concurrent operations on same plugin
|
||||
- Operation status tracking
|
||||
- Operation cancellation
|
||||
- Operation history
|
||||
- In-memory history of finished operations
|
||||
|
||||
The history is not persisted. The web UI's operation history comes from
|
||||
OperationHistory (operation_history.py), which has its own file; a copy
|
||||
written here was never read back by anything.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
history_file: Optional[str] = None,
|
||||
max_history: int = 100,
|
||||
lazy_load: bool = False
|
||||
):
|
||||
def __init__(self, max_history: int = 100):
|
||||
"""
|
||||
Initialize operation queue.
|
||||
|
||||
Args:
|
||||
history_file: Optional path to file for persisting operation history
|
||||
max_history: Maximum number of operations to keep in history
|
||||
lazy_load: If True, defer loading history file until first access
|
||||
"""
|
||||
self.logger = get_logger(__name__)
|
||||
self.history_file = Path(history_file) if history_file else None
|
||||
self.max_history = max_history
|
||||
self._lazy_load = lazy_load
|
||||
self._history_loaded = False
|
||||
|
||||
# Operation tracking
|
||||
self._operations: Dict[str, PluginOperation] = {}
|
||||
@@ -62,20 +54,8 @@ class PluginOperationQueue:
|
||||
self._worker_thread: Optional[threading.Thread] = None
|
||||
self._stop_event = threading.Event()
|
||||
|
||||
# Load history from file if it exists (unless lazy loading)
|
||||
if not self._lazy_load and self.history_file and self.history_file.exists():
|
||||
self._load_history()
|
||||
self._history_loaded = True
|
||||
|
||||
# Start worker thread
|
||||
self._start_worker()
|
||||
|
||||
def _ensure_loaded(self) -> None:
|
||||
"""Ensure history is loaded (for lazy loading)."""
|
||||
if not self._history_loaded and self.history_file and self.history_file.exists():
|
||||
self._load_history()
|
||||
self._history_loaded = True
|
||||
|
||||
def enqueue_operation(
|
||||
self,
|
||||
operation_type: OperationType,
|
||||
@@ -139,7 +119,6 @@ class PluginOperationQueue:
|
||||
Returns:
|
||||
PluginOperation if found, None otherwise
|
||||
"""
|
||||
self._ensure_loaded()
|
||||
with self._lock:
|
||||
return self._operations.get(operation_id)
|
||||
|
||||
@@ -184,7 +163,6 @@ class PluginOperationQueue:
|
||||
Returns:
|
||||
List of operations, sorted by creation time (newest first)
|
||||
"""
|
||||
self._ensure_loaded()
|
||||
with self._lock:
|
||||
# Sort by creation time (newest first)
|
||||
history = sorted(
|
||||
@@ -314,12 +292,8 @@ class PluginOperationQueue:
|
||||
if self._active_operations[operation.plugin_id].operation_id == operation.operation_id:
|
||||
del self._active_operations[operation.plugin_id]
|
||||
|
||||
# Add to history
|
||||
self._add_to_history(operation)
|
||||
|
||||
# Save history to file
|
||||
self._save_history()
|
||||
|
||||
def _add_to_history(self, operation: PluginOperation) -> None:
|
||||
"""Add operation to history, maintaining max_history limit."""
|
||||
self._operation_history.append(operation)
|
||||
@@ -330,46 +304,6 @@ class PluginOperationQueue:
|
||||
self._operation_history.sort(key=lambda op: op.created_at)
|
||||
self._operation_history = self._operation_history[-self.max_history:]
|
||||
|
||||
def _save_history(self) -> None:
|
||||
"""Save operation history to file."""
|
||||
if not self.history_file:
|
||||
return
|
||||
|
||||
try:
|
||||
with self._lock:
|
||||
# Convert operations to dicts
|
||||
history_data = [op.to_dict() for op in self._operation_history]
|
||||
|
||||
# Ensure directory exists
|
||||
self.history_file.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Write to file
|
||||
with open(self.history_file, 'w') as f:
|
||||
json.dump(history_data, f, indent=2)
|
||||
|
||||
except Exception as e:
|
||||
self.logger.warning(f"Error saving operation history: {e}")
|
||||
|
||||
def _load_history(self) -> None:
|
||||
"""Load operation history from file."""
|
||||
if not self.history_file or not self.history_file.exists():
|
||||
return
|
||||
|
||||
try:
|
||||
with open(self.history_file, 'r') as f:
|
||||
history_data = json.load(f)
|
||||
|
||||
with self._lock:
|
||||
self._operation_history = [
|
||||
PluginOperation.from_dict(op_data)
|
||||
for op_data in history_data
|
||||
]
|
||||
|
||||
self.logger.info(f"Loaded {len(self._operation_history)} operations from history")
|
||||
|
||||
except Exception as e:
|
||||
self.logger.warning(f"Error loading operation history: {e}")
|
||||
|
||||
def shutdown(self) -> None:
|
||||
"""Shutdown the operation queue and worker thread."""
|
||||
self.logger.info("Shutting down plugin operation queue")
|
||||
@@ -378,6 +312,3 @@ class PluginOperationQueue:
|
||||
if self._worker_thread and self._worker_thread.is_alive():
|
||||
self._worker_thread.join(timeout=5.0)
|
||||
|
||||
# Save history one last time
|
||||
self._save_history()
|
||||
|
||||
|
||||
@@ -5,6 +5,7 @@ Handles plugin module imports, dependency installation, and class instantiation.
|
||||
Extracted from PluginManager to improve separation of concerns.
|
||||
"""
|
||||
|
||||
import errno
|
||||
import importlib
|
||||
import importlib.metadata
|
||||
import importlib.util
|
||||
@@ -184,6 +185,46 @@ def find_trusted_subdir(trusted_dir: str, name: str) -> Optional[str]:
|
||||
return None
|
||||
|
||||
|
||||
def contained_plugin_dir(plugin_dir: Path, plugins_dir: Path) -> Optional[str]:
|
||||
"""``plugin_dir`` rebuilt from an entry enumerated under ``plugins_dir``.
|
||||
|
||||
Returns None when ``plugin_dir`` is not a subdirectory of ``plugins_dir``.
|
||||
Callers derive ``plugin_dir`` from a manifest-declared id, so the path is
|
||||
rebuilt from :func:`find_trusted_subdir`'s answer rather than trusted: a
|
||||
name that came out of ``os.scandir()`` on the trusted root carries no
|
||||
taint, which is a real containment guarantee (and one CodeQL's
|
||||
path-injection query can follow), not a string sanitiser.
|
||||
"""
|
||||
plugin_dir_real = os.path.realpath(str(plugin_dir))
|
||||
plugins_dir_real = os.path.realpath(str(plugins_dir))
|
||||
matched_name = find_trusted_subdir(plugins_dir_real, os.path.basename(plugin_dir_real))
|
||||
if matched_name is None:
|
||||
return None
|
||||
return os.path.join(plugins_dir_real, matched_name)
|
||||
|
||||
|
||||
def requirements_to_install(plugin_dir: str, logger: logging.Logger,
|
||||
label: str) -> Optional[str]:
|
||||
"""The plugin's requirements.txt if pip has work to do, else None.
|
||||
|
||||
None when there is no requirements.txt, when it lists nothing (plugins
|
||||
whose dependencies ship with core often keep an all-comments file), or
|
||||
when every requirement is already installed. Shared by the loader and the
|
||||
store so both skip pip for the same reasons; they differ only in how they
|
||||
run it.
|
||||
"""
|
||||
requirements_file = os.path.join(plugin_dir, "requirements.txt")
|
||||
if not os.path.isfile(requirements_file):
|
||||
return None
|
||||
if not requirements_has_real_deps(requirements_file):
|
||||
logger.debug("requirements.txt for %s has no real dependencies, skipping pip", label)
|
||||
return None
|
||||
if requirements_are_satisfied(requirements_file):
|
||||
logger.debug("Dependencies for %s already satisfied, skipping pip", label)
|
||||
return None
|
||||
return requirements_file
|
||||
|
||||
|
||||
class PluginLoader:
|
||||
"""Handles plugin module loading and class instantiation."""
|
||||
|
||||
@@ -273,43 +314,15 @@ class PluginLoader:
|
||||
if not plugin_id:
|
||||
return False
|
||||
|
||||
# Resolve to a canonical absolute path (normalises .. and symlinks)
|
||||
plugin_dir_real = os.path.realpath(str(plugin_dir))
|
||||
plugins_dir_real = os.path.realpath(str(plugins_dir))
|
||||
requested_name = os.path.basename(plugin_dir_real)
|
||||
|
||||
# Match the requested directory against an entry actually enumerated
|
||||
# from the trusted plugins_dir, and build the path from that entry --
|
||||
# not from requested_name. A name that came out of os.scandir() on a
|
||||
# trusted root carries no taint regardless of what the caller asked
|
||||
# for, so this is a real containment guarantee (an allowlist check
|
||||
# against a trusted source), not a string-sanitisation of untrusted
|
||||
# input that a static analyzer has to trust blindly.
|
||||
matched_name = find_trusted_subdir(plugins_dir_real, requested_name)
|
||||
if matched_name is None:
|
||||
safe_plugin_dir = contained_plugin_dir(plugin_dir, plugins_dir)
|
||||
if safe_plugin_dir is None:
|
||||
self.logger.error(
|
||||
"Plugin directory for %s not found inside plugins dir", plugin_id
|
||||
)
|
||||
return False
|
||||
|
||||
safe_plugin_dir = os.path.join(plugins_dir_real, matched_name)
|
||||
requirements_file = os.path.join(safe_plugin_dir, "requirements.txt")
|
||||
|
||||
if not os.path.isfile(requirements_file):
|
||||
return True # No dependencies needed
|
||||
|
||||
if not requirements_has_real_deps(requirements_file):
|
||||
self.logger.debug(
|
||||
"requirements.txt for %s has no real dependencies (comments/blank only), skipping pip",
|
||||
plugin_id
|
||||
)
|
||||
return True
|
||||
|
||||
if requirements_are_satisfied(requirements_file):
|
||||
self.logger.debug(
|
||||
"Dependencies for %s already satisfied in current environment, skipping pip",
|
||||
plugin_id
|
||||
)
|
||||
requirements_file = requirements_to_install(safe_plugin_dir, self.logger, plugin_id)
|
||||
if requirements_file is None:
|
||||
return True
|
||||
|
||||
try:
|
||||
@@ -348,8 +361,8 @@ class PluginLoader:
|
||||
# below).
|
||||
try:
|
||||
# sys.executable is this process's own interpreter (not
|
||||
# attacker-influenced), and requirements_file is a path
|
||||
# built internally by find_plugin_directory, never raw
|
||||
# attacker-influenced), and requirements_file is rebuilt
|
||||
# by contained_plugin_dir() from a trusted listing, never raw
|
||||
# external input.
|
||||
retry_result = subprocess.run( # nosec B603 - no shell invoked (list-form argv) # nosemgrep
|
||||
[sys.executable, "-m", "pip", "install", "--break-system-packages",
|
||||
@@ -384,10 +397,10 @@ class PluginLoader:
|
||||
except FileNotFoundError:
|
||||
self.logger.warning("pip not found. Skipping dependency installation for %s", plugin_id)
|
||||
return True
|
||||
except (BrokenPipeError, OSError) as e:
|
||||
# Handle broken pipe errors (errno 32) which can occur during pip downloads
|
||||
# Often caused by network interruptions or output buffer issues
|
||||
if isinstance(e, OSError) and e.errno == 32:
|
||||
except OSError as e:
|
||||
# A broken pipe (EPIPE) happens when pip's output pipe closes
|
||||
# mid-download, usually a network interruption.
|
||||
if e.errno == errno.EPIPE:
|
||||
self.logger.error(
|
||||
"Broken pipe error during dependency installation for %s. "
|
||||
"This usually indicates a network interruption or pip output buffer issue. "
|
||||
@@ -528,7 +541,7 @@ class PluginLoader:
|
||||
plugin_id: str,
|
||||
plugin_dir: Path,
|
||||
entry_point: str
|
||||
) -> Optional[Any]:
|
||||
) -> Any:
|
||||
"""
|
||||
Load a plugin module from file.
|
||||
|
||||
@@ -547,7 +560,12 @@ class PluginLoader:
|
||||
entry_point: Entry point filename (e.g., 'manager.py')
|
||||
|
||||
Returns:
|
||||
Loaded module or None on error
|
||||
The loaded module
|
||||
|
||||
Raises:
|
||||
PluginError: If the plugin id, directory or entry point is
|
||||
invalid. Whatever the module raises while executing
|
||||
propagates unchanged.
|
||||
"""
|
||||
plugin_id = os.path.basename(plugin_id or '')
|
||||
if not plugin_id:
|
||||
@@ -782,8 +800,6 @@ class PluginLoader:
|
||||
# Load module
|
||||
entry_point = manifest.get('entry_point', 'manager.py')
|
||||
module = self.load_module(plugin_id, plugin_dir, entry_point)
|
||||
if module is None:
|
||||
raise PluginError(f"Failed to load module for plugin {plugin_id}", plugin_id=plugin_id)
|
||||
|
||||
# Get plugin class
|
||||
class_name = manifest.get('class_name')
|
||||
|
||||
@@ -2,7 +2,8 @@
|
||||
Plugin Manager
|
||||
|
||||
Manages plugin discovery, loading, and lifecycle for the LEDMatrix system.
|
||||
Handles dynamic plugin loading from the plugins/ directory.
|
||||
Loads plugins from the configured plugins directory
|
||||
(``plugin_system.plugins_directory``, ``plugin-repos/`` by default).
|
||||
|
||||
API Version: 1.0.0
|
||||
"""
|
||||
@@ -40,7 +41,7 @@ class PluginManager:
|
||||
Manages plugin discovery, loading, and lifecycle.
|
||||
|
||||
The PluginManager is responsible for:
|
||||
- Discovering plugins in the plugins/ directory
|
||||
- Discovering plugins in the configured plugins directory
|
||||
- Loading plugin modules and instantiating plugin classes
|
||||
- Managing plugin lifecycle (load, unload, reload)
|
||||
- Providing access to loaded plugins
|
||||
@@ -99,10 +100,9 @@ class PluginManager:
|
||||
self.plugin_directories: Dict[str, Path] = {}
|
||||
self.plugin_last_update: Dict[str, float] = {}
|
||||
|
||||
# Cached data-fetch intervals per plugin_id.
|
||||
# _get_plugin_update_interval falls back to config_manager.get_config()
|
||||
# (a full dict copy) when the manifest lacks an interval — caching avoids
|
||||
# that copy on every 30-fps tick. Cleared on load/unload.
|
||||
# Cached static data-fetch intervals per plugin_id, so the render
|
||||
# loop's scheduling tick does not repeat the manifest/config lookup
|
||||
# for every plugin. Cleared on load/unload.
|
||||
self._update_interval_cache: Dict[str, Optional[float]] = {}
|
||||
|
||||
# Health tracking (optional, set by display_controller if available)
|
||||
@@ -110,14 +110,12 @@ class PluginManager:
|
||||
self.resource_monitor = None
|
||||
|
||||
# --- Asynchronous plugin updates -------------------------------
|
||||
# update() used to run inline in the render loop (execute_update's
|
||||
# internal thread.join(timeout=30) blocked it), so one slow plugin
|
||||
# HTTP fetch froze scrolling for the whole fetch. Scheduling still
|
||||
# happens on the render thread (run_scheduled_updates), but
|
||||
# execution moves to this single background worker. Per-plugin
|
||||
# locks keep a plugin's update() and display() mutually exclusive —
|
||||
# today's implicit guarantee, now explicit (and, unlike today,
|
||||
# also held across the post-timeout window).
|
||||
# Run inline in the render loop, one slow plugin HTTP fetch in
|
||||
# update() freezes scrolling for the whole fetch. Scheduling happens
|
||||
# on the render thread (run_scheduled_updates); execution happens on
|
||||
# this single background worker. Per-plugin locks keep a plugin's
|
||||
# update() and display() mutually exclusive, including across the
|
||||
# post-timeout window.
|
||||
# Kill switch: plugin_system.synchronous_updates: true restores the
|
||||
# inline path.
|
||||
self._update_queue: "queue.Queue[Optional[Tuple[str, float]]]" = queue.Queue()
|
||||
@@ -661,9 +659,15 @@ class PluginManager:
|
||||
if not self.unload_plugin(plugin_id):
|
||||
return False
|
||||
|
||||
# Re-discover to get updated manifest
|
||||
manifest_path = self.plugins_dir / plugin_id / "manifest.json"
|
||||
if manifest_path.exists():
|
||||
# Re-read the manifest so an edit to it takes effect, from the
|
||||
# directory discovery found the plugin in: a directory's name need not
|
||||
# be the id its manifest declares.
|
||||
with self._discovery_lock:
|
||||
directories = dict(self.plugin_directories)
|
||||
plugin_dir = self.plugin_loader.find_plugin_directory(
|
||||
plugin_id, self.plugins_dir, directories)
|
||||
manifest_path = plugin_dir / "manifest.json" if plugin_dir is not None else None
|
||||
if manifest_path is not None and manifest_path.exists():
|
||||
try:
|
||||
with open(manifest_path, 'r', encoding='utf-8') as f:
|
||||
manifest = json.load(f)
|
||||
@@ -881,11 +885,12 @@ class PluginManager:
|
||||
updating, since a scheduler that propagates a plugin bug stops every
|
||||
other plugin too.
|
||||
|
||||
The static result is cached per plugin_id after the first lookup to
|
||||
avoid calling config_manager.get_config() — which returns a full dict
|
||||
copy — on every tick of the 30-fps display loop. The cache is
|
||||
invalidated when a plugin is loaded or unloaded. The dynamic hook is
|
||||
deliberately *not* cached: caching it would defeat its only purpose.
|
||||
The static result is cached per plugin_id after the first lookup, so
|
||||
the manifest/config resolution is not repeated on every scheduling
|
||||
tick of the display loop. A change to ``update_interval`` in
|
||||
config.json therefore takes effect when the plugin is next loaded or
|
||||
unloaded, which clears the cache. The dynamic hook is deliberately
|
||||
*not* cached: caching it would defeat its only purpose.
|
||||
"""
|
||||
dynamic = self._dynamic_update_interval(plugin_id, plugin_instance)
|
||||
if dynamic is not None:
|
||||
|
||||
@@ -41,7 +41,6 @@ class PluginStateManager:
|
||||
self._state_transition_counts: Dict[str, int] = {}
|
||||
self._error_info: Dict[str, Dict[str, Any]] = {}
|
||||
self._last_update: Dict[str, datetime] = {}
|
||||
self._last_display: Dict[str, datetime] = {}
|
||||
|
||||
def _record_transition(self, plugin_id: str) -> None:
|
||||
"""Count a state transition. Callers must already hold ``_lock``."""
|
||||
@@ -182,10 +181,6 @@ class PluginStateManager:
|
||||
"""Get timestamp of last update() call."""
|
||||
return self._last_update.get(plugin_id)
|
||||
|
||||
def get_last_display(self, plugin_id: str) -> Optional[datetime]:
|
||||
"""Get timestamp of last display() call."""
|
||||
return self._last_display.get(plugin_id)
|
||||
|
||||
def get_state_info(self, plugin_id: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Get comprehensive state information for a plugin.
|
||||
@@ -212,7 +207,6 @@ class PluginStateManager:
|
||||
'is_error': self.is_error(plugin_id),
|
||||
'can_execute': self.can_execute(plugin_id),
|
||||
'last_update': self.get_last_update(plugin_id),
|
||||
'last_display': self.get_last_display(plugin_id),
|
||||
'error_info': self.get_error_info(plugin_id),
|
||||
'state_history_count': self._state_transition_counts.get(plugin_id, 0)
|
||||
}
|
||||
@@ -231,5 +225,4 @@ class PluginStateManager:
|
||||
self._state_transition_counts.pop(plugin_id, None)
|
||||
self._error_info.pop(plugin_id, None)
|
||||
self._last_update.pop(plugin_id, None)
|
||||
self._last_display.pop(plugin_id, None)
|
||||
|
||||
|
||||
@@ -0,0 +1,70 @@
|
||||
"""
|
||||
Repository URL helpers shared by the plugin store and saved repositories.
|
||||
|
||||
One definition of "the same repository URL", of how a GitHub URL maps to
|
||||
``owner/repo``, and of the headers sent to the GitHub API.
|
||||
"""
|
||||
|
||||
from typing import Dict, Optional, Tuple
|
||||
from urllib.parse import urlparse
|
||||
|
||||
#: Hosts whose URLs name a GitHub repository. Matched against
|
||||
#: ``urlparse(url).hostname``, never by substring: a substring test accepts
|
||||
#: ``https://github.com.example.org/...`` as GitHub.
|
||||
GITHUB_HOSTS = frozenset({'github.com', 'www.github.com'})
|
||||
|
||||
#: Sent on every request the store makes to GitHub.
|
||||
USER_AGENT = 'LEDMatrix-Plugin-Manager/1.0'
|
||||
|
||||
|
||||
def normalize_repo_url(url: str) -> str:
|
||||
"""``url`` without surrounding whitespace, trailing slashes or a trailing ``.git``.
|
||||
|
||||
Only a *trailing* ``.git`` is removed. The unanchored
|
||||
``url.replace('.git', '')`` this replaces turned
|
||||
``https://github.com/user/my.github.io`` into ``.../myhub.io``.
|
||||
Case is preserved; compare with :func:`same_repo`.
|
||||
"""
|
||||
url = url.strip().rstrip('/')
|
||||
if url.endswith('.git'):
|
||||
url = url[:-4]
|
||||
return url
|
||||
|
||||
|
||||
def same_repo(url_a: str, url_b: str) -> bool:
|
||||
"""Whether two URLs name the same repository.
|
||||
|
||||
GitHub owner and repository names are case-insensitive, so
|
||||
``ChuckBuilds/LEDMatrix-Plugins`` and ``chuckbuilds/ledmatrix-plugins``
|
||||
are one repository.
|
||||
"""
|
||||
return normalize_repo_url(url_a).lower() == normalize_repo_url(url_b).lower()
|
||||
|
||||
|
||||
def github_owner_repo(url: str) -> Optional[Tuple[str, str]]:
|
||||
"""``(owner, repo)`` for a github.com repository URL, else None.
|
||||
|
||||
The first two path segments, so a URL that points inside the repository
|
||||
(``.../owner/repo/tree/main/plugins/x``) still names ``owner/repo``.
|
||||
"""
|
||||
parsed = urlparse(normalize_repo_url(url))
|
||||
if parsed.hostname not in GITHUB_HOSTS:
|
||||
return None
|
||||
parts = [part for part in parsed.path.split('/') if part]
|
||||
if len(parts) < 2:
|
||||
return None
|
||||
return parts[0], normalize_repo_url(parts[1])
|
||||
|
||||
|
||||
def github_api_headers(token: Optional[str] = None) -> Dict[str, str]:
|
||||
"""Headers for a GitHub REST API request, authenticated when ``token`` is set.
|
||||
|
||||
An authenticated request gets 5000 requests an hour instead of 60.
|
||||
"""
|
||||
headers = {
|
||||
'Accept': 'application/vnd.github.v3+json',
|
||||
'User-Agent': USER_AGENT,
|
||||
}
|
||||
if token:
|
||||
headers['Authorization'] = f'token {token}'
|
||||
return headers
|
||||
@@ -33,7 +33,14 @@ class ResourceLimits:
|
||||
|
||||
@dataclass
|
||||
class ResourceMetrics:
|
||||
"""Resource usage metrics for a plugin."""
|
||||
"""Resource usage metrics for a plugin.
|
||||
|
||||
``memory_mb`` is the largest growth in this *process's* resident memory
|
||||
seen across a single monitored call -- a high-water mark, not current
|
||||
usage, and not the plugin's own footprint (another thread allocating
|
||||
during the call counts too). ``cpu_percent`` is the whole process's CPU
|
||||
use since the previous sample.
|
||||
"""
|
||||
memory_mb: float = 0.0
|
||||
cpu_percent: float = 0.0
|
||||
execution_time: float = 0.0
|
||||
@@ -43,11 +50,6 @@ class ResourceMetrics:
|
||||
min_execution_time: float = float('inf')
|
||||
last_update_time: float = field(default_factory=time.time)
|
||||
|
||||
def update_average_execution_time(self):
|
||||
"""Update average execution time."""
|
||||
if self.call_count > 0:
|
||||
self.total_execution_time = self.total_execution_time / self.call_count
|
||||
|
||||
|
||||
#: How often a plugin's metrics are written to the cache, in seconds.
|
||||
#:
|
||||
@@ -287,6 +289,7 @@ class PluginResourceMonitor:
|
||||
|
||||
# Calculate execution time
|
||||
execution_time = time.time() - start_time
|
||||
memory_growth_mb = 0.0
|
||||
|
||||
# Update metrics
|
||||
with self._lock:
|
||||
@@ -302,17 +305,17 @@ class PluginResourceMonitor:
|
||||
|
||||
# Update memory and CPU if monitoring enabled
|
||||
if self.enable_monitoring:
|
||||
end_memory = self._get_process_memory_mb()
|
||||
metrics.memory_mb = max(metrics.memory_mb, end_memory - start_memory)
|
||||
memory_growth_mb = self._get_process_memory_mb() - start_memory
|
||||
metrics.memory_mb = max(metrics.memory_mb, memory_growth_mb)
|
||||
# CPU is harder to measure per-call, so we track it separately
|
||||
metrics.cpu_percent = self._get_process_cpu_percent()
|
||||
|
||||
# Persist metrics, at most once per interval per plugin.
|
||||
self._persist_metrics(plugin_id, metrics)
|
||||
|
||||
# Check limits
|
||||
if limits:
|
||||
self._check_limits(plugin_id, metrics, limits, execution_time)
|
||||
self._check_limits(plugin_id, metrics, limits, execution_time,
|
||||
memory_growth_mb)
|
||||
|
||||
return result
|
||||
|
||||
@@ -327,8 +330,16 @@ class PluginResourceMonitor:
|
||||
raise
|
||||
|
||||
def _check_limits(self, plugin_id: str, metrics: ResourceMetrics,
|
||||
limits: ResourceLimits, execution_time: float) -> None:
|
||||
"""Check if plugin has exceeded resource limits."""
|
||||
limits: ResourceLimits, execution_time: float,
|
||||
memory_growth_mb: float) -> None:
|
||||
"""Raise ResourceLimitExceeded if this call went over a limit.
|
||||
|
||||
Execution time and memory growth are this call's own; CPU is the
|
||||
latest process sample. Judging memory by the stored high-water mark
|
||||
(``metrics.memory_mb``) instead would fail every call after the first
|
||||
expensive one, so the health tracker's circuit breaker would reopen
|
||||
on every recovery probe and the plugin would never update again.
|
||||
"""
|
||||
warnings = []
|
||||
errors = []
|
||||
|
||||
@@ -343,13 +354,13 @@ class PluginResourceMonitor:
|
||||
)
|
||||
|
||||
# Check memory
|
||||
if limits.max_memory_mb and metrics.memory_mb > limits.max_memory_mb:
|
||||
if limits.max_memory_mb and memory_growth_mb > limits.max_memory_mb:
|
||||
errors.append(
|
||||
f"Memory usage {metrics.memory_mb:.2f}MB exceeds limit {limits.max_memory_mb:.2f}MB"
|
||||
f"Memory growth {memory_growth_mb:.2f}MB exceeds limit {limits.max_memory_mb:.2f}MB"
|
||||
)
|
||||
elif limits.max_memory_mb and metrics.memory_mb > limits.max_memory_mb * limits.warning_threshold:
|
||||
elif limits.max_memory_mb and memory_growth_mb > limits.max_memory_mb * limits.warning_threshold:
|
||||
warnings.append(
|
||||
f"Memory usage {metrics.memory_mb:.2f}MB approaching limit {limits.max_memory_mb:.2f}MB"
|
||||
f"Memory growth {memory_growth_mb:.2f}MB approaching limit {limits.max_memory_mb:.2f}MB"
|
||||
)
|
||||
|
||||
# Check CPU
|
||||
|
||||
@@ -10,6 +10,8 @@ import os
|
||||
from pathlib import Path
|
||||
from typing import List, Dict, Optional
|
||||
|
||||
from src.plugin_system.repo_urls import normalize_repo_url
|
||||
|
||||
|
||||
class SavedRepositoriesManager:
|
||||
"""Manages saved GitHub repository URLs."""
|
||||
@@ -71,18 +73,6 @@ class SavedRepositoriesManager:
|
||||
pass
|
||||
return False
|
||||
|
||||
@staticmethod
|
||||
def _clean_url(repo_url: str) -> str:
|
||||
"""Normalize a repo URL: strip whitespace, trailing slashes, and a
|
||||
trailing ``.git`` suffix ONLY. (The old ``.replace('.git', '')``
|
||||
was an unanchored substring replace that mangled URLs merely
|
||||
containing ``.git``, e.g. ``https://github.com/user/my.github.io``.)
|
||||
"""
|
||||
repo_url = repo_url.strip().rstrip('/')
|
||||
if repo_url.endswith('.git'):
|
||||
repo_url = repo_url[:-4]
|
||||
return repo_url
|
||||
|
||||
def get_all(self) -> List[Dict[str, str]]:
|
||||
"""Get all saved repositories."""
|
||||
return self.repositories.copy()
|
||||
@@ -98,7 +88,7 @@ class SavedRepositoriesManager:
|
||||
Returns:
|
||||
True if added successfully
|
||||
"""
|
||||
repo_url = self._clean_url(repo_url)
|
||||
repo_url = normalize_repo_url(repo_url)
|
||||
|
||||
# Check if already exists
|
||||
for repo in self.repositories:
|
||||
@@ -138,7 +128,7 @@ class SavedRepositoriesManager:
|
||||
Returns:
|
||||
True if removed successfully
|
||||
"""
|
||||
repo_url = self._clean_url(repo_url)
|
||||
repo_url = normalize_repo_url(repo_url)
|
||||
|
||||
previous = self.repositories
|
||||
remaining = [r for r in previous if r.get('url') != repo_url]
|
||||
@@ -156,7 +146,7 @@ class SavedRepositoriesManager:
|
||||
|
||||
def has(self, repo_url: str) -> bool:
|
||||
"""Check if a repository is already saved."""
|
||||
repo_url = self._clean_url(repo_url)
|
||||
repo_url = normalize_repo_url(repo_url)
|
||||
return any(r.get('url') == repo_url for r in self.repositories)
|
||||
|
||||
def get_registry_repositories(self) -> List[Dict[str, str]]:
|
||||
|
||||
@@ -14,6 +14,7 @@ import jsonschema
|
||||
from jsonschema import Draft7Validator, ValidationError
|
||||
|
||||
from src.core_config_keys import CORE_CONFIG_KEYS
|
||||
from src.element_style import expand_style_elements
|
||||
|
||||
|
||||
def _renders_as_object(prop: Dict[str, Any]) -> bool:
|
||||
@@ -452,11 +453,7 @@ class SchemaManager:
|
||||
# full per-element style blocks (font/size/color + layout
|
||||
# offsets) the web-UI config form renders. No-op for schemas
|
||||
# without the declaration; never raises.
|
||||
try:
|
||||
from src.element_style import expand_style_elements
|
||||
schema = expand_style_elements(schema)
|
||||
except ImportError:
|
||||
pass
|
||||
schema = expand_style_elements(schema)
|
||||
|
||||
# Cache the schema
|
||||
self._schema_cache[plugin_id] = schema
|
||||
@@ -643,19 +640,11 @@ class SchemaManager:
|
||||
[name for name in CORE_PLUGIN_PROPERTIES if name not in declared]
|
||||
)
|
||||
|
||||
# Create validator with enhanced schema
|
||||
# iter_errors reports every violation, including one ``required``
|
||||
# error per missing field at every depth.
|
||||
validator = Draft7Validator(enhanced_schema)
|
||||
|
||||
# Collect all validation errors
|
||||
for error in validator.iter_errors(config):
|
||||
error_msg = self._format_validation_error(error, plugin_id)
|
||||
errors.append(error_msg)
|
||||
|
||||
# Check required fields
|
||||
required_fields = enhanced_schema.get('required', [])
|
||||
for field in required_fields:
|
||||
if field not in config:
|
||||
errors.append(f"Missing required field: '{field}'")
|
||||
errors.append(self._format_validation_error(error, plugin_id))
|
||||
|
||||
if errors:
|
||||
return False, errors
|
||||
@@ -687,7 +676,15 @@ class SchemaManager:
|
||||
field_path = f"'{path}'" if path else "root"
|
||||
|
||||
if error.validator == 'required':
|
||||
missing = error.validator_value
|
||||
# validator_value is the schema's whole ``required`` list; the
|
||||
# error itself is about one field, which jsonschema names only in
|
||||
# its message ("'api_key' is a required property").
|
||||
missing = next(
|
||||
(name for name in error.validator_value
|
||||
if error.message.startswith(f"{name!r} ")),
|
||||
None)
|
||||
if missing is None:
|
||||
return f"Field {field_path}: {error.message}"
|
||||
return f"Field {field_path}: Missing required property '{missing}'"
|
||||
elif error.validator == 'type':
|
||||
expected = error.validator_value
|
||||
|
||||
@@ -34,7 +34,9 @@ class PluginState:
|
||||
version: Optional[str] = None
|
||||
installed_at: Optional[datetime] = None
|
||||
last_updated: Optional[datetime] = None
|
||||
config_version: int = 1 # For detecting state corruption
|
||||
# Bumped on every update_plugin_state(). Nothing reads it; it stays so
|
||||
# plugin_state.json keeps the shape older releases load with cls(**data).
|
||||
config_version: int = 1
|
||||
metadata: Dict[str, Any] = None
|
||||
|
||||
def __post_init__(self):
|
||||
@@ -100,6 +102,8 @@ class PluginStateManager:
|
||||
|
||||
# State storage
|
||||
self._states: Dict[str, PluginState] = {}
|
||||
# The file's top-level "version", written back as read. Nothing
|
||||
# checks it yet; it is there for a future format change to branch on.
|
||||
self._state_version = 1
|
||||
|
||||
# Threading
|
||||
@@ -193,7 +197,6 @@ class PluginStateManager:
|
||||
current_state.metadata = {}
|
||||
current_state.metadata.update(updates['metadata'])
|
||||
|
||||
# Increment config version
|
||||
current_state.config_version += 1
|
||||
|
||||
# Store updated state
|
||||
|
||||
+306
-425
File diff suppressed because it is too large
Load Diff
@@ -299,3 +299,48 @@ class TestDefaultMerging:
|
||||
|
||||
assert merged["enabled"] is False
|
||||
assert merged["display_duration"] == 60
|
||||
|
||||
|
||||
class TestMissingRequiredFields:
|
||||
"""One message per missing field, naming that field.
|
||||
|
||||
A manual ``required`` loop used to run after Draft7Validator, which already
|
||||
reports ``required``, so every missing top-level field was listed twice --
|
||||
and the validator's copy printed the schema's whole ``required`` list as
|
||||
if it were the field name.
|
||||
"""
|
||||
|
||||
SCHEMA = {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"api_key": {"type": "string"},
|
||||
"city": {"type": "string"},
|
||||
"units": {"type": "string"},
|
||||
},
|
||||
"required": ["api_key", "city", "units"],
|
||||
}
|
||||
|
||||
def test_each_missing_field_is_reported_once_by_name(self):
|
||||
ok, errors = SchemaManager().validate_config_against_schema(
|
||||
{"units": "metric"}, self.SCHEMA, "test-plugin")
|
||||
|
||||
assert not ok
|
||||
assert errors == [
|
||||
"Field root: Missing required property 'api_key'",
|
||||
"Field root: Missing required property 'city'",
|
||||
]
|
||||
|
||||
def test_nested_missing_field_names_the_field_and_its_parent(self):
|
||||
schema = {
|
||||
"type": "object",
|
||||
"properties": {"nfl": {
|
||||
"type": "object",
|
||||
"properties": {"api_key": {"type": "string"}},
|
||||
"required": ["api_key"],
|
||||
}},
|
||||
}
|
||||
ok, errors = SchemaManager().validate_config_against_schema(
|
||||
{"nfl": {}}, schema, "test-plugin")
|
||||
|
||||
assert not ok
|
||||
assert errors == ["Field 'nfl': Missing required property 'api_key'"]
|
||||
|
||||
@@ -116,10 +116,14 @@ def test_values_of_the_wrong_type_fall_back_to_usable_defaults(bad):
|
||||
assert isinstance(getattr(metrics, field_name), (int, float)), \
|
||||
f"{field_name} came back as {getattr(metrics, field_name)!r}"
|
||||
|
||||
# The real proof: arithmetic on the loaded metrics must not explode.
|
||||
# The real proof: the arithmetic monitor_call and get_metrics_summary do
|
||||
# on the loaded metrics must not explode.
|
||||
metrics.call_count += 1
|
||||
metrics.total_execution_time += 0.5
|
||||
metrics.update_average_execution_time()
|
||||
metrics.max_execution_time = max(metrics.max_execution_time, 0.5)
|
||||
metrics.min_execution_time = min(metrics.min_execution_time, 0.5)
|
||||
metrics.memory_mb = max(metrics.memory_mb, 1.0)
|
||||
assert metrics.total_execution_time / metrics.call_count >= 0
|
||||
|
||||
|
||||
def test_a_numeric_string_is_accepted_rather_than_discarded():
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
"""reload_plugin re-reads the manifest from the plugin's actual directory.
|
||||
|
||||
It read ``plugins_dir / plugin_id / manifest.json``, but a plugin directory's
|
||||
name need not be the id its manifest declares -- discovery maps ids to
|
||||
directories for exactly that reason. For such a plugin the path did not exist,
|
||||
the re-read was skipped silently, and the reload kept the stale manifest.
|
||||
"""
|
||||
|
||||
import json
|
||||
|
||||
import pytest
|
||||
|
||||
from src.plugin_system.plugin_manager import PluginManager
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def pm_with_renamed_dir(tmp_path):
|
||||
plugins_dir = tmp_path / "plugins"
|
||||
plugin_dir = plugins_dir / "stock-ticker-v2"
|
||||
plugin_dir.mkdir(parents=True)
|
||||
manifest_path = plugin_dir / "manifest.json"
|
||||
manifest_path.write_text(json.dumps({"id": "stocks", "version": "1.0.0"}))
|
||||
pm = PluginManager(plugins_dir=str(plugins_dir))
|
||||
assert pm.discover_plugins() == ["stocks"]
|
||||
return pm, manifest_path
|
||||
|
||||
|
||||
def test_reload_picks_up_an_edited_manifest(pm_with_renamed_dir, monkeypatch):
|
||||
pm, manifest_path = pm_with_renamed_dir
|
||||
manifest_path.write_text(json.dumps({"id": "stocks", "version": "2.0.0"}))
|
||||
loaded = []
|
||||
monkeypatch.setattr(pm, "load_plugin", lambda pid: loaded.append(pid) or True)
|
||||
|
||||
assert pm.reload_plugin("stocks") is True
|
||||
assert pm.plugin_manifests["stocks"]["version"] == "2.0.0"
|
||||
assert loaded == ["stocks"]
|
||||
@@ -115,5 +115,22 @@ def test_get_state_info_is_a_consistent_snapshot():
|
||||
assert not inconsistent, f"observed a torn snapshot: {inconsistent[:1]}"
|
||||
|
||||
|
||||
def test_state_info_reports_only_what_something_records():
|
||||
"""No field that is always null.
|
||||
|
||||
``last_display`` was reported here, but nothing ever recorded a display()
|
||||
call, so it was null for every plugin. Its only reader is the web process,
|
||||
whose PluginManager never calls display(), so recording it in the display
|
||||
process could not have filled it either.
|
||||
"""
|
||||
manager = PluginStateManager()
|
||||
manager.set_state("clock", PluginState.ENABLED)
|
||||
manager.record_update("clock")
|
||||
|
||||
info = manager.get_state_info("clock")
|
||||
assert "last_display" not in info
|
||||
assert info["last_update"] is not None
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(pytest.main([__file__, "-v"]))
|
||||
|
||||
@@ -0,0 +1,95 @@
|
||||
"""Repository URL handling shared by the plugin store and saved repositories.
|
||||
|
||||
The store cleaned URLs with ``url.rstrip('/').replace('.git', '')`` in two
|
||||
places, which removes ``.git`` anywhere in the URL:
|
||||
``https://github.com/user/my.github.io`` became ``.../myhub.io``, so installing
|
||||
or browsing such a repository asked GitHub for one that does not exist.
|
||||
"""
|
||||
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from src.plugin_system.repo_urls import (
|
||||
github_api_headers, github_owner_repo, normalize_repo_url, same_repo,
|
||||
)
|
||||
from src.plugin_system.store_manager import PluginStoreManager
|
||||
|
||||
PAGES_REPO = "https://github.com/user/my.github.io"
|
||||
|
||||
|
||||
class TestNormalizeRepoUrl:
|
||||
@pytest.mark.parametrize("raw, expected", [
|
||||
(PAGES_REPO, PAGES_REPO),
|
||||
(PAGES_REPO + ".git", PAGES_REPO),
|
||||
("https://github.com/user/repo.git/", "https://github.com/user/repo"),
|
||||
(" https://github.com/user/repo/ ", "https://github.com/user/repo"),
|
||||
])
|
||||
def test_only_a_trailing_dot_git_is_removed(self, raw, expected):
|
||||
assert normalize_repo_url(raw) == expected
|
||||
|
||||
def test_same_repo_ignores_case_and_suffix(self):
|
||||
assert same_repo("https://github.com/Owner/Repo.git",
|
||||
"https://github.com/owner/repo/")
|
||||
assert not same_repo("https://github.com/owner/repo",
|
||||
"https://github.com/owner/other")
|
||||
|
||||
|
||||
class TestGithubOwnerRepo:
|
||||
@pytest.mark.parametrize("url, expected", [
|
||||
(PAGES_REPO + ".git", ("user", "my.github.io")),
|
||||
("https://www.github.com/owner/repo", ("owner", "repo")),
|
||||
("https://github.com/owner/repo/tree/main/plugins/x", ("owner", "repo")),
|
||||
])
|
||||
def test_github_urls(self, url, expected):
|
||||
assert github_owner_repo(url) == expected
|
||||
|
||||
@pytest.mark.parametrize("url", [
|
||||
"https://github.com.example.org/owner/repo",
|
||||
"https://gitlab.com/owner/repo",
|
||||
"https://github.com/owner",
|
||||
"github.com/owner/repo",
|
||||
])
|
||||
def test_anything_else_is_not_a_github_repo(self, url):
|
||||
assert github_owner_repo(url) is None
|
||||
|
||||
def test_headers_carry_the_token_only_when_given(self):
|
||||
assert "Authorization" not in github_api_headers(None)
|
||||
assert github_api_headers("abc")["Authorization"] == "token abc"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def store(tmp_path):
|
||||
return PluginStoreManager(
|
||||
plugins_dir=str(tmp_path / "plugins"),
|
||||
uninstalled_registry_path=str(tmp_path / "uninstalled.json"))
|
||||
|
||||
|
||||
def test_install_from_url_keeps_an_interior_dot_git(store, monkeypatch):
|
||||
cloned_from = []
|
||||
monkeypatch.setattr(store, "_install_via_git",
|
||||
lambda url, *a, **k: cloned_from.append(url))
|
||||
downloaded = []
|
||||
monkeypatch.setattr(store, "_install_via_download",
|
||||
lambda url, *a, **k: downloaded.append(url) or False)
|
||||
|
||||
result = store.install_from_url(PAGES_REPO + ".git")
|
||||
|
||||
assert result["success"] is False
|
||||
assert cloned_from == [PAGES_REPO]
|
||||
assert all(url.startswith(PAGES_REPO + "/archive/") for url in downloaded)
|
||||
|
||||
|
||||
def test_fetch_registry_from_url_asks_for_the_named_repository(store, monkeypatch):
|
||||
requested = []
|
||||
|
||||
def fake_get(url, **kwargs):
|
||||
requested.append(url)
|
||||
return MagicMock(status_code=404)
|
||||
|
||||
monkeypatch.setattr(store, "_http_get_with_retries", fake_get)
|
||||
|
||||
assert store.fetch_registry_from_url(PAGES_REPO) is None
|
||||
assert requested
|
||||
assert all(url.startswith("https://raw.githubusercontent.com/user/my.github.io/")
|
||||
for url in requested)
|
||||
@@ -93,6 +93,27 @@ class TestResourceLimits:
|
||||
with pytest.raises(ResourceLimitExceeded):
|
||||
mon.monitor_call("p", lambda: time.sleep(0.02))
|
||||
|
||||
def test_memory_limit_judges_each_call_on_its_own_growth(self):
|
||||
"""One expensive call must not fail every call after it.
|
||||
|
||||
The check used to compare the stored high-water mark, which never
|
||||
decreases, so after one call grew memory past the limit every later
|
||||
call raised too and the plugin never updated again.
|
||||
"""
|
||||
mon = PluginResourceMonitor(_cache(), enable_monitoring=False)
|
||||
mon.enable_monitoring = True # measure without needing psutil
|
||||
readings = iter([100.0, 200.0, # first call grows RSS by 100 MB
|
||||
200.0, 201.0]) # second call grows it by 1 MB
|
||||
mon._get_process_memory_mb = lambda: next(readings)
|
||||
mon._get_process_cpu_percent = lambda: 0.0
|
||||
mon.set_limits("p", ResourceLimits(max_memory_mb=50))
|
||||
|
||||
with pytest.raises(ResourceLimitExceeded):
|
||||
mon.monitor_call("p", lambda: None)
|
||||
assert mon.monitor_call("p", lambda: "ok") == "ok"
|
||||
# The high-water mark is still reported.
|
||||
assert mon.get_metrics("p").memory_mb == 100.0
|
||||
|
||||
def test_reset_metrics_clears_counts(self):
|
||||
cache = _cache()
|
||||
mon = PluginResourceMonitor(cache, enable_monitoring=False)
|
||||
|
||||
@@ -5,7 +5,7 @@ SavedRepositoriesManager contract.
|
||||
Covers: the three accepted on-disk load shapes (bare list, wrapped
|
||||
{"repositories": [...]}, anything else -> []) and that saves always write
|
||||
the bare-list form; add/remove/has round trips through a fresh manager;
|
||||
URL normalization post-fix (_clean_url strips only a TRAILING '.git' after
|
||||
URL normalization post-fix (normalize_repo_url strips only a TRAILING '.git' after
|
||||
trailing slashes — the old unanchored .replace('.git', '') mangled URLs
|
||||
like my.github.io); name derivation and registry-vs-single type
|
||||
classification (the ledmatrix-plugins check is lowercased, the
|
||||
|
||||
@@ -0,0 +1,77 @@
|
||||
"""A repository whose only branch is neither main nor master still installs.
|
||||
|
||||
_install_via_git tries the candidate branches, then the repository's default
|
||||
branch -- but it returned None both for "every clone failed" and for "the
|
||||
default-branch clone succeeded". install_from_url took the None as failure,
|
||||
fell through to the archive download of main/master (which does not exist),
|
||||
and reported "Failed to clone or download repository" for a repository it had
|
||||
just cloned.
|
||||
"""
|
||||
|
||||
import json
|
||||
import shutil
|
||||
import subprocess
|
||||
|
||||
import pytest
|
||||
|
||||
from src.plugin_system.store_manager import PluginStoreManager
|
||||
|
||||
pytestmark = pytest.mark.skipif(shutil.which("git") is None, reason="git not installed")
|
||||
|
||||
MANIFEST = {
|
||||
"id": "develop-only", "name": "Develop Only", "class_name": "P",
|
||||
"display_modes": ["develop_only"], "version": "1.0.0",
|
||||
}
|
||||
|
||||
|
||||
def _git(*args, cwd):
|
||||
subprocess.run(
|
||||
["git", "-c", "user.name=t", "-c", "user.email=t@example.invalid", *args],
|
||||
cwd=cwd, check=True, capture_output=True)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def develop_only_repo(tmp_path):
|
||||
repo = tmp_path / "upstream"
|
||||
repo.mkdir()
|
||||
_git("init", "-q", "-b", "develop", cwd=repo)
|
||||
(repo / "manifest.json").write_text(json.dumps(MANIFEST))
|
||||
(repo / "manager.py").write_text("class P: pass\n")
|
||||
_git("add", ".", cwd=repo)
|
||||
_git("commit", "-q", "-m", "init", cwd=repo)
|
||||
return repo.as_uri()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def store(tmp_path, monkeypatch):
|
||||
mgr = PluginStoreManager(
|
||||
plugins_dir=str(tmp_path / "plugins"),
|
||||
uninstalled_registry_path=str(tmp_path / "uninstalled.json"))
|
||||
monkeypatch.setattr(mgr, "_install_dependencies", lambda *a, **k: True)
|
||||
downloads = []
|
||||
monkeypatch.setattr(mgr, "_install_via_download",
|
||||
lambda url, *a, **k: downloads.append(url) or False)
|
||||
mgr.downloads = downloads
|
||||
return mgr
|
||||
|
||||
|
||||
def test_a_default_branch_clone_reports_its_branch(store, develop_only_repo, tmp_path):
|
||||
target = tmp_path / "clone"
|
||||
assert store._install_via_git(develop_only_repo, target, ["main", "master"]) == "develop"
|
||||
assert (target / "manifest.json").exists()
|
||||
|
||||
|
||||
def test_a_failed_clone_reports_none(store, tmp_path):
|
||||
missing = (tmp_path / "no-such-repo").as_uri()
|
||||
target = tmp_path / "clone"
|
||||
assert store._install_via_git(missing, target, ["main"]) is None
|
||||
assert not target.exists()
|
||||
|
||||
|
||||
def test_install_from_url_installs_a_develop_only_repository(store, develop_only_repo):
|
||||
result = store.install_from_url(develop_only_repo)
|
||||
|
||||
assert result == {"success": True, "plugin_id": "develop-only",
|
||||
"name": "Develop Only", "branch": "develop"}
|
||||
assert (store.plugins_dir / "develop-only" / "manifest.json").exists()
|
||||
assert store.downloads == []
|
||||
@@ -0,0 +1,62 @@
|
||||
"""update_plugin must not borrow the enclosing LEDMatrix checkout's remote.
|
||||
|
||||
Plugins live in ``plugin-repos/`` inside the LEDMatrix git checkout. For a
|
||||
plugin installed from a ZIP (no ``.git`` of its own), ``git -C <plugin>``
|
||||
walks up to the LEDMatrix repository, and ``git config --local --get
|
||||
remote.origin.url`` answers with LEDMatrix's own URL. update_plugin then tried
|
||||
to "reinstall" the plugin from the LEDMatrix repository.
|
||||
"""
|
||||
|
||||
import json
|
||||
import shutil
|
||||
import subprocess
|
||||
|
||||
import pytest
|
||||
|
||||
from src.plugin_system.store_manager import PluginStoreManager
|
||||
|
||||
pytestmark = pytest.mark.skipif(shutil.which("git") is None, reason="git not installed")
|
||||
|
||||
PLUGIN_ID = "zip-installed"
|
||||
PARENT_REMOTE = "https://github.com/example/LEDMatrix"
|
||||
|
||||
|
||||
def _git(*args, cwd):
|
||||
subprocess.run(["git", *args], cwd=cwd, check=True, capture_output=True)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def store_inside_checkout(tmp_path):
|
||||
checkout = tmp_path / "LEDMatrix"
|
||||
checkout.mkdir()
|
||||
_git("init", "-q", cwd=checkout)
|
||||
_git("remote", "add", "origin", PARENT_REMOTE, cwd=checkout)
|
||||
|
||||
plugins_dir = checkout / "plugin-repos"
|
||||
plugin_dir = plugins_dir / PLUGIN_ID
|
||||
plugin_dir.mkdir(parents=True)
|
||||
(plugin_dir / "manifest.json").write_text(json.dumps(
|
||||
{"id": PLUGIN_ID, "name": "Zip", "version": "1.0.0"}))
|
||||
|
||||
store = PluginStoreManager(
|
||||
plugins_dir=str(plugins_dir),
|
||||
uninstalled_registry_path=str(tmp_path / "uninstalled.json"))
|
||||
return store, plugin_dir
|
||||
|
||||
|
||||
def test_a_plugin_without_its_own_git_has_no_remote(store_inside_checkout, monkeypatch):
|
||||
store, plugin_dir = store_inside_checkout
|
||||
# The premise: git itself does report the parent's remote here.
|
||||
parent_view = subprocess.run(
|
||||
["git", "-C", str(plugin_dir), "config", "--local", "--get", "remote.origin.url"],
|
||||
capture_output=True, text=True)
|
||||
assert parent_view.stdout.strip() == PARENT_REMOTE
|
||||
|
||||
monkeypatch.setattr(store, "fetch_registry", lambda *a, **k: {"plugins": []})
|
||||
monkeypatch.setattr(store, "get_plugin_info", lambda *a, **k: None)
|
||||
install_calls = []
|
||||
monkeypatch.setattr(store, "install_from_url",
|
||||
lambda *a, **k: install_calls.append((a, k)) or {"success": True})
|
||||
|
||||
assert store.update_plugin(PLUGIN_ID) is False
|
||||
assert install_calls == []
|
||||
@@ -21,10 +21,7 @@ class TestPluginOperationsIntegration(unittest.TestCase):
|
||||
self.temp_dir = Path(tempfile.mkdtemp())
|
||||
|
||||
# Initialize components
|
||||
self.operation_queue = PluginOperationQueue(
|
||||
history_file=str(self.temp_dir / "operations.json"),
|
||||
max_history=100
|
||||
)
|
||||
self.operation_queue = PluginOperationQueue(max_history=100)
|
||||
|
||||
self.state_manager = PluginStateManager(
|
||||
state_file=str(self.temp_dir / "state.json"),
|
||||
|
||||
@@ -144,12 +144,7 @@ schema_manager = SchemaManager(
|
||||
)
|
||||
|
||||
# Initialize operation queue for plugin operations
|
||||
# Use lazy_load=True to defer file loading until first use (improves startup time)
|
||||
operation_queue = PluginOperationQueue(
|
||||
history_file=str(project_root / "data" / "plugin_operations.json"),
|
||||
max_history=500,
|
||||
lazy_load=True
|
||||
)
|
||||
operation_queue = PluginOperationQueue(max_history=500)
|
||||
|
||||
# Initialize plugin state manager
|
||||
# Use lazy_load=True to defer file loading until first use (improves startup time)
|
||||
|
||||
Reference in New Issue
Block a user