mirror of
https://github.com/ChuckBuilds/LEDMatrix.git
synced 2026-10-05 06:45:09 +00:00
fix(plugins): store and plugin-manager bugs; tidy src/plugin_system (#635)
* fix(store): don't read a ZIP-installed plugin's remote from the LEDMatrix repo update_plugin looked up remote.origin.url with `git -C <plugin> config --local` for plugins that are not git checkouts. Under plugin-repos/ git walks up to the enclosing LEDMatrix repository, so the lookup returned LEDMatrix's own URL and a plugin missing from the registry was "reinstalled" from the LEDMatrix repo. Only ask git when the plugin directory has its own .git, the test _get_local_git_info already uses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(schema): report each missing required field once, by name validate_config_against_schema ran its own required-fields loop after Draft7Validator.iter_errors, which already yields one `required` error per missing field, so every missing top-level field was listed twice. The validator's copy also printed the schema's whole `required` list ("Missing required property '['api_key', 'city']'") instead of the field. Drop the loop and take the field name from the error itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(store): stop mangling repository URLs that contain ".git" install_from_url and fetch_registry_from_url cleaned URLs with `rstrip('/').replace('.git', '')`, which removes ".git" anywhere: https://github.com/user/my.github.io became .../myhub.io, so installing or browsing that repository asked GitHub for one that does not exist. Add src/plugin_system/repo_urls.py with one anchored normalize_repo_url(), same_repo() for comparisons, github_owner_repo() and github_api_headers(), and use them for the five copies of the owner/repo parsing and GitHub headers in the store and for saved repositories. GitHub URLs are now recognised by urlparse().hostname everywhere: _get_latest_commit_info used a substring test, and _install_from_monorepo_api parsed any host. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(store): install a repository whose only branch is not main/master _install_via_git returned None both when every clone failed and when the last-resort clone of the repository's default branch succeeded. _install_plugin_impl papered over it with `and not plugin_path.exists()`; install_from_url did not, so a repository whose only branch is e.g. `develop` was cloned, then treated as a failure, then "downloaded" from main/master archives that do not exist. After a default-branch clone, return the branch the clone checked out (read from .git/HEAD), so None means failure and nothing else, and give both callers the same `branch_used is None` fallback. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(plugins): judge the memory limit on each call's own growth monitor_call stores `metrics.memory_mb = max(previous, growth)`, and _check_limits compared that high-water mark with max_memory_mb. It never decreases, so once one update() grew the process past the limit every later call raised ResourceLimitExceeded and the circuit breaker kept reopening. Pass the call's own RSS growth to _check_limits; keep the high-water mark for reporting and document what it measures. Remove ResourceMetrics.update_average_execution_time: nothing called it, and it overwrote the running total with the average. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(plugins): reload_plugin re-reads the manifest from the discovered directory reload_plugin read `plugins_dir / plugin_id / "manifest.json"`, ignoring the discovery map and the plugin_dirs rules. For a plugin whose directory name differs from its manifest id the path did not exist, the re-read was skipped without a word, and the reload kept the stale manifest. Resolve the directory with find_plugin_directory, as load_plugin does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(plugins): drop the always-null last_display from plugin state info PluginStateManager reported `last_display` from `_last_display`, which nothing ever wrote, so it was null for every plugin. Recording it in PluginExecutor.execute_display would not help: get_state_info's only reader is the web process, whose PluginManager never calls display(). Remove the field, its dict and get_last_display() (no caller in core, the web UI or the plugin monorepo). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(store): share the rollback and requirements helpers, drop dead code - install_plugin and _reinstall_with_rollback set aside, discard and restore the old copy through _set_aside/_discard_backup/_restore_backup instead of two copies of the same blocks. - The loader and the store run the same pre-pip checks through contained_plugin_dir() and requirements_to_install() in plugin_loader. They still invoke pip differently (sys.executable -m pip vs. the sudo wrapper). `except (BrokenPipeError, OSError)` + `isinstance(e, OSError)` becomes `except OSError` checking errno.EPIPE. - load_module never returns None, so load_plugin's check is gone and the docstring says what it raises. - Remove the always-true JSONSCHEMA_AVAILABLE, the inline re-imports of re and permission_utils, the fake status_result object nobody reads, hasattr(git_error, 'cmd'), a redundant "merge conflict" test and `import traceback` (exc_info=True does it). - Correct comments: install_from_url names the directory for the caller's id when given (not always the manifest id), _get_local_git_info saves one git subprocess (not four), _enrich calls two helpers, search_plugins documents all its arguments, _find_plugin_path states its behaviour instead of a TODO, and history narration is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(plugins): tidy base_plugin, correct plugin_manager/state comments - base_plugin: drop the unused `import logging`; get_display_duration runs the instance value and the config value through one _positive_seconds() helper instead of two copies of the coercion; the 'static'/'none'/fallback branches of get_vegas_display_mode, which all returned FIXED_SEGMENT, are one; fix the mis-indented validate_config example; say that get_supported_vegas_modes/get_vegas_segment_width are not consulted by core (kept, plugins override them). - schema_manager: import expand_style_elements normally rather than swallowing an ImportError of a core module. - plugin_manager: the plugins directory is the configured one (plugin-repos/ by default), not plugins/; get_config() returns the live dict, not a copy, so the interval cache comments say what it saves. - state_manager: config_version and the file version are not used to detect corruption; say what they are. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(plugins): stop writing data/plugin_operations.json PluginOperationQueue wrote its finished-operation history to data/plugin_operations.json after every operation, and read it back only into its own in-memory list, which only get_operation_history() exposes -- and nothing calls that. The operation-history endpoint reads OperationHistory (data/operation_history.json). No code in src/, web_interface/, scripts/ or test/ reads the file. Drop the history_file/lazy_load parameters and the load/save code; the bounded in-memory history stays. web_interface/app.py and the integration test stop passing the removed arguments. An existing data/plugin_operations.json is left in place (data/* is gitignored). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(changelog): plugin-system Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
+4
-4
@@ -81,10 +81,10 @@ assets/stocks/crypto_icons/
|
|||||||
|
|
||||||
# Plugin operation state written at runtime.
|
# Plugin operation state written at runtime.
|
||||||
#
|
#
|
||||||
# web_interface/app.py writes data/plugin_operations.json, data/plugin_state.json
|
# web_interface/app.py writes data/plugin_state.json and data/operation_history.json
|
||||||
# and data/operation_history.json as the web interface runs, into a directory that
|
# (older releases also data/plugin_operations.json) as the web interface runs, into
|
||||||
# ships tracked (data/.gitkeep) and was otherwise unignored. So every rig that ever
|
# a directory that ships tracked (data/.gitkeep). Unignored, every rig that ever
|
||||||
# opened the web UI -- and every test run that constructs the app -- left three
|
# opened the web UI -- and every test run that constructs the app -- would leave
|
||||||
# untracked files behind and a permanently dirty `git status`. Same reasoning as
|
# untracked files behind and a permanently dirty `git status`. Same reasoning as
|
||||||
# the logo rule above: a checkout that is always dirty is a checkout nobody reads.
|
# the logo rule above: a checkout that is always dirty is a checkout nobody reads.
|
||||||
data/*
|
data/*
|
||||||
|
|||||||
@@ -33,6 +33,15 @@ accepts both, but the store flags the old spelling as deprecated
|
|||||||
- `src/common/README.md` covers every module.
|
- `src/common/README.md` covers every module.
|
||||||
- Stale setup, service and troubleshooting claims are corrected.
|
- Stale setup, service and troubleshooting claims are corrected.
|
||||||
|
|
||||||
|
- Plugin store and plugin manager fixes:
|
||||||
|
- Updating a plugin that was installed from a ZIP no longer tries to reinstall it from the LEDMatrix repository's own URL.
|
||||||
|
- Repository URLs with `.git` in the middle are no longer mangled. The URL helpers now live in `src/plugin_system/repo_urls.py`.
|
||||||
|
- Installing from a URL works when the repository's only branch isn't `main` or `master`.
|
||||||
|
- A missing required config field is reported once, by name.
|
||||||
|
- A plugin that went over `max_memory_mb` once is no longer refused on every call after that.
|
||||||
|
- `reload_plugin` reads the manifest from the plugin's discovered directory.
|
||||||
|
- Removed: `last_display` from plugin state info and `get_last_display()` (nothing recorded them); `PluginOperationQueue`'s `history_file` and `lazy_load` arguments; and `data/plugin_operations.json`, which nothing read.
|
||||||
|
|
||||||
- The web service (`ledmatrix-web`) logs through `src.logging_config` like the
|
- The web service (`ledmatrix-web`) logs through `src.logging_config` like the
|
||||||
display service, so `journalctl -p err -u ledmatrix-web` works. Successful
|
display service, so `journalctl -p err -u ledmatrix-web` works. Successful
|
||||||
GET/HEAD/OPTIONS requests (the UI's polling) are logged at DEBUG instead of
|
GET/HEAD/OPTIONS requests (the UI's polling) are logged at DEBUG instead of
|
||||||
|
|||||||
@@ -11,7 +11,6 @@ Stability: Stable - maintains backward compatibility
|
|||||||
from abc import ABC, abstractmethod
|
from abc import ABC, abstractmethod
|
||||||
from enum import Enum
|
from enum import Enum
|
||||||
from typing import Dict, Any, Optional, List
|
from typing import Dict, Any, Optional, List
|
||||||
import logging
|
|
||||||
import os
|
import os
|
||||||
import sys
|
import sys
|
||||||
from src.logging_config import get_logger
|
from src.logging_config import get_logger
|
||||||
@@ -511,104 +510,53 @@ class BasePlugin(ABC):
|
|||||||
"""
|
"""
|
||||||
Get the display duration for this plugin instance.
|
Get the display duration for this plugin instance.
|
||||||
|
|
||||||
Automatically detects duration from:
|
Uses, in order, the first positive number among:
|
||||||
1. self.display_duration instance variable (if exists)
|
1. ``self.display_duration`` (a common pattern in scoreboard plugins)
|
||||||
2. self.config.get("display_duration", 15.0) (fallback)
|
2. ``self.config["display_duration"]``
|
||||||
|
3. 15.0
|
||||||
|
|
||||||
Can be overridden by plugins to provide dynamic durations based
|
Numeric strings count as numbers. Can be overridden by plugins to
|
||||||
on content (e.g., longer duration for more complex displays).
|
provide dynamic durations based on content (e.g., longer duration for
|
||||||
|
more complex displays).
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
Duration in seconds to display this plugin's content
|
Duration in seconds to display this plugin's content
|
||||||
"""
|
"""
|
||||||
# Check for instance variable first (common pattern in scoreboard plugins)
|
|
||||||
if hasattr(self, 'display_duration'):
|
|
||||||
try:
|
|
||||||
duration = getattr(self, 'display_duration')
|
|
||||||
# Handle None case
|
|
||||||
if duration is None:
|
|
||||||
pass # Fall through to config
|
|
||||||
# Try to convert to float if it's a number or numeric string.
|
|
||||||
# bool is excluded: it's an int subclass, and True would
|
|
||||||
# otherwise read as a 1-second duration.
|
|
||||||
elif isinstance(duration, (int, float)) and not isinstance(duration, bool):
|
|
||||||
if duration > 0:
|
|
||||||
return float(duration)
|
|
||||||
else:
|
|
||||||
self.logger.debug(
|
|
||||||
"display_duration instance variable is non-positive (%s), using config fallback",
|
|
||||||
duration
|
|
||||||
)
|
|
||||||
# Try converting string representations of numbers
|
|
||||||
elif isinstance(duration, str):
|
|
||||||
try:
|
|
||||||
duration_float = float(duration)
|
|
||||||
if duration_float > 0:
|
|
||||||
return duration_float
|
|
||||||
else:
|
|
||||||
self.logger.debug(
|
|
||||||
"display_duration string value is non-positive (%s), using config fallback",
|
|
||||||
duration
|
|
||||||
)
|
|
||||||
except (ValueError, TypeError):
|
|
||||||
self.logger.warning(
|
|
||||||
"display_duration instance variable has invalid string value '%s', using config fallback",
|
|
||||||
duration
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
self.logger.warning(
|
|
||||||
"display_duration instance variable has unexpected type %s (value: %s), using config fallback",
|
|
||||||
type(duration).__name__, duration
|
|
||||||
)
|
|
||||||
except (TypeError, ValueError, AttributeError) as e:
|
|
||||||
self.logger.warning(
|
|
||||||
"Error reading display_duration instance variable: %s, using config fallback",
|
|
||||||
e
|
|
||||||
)
|
|
||||||
|
|
||||||
# Fall back to config
|
|
||||||
config_duration = self.config.get("display_duration", 15.0)
|
|
||||||
try:
|
try:
|
||||||
# Ensure config value is also a valid float (bool excluded — an
|
duration = getattr(self, 'display_duration', None)
|
||||||
# int subclass that would otherwise read True as 1 second)
|
except (TypeError, ValueError, AttributeError) as e:
|
||||||
if isinstance(config_duration, (int, float)) and not isinstance(config_duration, bool):
|
# A plugin may define display_duration as a property that raises.
|
||||||
if config_duration > 0:
|
|
||||||
return float(config_duration)
|
|
||||||
else:
|
|
||||||
self.logger.debug(
|
|
||||||
"Config display_duration is non-positive (%s), using default 15.0",
|
|
||||||
config_duration
|
|
||||||
)
|
|
||||||
return 15.0
|
|
||||||
elif isinstance(config_duration, str):
|
|
||||||
try:
|
|
||||||
duration_float = float(config_duration)
|
|
||||||
if duration_float > 0:
|
|
||||||
return duration_float
|
|
||||||
else:
|
|
||||||
self.logger.debug(
|
|
||||||
"Config display_duration string is non-positive (%s), using default 15.0",
|
|
||||||
config_duration
|
|
||||||
)
|
|
||||||
return 15.0
|
|
||||||
except ValueError:
|
|
||||||
self.logger.warning(
|
|
||||||
"Config display_duration has invalid string value '%s', using default 15.0",
|
|
||||||
config_duration
|
|
||||||
)
|
|
||||||
return 15.0
|
|
||||||
else:
|
|
||||||
self.logger.warning(
|
|
||||||
"Config display_duration has unexpected type %s (value: %s), using default 15.0",
|
|
||||||
type(config_duration).__name__, config_duration
|
|
||||||
)
|
|
||||||
except (ValueError, TypeError) as e:
|
|
||||||
self.logger.warning(
|
self.logger.warning(
|
||||||
"Error processing config display_duration: %s, using default 15.0",
|
"Error reading display_duration instance variable: %s, using config fallback", e)
|
||||||
e
|
duration = None
|
||||||
)
|
if duration is not None:
|
||||||
|
seconds = self._positive_seconds(duration, "display_duration instance variable")
|
||||||
|
if seconds is not None:
|
||||||
|
return seconds
|
||||||
|
|
||||||
return 15.0
|
seconds = self._positive_seconds(
|
||||||
|
self.config.get("display_duration", 15.0), "config display_duration")
|
||||||
|
return seconds if seconds is not None else 15.0
|
||||||
|
|
||||||
|
def _positive_seconds(self, value: Any, source: str) -> Optional[float]:
|
||||||
|
"""``value`` as a positive float, or None (with a log line) if it is not one.
|
||||||
|
|
||||||
|
bool is rejected although it is an int subclass: True would otherwise
|
||||||
|
read as a 1-second duration.
|
||||||
|
"""
|
||||||
|
if isinstance(value, bool) or not isinstance(value, (int, float, str)):
|
||||||
|
self.logger.warning("%s has unexpected type %s (value: %s), ignoring it",
|
||||||
|
source, type(value).__name__, value)
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
seconds = float(value)
|
||||||
|
except ValueError:
|
||||||
|
self.logger.warning("%s has invalid value %r, ignoring it", source, value)
|
||||||
|
return None
|
||||||
|
if seconds > 0:
|
||||||
|
return seconds
|
||||||
|
self.logger.debug("%s is non-positive (%s), ignoring it", source, value)
|
||||||
|
return None
|
||||||
|
|
||||||
# ---------------------------------------------------------------------
|
# ---------------------------------------------------------------------
|
||||||
# Dynamic duration support hooks
|
# Dynamic duration support hooks
|
||||||
@@ -926,25 +874,20 @@ class BasePlugin(ABC):
|
|||||||
config_mode, self.plugin_id
|
config_mode, self.plugin_id
|
||||||
)
|
)
|
||||||
|
|
||||||
# Fall back to mapping legacy content_type
|
# Fall back to mapping legacy content_type. 'none' (excluded from
|
||||||
content_type = self.get_vegas_content_type()
|
# Vegas) also maps to FIXED_SEGMENT: exclusion is decided by checking
|
||||||
if content_type == 'multi':
|
# get_vegas_content_type() separately.
|
||||||
|
if self.get_vegas_content_type() == 'multi':
|
||||||
return VegasDisplayMode.SCROLL
|
return VegasDisplayMode.SCROLL
|
||||||
elif content_type == 'static':
|
|
||||||
return VegasDisplayMode.FIXED_SEGMENT
|
|
||||||
elif content_type == 'none':
|
|
||||||
# 'none' means excluded - return FIXED_SEGMENT as default
|
|
||||||
# The exclusion is handled by checking get_vegas_content_type() separately
|
|
||||||
return VegasDisplayMode.FIXED_SEGMENT
|
|
||||||
|
|
||||||
return VegasDisplayMode.FIXED_SEGMENT
|
return VegasDisplayMode.FIXED_SEGMENT
|
||||||
|
|
||||||
def get_supported_vegas_modes(self) -> List[VegasDisplayMode]:
|
def get_supported_vegas_modes(self) -> List[VegasDisplayMode]:
|
||||||
"""
|
"""
|
||||||
Return list of Vegas display modes this plugin supports.
|
Return list of Vegas display modes this plugin supports.
|
||||||
|
|
||||||
Used by the web UI to show available mode options for user configuration.
|
Not currently consulted by core: neither Vegas mode nor the web UI
|
||||||
Override to customize which modes are available for this plugin.
|
calls it. It is kept, and plugins override it, as the declared set of
|
||||||
|
modes a future mode picker would offer.
|
||||||
|
|
||||||
By default:
|
By default:
|
||||||
- 'multi' content type plugins support SCROLL and FIXED_SEGMENT
|
- 'multi' content type plugins support SCROLL and FIXED_SEGMENT
|
||||||
@@ -972,6 +915,10 @@ class BasePlugin(ABC):
|
|||||||
"""
|
"""
|
||||||
Get the preferred width for this plugin in Vegas FIXED_SEGMENT mode.
|
Get the preferred width for this plugin in Vegas FIXED_SEGMENT mode.
|
||||||
|
|
||||||
|
Not currently consulted by core: Vegas mode sizes a card from the
|
||||||
|
``vegas_width_pct`` / ``vegas_scroll.render_width_pct`` settings
|
||||||
|
(see get_vegas_render_width()). Kept because plugins override it.
|
||||||
|
|
||||||
Returns the number of panels this plugin should occupy when displayed
|
Returns the number of panels this plugin should occupy when displayed
|
||||||
as a fixed segment. The actual pixel width is calculated as:
|
as a fixed segment. The actual pixel width is calculated as:
|
||||||
width = panels * single_panel_width
|
width = panels * single_panel_width
|
||||||
@@ -1025,7 +972,7 @@ class BasePlugin(ABC):
|
|||||||
required_fields = ['api_key', 'city']
|
required_fields = ['api_key', 'city']
|
||||||
for field in required_fields:
|
for field in required_fields:
|
||||||
if field not in self.config:
|
if field not in self.config:
|
||||||
self.logger.error("Missing required field: %s", field)
|
self.logger.error("Missing required field: %s", field)
|
||||||
return False
|
return False
|
||||||
return True
|
return True
|
||||||
"""
|
"""
|
||||||
|
|||||||
@@ -9,8 +9,6 @@ import threading
|
|||||||
import queue
|
import queue
|
||||||
from typing import Dict, Optional, List, Callable, Any
|
from typing import Dict, Optional, List, Callable, Any
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
from pathlib import Path
|
|
||||||
import json
|
|
||||||
|
|
||||||
from src.plugin_system.operation_types import (
|
from src.plugin_system.operation_types import (
|
||||||
PluginOperation, OperationType, OperationStatus
|
PluginOperation, OperationType, OperationStatus
|
||||||
@@ -28,28 +26,22 @@ class PluginOperationQueue:
|
|||||||
- Prevents concurrent operations on same plugin
|
- Prevents concurrent operations on same plugin
|
||||||
- Operation status tracking
|
- Operation status tracking
|
||||||
- Operation cancellation
|
- Operation cancellation
|
||||||
- Operation history
|
- In-memory history of finished operations
|
||||||
|
|
||||||
|
The history is not persisted. The web UI's operation history comes from
|
||||||
|
OperationHistory (operation_history.py), which has its own file; a copy
|
||||||
|
written here was never read back by anything.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(
|
def __init__(self, max_history: int = 100):
|
||||||
self,
|
|
||||||
history_file: Optional[str] = None,
|
|
||||||
max_history: int = 100,
|
|
||||||
lazy_load: bool = False
|
|
||||||
):
|
|
||||||
"""
|
"""
|
||||||
Initialize operation queue.
|
Initialize operation queue.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
history_file: Optional path to file for persisting operation history
|
|
||||||
max_history: Maximum number of operations to keep in history
|
max_history: Maximum number of operations to keep in history
|
||||||
lazy_load: If True, defer loading history file until first access
|
|
||||||
"""
|
"""
|
||||||
self.logger = get_logger(__name__)
|
self.logger = get_logger(__name__)
|
||||||
self.history_file = Path(history_file) if history_file else None
|
|
||||||
self.max_history = max_history
|
self.max_history = max_history
|
||||||
self._lazy_load = lazy_load
|
|
||||||
self._history_loaded = False
|
|
||||||
|
|
||||||
# Operation tracking
|
# Operation tracking
|
||||||
self._operations: Dict[str, PluginOperation] = {}
|
self._operations: Dict[str, PluginOperation] = {}
|
||||||
@@ -62,20 +54,8 @@ class PluginOperationQueue:
|
|||||||
self._worker_thread: Optional[threading.Thread] = None
|
self._worker_thread: Optional[threading.Thread] = None
|
||||||
self._stop_event = threading.Event()
|
self._stop_event = threading.Event()
|
||||||
|
|
||||||
# Load history from file if it exists (unless lazy loading)
|
|
||||||
if not self._lazy_load and self.history_file and self.history_file.exists():
|
|
||||||
self._load_history()
|
|
||||||
self._history_loaded = True
|
|
||||||
|
|
||||||
# Start worker thread
|
|
||||||
self._start_worker()
|
self._start_worker()
|
||||||
|
|
||||||
def _ensure_loaded(self) -> None:
|
|
||||||
"""Ensure history is loaded (for lazy loading)."""
|
|
||||||
if not self._history_loaded and self.history_file and self.history_file.exists():
|
|
||||||
self._load_history()
|
|
||||||
self._history_loaded = True
|
|
||||||
|
|
||||||
def enqueue_operation(
|
def enqueue_operation(
|
||||||
self,
|
self,
|
||||||
operation_type: OperationType,
|
operation_type: OperationType,
|
||||||
@@ -139,7 +119,6 @@ class PluginOperationQueue:
|
|||||||
Returns:
|
Returns:
|
||||||
PluginOperation if found, None otherwise
|
PluginOperation if found, None otherwise
|
||||||
"""
|
"""
|
||||||
self._ensure_loaded()
|
|
||||||
with self._lock:
|
with self._lock:
|
||||||
return self._operations.get(operation_id)
|
return self._operations.get(operation_id)
|
||||||
|
|
||||||
@@ -184,7 +163,6 @@ class PluginOperationQueue:
|
|||||||
Returns:
|
Returns:
|
||||||
List of operations, sorted by creation time (newest first)
|
List of operations, sorted by creation time (newest first)
|
||||||
"""
|
"""
|
||||||
self._ensure_loaded()
|
|
||||||
with self._lock:
|
with self._lock:
|
||||||
# Sort by creation time (newest first)
|
# Sort by creation time (newest first)
|
||||||
history = sorted(
|
history = sorted(
|
||||||
@@ -314,12 +292,8 @@ class PluginOperationQueue:
|
|||||||
if self._active_operations[operation.plugin_id].operation_id == operation.operation_id:
|
if self._active_operations[operation.plugin_id].operation_id == operation.operation_id:
|
||||||
del self._active_operations[operation.plugin_id]
|
del self._active_operations[operation.plugin_id]
|
||||||
|
|
||||||
# Add to history
|
|
||||||
self._add_to_history(operation)
|
self._add_to_history(operation)
|
||||||
|
|
||||||
# Save history to file
|
|
||||||
self._save_history()
|
|
||||||
|
|
||||||
def _add_to_history(self, operation: PluginOperation) -> None:
|
def _add_to_history(self, operation: PluginOperation) -> None:
|
||||||
"""Add operation to history, maintaining max_history limit."""
|
"""Add operation to history, maintaining max_history limit."""
|
||||||
self._operation_history.append(operation)
|
self._operation_history.append(operation)
|
||||||
@@ -330,46 +304,6 @@ class PluginOperationQueue:
|
|||||||
self._operation_history.sort(key=lambda op: op.created_at)
|
self._operation_history.sort(key=lambda op: op.created_at)
|
||||||
self._operation_history = self._operation_history[-self.max_history:]
|
self._operation_history = self._operation_history[-self.max_history:]
|
||||||
|
|
||||||
def _save_history(self) -> None:
|
|
||||||
"""Save operation history to file."""
|
|
||||||
if not self.history_file:
|
|
||||||
return
|
|
||||||
|
|
||||||
try:
|
|
||||||
with self._lock:
|
|
||||||
# Convert operations to dicts
|
|
||||||
history_data = [op.to_dict() for op in self._operation_history]
|
|
||||||
|
|
||||||
# Ensure directory exists
|
|
||||||
self.history_file.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
|
|
||||||
# Write to file
|
|
||||||
with open(self.history_file, 'w') as f:
|
|
||||||
json.dump(history_data, f, indent=2)
|
|
||||||
|
|
||||||
except Exception as e:
|
|
||||||
self.logger.warning(f"Error saving operation history: {e}")
|
|
||||||
|
|
||||||
def _load_history(self) -> None:
|
|
||||||
"""Load operation history from file."""
|
|
||||||
if not self.history_file or not self.history_file.exists():
|
|
||||||
return
|
|
||||||
|
|
||||||
try:
|
|
||||||
with open(self.history_file, 'r') as f:
|
|
||||||
history_data = json.load(f)
|
|
||||||
|
|
||||||
with self._lock:
|
|
||||||
self._operation_history = [
|
|
||||||
PluginOperation.from_dict(op_data)
|
|
||||||
for op_data in history_data
|
|
||||||
]
|
|
||||||
|
|
||||||
self.logger.info(f"Loaded {len(self._operation_history)} operations from history")
|
|
||||||
|
|
||||||
except Exception as e:
|
|
||||||
self.logger.warning(f"Error loading operation history: {e}")
|
|
||||||
|
|
||||||
def shutdown(self) -> None:
|
def shutdown(self) -> None:
|
||||||
"""Shutdown the operation queue and worker thread."""
|
"""Shutdown the operation queue and worker thread."""
|
||||||
self.logger.info("Shutting down plugin operation queue")
|
self.logger.info("Shutting down plugin operation queue")
|
||||||
@@ -378,6 +312,3 @@ class PluginOperationQueue:
|
|||||||
if self._worker_thread and self._worker_thread.is_alive():
|
if self._worker_thread and self._worker_thread.is_alive():
|
||||||
self._worker_thread.join(timeout=5.0)
|
self._worker_thread.join(timeout=5.0)
|
||||||
|
|
||||||
# Save history one last time
|
|
||||||
self._save_history()
|
|
||||||
|
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ Handles plugin module imports, dependency installation, and class instantiation.
|
|||||||
Extracted from PluginManager to improve separation of concerns.
|
Extracted from PluginManager to improve separation of concerns.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
import errno
|
||||||
import importlib
|
import importlib
|
||||||
import importlib.metadata
|
import importlib.metadata
|
||||||
import importlib.util
|
import importlib.util
|
||||||
@@ -184,6 +185,46 @@ def find_trusted_subdir(trusted_dir: str, name: str) -> Optional[str]:
|
|||||||
return None
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def contained_plugin_dir(plugin_dir: Path, plugins_dir: Path) -> Optional[str]:
|
||||||
|
"""``plugin_dir`` rebuilt from an entry enumerated under ``plugins_dir``.
|
||||||
|
|
||||||
|
Returns None when ``plugin_dir`` is not a subdirectory of ``plugins_dir``.
|
||||||
|
Callers derive ``plugin_dir`` from a manifest-declared id, so the path is
|
||||||
|
rebuilt from :func:`find_trusted_subdir`'s answer rather than trusted: a
|
||||||
|
name that came out of ``os.scandir()`` on the trusted root carries no
|
||||||
|
taint, which is a real containment guarantee (and one CodeQL's
|
||||||
|
path-injection query can follow), not a string sanitiser.
|
||||||
|
"""
|
||||||
|
plugin_dir_real = os.path.realpath(str(plugin_dir))
|
||||||
|
plugins_dir_real = os.path.realpath(str(plugins_dir))
|
||||||
|
matched_name = find_trusted_subdir(plugins_dir_real, os.path.basename(plugin_dir_real))
|
||||||
|
if matched_name is None:
|
||||||
|
return None
|
||||||
|
return os.path.join(plugins_dir_real, matched_name)
|
||||||
|
|
||||||
|
|
||||||
|
def requirements_to_install(plugin_dir: str, logger: logging.Logger,
|
||||||
|
label: str) -> Optional[str]:
|
||||||
|
"""The plugin's requirements.txt if pip has work to do, else None.
|
||||||
|
|
||||||
|
None when there is no requirements.txt, when it lists nothing (plugins
|
||||||
|
whose dependencies ship with core often keep an all-comments file), or
|
||||||
|
when every requirement is already installed. Shared by the loader and the
|
||||||
|
store so both skip pip for the same reasons; they differ only in how they
|
||||||
|
run it.
|
||||||
|
"""
|
||||||
|
requirements_file = os.path.join(plugin_dir, "requirements.txt")
|
||||||
|
if not os.path.isfile(requirements_file):
|
||||||
|
return None
|
||||||
|
if not requirements_has_real_deps(requirements_file):
|
||||||
|
logger.debug("requirements.txt for %s has no real dependencies, skipping pip", label)
|
||||||
|
return None
|
||||||
|
if requirements_are_satisfied(requirements_file):
|
||||||
|
logger.debug("Dependencies for %s already satisfied, skipping pip", label)
|
||||||
|
return None
|
||||||
|
return requirements_file
|
||||||
|
|
||||||
|
|
||||||
class PluginLoader:
|
class PluginLoader:
|
||||||
"""Handles plugin module loading and class instantiation."""
|
"""Handles plugin module loading and class instantiation."""
|
||||||
|
|
||||||
@@ -273,43 +314,15 @@ class PluginLoader:
|
|||||||
if not plugin_id:
|
if not plugin_id:
|
||||||
return False
|
return False
|
||||||
|
|
||||||
# Resolve to a canonical absolute path (normalises .. and symlinks)
|
safe_plugin_dir = contained_plugin_dir(plugin_dir, plugins_dir)
|
||||||
plugin_dir_real = os.path.realpath(str(plugin_dir))
|
if safe_plugin_dir is None:
|
||||||
plugins_dir_real = os.path.realpath(str(plugins_dir))
|
|
||||||
requested_name = os.path.basename(plugin_dir_real)
|
|
||||||
|
|
||||||
# Match the requested directory against an entry actually enumerated
|
|
||||||
# from the trusted plugins_dir, and build the path from that entry --
|
|
||||||
# not from requested_name. A name that came out of os.scandir() on a
|
|
||||||
# trusted root carries no taint regardless of what the caller asked
|
|
||||||
# for, so this is a real containment guarantee (an allowlist check
|
|
||||||
# against a trusted source), not a string-sanitisation of untrusted
|
|
||||||
# input that a static analyzer has to trust blindly.
|
|
||||||
matched_name = find_trusted_subdir(plugins_dir_real, requested_name)
|
|
||||||
if matched_name is None:
|
|
||||||
self.logger.error(
|
self.logger.error(
|
||||||
"Plugin directory for %s not found inside plugins dir", plugin_id
|
"Plugin directory for %s not found inside plugins dir", plugin_id
|
||||||
)
|
)
|
||||||
return False
|
return False
|
||||||
|
|
||||||
safe_plugin_dir = os.path.join(plugins_dir_real, matched_name)
|
requirements_file = requirements_to_install(safe_plugin_dir, self.logger, plugin_id)
|
||||||
requirements_file = os.path.join(safe_plugin_dir, "requirements.txt")
|
if requirements_file is None:
|
||||||
|
|
||||||
if not os.path.isfile(requirements_file):
|
|
||||||
return True # No dependencies needed
|
|
||||||
|
|
||||||
if not requirements_has_real_deps(requirements_file):
|
|
||||||
self.logger.debug(
|
|
||||||
"requirements.txt for %s has no real dependencies (comments/blank only), skipping pip",
|
|
||||||
plugin_id
|
|
||||||
)
|
|
||||||
return True
|
|
||||||
|
|
||||||
if requirements_are_satisfied(requirements_file):
|
|
||||||
self.logger.debug(
|
|
||||||
"Dependencies for %s already satisfied in current environment, skipping pip",
|
|
||||||
plugin_id
|
|
||||||
)
|
|
||||||
return True
|
return True
|
||||||
|
|
||||||
try:
|
try:
|
||||||
@@ -348,8 +361,8 @@ class PluginLoader:
|
|||||||
# below).
|
# below).
|
||||||
try:
|
try:
|
||||||
# sys.executable is this process's own interpreter (not
|
# sys.executable is this process's own interpreter (not
|
||||||
# attacker-influenced), and requirements_file is a path
|
# attacker-influenced), and requirements_file is rebuilt
|
||||||
# built internally by find_plugin_directory, never raw
|
# by contained_plugin_dir() from a trusted listing, never raw
|
||||||
# external input.
|
# external input.
|
||||||
retry_result = subprocess.run( # nosec B603 - no shell invoked (list-form argv) # nosemgrep
|
retry_result = subprocess.run( # nosec B603 - no shell invoked (list-form argv) # nosemgrep
|
||||||
[sys.executable, "-m", "pip", "install", "--break-system-packages",
|
[sys.executable, "-m", "pip", "install", "--break-system-packages",
|
||||||
@@ -384,10 +397,10 @@ class PluginLoader:
|
|||||||
except FileNotFoundError:
|
except FileNotFoundError:
|
||||||
self.logger.warning("pip not found. Skipping dependency installation for %s", plugin_id)
|
self.logger.warning("pip not found. Skipping dependency installation for %s", plugin_id)
|
||||||
return True
|
return True
|
||||||
except (BrokenPipeError, OSError) as e:
|
except OSError as e:
|
||||||
# Handle broken pipe errors (errno 32) which can occur during pip downloads
|
# A broken pipe (EPIPE) happens when pip's output pipe closes
|
||||||
# Often caused by network interruptions or output buffer issues
|
# mid-download, usually a network interruption.
|
||||||
if isinstance(e, OSError) and e.errno == 32:
|
if e.errno == errno.EPIPE:
|
||||||
self.logger.error(
|
self.logger.error(
|
||||||
"Broken pipe error during dependency installation for %s. "
|
"Broken pipe error during dependency installation for %s. "
|
||||||
"This usually indicates a network interruption or pip output buffer issue. "
|
"This usually indicates a network interruption or pip output buffer issue. "
|
||||||
@@ -528,7 +541,7 @@ class PluginLoader:
|
|||||||
plugin_id: str,
|
plugin_id: str,
|
||||||
plugin_dir: Path,
|
plugin_dir: Path,
|
||||||
entry_point: str
|
entry_point: str
|
||||||
) -> Optional[Any]:
|
) -> Any:
|
||||||
"""
|
"""
|
||||||
Load a plugin module from file.
|
Load a plugin module from file.
|
||||||
|
|
||||||
@@ -547,7 +560,12 @@ class PluginLoader:
|
|||||||
entry_point: Entry point filename (e.g., 'manager.py')
|
entry_point: Entry point filename (e.g., 'manager.py')
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
Loaded module or None on error
|
The loaded module
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
PluginError: If the plugin id, directory or entry point is
|
||||||
|
invalid. Whatever the module raises while executing
|
||||||
|
propagates unchanged.
|
||||||
"""
|
"""
|
||||||
plugin_id = os.path.basename(plugin_id or '')
|
plugin_id = os.path.basename(plugin_id or '')
|
||||||
if not plugin_id:
|
if not plugin_id:
|
||||||
@@ -782,8 +800,6 @@ class PluginLoader:
|
|||||||
# Load module
|
# Load module
|
||||||
entry_point = manifest.get('entry_point', 'manager.py')
|
entry_point = manifest.get('entry_point', 'manager.py')
|
||||||
module = self.load_module(plugin_id, plugin_dir, entry_point)
|
module = self.load_module(plugin_id, plugin_dir, entry_point)
|
||||||
if module is None:
|
|
||||||
raise PluginError(f"Failed to load module for plugin {plugin_id}", plugin_id=plugin_id)
|
|
||||||
|
|
||||||
# Get plugin class
|
# Get plugin class
|
||||||
class_name = manifest.get('class_name')
|
class_name = manifest.get('class_name')
|
||||||
|
|||||||
@@ -2,7 +2,8 @@
|
|||||||
Plugin Manager
|
Plugin Manager
|
||||||
|
|
||||||
Manages plugin discovery, loading, and lifecycle for the LEDMatrix system.
|
Manages plugin discovery, loading, and lifecycle for the LEDMatrix system.
|
||||||
Handles dynamic plugin loading from the plugins/ directory.
|
Loads plugins from the configured plugins directory
|
||||||
|
(``plugin_system.plugins_directory``, ``plugin-repos/`` by default).
|
||||||
|
|
||||||
API Version: 1.0.0
|
API Version: 1.0.0
|
||||||
"""
|
"""
|
||||||
@@ -40,7 +41,7 @@ class PluginManager:
|
|||||||
Manages plugin discovery, loading, and lifecycle.
|
Manages plugin discovery, loading, and lifecycle.
|
||||||
|
|
||||||
The PluginManager is responsible for:
|
The PluginManager is responsible for:
|
||||||
- Discovering plugins in the plugins/ directory
|
- Discovering plugins in the configured plugins directory
|
||||||
- Loading plugin modules and instantiating plugin classes
|
- Loading plugin modules and instantiating plugin classes
|
||||||
- Managing plugin lifecycle (load, unload, reload)
|
- Managing plugin lifecycle (load, unload, reload)
|
||||||
- Providing access to loaded plugins
|
- Providing access to loaded plugins
|
||||||
@@ -99,10 +100,9 @@ class PluginManager:
|
|||||||
self.plugin_directories: Dict[str, Path] = {}
|
self.plugin_directories: Dict[str, Path] = {}
|
||||||
self.plugin_last_update: Dict[str, float] = {}
|
self.plugin_last_update: Dict[str, float] = {}
|
||||||
|
|
||||||
# Cached data-fetch intervals per plugin_id.
|
# Cached static data-fetch intervals per plugin_id, so the render
|
||||||
# _get_plugin_update_interval falls back to config_manager.get_config()
|
# loop's scheduling tick does not repeat the manifest/config lookup
|
||||||
# (a full dict copy) when the manifest lacks an interval — caching avoids
|
# for every plugin. Cleared on load/unload.
|
||||||
# that copy on every 30-fps tick. Cleared on load/unload.
|
|
||||||
self._update_interval_cache: Dict[str, Optional[float]] = {}
|
self._update_interval_cache: Dict[str, Optional[float]] = {}
|
||||||
|
|
||||||
# Health tracking (optional, set by display_controller if available)
|
# Health tracking (optional, set by display_controller if available)
|
||||||
@@ -110,14 +110,12 @@ class PluginManager:
|
|||||||
self.resource_monitor = None
|
self.resource_monitor = None
|
||||||
|
|
||||||
# --- Asynchronous plugin updates -------------------------------
|
# --- Asynchronous plugin updates -------------------------------
|
||||||
# update() used to run inline in the render loop (execute_update's
|
# Run inline in the render loop, one slow plugin HTTP fetch in
|
||||||
# internal thread.join(timeout=30) blocked it), so one slow plugin
|
# update() freezes scrolling for the whole fetch. Scheduling happens
|
||||||
# HTTP fetch froze scrolling for the whole fetch. Scheduling still
|
# on the render thread (run_scheduled_updates); execution happens on
|
||||||
# happens on the render thread (run_scheduled_updates), but
|
# this single background worker. Per-plugin locks keep a plugin's
|
||||||
# execution moves to this single background worker. Per-plugin
|
# update() and display() mutually exclusive, including across the
|
||||||
# locks keep a plugin's update() and display() mutually exclusive —
|
# post-timeout window.
|
||||||
# today's implicit guarantee, now explicit (and, unlike today,
|
|
||||||
# also held across the post-timeout window).
|
|
||||||
# Kill switch: plugin_system.synchronous_updates: true restores the
|
# Kill switch: plugin_system.synchronous_updates: true restores the
|
||||||
# inline path.
|
# inline path.
|
||||||
self._update_queue: "queue.Queue[Optional[Tuple[str, float]]]" = queue.Queue()
|
self._update_queue: "queue.Queue[Optional[Tuple[str, float]]]" = queue.Queue()
|
||||||
@@ -661,9 +659,15 @@ class PluginManager:
|
|||||||
if not self.unload_plugin(plugin_id):
|
if not self.unload_plugin(plugin_id):
|
||||||
return False
|
return False
|
||||||
|
|
||||||
# Re-discover to get updated manifest
|
# Re-read the manifest so an edit to it takes effect, from the
|
||||||
manifest_path = self.plugins_dir / plugin_id / "manifest.json"
|
# directory discovery found the plugin in: a directory's name need not
|
||||||
if manifest_path.exists():
|
# be the id its manifest declares.
|
||||||
|
with self._discovery_lock:
|
||||||
|
directories = dict(self.plugin_directories)
|
||||||
|
plugin_dir = self.plugin_loader.find_plugin_directory(
|
||||||
|
plugin_id, self.plugins_dir, directories)
|
||||||
|
manifest_path = plugin_dir / "manifest.json" if plugin_dir is not None else None
|
||||||
|
if manifest_path is not None and manifest_path.exists():
|
||||||
try:
|
try:
|
||||||
with open(manifest_path, 'r', encoding='utf-8') as f:
|
with open(manifest_path, 'r', encoding='utf-8') as f:
|
||||||
manifest = json.load(f)
|
manifest = json.load(f)
|
||||||
@@ -881,11 +885,12 @@ class PluginManager:
|
|||||||
updating, since a scheduler that propagates a plugin bug stops every
|
updating, since a scheduler that propagates a plugin bug stops every
|
||||||
other plugin too.
|
other plugin too.
|
||||||
|
|
||||||
The static result is cached per plugin_id after the first lookup to
|
The static result is cached per plugin_id after the first lookup, so
|
||||||
avoid calling config_manager.get_config() — which returns a full dict
|
the manifest/config resolution is not repeated on every scheduling
|
||||||
copy — on every tick of the 30-fps display loop. The cache is
|
tick of the display loop. A change to ``update_interval`` in
|
||||||
invalidated when a plugin is loaded or unloaded. The dynamic hook is
|
config.json therefore takes effect when the plugin is next loaded or
|
||||||
deliberately *not* cached: caching it would defeat its only purpose.
|
unloaded, which clears the cache. The dynamic hook is deliberately
|
||||||
|
*not* cached: caching it would defeat its only purpose.
|
||||||
"""
|
"""
|
||||||
dynamic = self._dynamic_update_interval(plugin_id, plugin_instance)
|
dynamic = self._dynamic_update_interval(plugin_id, plugin_instance)
|
||||||
if dynamic is not None:
|
if dynamic is not None:
|
||||||
|
|||||||
@@ -41,7 +41,6 @@ class PluginStateManager:
|
|||||||
self._state_transition_counts: Dict[str, int] = {}
|
self._state_transition_counts: Dict[str, int] = {}
|
||||||
self._error_info: Dict[str, Dict[str, Any]] = {}
|
self._error_info: Dict[str, Dict[str, Any]] = {}
|
||||||
self._last_update: Dict[str, datetime] = {}
|
self._last_update: Dict[str, datetime] = {}
|
||||||
self._last_display: Dict[str, datetime] = {}
|
|
||||||
|
|
||||||
def _record_transition(self, plugin_id: str) -> None:
|
def _record_transition(self, plugin_id: str) -> None:
|
||||||
"""Count a state transition. Callers must already hold ``_lock``."""
|
"""Count a state transition. Callers must already hold ``_lock``."""
|
||||||
@@ -182,10 +181,6 @@ class PluginStateManager:
|
|||||||
"""Get timestamp of last update() call."""
|
"""Get timestamp of last update() call."""
|
||||||
return self._last_update.get(plugin_id)
|
return self._last_update.get(plugin_id)
|
||||||
|
|
||||||
def get_last_display(self, plugin_id: str) -> Optional[datetime]:
|
|
||||||
"""Get timestamp of last display() call."""
|
|
||||||
return self._last_display.get(plugin_id)
|
|
||||||
|
|
||||||
def get_state_info(self, plugin_id: str) -> Dict[str, Any]:
|
def get_state_info(self, plugin_id: str) -> Dict[str, Any]:
|
||||||
"""
|
"""
|
||||||
Get comprehensive state information for a plugin.
|
Get comprehensive state information for a plugin.
|
||||||
@@ -212,7 +207,6 @@ class PluginStateManager:
|
|||||||
'is_error': self.is_error(plugin_id),
|
'is_error': self.is_error(plugin_id),
|
||||||
'can_execute': self.can_execute(plugin_id),
|
'can_execute': self.can_execute(plugin_id),
|
||||||
'last_update': self.get_last_update(plugin_id),
|
'last_update': self.get_last_update(plugin_id),
|
||||||
'last_display': self.get_last_display(plugin_id),
|
|
||||||
'error_info': self.get_error_info(plugin_id),
|
'error_info': self.get_error_info(plugin_id),
|
||||||
'state_history_count': self._state_transition_counts.get(plugin_id, 0)
|
'state_history_count': self._state_transition_counts.get(plugin_id, 0)
|
||||||
}
|
}
|
||||||
@@ -231,5 +225,4 @@ class PluginStateManager:
|
|||||||
self._state_transition_counts.pop(plugin_id, None)
|
self._state_transition_counts.pop(plugin_id, None)
|
||||||
self._error_info.pop(plugin_id, None)
|
self._error_info.pop(plugin_id, None)
|
||||||
self._last_update.pop(plugin_id, None)
|
self._last_update.pop(plugin_id, None)
|
||||||
self._last_display.pop(plugin_id, None)
|
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,70 @@
|
|||||||
|
"""
|
||||||
|
Repository URL helpers shared by the plugin store and saved repositories.
|
||||||
|
|
||||||
|
One definition of "the same repository URL", of how a GitHub URL maps to
|
||||||
|
``owner/repo``, and of the headers sent to the GitHub API.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from typing import Dict, Optional, Tuple
|
||||||
|
from urllib.parse import urlparse
|
||||||
|
|
||||||
|
#: Hosts whose URLs name a GitHub repository. Matched against
|
||||||
|
#: ``urlparse(url).hostname``, never by substring: a substring test accepts
|
||||||
|
#: ``https://github.com.example.org/...`` as GitHub.
|
||||||
|
GITHUB_HOSTS = frozenset({'github.com', 'www.github.com'})
|
||||||
|
|
||||||
|
#: Sent on every request the store makes to GitHub.
|
||||||
|
USER_AGENT = 'LEDMatrix-Plugin-Manager/1.0'
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_repo_url(url: str) -> str:
|
||||||
|
"""``url`` without surrounding whitespace, trailing slashes or a trailing ``.git``.
|
||||||
|
|
||||||
|
Only a *trailing* ``.git`` is removed. The unanchored
|
||||||
|
``url.replace('.git', '')`` this replaces turned
|
||||||
|
``https://github.com/user/my.github.io`` into ``.../myhub.io``.
|
||||||
|
Case is preserved; compare with :func:`same_repo`.
|
||||||
|
"""
|
||||||
|
url = url.strip().rstrip('/')
|
||||||
|
if url.endswith('.git'):
|
||||||
|
url = url[:-4]
|
||||||
|
return url
|
||||||
|
|
||||||
|
|
||||||
|
def same_repo(url_a: str, url_b: str) -> bool:
|
||||||
|
"""Whether two URLs name the same repository.
|
||||||
|
|
||||||
|
GitHub owner and repository names are case-insensitive, so
|
||||||
|
``ChuckBuilds/LEDMatrix-Plugins`` and ``chuckbuilds/ledmatrix-plugins``
|
||||||
|
are one repository.
|
||||||
|
"""
|
||||||
|
return normalize_repo_url(url_a).lower() == normalize_repo_url(url_b).lower()
|
||||||
|
|
||||||
|
|
||||||
|
def github_owner_repo(url: str) -> Optional[Tuple[str, str]]:
|
||||||
|
"""``(owner, repo)`` for a github.com repository URL, else None.
|
||||||
|
|
||||||
|
The first two path segments, so a URL that points inside the repository
|
||||||
|
(``.../owner/repo/tree/main/plugins/x``) still names ``owner/repo``.
|
||||||
|
"""
|
||||||
|
parsed = urlparse(normalize_repo_url(url))
|
||||||
|
if parsed.hostname not in GITHUB_HOSTS:
|
||||||
|
return None
|
||||||
|
parts = [part for part in parsed.path.split('/') if part]
|
||||||
|
if len(parts) < 2:
|
||||||
|
return None
|
||||||
|
return parts[0], normalize_repo_url(parts[1])
|
||||||
|
|
||||||
|
|
||||||
|
def github_api_headers(token: Optional[str] = None) -> Dict[str, str]:
|
||||||
|
"""Headers for a GitHub REST API request, authenticated when ``token`` is set.
|
||||||
|
|
||||||
|
An authenticated request gets 5000 requests an hour instead of 60.
|
||||||
|
"""
|
||||||
|
headers = {
|
||||||
|
'Accept': 'application/vnd.github.v3+json',
|
||||||
|
'User-Agent': USER_AGENT,
|
||||||
|
}
|
||||||
|
if token:
|
||||||
|
headers['Authorization'] = f'token {token}'
|
||||||
|
return headers
|
||||||
@@ -33,7 +33,14 @@ class ResourceLimits:
|
|||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class ResourceMetrics:
|
class ResourceMetrics:
|
||||||
"""Resource usage metrics for a plugin."""
|
"""Resource usage metrics for a plugin.
|
||||||
|
|
||||||
|
``memory_mb`` is the largest growth in this *process's* resident memory
|
||||||
|
seen across a single monitored call -- a high-water mark, not current
|
||||||
|
usage, and not the plugin's own footprint (another thread allocating
|
||||||
|
during the call counts too). ``cpu_percent`` is the whole process's CPU
|
||||||
|
use since the previous sample.
|
||||||
|
"""
|
||||||
memory_mb: float = 0.0
|
memory_mb: float = 0.0
|
||||||
cpu_percent: float = 0.0
|
cpu_percent: float = 0.0
|
||||||
execution_time: float = 0.0
|
execution_time: float = 0.0
|
||||||
@@ -43,11 +50,6 @@ class ResourceMetrics:
|
|||||||
min_execution_time: float = float('inf')
|
min_execution_time: float = float('inf')
|
||||||
last_update_time: float = field(default_factory=time.time)
|
last_update_time: float = field(default_factory=time.time)
|
||||||
|
|
||||||
def update_average_execution_time(self):
|
|
||||||
"""Update average execution time."""
|
|
||||||
if self.call_count > 0:
|
|
||||||
self.total_execution_time = self.total_execution_time / self.call_count
|
|
||||||
|
|
||||||
|
|
||||||
#: How often a plugin's metrics are written to the cache, in seconds.
|
#: How often a plugin's metrics are written to the cache, in seconds.
|
||||||
#:
|
#:
|
||||||
@@ -287,6 +289,7 @@ class PluginResourceMonitor:
|
|||||||
|
|
||||||
# Calculate execution time
|
# Calculate execution time
|
||||||
execution_time = time.time() - start_time
|
execution_time = time.time() - start_time
|
||||||
|
memory_growth_mb = 0.0
|
||||||
|
|
||||||
# Update metrics
|
# Update metrics
|
||||||
with self._lock:
|
with self._lock:
|
||||||
@@ -302,17 +305,17 @@ class PluginResourceMonitor:
|
|||||||
|
|
||||||
# Update memory and CPU if monitoring enabled
|
# Update memory and CPU if monitoring enabled
|
||||||
if self.enable_monitoring:
|
if self.enable_monitoring:
|
||||||
end_memory = self._get_process_memory_mb()
|
memory_growth_mb = self._get_process_memory_mb() - start_memory
|
||||||
metrics.memory_mb = max(metrics.memory_mb, end_memory - start_memory)
|
metrics.memory_mb = max(metrics.memory_mb, memory_growth_mb)
|
||||||
# CPU is harder to measure per-call, so we track it separately
|
# CPU is harder to measure per-call, so we track it separately
|
||||||
metrics.cpu_percent = self._get_process_cpu_percent()
|
metrics.cpu_percent = self._get_process_cpu_percent()
|
||||||
|
|
||||||
# Persist metrics, at most once per interval per plugin.
|
# Persist metrics, at most once per interval per plugin.
|
||||||
self._persist_metrics(plugin_id, metrics)
|
self._persist_metrics(plugin_id, metrics)
|
||||||
|
|
||||||
# Check limits
|
|
||||||
if limits:
|
if limits:
|
||||||
self._check_limits(plugin_id, metrics, limits, execution_time)
|
self._check_limits(plugin_id, metrics, limits, execution_time,
|
||||||
|
memory_growth_mb)
|
||||||
|
|
||||||
return result
|
return result
|
||||||
|
|
||||||
@@ -327,8 +330,16 @@ class PluginResourceMonitor:
|
|||||||
raise
|
raise
|
||||||
|
|
||||||
def _check_limits(self, plugin_id: str, metrics: ResourceMetrics,
|
def _check_limits(self, plugin_id: str, metrics: ResourceMetrics,
|
||||||
limits: ResourceLimits, execution_time: float) -> None:
|
limits: ResourceLimits, execution_time: float,
|
||||||
"""Check if plugin has exceeded resource limits."""
|
memory_growth_mb: float) -> None:
|
||||||
|
"""Raise ResourceLimitExceeded if this call went over a limit.
|
||||||
|
|
||||||
|
Execution time and memory growth are this call's own; CPU is the
|
||||||
|
latest process sample. Judging memory by the stored high-water mark
|
||||||
|
(``metrics.memory_mb``) instead would fail every call after the first
|
||||||
|
expensive one, so the health tracker's circuit breaker would reopen
|
||||||
|
on every recovery probe and the plugin would never update again.
|
||||||
|
"""
|
||||||
warnings = []
|
warnings = []
|
||||||
errors = []
|
errors = []
|
||||||
|
|
||||||
@@ -343,13 +354,13 @@ class PluginResourceMonitor:
|
|||||||
)
|
)
|
||||||
|
|
||||||
# Check memory
|
# Check memory
|
||||||
if limits.max_memory_mb and metrics.memory_mb > limits.max_memory_mb:
|
if limits.max_memory_mb and memory_growth_mb > limits.max_memory_mb:
|
||||||
errors.append(
|
errors.append(
|
||||||
f"Memory usage {metrics.memory_mb:.2f}MB exceeds limit {limits.max_memory_mb:.2f}MB"
|
f"Memory growth {memory_growth_mb:.2f}MB exceeds limit {limits.max_memory_mb:.2f}MB"
|
||||||
)
|
)
|
||||||
elif limits.max_memory_mb and metrics.memory_mb > limits.max_memory_mb * limits.warning_threshold:
|
elif limits.max_memory_mb and memory_growth_mb > limits.max_memory_mb * limits.warning_threshold:
|
||||||
warnings.append(
|
warnings.append(
|
||||||
f"Memory usage {metrics.memory_mb:.2f}MB approaching limit {limits.max_memory_mb:.2f}MB"
|
f"Memory growth {memory_growth_mb:.2f}MB approaching limit {limits.max_memory_mb:.2f}MB"
|
||||||
)
|
)
|
||||||
|
|
||||||
# Check CPU
|
# Check CPU
|
||||||
|
|||||||
@@ -10,6 +10,8 @@ import os
|
|||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import List, Dict, Optional
|
from typing import List, Dict, Optional
|
||||||
|
|
||||||
|
from src.plugin_system.repo_urls import normalize_repo_url
|
||||||
|
|
||||||
|
|
||||||
class SavedRepositoriesManager:
|
class SavedRepositoriesManager:
|
||||||
"""Manages saved GitHub repository URLs."""
|
"""Manages saved GitHub repository URLs."""
|
||||||
@@ -71,18 +73,6 @@ class SavedRepositoriesManager:
|
|||||||
pass
|
pass
|
||||||
return False
|
return False
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _clean_url(repo_url: str) -> str:
|
|
||||||
"""Normalize a repo URL: strip whitespace, trailing slashes, and a
|
|
||||||
trailing ``.git`` suffix ONLY. (The old ``.replace('.git', '')``
|
|
||||||
was an unanchored substring replace that mangled URLs merely
|
|
||||||
containing ``.git``, e.g. ``https://github.com/user/my.github.io``.)
|
|
||||||
"""
|
|
||||||
repo_url = repo_url.strip().rstrip('/')
|
|
||||||
if repo_url.endswith('.git'):
|
|
||||||
repo_url = repo_url[:-4]
|
|
||||||
return repo_url
|
|
||||||
|
|
||||||
def get_all(self) -> List[Dict[str, str]]:
|
def get_all(self) -> List[Dict[str, str]]:
|
||||||
"""Get all saved repositories."""
|
"""Get all saved repositories."""
|
||||||
return self.repositories.copy()
|
return self.repositories.copy()
|
||||||
@@ -98,7 +88,7 @@ class SavedRepositoriesManager:
|
|||||||
Returns:
|
Returns:
|
||||||
True if added successfully
|
True if added successfully
|
||||||
"""
|
"""
|
||||||
repo_url = self._clean_url(repo_url)
|
repo_url = normalize_repo_url(repo_url)
|
||||||
|
|
||||||
# Check if already exists
|
# Check if already exists
|
||||||
for repo in self.repositories:
|
for repo in self.repositories:
|
||||||
@@ -138,7 +128,7 @@ class SavedRepositoriesManager:
|
|||||||
Returns:
|
Returns:
|
||||||
True if removed successfully
|
True if removed successfully
|
||||||
"""
|
"""
|
||||||
repo_url = self._clean_url(repo_url)
|
repo_url = normalize_repo_url(repo_url)
|
||||||
|
|
||||||
previous = self.repositories
|
previous = self.repositories
|
||||||
remaining = [r for r in previous if r.get('url') != repo_url]
|
remaining = [r for r in previous if r.get('url') != repo_url]
|
||||||
@@ -156,7 +146,7 @@ class SavedRepositoriesManager:
|
|||||||
|
|
||||||
def has(self, repo_url: str) -> bool:
|
def has(self, repo_url: str) -> bool:
|
||||||
"""Check if a repository is already saved."""
|
"""Check if a repository is already saved."""
|
||||||
repo_url = self._clean_url(repo_url)
|
repo_url = normalize_repo_url(repo_url)
|
||||||
return any(r.get('url') == repo_url for r in self.repositories)
|
return any(r.get('url') == repo_url for r in self.repositories)
|
||||||
|
|
||||||
def get_registry_repositories(self) -> List[Dict[str, str]]:
|
def get_registry_repositories(self) -> List[Dict[str, str]]:
|
||||||
|
|||||||
@@ -14,6 +14,7 @@ import jsonschema
|
|||||||
from jsonschema import Draft7Validator, ValidationError
|
from jsonschema import Draft7Validator, ValidationError
|
||||||
|
|
||||||
from src.core_config_keys import CORE_CONFIG_KEYS
|
from src.core_config_keys import CORE_CONFIG_KEYS
|
||||||
|
from src.element_style import expand_style_elements
|
||||||
|
|
||||||
|
|
||||||
def _renders_as_object(prop: Dict[str, Any]) -> bool:
|
def _renders_as_object(prop: Dict[str, Any]) -> bool:
|
||||||
@@ -452,11 +453,7 @@ class SchemaManager:
|
|||||||
# full per-element style blocks (font/size/color + layout
|
# full per-element style blocks (font/size/color + layout
|
||||||
# offsets) the web-UI config form renders. No-op for schemas
|
# offsets) the web-UI config form renders. No-op for schemas
|
||||||
# without the declaration; never raises.
|
# without the declaration; never raises.
|
||||||
try:
|
schema = expand_style_elements(schema)
|
||||||
from src.element_style import expand_style_elements
|
|
||||||
schema = expand_style_elements(schema)
|
|
||||||
except ImportError:
|
|
||||||
pass
|
|
||||||
|
|
||||||
# Cache the schema
|
# Cache the schema
|
||||||
self._schema_cache[plugin_id] = schema
|
self._schema_cache[plugin_id] = schema
|
||||||
@@ -643,19 +640,11 @@ class SchemaManager:
|
|||||||
[name for name in CORE_PLUGIN_PROPERTIES if name not in declared]
|
[name for name in CORE_PLUGIN_PROPERTIES if name not in declared]
|
||||||
)
|
)
|
||||||
|
|
||||||
# Create validator with enhanced schema
|
# iter_errors reports every violation, including one ``required``
|
||||||
|
# error per missing field at every depth.
|
||||||
validator = Draft7Validator(enhanced_schema)
|
validator = Draft7Validator(enhanced_schema)
|
||||||
|
|
||||||
# Collect all validation errors
|
|
||||||
for error in validator.iter_errors(config):
|
for error in validator.iter_errors(config):
|
||||||
error_msg = self._format_validation_error(error, plugin_id)
|
errors.append(self._format_validation_error(error, plugin_id))
|
||||||
errors.append(error_msg)
|
|
||||||
|
|
||||||
# Check required fields
|
|
||||||
required_fields = enhanced_schema.get('required', [])
|
|
||||||
for field in required_fields:
|
|
||||||
if field not in config:
|
|
||||||
errors.append(f"Missing required field: '{field}'")
|
|
||||||
|
|
||||||
if errors:
|
if errors:
|
||||||
return False, errors
|
return False, errors
|
||||||
@@ -687,7 +676,15 @@ class SchemaManager:
|
|||||||
field_path = f"'{path}'" if path else "root"
|
field_path = f"'{path}'" if path else "root"
|
||||||
|
|
||||||
if error.validator == 'required':
|
if error.validator == 'required':
|
||||||
missing = error.validator_value
|
# validator_value is the schema's whole ``required`` list; the
|
||||||
|
# error itself is about one field, which jsonschema names only in
|
||||||
|
# its message ("'api_key' is a required property").
|
||||||
|
missing = next(
|
||||||
|
(name for name in error.validator_value
|
||||||
|
if error.message.startswith(f"{name!r} ")),
|
||||||
|
None)
|
||||||
|
if missing is None:
|
||||||
|
return f"Field {field_path}: {error.message}"
|
||||||
return f"Field {field_path}: Missing required property '{missing}'"
|
return f"Field {field_path}: Missing required property '{missing}'"
|
||||||
elif error.validator == 'type':
|
elif error.validator == 'type':
|
||||||
expected = error.validator_value
|
expected = error.validator_value
|
||||||
|
|||||||
@@ -34,7 +34,9 @@ class PluginState:
|
|||||||
version: Optional[str] = None
|
version: Optional[str] = None
|
||||||
installed_at: Optional[datetime] = None
|
installed_at: Optional[datetime] = None
|
||||||
last_updated: Optional[datetime] = None
|
last_updated: Optional[datetime] = None
|
||||||
config_version: int = 1 # For detecting state corruption
|
# Bumped on every update_plugin_state(). Nothing reads it; it stays so
|
||||||
|
# plugin_state.json keeps the shape older releases load with cls(**data).
|
||||||
|
config_version: int = 1
|
||||||
metadata: Dict[str, Any] = None
|
metadata: Dict[str, Any] = None
|
||||||
|
|
||||||
def __post_init__(self):
|
def __post_init__(self):
|
||||||
@@ -100,6 +102,8 @@ class PluginStateManager:
|
|||||||
|
|
||||||
# State storage
|
# State storage
|
||||||
self._states: Dict[str, PluginState] = {}
|
self._states: Dict[str, PluginState] = {}
|
||||||
|
# The file's top-level "version", written back as read. Nothing
|
||||||
|
# checks it yet; it is there for a future format change to branch on.
|
||||||
self._state_version = 1
|
self._state_version = 1
|
||||||
|
|
||||||
# Threading
|
# Threading
|
||||||
@@ -193,7 +197,6 @@ class PluginStateManager:
|
|||||||
current_state.metadata = {}
|
current_state.metadata = {}
|
||||||
current_state.metadata.update(updates['metadata'])
|
current_state.metadata.update(updates['metadata'])
|
||||||
|
|
||||||
# Increment config version
|
|
||||||
current_state.config_version += 1
|
current_state.config_version += 1
|
||||||
|
|
||||||
# Store updated state
|
# Store updated state
|
||||||
|
|||||||
+306
-425
File diff suppressed because it is too large
Load Diff
@@ -299,3 +299,48 @@ class TestDefaultMerging:
|
|||||||
|
|
||||||
assert merged["enabled"] is False
|
assert merged["enabled"] is False
|
||||||
assert merged["display_duration"] == 60
|
assert merged["display_duration"] == 60
|
||||||
|
|
||||||
|
|
||||||
|
class TestMissingRequiredFields:
|
||||||
|
"""One message per missing field, naming that field.
|
||||||
|
|
||||||
|
A manual ``required`` loop used to run after Draft7Validator, which already
|
||||||
|
reports ``required``, so every missing top-level field was listed twice --
|
||||||
|
and the validator's copy printed the schema's whole ``required`` list as
|
||||||
|
if it were the field name.
|
||||||
|
"""
|
||||||
|
|
||||||
|
SCHEMA = {
|
||||||
|
"type": "object",
|
||||||
|
"properties": {
|
||||||
|
"api_key": {"type": "string"},
|
||||||
|
"city": {"type": "string"},
|
||||||
|
"units": {"type": "string"},
|
||||||
|
},
|
||||||
|
"required": ["api_key", "city", "units"],
|
||||||
|
}
|
||||||
|
|
||||||
|
def test_each_missing_field_is_reported_once_by_name(self):
|
||||||
|
ok, errors = SchemaManager().validate_config_against_schema(
|
||||||
|
{"units": "metric"}, self.SCHEMA, "test-plugin")
|
||||||
|
|
||||||
|
assert not ok
|
||||||
|
assert errors == [
|
||||||
|
"Field root: Missing required property 'api_key'",
|
||||||
|
"Field root: Missing required property 'city'",
|
||||||
|
]
|
||||||
|
|
||||||
|
def test_nested_missing_field_names_the_field_and_its_parent(self):
|
||||||
|
schema = {
|
||||||
|
"type": "object",
|
||||||
|
"properties": {"nfl": {
|
||||||
|
"type": "object",
|
||||||
|
"properties": {"api_key": {"type": "string"}},
|
||||||
|
"required": ["api_key"],
|
||||||
|
}},
|
||||||
|
}
|
||||||
|
ok, errors = SchemaManager().validate_config_against_schema(
|
||||||
|
{"nfl": {}}, schema, "test-plugin")
|
||||||
|
|
||||||
|
assert not ok
|
||||||
|
assert errors == ["Field 'nfl': Missing required property 'api_key'"]
|
||||||
|
|||||||
@@ -116,10 +116,14 @@ def test_values_of_the_wrong_type_fall_back_to_usable_defaults(bad):
|
|||||||
assert isinstance(getattr(metrics, field_name), (int, float)), \
|
assert isinstance(getattr(metrics, field_name), (int, float)), \
|
||||||
f"{field_name} came back as {getattr(metrics, field_name)!r}"
|
f"{field_name} came back as {getattr(metrics, field_name)!r}"
|
||||||
|
|
||||||
# The real proof: arithmetic on the loaded metrics must not explode.
|
# The real proof: the arithmetic monitor_call and get_metrics_summary do
|
||||||
|
# on the loaded metrics must not explode.
|
||||||
metrics.call_count += 1
|
metrics.call_count += 1
|
||||||
metrics.total_execution_time += 0.5
|
metrics.total_execution_time += 0.5
|
||||||
metrics.update_average_execution_time()
|
metrics.max_execution_time = max(metrics.max_execution_time, 0.5)
|
||||||
|
metrics.min_execution_time = min(metrics.min_execution_time, 0.5)
|
||||||
|
metrics.memory_mb = max(metrics.memory_mb, 1.0)
|
||||||
|
assert metrics.total_execution_time / metrics.call_count >= 0
|
||||||
|
|
||||||
|
|
||||||
def test_a_numeric_string_is_accepted_rather_than_discarded():
|
def test_a_numeric_string_is_accepted_rather_than_discarded():
|
||||||
|
|||||||
@@ -0,0 +1,36 @@
|
|||||||
|
"""reload_plugin re-reads the manifest from the plugin's actual directory.
|
||||||
|
|
||||||
|
It read ``plugins_dir / plugin_id / manifest.json``, but a plugin directory's
|
||||||
|
name need not be the id its manifest declares -- discovery maps ids to
|
||||||
|
directories for exactly that reason. For such a plugin the path did not exist,
|
||||||
|
the re-read was skipped silently, and the reload kept the stale manifest.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from src.plugin_system.plugin_manager import PluginManager
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def pm_with_renamed_dir(tmp_path):
|
||||||
|
plugins_dir = tmp_path / "plugins"
|
||||||
|
plugin_dir = plugins_dir / "stock-ticker-v2"
|
||||||
|
plugin_dir.mkdir(parents=True)
|
||||||
|
manifest_path = plugin_dir / "manifest.json"
|
||||||
|
manifest_path.write_text(json.dumps({"id": "stocks", "version": "1.0.0"}))
|
||||||
|
pm = PluginManager(plugins_dir=str(plugins_dir))
|
||||||
|
assert pm.discover_plugins() == ["stocks"]
|
||||||
|
return pm, manifest_path
|
||||||
|
|
||||||
|
|
||||||
|
def test_reload_picks_up_an_edited_manifest(pm_with_renamed_dir, monkeypatch):
|
||||||
|
pm, manifest_path = pm_with_renamed_dir
|
||||||
|
manifest_path.write_text(json.dumps({"id": "stocks", "version": "2.0.0"}))
|
||||||
|
loaded = []
|
||||||
|
monkeypatch.setattr(pm, "load_plugin", lambda pid: loaded.append(pid) or True)
|
||||||
|
|
||||||
|
assert pm.reload_plugin("stocks") is True
|
||||||
|
assert pm.plugin_manifests["stocks"]["version"] == "2.0.0"
|
||||||
|
assert loaded == ["stocks"]
|
||||||
@@ -115,5 +115,22 @@ def test_get_state_info_is_a_consistent_snapshot():
|
|||||||
assert not inconsistent, f"observed a torn snapshot: {inconsistent[:1]}"
|
assert not inconsistent, f"observed a torn snapshot: {inconsistent[:1]}"
|
||||||
|
|
||||||
|
|
||||||
|
def test_state_info_reports_only_what_something_records():
|
||||||
|
"""No field that is always null.
|
||||||
|
|
||||||
|
``last_display`` was reported here, but nothing ever recorded a display()
|
||||||
|
call, so it was null for every plugin. Its only reader is the web process,
|
||||||
|
whose PluginManager never calls display(), so recording it in the display
|
||||||
|
process could not have filled it either.
|
||||||
|
"""
|
||||||
|
manager = PluginStateManager()
|
||||||
|
manager.set_state("clock", PluginState.ENABLED)
|
||||||
|
manager.record_update("clock")
|
||||||
|
|
||||||
|
info = manager.get_state_info("clock")
|
||||||
|
assert "last_display" not in info
|
||||||
|
assert info["last_update"] is not None
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
sys.exit(pytest.main([__file__, "-v"]))
|
sys.exit(pytest.main([__file__, "-v"]))
|
||||||
|
|||||||
@@ -0,0 +1,95 @@
|
|||||||
|
"""Repository URL handling shared by the plugin store and saved repositories.
|
||||||
|
|
||||||
|
The store cleaned URLs with ``url.rstrip('/').replace('.git', '')`` in two
|
||||||
|
places, which removes ``.git`` anywhere in the URL:
|
||||||
|
``https://github.com/user/my.github.io`` became ``.../myhub.io``, so installing
|
||||||
|
or browsing such a repository asked GitHub for one that does not exist.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from unittest.mock import MagicMock
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from src.plugin_system.repo_urls import (
|
||||||
|
github_api_headers, github_owner_repo, normalize_repo_url, same_repo,
|
||||||
|
)
|
||||||
|
from src.plugin_system.store_manager import PluginStoreManager
|
||||||
|
|
||||||
|
PAGES_REPO = "https://github.com/user/my.github.io"
|
||||||
|
|
||||||
|
|
||||||
|
class TestNormalizeRepoUrl:
|
||||||
|
@pytest.mark.parametrize("raw, expected", [
|
||||||
|
(PAGES_REPO, PAGES_REPO),
|
||||||
|
(PAGES_REPO + ".git", PAGES_REPO),
|
||||||
|
("https://github.com/user/repo.git/", "https://github.com/user/repo"),
|
||||||
|
(" https://github.com/user/repo/ ", "https://github.com/user/repo"),
|
||||||
|
])
|
||||||
|
def test_only_a_trailing_dot_git_is_removed(self, raw, expected):
|
||||||
|
assert normalize_repo_url(raw) == expected
|
||||||
|
|
||||||
|
def test_same_repo_ignores_case_and_suffix(self):
|
||||||
|
assert same_repo("https://github.com/Owner/Repo.git",
|
||||||
|
"https://github.com/owner/repo/")
|
||||||
|
assert not same_repo("https://github.com/owner/repo",
|
||||||
|
"https://github.com/owner/other")
|
||||||
|
|
||||||
|
|
||||||
|
class TestGithubOwnerRepo:
|
||||||
|
@pytest.mark.parametrize("url, expected", [
|
||||||
|
(PAGES_REPO + ".git", ("user", "my.github.io")),
|
||||||
|
("https://www.github.com/owner/repo", ("owner", "repo")),
|
||||||
|
("https://github.com/owner/repo/tree/main/plugins/x", ("owner", "repo")),
|
||||||
|
])
|
||||||
|
def test_github_urls(self, url, expected):
|
||||||
|
assert github_owner_repo(url) == expected
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("url", [
|
||||||
|
"https://github.com.example.org/owner/repo",
|
||||||
|
"https://gitlab.com/owner/repo",
|
||||||
|
"https://github.com/owner",
|
||||||
|
"github.com/owner/repo",
|
||||||
|
])
|
||||||
|
def test_anything_else_is_not_a_github_repo(self, url):
|
||||||
|
assert github_owner_repo(url) is None
|
||||||
|
|
||||||
|
def test_headers_carry_the_token_only_when_given(self):
|
||||||
|
assert "Authorization" not in github_api_headers(None)
|
||||||
|
assert github_api_headers("abc")["Authorization"] == "token abc"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def store(tmp_path):
|
||||||
|
return PluginStoreManager(
|
||||||
|
plugins_dir=str(tmp_path / "plugins"),
|
||||||
|
uninstalled_registry_path=str(tmp_path / "uninstalled.json"))
|
||||||
|
|
||||||
|
|
||||||
|
def test_install_from_url_keeps_an_interior_dot_git(store, monkeypatch):
|
||||||
|
cloned_from = []
|
||||||
|
monkeypatch.setattr(store, "_install_via_git",
|
||||||
|
lambda url, *a, **k: cloned_from.append(url))
|
||||||
|
downloaded = []
|
||||||
|
monkeypatch.setattr(store, "_install_via_download",
|
||||||
|
lambda url, *a, **k: downloaded.append(url) or False)
|
||||||
|
|
||||||
|
result = store.install_from_url(PAGES_REPO + ".git")
|
||||||
|
|
||||||
|
assert result["success"] is False
|
||||||
|
assert cloned_from == [PAGES_REPO]
|
||||||
|
assert all(url.startswith(PAGES_REPO + "/archive/") for url in downloaded)
|
||||||
|
|
||||||
|
|
||||||
|
def test_fetch_registry_from_url_asks_for_the_named_repository(store, monkeypatch):
|
||||||
|
requested = []
|
||||||
|
|
||||||
|
def fake_get(url, **kwargs):
|
||||||
|
requested.append(url)
|
||||||
|
return MagicMock(status_code=404)
|
||||||
|
|
||||||
|
monkeypatch.setattr(store, "_http_get_with_retries", fake_get)
|
||||||
|
|
||||||
|
assert store.fetch_registry_from_url(PAGES_REPO) is None
|
||||||
|
assert requested
|
||||||
|
assert all(url.startswith("https://raw.githubusercontent.com/user/my.github.io/")
|
||||||
|
for url in requested)
|
||||||
@@ -93,6 +93,27 @@ class TestResourceLimits:
|
|||||||
with pytest.raises(ResourceLimitExceeded):
|
with pytest.raises(ResourceLimitExceeded):
|
||||||
mon.monitor_call("p", lambda: time.sleep(0.02))
|
mon.monitor_call("p", lambda: time.sleep(0.02))
|
||||||
|
|
||||||
|
def test_memory_limit_judges_each_call_on_its_own_growth(self):
|
||||||
|
"""One expensive call must not fail every call after it.
|
||||||
|
|
||||||
|
The check used to compare the stored high-water mark, which never
|
||||||
|
decreases, so after one call grew memory past the limit every later
|
||||||
|
call raised too and the plugin never updated again.
|
||||||
|
"""
|
||||||
|
mon = PluginResourceMonitor(_cache(), enable_monitoring=False)
|
||||||
|
mon.enable_monitoring = True # measure without needing psutil
|
||||||
|
readings = iter([100.0, 200.0, # first call grows RSS by 100 MB
|
||||||
|
200.0, 201.0]) # second call grows it by 1 MB
|
||||||
|
mon._get_process_memory_mb = lambda: next(readings)
|
||||||
|
mon._get_process_cpu_percent = lambda: 0.0
|
||||||
|
mon.set_limits("p", ResourceLimits(max_memory_mb=50))
|
||||||
|
|
||||||
|
with pytest.raises(ResourceLimitExceeded):
|
||||||
|
mon.monitor_call("p", lambda: None)
|
||||||
|
assert mon.monitor_call("p", lambda: "ok") == "ok"
|
||||||
|
# The high-water mark is still reported.
|
||||||
|
assert mon.get_metrics("p").memory_mb == 100.0
|
||||||
|
|
||||||
def test_reset_metrics_clears_counts(self):
|
def test_reset_metrics_clears_counts(self):
|
||||||
cache = _cache()
|
cache = _cache()
|
||||||
mon = PluginResourceMonitor(cache, enable_monitoring=False)
|
mon = PluginResourceMonitor(cache, enable_monitoring=False)
|
||||||
|
|||||||
@@ -5,7 +5,7 @@ SavedRepositoriesManager contract.
|
|||||||
Covers: the three accepted on-disk load shapes (bare list, wrapped
|
Covers: the three accepted on-disk load shapes (bare list, wrapped
|
||||||
{"repositories": [...]}, anything else -> []) and that saves always write
|
{"repositories": [...]}, anything else -> []) and that saves always write
|
||||||
the bare-list form; add/remove/has round trips through a fresh manager;
|
the bare-list form; add/remove/has round trips through a fresh manager;
|
||||||
URL normalization post-fix (_clean_url strips only a TRAILING '.git' after
|
URL normalization post-fix (normalize_repo_url strips only a TRAILING '.git' after
|
||||||
trailing slashes — the old unanchored .replace('.git', '') mangled URLs
|
trailing slashes — the old unanchored .replace('.git', '') mangled URLs
|
||||||
like my.github.io); name derivation and registry-vs-single type
|
like my.github.io); name derivation and registry-vs-single type
|
||||||
classification (the ledmatrix-plugins check is lowercased, the
|
classification (the ledmatrix-plugins check is lowercased, the
|
||||||
|
|||||||
@@ -0,0 +1,77 @@
|
|||||||
|
"""A repository whose only branch is neither main nor master still installs.
|
||||||
|
|
||||||
|
_install_via_git tries the candidate branches, then the repository's default
|
||||||
|
branch -- but it returned None both for "every clone failed" and for "the
|
||||||
|
default-branch clone succeeded". install_from_url took the None as failure,
|
||||||
|
fell through to the archive download of main/master (which does not exist),
|
||||||
|
and reported "Failed to clone or download repository" for a repository it had
|
||||||
|
just cloned.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import shutil
|
||||||
|
import subprocess
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from src.plugin_system.store_manager import PluginStoreManager
|
||||||
|
|
||||||
|
pytestmark = pytest.mark.skipif(shutil.which("git") is None, reason="git not installed")
|
||||||
|
|
||||||
|
MANIFEST = {
|
||||||
|
"id": "develop-only", "name": "Develop Only", "class_name": "P",
|
||||||
|
"display_modes": ["develop_only"], "version": "1.0.0",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _git(*args, cwd):
|
||||||
|
subprocess.run(
|
||||||
|
["git", "-c", "user.name=t", "-c", "user.email=t@example.invalid", *args],
|
||||||
|
cwd=cwd, check=True, capture_output=True)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def develop_only_repo(tmp_path):
|
||||||
|
repo = tmp_path / "upstream"
|
||||||
|
repo.mkdir()
|
||||||
|
_git("init", "-q", "-b", "develop", cwd=repo)
|
||||||
|
(repo / "manifest.json").write_text(json.dumps(MANIFEST))
|
||||||
|
(repo / "manager.py").write_text("class P: pass\n")
|
||||||
|
_git("add", ".", cwd=repo)
|
||||||
|
_git("commit", "-q", "-m", "init", cwd=repo)
|
||||||
|
return repo.as_uri()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def store(tmp_path, monkeypatch):
|
||||||
|
mgr = PluginStoreManager(
|
||||||
|
plugins_dir=str(tmp_path / "plugins"),
|
||||||
|
uninstalled_registry_path=str(tmp_path / "uninstalled.json"))
|
||||||
|
monkeypatch.setattr(mgr, "_install_dependencies", lambda *a, **k: True)
|
||||||
|
downloads = []
|
||||||
|
monkeypatch.setattr(mgr, "_install_via_download",
|
||||||
|
lambda url, *a, **k: downloads.append(url) or False)
|
||||||
|
mgr.downloads = downloads
|
||||||
|
return mgr
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_default_branch_clone_reports_its_branch(store, develop_only_repo, tmp_path):
|
||||||
|
target = tmp_path / "clone"
|
||||||
|
assert store._install_via_git(develop_only_repo, target, ["main", "master"]) == "develop"
|
||||||
|
assert (target / "manifest.json").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_failed_clone_reports_none(store, tmp_path):
|
||||||
|
missing = (tmp_path / "no-such-repo").as_uri()
|
||||||
|
target = tmp_path / "clone"
|
||||||
|
assert store._install_via_git(missing, target, ["main"]) is None
|
||||||
|
assert not target.exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_install_from_url_installs_a_develop_only_repository(store, develop_only_repo):
|
||||||
|
result = store.install_from_url(develop_only_repo)
|
||||||
|
|
||||||
|
assert result == {"success": True, "plugin_id": "develop-only",
|
||||||
|
"name": "Develop Only", "branch": "develop"}
|
||||||
|
assert (store.plugins_dir / "develop-only" / "manifest.json").exists()
|
||||||
|
assert store.downloads == []
|
||||||
@@ -0,0 +1,62 @@
|
|||||||
|
"""update_plugin must not borrow the enclosing LEDMatrix checkout's remote.
|
||||||
|
|
||||||
|
Plugins live in ``plugin-repos/`` inside the LEDMatrix git checkout. For a
|
||||||
|
plugin installed from a ZIP (no ``.git`` of its own), ``git -C <plugin>``
|
||||||
|
walks up to the LEDMatrix repository, and ``git config --local --get
|
||||||
|
remote.origin.url`` answers with LEDMatrix's own URL. update_plugin then tried
|
||||||
|
to "reinstall" the plugin from the LEDMatrix repository.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import shutil
|
||||||
|
import subprocess
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from src.plugin_system.store_manager import PluginStoreManager
|
||||||
|
|
||||||
|
pytestmark = pytest.mark.skipif(shutil.which("git") is None, reason="git not installed")
|
||||||
|
|
||||||
|
PLUGIN_ID = "zip-installed"
|
||||||
|
PARENT_REMOTE = "https://github.com/example/LEDMatrix"
|
||||||
|
|
||||||
|
|
||||||
|
def _git(*args, cwd):
|
||||||
|
subprocess.run(["git", *args], cwd=cwd, check=True, capture_output=True)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def store_inside_checkout(tmp_path):
|
||||||
|
checkout = tmp_path / "LEDMatrix"
|
||||||
|
checkout.mkdir()
|
||||||
|
_git("init", "-q", cwd=checkout)
|
||||||
|
_git("remote", "add", "origin", PARENT_REMOTE, cwd=checkout)
|
||||||
|
|
||||||
|
plugins_dir = checkout / "plugin-repos"
|
||||||
|
plugin_dir = plugins_dir / PLUGIN_ID
|
||||||
|
plugin_dir.mkdir(parents=True)
|
||||||
|
(plugin_dir / "manifest.json").write_text(json.dumps(
|
||||||
|
{"id": PLUGIN_ID, "name": "Zip", "version": "1.0.0"}))
|
||||||
|
|
||||||
|
store = PluginStoreManager(
|
||||||
|
plugins_dir=str(plugins_dir),
|
||||||
|
uninstalled_registry_path=str(tmp_path / "uninstalled.json"))
|
||||||
|
return store, plugin_dir
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_plugin_without_its_own_git_has_no_remote(store_inside_checkout, monkeypatch):
|
||||||
|
store, plugin_dir = store_inside_checkout
|
||||||
|
# The premise: git itself does report the parent's remote here.
|
||||||
|
parent_view = subprocess.run(
|
||||||
|
["git", "-C", str(plugin_dir), "config", "--local", "--get", "remote.origin.url"],
|
||||||
|
capture_output=True, text=True)
|
||||||
|
assert parent_view.stdout.strip() == PARENT_REMOTE
|
||||||
|
|
||||||
|
monkeypatch.setattr(store, "fetch_registry", lambda *a, **k: {"plugins": []})
|
||||||
|
monkeypatch.setattr(store, "get_plugin_info", lambda *a, **k: None)
|
||||||
|
install_calls = []
|
||||||
|
monkeypatch.setattr(store, "install_from_url",
|
||||||
|
lambda *a, **k: install_calls.append((a, k)) or {"success": True})
|
||||||
|
|
||||||
|
assert store.update_plugin(PLUGIN_ID) is False
|
||||||
|
assert install_calls == []
|
||||||
@@ -21,10 +21,7 @@ class TestPluginOperationsIntegration(unittest.TestCase):
|
|||||||
self.temp_dir = Path(tempfile.mkdtemp())
|
self.temp_dir = Path(tempfile.mkdtemp())
|
||||||
|
|
||||||
# Initialize components
|
# Initialize components
|
||||||
self.operation_queue = PluginOperationQueue(
|
self.operation_queue = PluginOperationQueue(max_history=100)
|
||||||
history_file=str(self.temp_dir / "operations.json"),
|
|
||||||
max_history=100
|
|
||||||
)
|
|
||||||
|
|
||||||
self.state_manager = PluginStateManager(
|
self.state_manager = PluginStateManager(
|
||||||
state_file=str(self.temp_dir / "state.json"),
|
state_file=str(self.temp_dir / "state.json"),
|
||||||
|
|||||||
@@ -144,12 +144,7 @@ schema_manager = SchemaManager(
|
|||||||
)
|
)
|
||||||
|
|
||||||
# Initialize operation queue for plugin operations
|
# Initialize operation queue for plugin operations
|
||||||
# Use lazy_load=True to defer file loading until first use (improves startup time)
|
operation_queue = PluginOperationQueue(max_history=500)
|
||||||
operation_queue = PluginOperationQueue(
|
|
||||||
history_file=str(project_root / "data" / "plugin_operations.json"),
|
|
||||||
max_history=500,
|
|
||||||
lazy_load=True
|
|
||||||
)
|
|
||||||
|
|
||||||
# Initialize plugin state manager
|
# Initialize plugin state manager
|
||||||
# Use lazy_load=True to defer file loading until first use (improves startup time)
|
# Use lazy_load=True to defer file loading until first use (improves startup time)
|
||||||
|
|||||||
Reference in New Issue
Block a user