rick/LEDMatrix
1
0
Fork 0
mirror of https://github.com/ChuckBuilds/LEDMatrix.git synced 2026-10-04 14:25:08 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(plugins): store and plugin-manager bugs; tidy src/plugin_system (#635)

Browse Source 
* fix(store): don't read a ZIP-installed plugin's remote from the LEDMatrix repo

update_plugin looked up remote.origin.url with `git -C <plugin> config
--local` for plugins that are not git checkouts. Under plugin-repos/ git
walks up to the enclosing LEDMatrix repository, so the lookup returned
LEDMatrix's own URL and a plugin missing from the registry was
"reinstalled" from the LEDMatrix repo. Only ask git when the plugin
directory has its own .git, the test _get_local_git_info already uses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(schema): report each missing required field once, by name

validate_config_against_schema ran its own required-fields loop after
Draft7Validator.iter_errors, which already yields one `required` error
per missing field, so every missing top-level field was listed twice.
The validator's copy also printed the schema's whole `required` list
("Missing required property '['api_key', 'city']'") instead of the field.
Drop the loop and take the field name from the error itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(store): stop mangling repository URLs that contain ".git"

install_from_url and fetch_registry_from_url cleaned URLs with
`rstrip('/').replace('.git', '')`, which removes ".git" anywhere:
https://github.com/user/my.github.io became .../myhub.io, so installing
or browsing that repository asked GitHub for one that does not exist.

Add src/plugin_system/repo_urls.py with one anchored normalize_repo_url(),
same_repo() for comparisons, github_owner_repo() and github_api_headers(),
and use them for the five copies of the owner/repo parsing and GitHub
headers in the store and for saved repositories. GitHub URLs are now
recognised by urlparse().hostname everywhere: _get_latest_commit_info
used a substring test, and _install_from_monorepo_api parsed any host.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(store): install a repository whose only branch is not main/master

_install_via_git returned None both when every clone failed and when the
last-resort clone of the repository's default branch succeeded.
_install_plugin_impl papered over it with `and not plugin_path.exists()`;
install_from_url did not, so a repository whose only branch is e.g.
`develop` was cloned, then treated as a failure, then "downloaded" from
main/master archives that do not exist.

After a default-branch clone, return the branch the clone checked out
(read from .git/HEAD), so None means failure and nothing else, and give
both callers the same `branch_used is None` fallback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(plugins): judge the memory limit on each call's own growth

monitor_call stores `metrics.memory_mb = max(previous, growth)`, and
_check_limits compared that high-water mark with max_memory_mb. It never
decreases, so once one update() grew the process past the limit every
later call raised ResourceLimitExceeded and the circuit breaker kept
reopening. Pass the call's own RSS growth to _check_limits; keep the
high-water mark for reporting and document what it measures.

Remove ResourceMetrics.update_average_execution_time: nothing called it,
and it overwrote the running total with the average.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(plugins): reload_plugin re-reads the manifest from the discovered directory

reload_plugin read `plugins_dir / plugin_id / "manifest.json"`, ignoring
the discovery map and the plugin_dirs rules. For a plugin whose
directory name differs from its manifest id the path did not exist, the
re-read was skipped without a word, and the reload kept the stale
manifest. Resolve the directory with find_plugin_directory, as
load_plugin does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(plugins): drop the always-null last_display from plugin state info

PluginStateManager reported `last_display` from `_last_display`, which
nothing ever wrote, so it was null for every plugin. Recording it in
PluginExecutor.execute_display would not help: get_state_info's only
reader is the web process, whose PluginManager never calls display().
Remove the field, its dict and get_last_display() (no caller in core,
the web UI or the plugin monorepo).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(store): share the rollback and requirements helpers, drop dead code

- install_plugin and _reinstall_with_rollback set aside, discard and
  restore the old copy through _set_aside/_discard_backup/_restore_backup
  instead of two copies of the same blocks.
- The loader and the store run the same pre-pip checks through
  contained_plugin_dir() and requirements_to_install() in plugin_loader.
  They still invoke pip differently (sys.executable -m pip vs. the sudo
  wrapper). `except (BrokenPipeError, OSError)` + `isinstance(e, OSError)`
  becomes `except OSError` checking errno.EPIPE.
- load_module never returns None, so load_plugin's check is gone and the
  docstring says what it raises.
- Remove the always-true JSONSCHEMA_AVAILABLE, the inline re-imports of
  re and permission_utils, the fake status_result object nobody reads,
  hasattr(git_error, 'cmd'), a redundant "merge conflict" test and
  `import traceback` (exc_info=True does it).
- Correct comments: install_from_url names the directory for the
  caller's id when given (not always the manifest id), _get_local_git_info
  saves one git subprocess (not four), _enrich calls two helpers,
  search_plugins documents all its arguments, _find_plugin_path states
  its behaviour instead of a TODO, and history narration is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(plugins): tidy base_plugin, correct plugin_manager/state comments

- base_plugin: drop the unused `import logging`; get_display_duration
  runs the instance value and the config value through one
  _positive_seconds() helper instead of two copies of the coercion; the
  'static'/'none'/fallback branches of get_vegas_display_mode, which all
  returned FIXED_SEGMENT, are one; fix the mis-indented validate_config
  example; say that get_supported_vegas_modes/get_vegas_segment_width
  are not consulted by core (kept, plugins override them).
- schema_manager: import expand_style_elements normally rather than
  swallowing an ImportError of a core module.
- plugin_manager: the plugins directory is the configured one
  (plugin-repos/ by default), not plugins/; get_config() returns the live
  dict, not a copy, so the interval cache comments say what it saves.
- state_manager: config_version and the file version are not used to
  detect corruption; say what they are.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(plugins): stop writing data/plugin_operations.json

PluginOperationQueue wrote its finished-operation history to
data/plugin_operations.json after every operation, and read it back only
into its own in-memory list, which only get_operation_history() exposes
-- and nothing calls that. The operation-history endpoint reads
OperationHistory (data/operation_history.json). No code in src/,
web_interface/, scripts/ or test/ reads the file.

Drop the history_file/lazy_load parameters and the load/save code; the
bounded in-memory history stays. web_interface/app.py and the
integration test stop passing the removed arguments. An existing
data/plugin_operations.json is left in place (data/* is gitignored).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(changelog): plugin-system

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Chuck
2026-09-24 17:32:02 -04:00
committed by GitHub
co-authored by Claude Opus 5.5
parent 3967a6cffc
commit b11bcfa204
24 changed files with 958 additions and 756 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/plugin_system/base_plugin.py
+51 -104
View File
@@ -11,7 +11,6 @@ Stability: Stable - maintains backward compatibility
from abc import ABC, abstractmethod
from enum import Enum
from typing import Dict, Any, Optional, List
import logging
import os
import sys
from src.logging_config import get_logger
@@ -511,104 +510,53 @@ class BasePlugin(ABC):
"""
Get the display duration for this plugin instance.
Automatically detects duration from:
1. self.display_duration instance variable (if exists)
2. self.config.get("display_duration", 15.0) (fallback)
Uses, in order, the first positive number among:
1. ``self.display_duration`` (a common pattern in scoreboard plugins)
2. ``self.config["display_duration"]``
3. 15.0
Can be overridden by plugins to provide dynamic durations based
on content (e.g., longer duration for more complex displays).
Numeric strings count as numbers. Can be overridden by plugins to
provide dynamic durations based on content (e.g., longer duration for
more complex displays).
Returns:
Duration in seconds to display this plugin's content
"""
# Check for instance variable first (common pattern in scoreboard plugins)
if hasattr(self, 'display_duration'):
try:
duration = getattr(self, 'display_duration')
# Handle None case
if duration is None:
pass # Fall through to config
# Try to convert to float if it's a number or numeric string.
# bool is excluded: it's an int subclass, and True would
# otherwise read as a 1-second duration.
elif isinstance(duration, (int, float)) and not isinstance(duration, bool):
if duration > 0:
return float(duration)
else:
self.logger.debug(
"display_duration instance variable is non-positive (%s), using config fallback",
duration
)
# Try converting string representations of numbers
elif isinstance(duration, str):
try:
duration_float = float(duration)
if duration_float > 0:
return duration_float
else:
self.logger.debug(
"display_duration string value is non-positive (%s), using config fallback",
duration
)
except (ValueError, TypeError):
self.logger.warning(
"display_duration instance variable has invalid string value '%s', using config fallback",
duration
)
else:
self.logger.warning(
"display_duration instance variable has unexpected type %s (value: %s), using config fallback",
type(duration).__name__, duration
)
except (TypeError, ValueError, AttributeError) as e:
self.logger.warning(
"Error reading display_duration instance variable: %s, using config fallback",
e
)
# Fall back to config
config_duration = self.config.get("display_duration", 15.0)
try:
# Ensure config value is also a valid float (bool excluded — an
# int subclass that would otherwise read True as 1 second)
if isinstance(config_duration, (int, float)) and not isinstance(config_duration, bool):
if config_duration > 0:
return float(config_duration)
else:
self.logger.debug(
"Config display_duration is non-positive (%s), using default 15.0",
config_duration
)
return 15.0
elif isinstance(config_duration, str):
try:
duration_float = float(config_duration)
if duration_float > 0:
return duration_float
else:
self.logger.debug(
"Config display_duration string is non-positive (%s), using default 15.0",
config_duration
)
return 15.0
except ValueError:
self.logger.warning(
"Config display_duration has invalid string value '%s', using default 15.0",
config_duration
)
return 15.0
else:
self.logger.warning(
"Config display_duration has unexpected type %s (value: %s), using default 15.0",
type(config_duration).__name__, config_duration
)
except (ValueError, TypeError) as e:
duration = getattr(self, 'display_duration', None)
except (TypeError, ValueError, AttributeError) as e:
# A plugin may define display_duration as a property that raises.
self.logger.warning(
"Error processing config display_duration: %s, using default 15.0",
e
)
"Error reading display_duration instance variable: %s, using config fallback", e)
duration = None
if duration is not None:
seconds = self._positive_seconds(duration, "display_duration instance variable")
if seconds is not None:
return seconds
return 15.0
seconds = self._positive_seconds(
self.config.get("display_duration", 15.0), "config display_duration")
return seconds if seconds is not None else 15.0
def _positive_seconds(self, value: Any, source: str) -> Optional[float]:
"""``value`` as a positive float, or None (with a log line) if it is not one.
bool is rejected although it is an int subclass: True would otherwise
read as a 1-second duration.
"""
if isinstance(value, bool) or not isinstance(value, (int, float, str)):
self.logger.warning("%s has unexpected type %s (value: %s), ignoring it",
source, type(value).__name__, value)
return None
try:
seconds = float(value)
except ValueError:
self.logger.warning("%s has invalid value %r, ignoring it", source, value)
return None
if seconds > 0:
return seconds
self.logger.debug("%s is non-positive (%s), ignoring it", source, value)
return None
# ---------------------------------------------------------------------
# Dynamic duration support hooks
@@ -926,25 +874,20 @@ class BasePlugin(ABC):
config_mode, self.plugin_id
)
# Fall back to mapping legacy content_type
content_type = self.get_vegas_content_type()
if content_type == 'multi':
# Fall back to mapping legacy content_type. 'none' (excluded from
# Vegas) also maps to FIXED_SEGMENT: exclusion is decided by checking
# get_vegas_content_type() separately.
if self.get_vegas_content_type() == 'multi':
return VegasDisplayMode.SCROLL
elif content_type == 'static':
return VegasDisplayMode.FIXED_SEGMENT
elif content_type == 'none':
# 'none' means excluded - return FIXED_SEGMENT as default
# The exclusion is handled by checking get_vegas_content_type() separately
return VegasDisplayMode.FIXED_SEGMENT
return VegasDisplayMode.FIXED_SEGMENT
def get_supported_vegas_modes(self) -> List[VegasDisplayMode]:
"""
Return list of Vegas display modes this plugin supports.
Used by the web UI to show available mode options for user configuration.
Override to customize which modes are available for this plugin.
Not currently consulted by core: neither Vegas mode nor the web UI
calls it. It is kept, and plugins override it, as the declared set of
modes a future mode picker would offer.
By default:
- 'multi' content type plugins support SCROLL and FIXED_SEGMENT
@@ -972,6 +915,10 @@ class BasePlugin(ABC):
"""
Get the preferred width for this plugin in Vegas FIXED_SEGMENT mode.
Not currently consulted by core: Vegas mode sizes a card from the
``vegas_width_pct`` / ``vegas_scroll.render_width_pct`` settings
(see get_vegas_render_width()). Kept because plugins override it.
Returns the number of panels this plugin should occupy when displayed
as a fixed segment. The actual pixel width is calculated as:
width = panels * single_panel_width
@@ -1025,7 +972,7 @@ class BasePlugin(ABC):
required_fields = ['api_key', 'city']
for field in required_fields:
if field not in self.config:
self.logger.error("Missing required field: %s", field)
self.logger.error("Missing required field: %s", field)
return False
return True
"""
src/plugin_system/operation_queue.py
+9 -78
View File
@@ -9,8 +9,6 @@ import threading
import queue
from typing import Dict, Optional, List, Callable, Any
from datetime import datetime
from pathlib import Path
import json
from src.plugin_system.operation_types import (
PluginOperation, OperationType, OperationStatus
@@ -28,28 +26,22 @@ class PluginOperationQueue:
- Prevents concurrent operations on same plugin
- Operation status tracking
- Operation cancellation
- Operation history
- In-memory history of finished operations
The history is not persisted. The web UI's operation history comes from
OperationHistory (operation_history.py), which has its own file; a copy
written here was never read back by anything.
"""
def __init__(
self,
history_file: Optional[str] = None,
max_history: int = 100,
lazy_load: bool = False
):
def __init__(self, max_history: int = 100):
"""
Initialize operation queue.
Args:
history_file: Optional path to file for persisting operation history
max_history: Maximum number of operations to keep in history
lazy_load: If True, defer loading history file until first access
"""
self.logger = get_logger(__name__)
self.history_file = Path(history_file) if history_file else None
self.max_history = max_history
self._lazy_load = lazy_load
self._history_loaded = False
# Operation tracking
self._operations: Dict[str, PluginOperation] = {}
@@ -62,20 +54,8 @@ class PluginOperationQueue:
self._worker_thread: Optional[threading.Thread] = None
self._stop_event = threading.Event()
# Load history from file if it exists (unless lazy loading)
if not self._lazy_load and self.history_file and self.history_file.exists():
self._load_history()
self._history_loaded = True
# Start worker thread
self._start_worker()
def _ensure_loaded(self) -> None:
"""Ensure history is loaded (for lazy loading)."""
if not self._history_loaded and self.history_file and self.history_file.exists():
self._load_history()
self._history_loaded = True
def enqueue_operation(
self,
operation_type: OperationType,
@@ -139,7 +119,6 @@ class PluginOperationQueue:
Returns:
PluginOperation if found, None otherwise
"""
self._ensure_loaded()
with self._lock:
return self._operations.get(operation_id)
@@ -184,7 +163,6 @@ class PluginOperationQueue:
Returns:
List of operations, sorted by creation time (newest first)
"""
self._ensure_loaded()
with self._lock:
# Sort by creation time (newest first)
history = sorted(
@@ -314,11 +292,7 @@ class PluginOperationQueue:
if self._active_operations[operation.plugin_id].operation_id == operation.operation_id:
del self._active_operations[operation.plugin_id]
# Add to history
self._add_to_history(operation)
# Save history to file
self._save_history()
def _add_to_history(self, operation: PluginOperation) -> None:
"""Add operation to history, maintaining max_history limit."""
@@ -330,46 +304,6 @@ class PluginOperationQueue:
self._operation_history.sort(key=lambda op: op.created_at)
self._operation_history = self._operation_history[-self.max_history:]
def _save_history(self) -> None:
"""Save operation history to file."""
if not self.history_file:
return
try:
with self._lock:
# Convert operations to dicts
history_data = [op.to_dict() for op in self._operation_history]
# Ensure directory exists
self.history_file.parent.mkdir(parents=True, exist_ok=True)
# Write to file
with open(self.history_file, 'w') as f:
json.dump(history_data, f, indent=2)
except Exception as e:
self.logger.warning(f"Error saving operation history: {e}")
def _load_history(self) -> None:
"""Load operation history from file."""
if not self.history_file or not self.history_file.exists():
return
try:
with open(self.history_file, 'r') as f:
history_data = json.load(f)
with self._lock:
self._operation_history = [
PluginOperation.from_dict(op_data)
for op_data in history_data
]
self.logger.info(f"Loaded {len(self._operation_history)} operations from history")
except Exception as e:
self.logger.warning(f"Error loading operation history: {e}")
def shutdown(self) -> None:
"""Shutdown the operation queue and worker thread."""
self.logger.info("Shutting down plugin operation queue")
@@ -377,7 +311,4 @@ class PluginOperationQueue:
if self._worker_thread and self._worker_thread.is_alive():
self._worker_thread.join(timeout=5.0)
# Save history one last time
self._save_history()
src/plugin_system/plugin_loader.py
+59 -43
View File
@@ -5,6 +5,7 @@ Handles plugin module imports, dependency installation, and class instantiation.
Extracted from PluginManager to improve separation of concerns.
"""
import errno
import importlib
import importlib.metadata
import importlib.util
@@ -184,6 +185,46 @@ def find_trusted_subdir(trusted_dir: str, name: str) -> Optional[str]:
return None
def contained_plugin_dir(plugin_dir: Path, plugins_dir: Path) -> Optional[str]:
"""``plugin_dir`` rebuilt from an entry enumerated under ``plugins_dir``.
Returns None when ``plugin_dir`` is not a subdirectory of ``plugins_dir``.
Callers derive ``plugin_dir`` from a manifest-declared id, so the path is
rebuilt from :func:`find_trusted_subdir`'s answer rather than trusted: a
name that came out of ``os.scandir()`` on the trusted root carries no
taint, which is a real containment guarantee (and one CodeQL's
path-injection query can follow), not a string sanitiser.
"""
plugin_dir_real = os.path.realpath(str(plugin_dir))
plugins_dir_real = os.path.realpath(str(plugins_dir))
matched_name = find_trusted_subdir(plugins_dir_real, os.path.basename(plugin_dir_real))
if matched_name is None:
return None
return os.path.join(plugins_dir_real, matched_name)
def requirements_to_install(plugin_dir: str, logger: logging.Logger,
label: str) -> Optional[str]:
"""The plugin's requirements.txt if pip has work to do, else None.
None when there is no requirements.txt, when it lists nothing (plugins
whose dependencies ship with core often keep an all-comments file), or
when every requirement is already installed. Shared by the loader and the
store so both skip pip for the same reasons; they differ only in how they
run it.
"""
requirements_file = os.path.join(plugin_dir, "requirements.txt")
if not os.path.isfile(requirements_file):
return None
if not requirements_has_real_deps(requirements_file):
logger.debug("requirements.txt for %s has no real dependencies, skipping pip", label)
return None
if requirements_are_satisfied(requirements_file):
logger.debug("Dependencies for %s already satisfied, skipping pip", label)
return None
return requirements_file
class PluginLoader:
"""Handles plugin module loading and class instantiation."""
@@ -273,43 +314,15 @@ class PluginLoader:
if not plugin_id:
return False
# Resolve to a canonical absolute path (normalises .. and symlinks)
plugin_dir_real = os.path.realpath(str(plugin_dir))
plugins_dir_real = os.path.realpath(str(plugins_dir))
requested_name = os.path.basename(plugin_dir_real)
# Match the requested directory against an entry actually enumerated
# from the trusted plugins_dir, and build the path from that entry --
# not from requested_name. A name that came out of os.scandir() on a
# trusted root carries no taint regardless of what the caller asked
# for, so this is a real containment guarantee (an allowlist check
# against a trusted source), not a string-sanitisation of untrusted
# input that a static analyzer has to trust blindly.
matched_name = find_trusted_subdir(plugins_dir_real, requested_name)
if matched_name is None:
safe_plugin_dir = contained_plugin_dir(plugin_dir, plugins_dir)
if safe_plugin_dir is None:
self.logger.error(
"Plugin directory for %s not found inside plugins dir", plugin_id
)
return False
safe_plugin_dir = os.path.join(plugins_dir_real, matched_name)
requirements_file = os.path.join(safe_plugin_dir, "requirements.txt")
if not os.path.isfile(requirements_file):
return True # No dependencies needed
if not requirements_has_real_deps(requirements_file):
self.logger.debug(
"requirements.txt for %s has no real dependencies (comments/blank only), skipping pip",
plugin_id
)
return True
if requirements_are_satisfied(requirements_file):
self.logger.debug(
"Dependencies for %s already satisfied in current environment, skipping pip",
plugin_id
)
requirements_file = requirements_to_install(safe_plugin_dir, self.logger, plugin_id)
if requirements_file is None:
return True
try:
@@ -348,8 +361,8 @@ class PluginLoader:
# below).
try:
# sys.executable is this process's own interpreter (not
# attacker-influenced), and requirements_file is a path
# built internally by find_plugin_directory, never raw
# attacker-influenced), and requirements_file is rebuilt
# by contained_plugin_dir() from a trusted listing, never raw
# external input.
retry_result = subprocess.run( # nosec B603 - no shell invoked (list-form argv) # nosemgrep
[sys.executable, "-m", "pip", "install", "--break-system-packages",
@@ -384,10 +397,10 @@ class PluginLoader:
except FileNotFoundError:
self.logger.warning("pip not found. Skipping dependency installation for %s", plugin_id)
return True
except (BrokenPipeError, OSError) as e:
# Handle broken pipe errors (errno 32) which can occur during pip downloads
# Often caused by network interruptions or output buffer issues
if isinstance(e, OSError) and e.errno == 32:
except OSError as e:
# A broken pipe (EPIPE) happens when pip's output pipe closes
# mid-download, usually a network interruption.
if e.errno == errno.EPIPE:
self.logger.error(
"Broken pipe error during dependency installation for %s. "
"This usually indicates a network interruption or pip output buffer issue. "
@@ -528,7 +541,7 @@ class PluginLoader:
plugin_id: str,
plugin_dir: Path,
entry_point: str
) -> Optional[Any]:
) -> Any:
"""
Load a plugin module from file.
@@ -547,7 +560,12 @@ class PluginLoader:
entry_point: Entry point filename (e.g., 'manager.py')
Returns:
Loaded module or None on error
The loaded module
Raises:
PluginError: If the plugin id, directory or entry point is
invalid. Whatever the module raises while executing
propagates unchanged.
"""
plugin_id = os.path.basename(plugin_id or '')
if not plugin_id:
@@ -782,9 +800,7 @@ class PluginLoader:
# Load module
entry_point = manifest.get('entry_point', 'manager.py')
module = self.load_module(plugin_id, plugin_dir, entry_point)
if module is None:
raise PluginError(f"Failed to load module for plugin {plugin_id}", plugin_id=plugin_id)
# Get plugin class
class_name = manifest.get('class_name')
if not class_name:
src/plugin_system/plugin_manager.py
+27 -22
View File
@@ -2,7 +2,8 @@
Plugin Manager
Manages plugin discovery, loading, and lifecycle for the LEDMatrix system.
Handles dynamic plugin loading from the plugins/ directory.
Loads plugins from the configured plugins directory
(``plugin_system.plugins_directory``, ``plugin-repos/`` by default).
API Version: 1.0.0
"""
@@ -40,7 +41,7 @@ class PluginManager:
Manages plugin discovery, loading, and lifecycle.
The PluginManager is responsible for:
- Discovering plugins in the plugins/ directory
- Discovering plugins in the configured plugins directory
- Loading plugin modules and instantiating plugin classes
- Managing plugin lifecycle (load, unload, reload)
- Providing access to loaded plugins
@@ -99,10 +100,9 @@ class PluginManager:
self.plugin_directories: Dict[str, Path] = {}
self.plugin_last_update: Dict[str, float] = {}
# Cached data-fetch intervals per plugin_id.
# _get_plugin_update_interval falls back to config_manager.get_config()
# (a full dict copy) when the manifest lacks an interval — caching avoids
# that copy on every 30-fps tick. Cleared on load/unload.
# Cached static data-fetch intervals per plugin_id, so the render
# loop's scheduling tick does not repeat the manifest/config lookup
# for every plugin. Cleared on load/unload.
self._update_interval_cache: Dict[str, Optional[float]] = {}
# Health tracking (optional, set by display_controller if available)
@@ -110,14 +110,12 @@ class PluginManager:
self.resource_monitor = None
# --- Asynchronous plugin updates -------------------------------
# update() used to run inline in the render loop (execute_update's
# internal thread.join(timeout=30) blocked it), so one slow plugin
# HTTP fetch froze scrolling for the whole fetch. Scheduling still
# happens on the render thread (run_scheduled_updates), but
# execution moves to this single background worker. Per-plugin
# locks keep a plugin's update() and display() mutually exclusive —
# today's implicit guarantee, now explicit (and, unlike today,
# also held across the post-timeout window).
# Run inline in the render loop, one slow plugin HTTP fetch in
# update() freezes scrolling for the whole fetch. Scheduling happens
# on the render thread (run_scheduled_updates); execution happens on
# this single background worker. Per-plugin locks keep a plugin's
# update() and display() mutually exclusive, including across the
# post-timeout window.
# Kill switch: plugin_system.synchronous_updates: true restores the
# inline path.
self._update_queue: "queue.Queue[Optional[Tuple[str, float]]]" = queue.Queue()
@@ -661,9 +659,15 @@ class PluginManager:
if not self.unload_plugin(plugin_id):
return False
# Re-discover to get updated manifest
manifest_path = self.plugins_dir / plugin_id / "manifest.json"
if manifest_path.exists():
# Re-read the manifest so an edit to it takes effect, from the
# directory discovery found the plugin in: a directory's name need not
# be the id its manifest declares.
with self._discovery_lock:
directories = dict(self.plugin_directories)
plugin_dir = self.plugin_loader.find_plugin_directory(
plugin_id, self.plugins_dir, directories)
manifest_path = plugin_dir / "manifest.json" if plugin_dir is not None else None
if manifest_path is not None and manifest_path.exists():
try:
with open(manifest_path, 'r', encoding='utf-8') as f:
manifest = json.load(f)
@@ -881,11 +885,12 @@ class PluginManager:
updating, since a scheduler that propagates a plugin bug stops every
other plugin too.
The static result is cached per plugin_id after the first lookup to
avoid calling config_manager.get_config() — which returns a full dict
copy — on every tick of the 30-fps display loop. The cache is
invalidated when a plugin is loaded or unloaded. The dynamic hook is
deliberately *not* cached: caching it would defeat its only purpose.
The static result is cached per plugin_id after the first lookup, so
the manifest/config resolution is not repeated on every scheduling
tick of the display loop. A change to ``update_interval`` in
config.json therefore takes effect when the plugin is next loaded or
unloaded, which clears the cache. The dynamic hook is deliberately
*not* cached: caching it would defeat its only purpose.
"""
dynamic = self._dynamic_update_interval(plugin_id, plugin_instance)
if dynamic is not None:
src/plugin_system/plugin_state.py
+1 -8
View File
@@ -41,7 +41,6 @@ class PluginStateManager:
self._state_transition_counts: Dict[str, int] = {}
self._error_info: Dict[str, Dict[str, Any]] = {}
self._last_update: Dict[str, datetime] = {}
self._last_display: Dict[str, datetime] = {}
def _record_transition(self, plugin_id: str) -> None:
"""Count a state transition. Callers must already hold ``_lock``."""
@@ -181,11 +180,7 @@ class PluginStateManager:
def get_last_update(self, plugin_id: str) -> Optional[datetime]:
"""Get timestamp of last update() call."""
return self._last_update.get(plugin_id)
def get_last_display(self, plugin_id: str) -> Optional[datetime]:
"""Get timestamp of last display() call."""
return self._last_display.get(plugin_id)
def get_state_info(self, plugin_id: str) -> Dict[str, Any]:
"""
Get comprehensive state information for a plugin.
@@ -212,7 +207,6 @@ class PluginStateManager:
'is_error': self.is_error(plugin_id),
'can_execute': self.can_execute(plugin_id),
'last_update': self.get_last_update(plugin_id),
'last_display': self.get_last_display(plugin_id),
'error_info': self.get_error_info(plugin_id),
'state_history_count': self._state_transition_counts.get(plugin_id, 0)
}
@@ -231,5 +225,4 @@ class PluginStateManager:
self._state_transition_counts.pop(plugin_id, None)
self._error_info.pop(plugin_id, None)
self._last_update.pop(plugin_id, None)
self._last_display.pop(plugin_id, None)
src/plugin_system/repo_urls.py
+70
View File
@@ -0,0 +1,70 @@
"""
Repository URL helpers shared by the plugin store and saved repositories.
One definition of "the same repository URL", of how a GitHub URL maps to
``owner/repo``, and of the headers sent to the GitHub API.
"""
from typing import Dict, Optional, Tuple
from urllib.parse import urlparse
#: Hosts whose URLs name a GitHub repository. Matched against
#: ``urlparse(url).hostname``, never by substring: a substring test accepts
#: ``https://github.com.example.org/...`` as GitHub.
GITHUB_HOSTS = frozenset({'github.com', 'www.github.com'})
#: Sent on every request the store makes to GitHub.
USER_AGENT = 'LEDMatrix-Plugin-Manager/1.0'
def normalize_repo_url(url: str) -> str:
"""``url`` without surrounding whitespace, trailing slashes or a trailing ``.git``.
Only a *trailing* ``.git`` is removed. The unanchored
``url.replace('.git', '')`` this replaces turned
``https://github.com/user/my.github.io`` into ``.../myhub.io``.
Case is preserved; compare with :func:`same_repo`.
"""
url = url.strip().rstrip('/')
if url.endswith('.git'):
url = url[:-4]
return url
def same_repo(url_a: str, url_b: str) -> bool:
"""Whether two URLs name the same repository.
GitHub owner and repository names are case-insensitive, so
``ChuckBuilds/LEDMatrix-Plugins`` and ``chuckbuilds/ledmatrix-plugins``
are one repository.
"""
return normalize_repo_url(url_a).lower() == normalize_repo_url(url_b).lower()
def github_owner_repo(url: str) -> Optional[Tuple[str, str]]:
"""``(owner, repo)`` for a github.com repository URL, else None.
The first two path segments, so a URL that points inside the repository
(``.../owner/repo/tree/main/plugins/x``) still names ``owner/repo``.
"""
parsed = urlparse(normalize_repo_url(url))
if parsed.hostname not in GITHUB_HOSTS:
return None
parts = [part for part in parsed.path.split('/') if part]
if len(parts) < 2:
return None
return parts[0], normalize_repo_url(parts[1])
def github_api_headers(token: Optional[str] = None) -> Dict[str, str]:
"""Headers for a GitHub REST API request, authenticated when ``token`` is set.
An authenticated request gets 5000 requests an hour instead of 60.
"""
headers = {
'Accept': 'application/vnd.github.v3+json',
'User-Agent': USER_AGENT,
}
if token:
headers['Authorization'] = f'token {token}'
return headers
src/plugin_system/resource_monitor.py
+31 -20
View File
@@ -33,7 +33,14 @@ class ResourceLimits:
@dataclass
class ResourceMetrics:
"""Resource usage metrics for a plugin."""
"""Resource usage metrics for a plugin.
``memory_mb`` is the largest growth in this *process's* resident memory
seen across a single monitored call -- a high-water mark, not current
usage, and not the plugin's own footprint (another thread allocating
during the call counts too). ``cpu_percent`` is the whole process's CPU
use since the previous sample.
"""
memory_mb: float = 0.0
cpu_percent: float = 0.0
execution_time: float = 0.0
@@ -42,11 +49,6 @@ class ResourceMetrics:
max_execution_time: float = 0.0
min_execution_time: float = float('inf')
last_update_time: float = field(default_factory=time.time)
def update_average_execution_time(self):
"""Update average execution time."""
if self.call_count > 0:
self.total_execution_time = self.total_execution_time / self.call_count
#: How often a plugin's metrics are written to the cache, in seconds.
@@ -287,7 +289,8 @@ class PluginResourceMonitor:
# Calculate execution time
execution_time = time.time() - start_time
memory_growth_mb = 0.0
# Update metrics
with self._lock:
metrics.execution_time = execution_time
@@ -302,17 +305,17 @@ class PluginResourceMonitor:
# Update memory and CPU if monitoring enabled
if self.enable_monitoring:
end_memory = self._get_process_memory_mb()
metrics.memory_mb = max(metrics.memory_mb, end_memory - start_memory)
memory_growth_mb = self._get_process_memory_mb() - start_memory
metrics.memory_mb = max(metrics.memory_mb, memory_growth_mb)
# CPU is harder to measure per-call, so we track it separately
metrics.cpu_percent = self._get_process_cpu_percent()
# Persist metrics, at most once per interval per plugin.
self._persist_metrics(plugin_id, metrics)
# Check limits
if limits:
self._check_limits(plugin_id, metrics, limits, execution_time)
self._check_limits(plugin_id, metrics, limits, execution_time,
memory_growth_mb)
return result
@@ -326,9 +329,17 @@ class PluginResourceMonitor:
metrics.last_update_time = time.time()
raise
def _check_limits(self, plugin_id: str, metrics: ResourceMetrics,
limits: ResourceLimits, execution_time: float) -> None:
"""Check if plugin has exceeded resource limits."""
def _check_limits(self, plugin_id: str, metrics: ResourceMetrics,
limits: ResourceLimits, execution_time: float,
memory_growth_mb: float) -> None:
"""Raise ResourceLimitExceeded if this call went over a limit.
Execution time and memory growth are this call's own; CPU is the
latest process sample. Judging memory by the stored high-water mark
(``metrics.memory_mb``) instead would fail every call after the first
expensive one, so the health tracker's circuit breaker would reopen
on every recovery probe and the plugin would never update again.
"""
warnings = []
errors = []
@@ -343,13 +354,13 @@ class PluginResourceMonitor:
)
# Check memory
if limits.max_memory_mb and metrics.memory_mb > limits.max_memory_mb:
if limits.max_memory_mb and memory_growth_mb > limits.max_memory_mb:
errors.append(
f"Memory usage {metrics.memory_mb:.2f}MB exceeds limit {limits.max_memory_mb:.2f}MB"
f"Memory growth {memory_growth_mb:.2f}MB exceeds limit {limits.max_memory_mb:.2f}MB"
)
elif limits.max_memory_mb and metrics.memory_mb > limits.max_memory_mb * limits.warning_threshold:
elif limits.max_memory_mb and memory_growth_mb > limits.max_memory_mb * limits.warning_threshold:
warnings.append(
f"Memory usage {metrics.memory_mb:.2f}MB approaching limit {limits.max_memory_mb:.2f}MB"
f"Memory growth {memory_growth_mb:.2f}MB approaching limit {limits.max_memory_mb:.2f}MB"
)
# Check CPU
src/plugin_system/saved_repositories.py
+5 -15
View File
@@ -10,6 +10,8 @@ import os
from pathlib import Path
from typing import List, Dict, Optional
from src.plugin_system.repo_urls import normalize_repo_url
class SavedRepositoriesManager:
"""Manages saved GitHub repository URLs."""
@@ -71,18 +73,6 @@ class SavedRepositoriesManager:
pass
return False
@staticmethod
def _clean_url(repo_url: str) -> str:
"""Normalize a repo URL: strip whitespace, trailing slashes, and a
trailing ``.git`` suffix ONLY. (The old ``.replace('.git', '')``
was an unanchored substring replace that mangled URLs merely
containing ``.git``, e.g. ``https://github.com/user/my.github.io``.)
"""
repo_url = repo_url.strip().rstrip('/')
if repo_url.endswith('.git'):
repo_url = repo_url[:-4]
return repo_url
def get_all(self) -> List[Dict[str, str]]:
"""Get all saved repositories."""
return self.repositories.copy()
@@ -98,7 +88,7 @@ class SavedRepositoriesManager:
Returns:
True if added successfully
"""
repo_url = self._clean_url(repo_url)
repo_url = normalize_repo_url(repo_url)
# Check if already exists
for repo in self.repositories:
@@ -138,7 +128,7 @@ class SavedRepositoriesManager:
Returns:
True if removed successfully
"""
repo_url = self._clean_url(repo_url)
repo_url = normalize_repo_url(repo_url)
previous = self.repositories
remaining = [r for r in previous if r.get('url') != repo_url]
@@ -156,7 +146,7 @@ class SavedRepositoriesManager:
def has(self, repo_url: str) -> bool:
"""Check if a repository is already saved."""
repo_url = self._clean_url(repo_url)
repo_url = normalize_repo_url(repo_url)
return any(r.get('url') == repo_url for r in self.repositories)
def get_registry_repositories(self) -> List[Dict[str, str]]:
src/plugin_system/schema_manager.py
+15 -18
View File
@@ -14,6 +14,7 @@ import jsonschema
from jsonschema import Draft7Validator, ValidationError
from src.core_config_keys import CORE_CONFIG_KEYS
from src.element_style import expand_style_elements
def _renders_as_object(prop: Dict[str, Any]) -> bool:
@@ -452,11 +453,7 @@ class SchemaManager:
# full per-element style blocks (font/size/color + layout
# offsets) the web-UI config form renders. No-op for schemas
# without the declaration; never raises.
try:
from src.element_style import expand_style_elements
schema = expand_style_elements(schema)
except ImportError:
pass
schema = expand_style_elements(schema)
# Cache the schema
self._schema_cache[plugin_id] = schema
@@ -643,20 +640,12 @@ class SchemaManager:
[name for name in CORE_PLUGIN_PROPERTIES if name not in declared]
)
# Create validator with enhanced schema
# iter_errors reports every violation, including one ``required``
# error per missing field at every depth.
validator = Draft7Validator(enhanced_schema)
# Collect all validation errors
for error in validator.iter_errors(config):
error_msg = self._format_validation_error(error, plugin_id)
errors.append(error_msg)
# Check required fields
required_fields = enhanced_schema.get('required', [])
for field in required_fields:
if field not in config:
errors.append(f"Missing required field: '{field}'")
errors.append(self._format_validation_error(error, plugin_id))
if errors:
return False, errors
@@ -687,7 +676,15 @@ class SchemaManager:
field_path = f"'{path}'" if path else "root"
if error.validator == 'required':
missing = error.validator_value
# validator_value is the schema's whole ``required`` list; the
# error itself is about one field, which jsonschema names only in
# its message ("'api_key' is a required property").
missing = next(
(name for name in error.validator_value
if error.message.startswith(f"{name!r} ")),
None)
if missing is None:
return f"Field {field_path}: {error.message}"
return f"Field {field_path}: Missing required property '{missing}'"
elif error.validator == 'type':
expected = error.validator_value
src/plugin_system/state_manager.py
+5 -2
View File
@@ -34,7 +34,9 @@ class PluginState:
version: Optional[str] = None
installed_at: Optional[datetime] = None
last_updated: Optional[datetime] = None
config_version: int = 1 # For detecting state corruption
# Bumped on every update_plugin_state(). Nothing reads it; it stays so
# plugin_state.json keeps the shape older releases load with cls(**data).
config_version: int = 1
metadata: Dict[str, Any] = None
def __post_init__(self):
@@ -100,6 +102,8 @@ class PluginStateManager:
# State storage
self._states: Dict[str, PluginState] = {}
# The file's top-level "version", written back as read. Nothing
# checks it yet; it is there for a future format change to branch on.
self._state_version = 1
# Threading
@@ -193,7 +197,6 @@ class PluginStateManager:
current_state.metadata = {}
current_state.metadata.update(updates['metadata'])
# Increment config version
current_state.config_version += 1
# Store updated state
src/plugin_system/store_manager.py
+310 -429
View File
File diff suppressed because it is too large Load Diff