mirror of
https://github.com/ChuckBuilds/LEDMatrix.git
synced 2026-05-15 10:03:31 +00:00
perf(plugins): dramatically speed up plugin manager tab load time (#333)
* fix(cache): check odds keys before generic live check in get_data_type_from_key Cache keys like odds_espn_basketball_nba_<id>_live contain both 'odds' and 'live'. The previous ordering matched the generic 'live' check first, returning 'sports_live' (30 s TTL) instead of the correct 'odds_live' (120 s TTL). This caused the ESPN odds API to be hit every 30 s per live game, frequently triggering the 3-second per-request timeout and returning no odds data. Moving the 'odds' check above the generic 'live' block restores the correct 120-second cache TTL for in-progress game odds. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(display): use single-quoted HTML attributes for JSON hidden inputs Placing |tojson output (which contains double quotes) inside a double-quoted HTML attribute broke the attribute — browsers closed the attribute at the first inner quote, leaving JS with an empty or truncated value. JSON.parse then failed silently, leaving excluded=[] so all Vegas scroll plugins appeared checked (included) regardless of the actual excluded_plugins config. Switch to single-quoted HTML attributes so the JSON double quotes are valid inside the attribute value. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * perf(plugins): dramatically speed up plugin manager tab load time ## Problem The Plugins tab loaded slowly and inconsistently (5–30s depending on cache state), with a blank spinner for the entire wait. Three root causes: 1. **N+1 subprocess per installed plugin** — `_get_local_git_info` ran 4 separate git subprocesses per plugin (rev-parse HEAD, abbrev-ref, config --get remote.origin.url, log --format=%cI). With 15 plugins that's 60 blocking subprocess spawns before the endpoint returned. 2. **Serial per-plugin loop** — the `/plugins/installed` endpoint processed each plugin sequentially: manifest read → git info → instance lookup → Vegas mode query, one plugin at a time. 3. **Serial JS loading** — the store search only started after installed plugins fully completed, so users waited for both round-trips back to back. No UI feedback during the wait. ## Changes ### Backend — src/plugin_system/store_manager.py - Consolidate 4 git subprocesses → 1: branch read from `.git/HEAD` (file I/O, no subprocess), remote URL parsed from `.git/config` (file I/O, no subprocess), SHA + commit date fetched together in a single `git log -1 --format=%H%n%cI` call - Existing signature-based cache already eliminates all subprocesses on warm hits; this change cuts cold-cache cost from 4 → 1 per plugin ### Backend — web_interface/blueprints/api_v3.py - Wrap per-plugin work in a `_build_plugin_entry()` helper and execute it across a `ThreadPoolExecutor(max_workers=8)` so all plugins are processed in parallel instead of sequentially - Fix double `get_plugin()` call per plugin (was called once for the enabled fallback and again for Vegas mode — now one shared call) ### Frontend — web_interface/static/v3/plugins_manager.js - Fire `searchPluginStore()` and `loadInstalledPlugins()` simultaneously instead of waiting for installed to complete before starting the store - After installed data arrives, call `applyStoreFiltersAndSort(true)` to refresh install/update/reinstall badges from already-cached store data (instant, no extra network call) ### Frontend — web_interface/templates/v3/partials/plugins.html - Add responsive skeleton cards to the installed plugins section that match real card proportions (removed automatically when data renders) - Replace the 5 featureless gray boxes in the store skeleton with 10 structured skeleton cards matching the real card layout ## Measured improvement on Pi 4 (11 installed plugins, ledpi-ticker) | Scenario | Before | After | |---|---|---| | Cold cache (first open) | ~8–15s | **0.9s** | | Warm cache (git cache hit) | ~1–2s | **55ms** | | UI feedback during load | blank spinner | skeleton cards | | Store waits for installed | yes (serial) | no (parallel) | Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(plugins): harden git metadata parsing and plugin entry building store_manager.py: - Detect worktree/submodule .git files (gitdir: <path>) and resolve to the actual git directory before reading HEAD or config - Wrap HEAD read_text in try/except OSError/NotADirectoryError so atypical repos return None instead of propagating exceptions - Guard config url line split with '=' presence check to avoid IndexError on malformed lines api_v3.py: - Wrap _build_plugin_entry body in a try/except via a thin outer wrapper so a single plugin's failure doesn't 500 the whole endpoint; failed entries return None and are filtered by the existing [r for r in results if r is not None] step - Narrow manifest except clause to FileNotFoundError, PermissionError, json.JSONDecodeError instead of bare Exception - Validate manifest is a dict before calling plugin_info.update() and log a debug message when it isn't Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
sarjent
@@ -1772,61 +1772,57 @@ def get_installed_plugins():
|
||||
# Load config once before the loop (not per-plugin)
|
||||
full_config = api_v3.config_manager.load_config() if api_v3.config_manager else {}
|
||||
|
||||
# Format for the web interface
|
||||
plugins = []
|
||||
for plugin_info in all_plugin_info:
|
||||
def _build_plugin_entry(plugin_info):
|
||||
plugin_id = plugin_info.get('id')
|
||||
try:
|
||||
return _build_plugin_entry_inner(plugin_info, plugin_id)
|
||||
except Exception:
|
||||
logger.exception("Error building plugin entry for %s — skipping", plugin_id)
|
||||
return None
|
||||
|
||||
def _build_plugin_entry_inner(plugin_info, plugin_id):
|
||||
# Re-read manifest from disk to ensure we have the latest metadata
|
||||
manifest_path = Path(api_v3.plugin_manager.plugins_dir) / plugin_id / "manifest.json"
|
||||
if manifest_path.exists():
|
||||
try:
|
||||
with open(manifest_path, 'r', encoding='utf-8') as f:
|
||||
fresh_manifest = json.load(f)
|
||||
# Update plugin_info with fresh manifest data
|
||||
plugin_info.update(fresh_manifest)
|
||||
except Exception as e:
|
||||
# If we can't read the fresh manifest, use the cached one
|
||||
print(f"Warning: Could not read fresh manifest for {plugin_id}: {e}")
|
||||
if isinstance(fresh_manifest, dict):
|
||||
plugin_info.update(fresh_manifest)
|
||||
else:
|
||||
logger.debug("Manifest for %s is not a dict (%s) — skipping merge",
|
||||
plugin_id, type(fresh_manifest).__name__)
|
||||
except (FileNotFoundError, PermissionError, json.JSONDecodeError) as e:
|
||||
logger.debug("Could not read fresh manifest for %s: %s", plugin_id, e)
|
||||
|
||||
# Get enabled status from config (source of truth)
|
||||
# Read from config file first, fall back to plugin instance if config doesn't have the key
|
||||
# Enabled status: config is source of truth, fall back to instance
|
||||
enabled = None
|
||||
if api_v3.config_manager:
|
||||
plugin_config = full_config.get(plugin_id, {})
|
||||
# Check if 'enabled' key exists in config (even if False)
|
||||
if 'enabled' in plugin_config:
|
||||
enabled = bool(plugin_config['enabled'])
|
||||
plugin_config = full_config.get(plugin_id, {})
|
||||
if 'enabled' in plugin_config:
|
||||
enabled = bool(plugin_config['enabled'])
|
||||
|
||||
# Fallback to plugin instance if config doesn't have enabled key
|
||||
# Single get_plugin() call shared for both enabled fallback and Vegas mode
|
||||
plugin_instance = api_v3.plugin_manager.get_plugin(plugin_id)
|
||||
if enabled is None:
|
||||
plugin_instance = api_v3.plugin_manager.get_plugin(plugin_id)
|
||||
if plugin_instance:
|
||||
enabled = plugin_instance.enabled
|
||||
else:
|
||||
# Default to True if no config key and plugin not loaded (matches BasePlugin default)
|
||||
enabled = True
|
||||
enabled = plugin_instance.enabled if plugin_instance else True
|
||||
|
||||
# Get verified status from store registry (no GitHub API calls needed)
|
||||
# Verified from registry (no network call)
|
||||
store_info = api_v3.plugin_store_manager.get_registry_info(plugin_id)
|
||||
verified = store_info.get('verified', False) if store_info else False
|
||||
|
||||
# Get local git info for installed plugin (actual installed commit)
|
||||
# Local git info (single subprocess on cache miss, zero on hit)
|
||||
plugin_path = Path(api_v3.plugin_manager.plugins_dir) / plugin_id
|
||||
local_git_info = api_v3.plugin_store_manager._get_local_git_info(plugin_path) if plugin_path.exists() else None
|
||||
|
||||
# Use local git info if available (actual installed commit), otherwise fall back to manifest/store info
|
||||
if local_git_info:
|
||||
last_commit = local_git_info.get('short_sha') or local_git_info.get('sha', '')[:7] if local_git_info.get('sha') else None
|
||||
sha = local_git_info.get('sha', '')
|
||||
last_commit = local_git_info.get('short_sha') or (sha[:7] if sha else None)
|
||||
branch = local_git_info.get('branch')
|
||||
# Use commit date from git if available
|
||||
last_updated = local_git_info.get('date_iso') or local_git_info.get('date')
|
||||
else:
|
||||
# Fall back to manifest/store info if no local git info
|
||||
last_updated = plugin_info.get('last_updated')
|
||||
last_commit = plugin_info.get('last_commit') or plugin_info.get('last_commit_sha')
|
||||
branch = plugin_info.get('branch')
|
||||
|
||||
if store_info:
|
||||
last_updated = last_updated or store_info.get('last_updated') or store_info.get('last_updated_iso')
|
||||
last_commit = last_commit or store_info.get('last_commit') or store_info.get('last_commit_sha')
|
||||
@@ -1836,35 +1832,26 @@ def get_installed_plugins():
|
||||
if store_info and not last_commit_message:
|
||||
last_commit_message = store_info.get('last_commit_message')
|
||||
|
||||
# Get web_ui_actions from manifest if available
|
||||
web_ui_actions = plugin_info.get('web_ui_actions', [])
|
||||
|
||||
# Get Vegas display mode info from plugin instance
|
||||
# Vegas mode from instance, overridden by explicit config value
|
||||
vegas_mode = None
|
||||
vegas_content_type = None
|
||||
plugin_instance = api_v3.plugin_manager.get_plugin(plugin_id)
|
||||
if plugin_instance:
|
||||
try:
|
||||
# Try to get the display mode enum
|
||||
if hasattr(plugin_instance, 'get_vegas_display_mode'):
|
||||
mode = plugin_instance.get_vegas_display_mode()
|
||||
vegas_mode = mode.value if hasattr(mode, 'value') else str(mode)
|
||||
except (AttributeError, TypeError, ValueError) as e:
|
||||
logger.debug("[%s] Failed to get vegas_display_mode: %s", plugin_id, e)
|
||||
try:
|
||||
# Get legacy content type as fallback
|
||||
if hasattr(plugin_instance, 'get_vegas_content_type'):
|
||||
vegas_content_type = plugin_instance.get_vegas_content_type()
|
||||
except (AttributeError, TypeError, ValueError) as e:
|
||||
logger.debug("[%s] Failed to get vegas_content_type: %s", plugin_id, e)
|
||||
|
||||
# Also check plugin config for explicit vegas_mode setting
|
||||
if api_v3.config_manager:
|
||||
plugin_cfg = full_config.get(plugin_id, {})
|
||||
if 'vegas_mode' in plugin_cfg:
|
||||
vegas_mode = plugin_cfg['vegas_mode']
|
||||
if 'vegas_mode' in plugin_config:
|
||||
vegas_mode = plugin_config['vegas_mode']
|
||||
|
||||
plugins.append({
|
||||
return {
|
||||
'id': plugin_id,
|
||||
'name': plugin_info.get('name', plugin_id),
|
||||
'version': plugin_info.get('version', ''),
|
||||
@@ -1879,10 +1866,15 @@ def get_installed_plugins():
|
||||
'last_commit': last_commit,
|
||||
'last_commit_message': last_commit_message,
|
||||
'branch': branch,
|
||||
'web_ui_actions': web_ui_actions,
|
||||
'web_ui_actions': plugin_info.get('web_ui_actions', []),
|
||||
'vegas_mode': vegas_mode,
|
||||
'vegas_content_type': vegas_content_type
|
||||
})
|
||||
'vegas_content_type': vegas_content_type,
|
||||
}
|
||||
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
with ThreadPoolExecutor(max_workers=8) as executor:
|
||||
results = list(executor.map(_build_plugin_entry, all_plugin_info))
|
||||
plugins = [r for r in results if r is not None]
|
||||
|
||||
return jsonify({'status': 'success', 'data': {'plugins': plugins}})
|
||||
except Exception as e:
|
||||
|
||||
@@ -1223,13 +1223,21 @@ function initializePlugins() {
|
||||
// during a re-swap, fetch fresh data including GitHub commit/version info.
|
||||
const isReswapWarm = !!window.pluginManager._reswap && !storeCacheExpired();
|
||||
window.pluginManager._reswap = false;
|
||||
// Await the installed-plugins fetch so window.installedPlugins is populated before
|
||||
// searchPluginStore renders Installed/Reinstall badges against it.
|
||||
loadInstalledPlugins().catch(err => {
|
||||
console.error('[PluginStore] loadInstalledPlugins failed:', err);
|
||||
}).finally(() => {
|
||||
searchPluginStore(!isReswapWarm);
|
||||
});
|
||||
|
||||
// Fire both requests in parallel so the store doesn't wait for installed plugins.
|
||||
// The store renders install/update badges using window.installedPlugins || [] so
|
||||
// it works with an empty list. When installed plugins finish loading we do a
|
||||
// lightweight re-render from the already-cached store data to refresh the badges.
|
||||
searchPluginStore(!isReswapWarm);
|
||||
loadInstalledPlugins()
|
||||
.catch(err => console.error('[PluginStore] loadInstalledPlugins failed:', err))
|
||||
.then(() => {
|
||||
// Re-render store from cache to update install/update/reinstall badges now
|
||||
// that window.installedPlugins is populated. No network call — instant.
|
||||
if (typeof applyStoreFiltersAndSort === 'function') {
|
||||
applyStoreFiltersAndSort(true);
|
||||
}
|
||||
});
|
||||
|
||||
// Setup search functionality (with guard against duplicate listeners)
|
||||
const searchInput = document.getElementById('plugin-search');
|
||||
@@ -1418,6 +1426,9 @@ function renderInstalledPlugins(plugins) {
|
||||
}
|
||||
}
|
||||
|
||||
// Remove skeleton cards before rendering real content
|
||||
container.querySelectorAll('.installed-skeleton').forEach(el => el.remove());
|
||||
|
||||
if (plugins.length === 0) {
|
||||
container.innerHTML = `
|
||||
<div class="col-span-full empty-state">
|
||||
|
||||
@@ -31,7 +31,52 @@
|
||||
</div>
|
||||
<div id="installed-plugins-content" class="block">
|
||||
<div id="installed-plugins-grid" class="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 xl:grid-cols-4 2xl:grid-cols-5 gap-6">
|
||||
<!-- Plugins will be loaded here -->
|
||||
<!-- Skeleton cards shown while installed plugins load -->
|
||||
<div class="installed-skeleton plugin-card animate-pulse">
|
||||
<div class="flex items-start justify-between mb-4">
|
||||
<div class="flex-1 min-w-0 space-y-2">
|
||||
<div class="h-4 bg-gray-200 rounded w-3/4"></div>
|
||||
<div class="h-3 bg-gray-200 rounded w-1/2"></div>
|
||||
<div class="h-3 bg-gray-200 rounded w-2/3"></div>
|
||||
</div>
|
||||
<div class="h-6 w-10 bg-gray-200 rounded-full ml-4 flex-shrink-0"></div>
|
||||
</div>
|
||||
<div class="space-y-2 mb-4">
|
||||
<div class="h-3 bg-gray-200 rounded w-full"></div>
|
||||
<div class="h-3 bg-gray-200 rounded w-5/6"></div>
|
||||
</div>
|
||||
<div class="h-8 bg-gray-200 rounded w-full mt-auto"></div>
|
||||
</div>
|
||||
<div class="installed-skeleton plugin-card animate-pulse hidden md:block">
|
||||
<div class="flex items-start justify-between mb-4">
|
||||
<div class="flex-1 min-w-0 space-y-2">
|
||||
<div class="h-4 bg-gray-200 rounded w-2/3"></div>
|
||||
<div class="h-3 bg-gray-200 rounded w-1/3"></div>
|
||||
<div class="h-3 bg-gray-200 rounded w-1/2"></div>
|
||||
</div>
|
||||
<div class="h-6 w-10 bg-gray-200 rounded-full ml-4 flex-shrink-0"></div>
|
||||
</div>
|
||||
<div class="space-y-2 mb-4">
|
||||
<div class="h-3 bg-gray-200 rounded w-full"></div>
|
||||
<div class="h-3 bg-gray-200 rounded w-4/5"></div>
|
||||
</div>
|
||||
<div class="h-8 bg-gray-200 rounded w-full mt-auto"></div>
|
||||
</div>
|
||||
<div class="installed-skeleton plugin-card animate-pulse hidden lg:block">
|
||||
<div class="flex items-start justify-between mb-4">
|
||||
<div class="flex-1 min-w-0 space-y-2">
|
||||
<div class="h-4 bg-gray-200 rounded w-4/5"></div>
|
||||
<div class="h-3 bg-gray-200 rounded w-2/5"></div>
|
||||
<div class="h-3 bg-gray-200 rounded w-3/5"></div>
|
||||
</div>
|
||||
<div class="h-6 w-10 bg-gray-200 rounded-full ml-4 flex-shrink-0"></div>
|
||||
</div>
|
||||
<div class="space-y-2 mb-4">
|
||||
<div class="h-3 bg-gray-200 rounded w-full"></div>
|
||||
<div class="h-3 bg-gray-200 rounded w-3/4"></div>
|
||||
</div>
|
||||
<div class="h-8 bg-gray-200 rounded w-full mt-auto"></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -203,14 +248,30 @@
|
||||
</div>
|
||||
|
||||
<div id="plugin-store-grid" class="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 xl:grid-cols-4 2xl:grid-cols-5 gap-6">
|
||||
<!-- Loading skeleton -->
|
||||
<!-- Loading skeleton — hidden by showStoreLoading(false) when data arrives -->
|
||||
<div class="store-loading col-span-full">
|
||||
<div class="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 xl:grid-cols-4 2xl:grid-cols-5 gap-4">
|
||||
<div class="bg-gray-200 rounded-lg p-4 h-48 animate-pulse"></div>
|
||||
<div class="bg-gray-200 rounded-lg p-4 h-48 animate-pulse"></div>
|
||||
<div class="bg-gray-200 rounded-lg p-4 h-48 animate-pulse"></div>
|
||||
<div class="bg-gray-200 rounded-lg p-4 h-48 animate-pulse"></div>
|
||||
<div class="bg-gray-200 rounded-lg p-4 h-48 animate-pulse"></div>
|
||||
<div class="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 xl:grid-cols-4 2xl:grid-cols-5 gap-6">
|
||||
{% for _ in range(10) %}
|
||||
<div class="plugin-card animate-pulse">
|
||||
<div class="flex items-start justify-between mb-4">
|
||||
<div class="flex-1 min-w-0 space-y-2">
|
||||
<div class="h-4 bg-gray-200 rounded w-3/4"></div>
|
||||
<div class="h-3 bg-gray-200 rounded w-1/2"></div>
|
||||
</div>
|
||||
<div class="h-5 w-14 bg-gray-200 rounded-full ml-3 flex-shrink-0"></div>
|
||||
</div>
|
||||
<div class="space-y-2 mb-4">
|
||||
<div class="h-3 bg-gray-200 rounded w-full"></div>
|
||||
<div class="h-3 bg-gray-200 rounded w-5/6"></div>
|
||||
<div class="h-3 bg-gray-200 rounded w-4/6"></div>
|
||||
</div>
|
||||
<div class="flex gap-2 mt-auto">
|
||||
<div class="h-3 bg-gray-200 rounded w-12"></div>
|
||||
<div class="h-3 bg-gray-200 rounded w-16"></div>
|
||||
</div>
|
||||
<div class="h-8 bg-gray-200 rounded w-full mt-3"></div>
|
||||
</div>
|
||||
{% endfor %}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
Reference in New Issue
Block a user