rick/LEDMatrix
1
0
Fork 0
mirror of https://github.com/ChuckBuilds/LEDMatrix.git synced 2026-05-15 18:03:32 +00:00
Code Issues Packages Projects Releases Wiki Activity

perf(plugins): dramatically speed up plugin manager tab load time (#333)

Browse Source 
* fix(cache): check odds keys before generic live check in get_data_type_from_key

Cache keys like odds_espn_basketball_nba_<id>_live contain both 'odds'
and 'live'. The previous ordering matched the generic 'live' check first,
returning 'sports_live' (30 s TTL) instead of the correct 'odds_live'
(120 s TTL). This caused the ESPN odds API to be hit every 30 s per live
game, frequently triggering the 3-second per-request timeout and returning
no odds data.

Moving the 'odds' check above the generic 'live' block restores the
correct 120-second cache TTL for in-progress game odds.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(display): use single-quoted HTML attributes for JSON hidden inputs

Placing |tojson output (which contains double quotes) inside a
double-quoted HTML attribute broke the attribute — browsers closed
the attribute at the first inner quote, leaving JS with an empty or
truncated value. JSON.parse then failed silently, leaving excluded=[]
so all Vegas scroll plugins appeared checked (included) regardless of
the actual excluded_plugins config.

Switch to single-quoted HTML attributes so the JSON double quotes
are valid inside the attribute value.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* perf(plugins): dramatically speed up plugin manager tab load time

## Problem

The Plugins tab loaded slowly and inconsistently (5–30s depending on
cache state), with a blank spinner for the entire wait. Three root
causes:

1. **N+1 subprocess per installed plugin** — `_get_local_git_info` ran
   4 separate git subprocesses per plugin (rev-parse HEAD, abbrev-ref,
   config --get remote.origin.url, log --format=%cI). With 15 plugins
   that's 60 blocking subprocess spawns before the endpoint returned.

2. **Serial per-plugin loop** — the `/plugins/installed` endpoint
   processed each plugin sequentially: manifest read → git info →
   instance lookup → Vegas mode query, one plugin at a time.

3. **Serial JS loading** — the store search only started after installed
   plugins fully completed, so users waited for both round-trips back
   to back. No UI feedback during the wait.

## Changes

### Backend — src/plugin_system/store_manager.py
- Consolidate 4 git subprocesses → 1: branch read from `.git/HEAD`
  (file I/O, no subprocess), remote URL parsed from `.git/config`
  (file I/O, no subprocess), SHA + commit date fetched together in a
  single `git log -1 --format=%H%n%cI` call
- Existing signature-based cache already eliminates all subprocesses on
  warm hits; this change cuts cold-cache cost from 4 → 1 per plugin

### Backend — web_interface/blueprints/api_v3.py
- Wrap per-plugin work in a `_build_plugin_entry()` helper and execute
  it across a `ThreadPoolExecutor(max_workers=8)` so all plugins are
  processed in parallel instead of sequentially
- Fix double `get_plugin()` call per plugin (was called once for the
  enabled fallback and again for Vegas mode — now one shared call)

### Frontend — web_interface/static/v3/plugins_manager.js
- Fire `searchPluginStore()` and `loadInstalledPlugins()` simultaneously
  instead of waiting for installed to complete before starting the store
- After installed data arrives, call `applyStoreFiltersAndSort(true)` to
  refresh install/update/reinstall badges from already-cached store data
  (instant, no extra network call)

### Frontend — web_interface/templates/v3/partials/plugins.html
- Add responsive skeleton cards to the installed plugins section that
  match real card proportions (removed automatically when data renders)
- Replace the 5 featureless gray boxes in the store skeleton with 10
  structured skeleton cards matching the real card layout

## Measured improvement on Pi 4 (11 installed plugins, ledpi-ticker)

| Scenario | Before | After |
|---|---|---|
| Cold cache (first open) | ~8–15s | **0.9s** |
| Warm cache (git cache hit) | ~1–2s | **55ms** |
| UI feedback during load | blank spinner | skeleton cards |
| Store waits for installed | yes (serial) | no (parallel) |

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(plugins): harden git metadata parsing and plugin entry building

store_manager.py:
- Detect worktree/submodule .git files (gitdir: <path>) and resolve
  to the actual git directory before reading HEAD or config
- Wrap HEAD read_text in try/except OSError/NotADirectoryError so
  atypical repos return None instead of propagating exceptions
- Guard config url line split with '=' presence check to avoid
  IndexError on malformed lines

api_v3.py:
- Wrap _build_plugin_entry body in a try/except via a thin outer
  wrapper so a single plugin's failure doesn't 500 the whole endpoint;
  failed entries return None and are filtered by the existing [r for r
  in results if r is not None] step
- Narrow manifest except clause to FileNotFoundError, PermissionError,
  json.JSONDecodeError instead of bare Exception
- Validate manifest is a dict before calling plugin_info.update() and
  log a debug message when it isn't

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
sarjent
2026-05-14 17:09:33 -05:00
committed by GitHub
parent 6a4644007d
commit 44d1a08db4
4 changed files with 177 additions and 99 deletions
Download Patch File Download Diff File Expand all files Collapse all files

82
web_interface/blueprints/api_v3.py
View File

@@ -1772,61 +1772,57 @@ def get_installed_plugins():
# Load config once before the loop (not per-plugin)
full_config = api_v3.config_manager.load_config() if api_v3.config_manager else {}
# Format for the web interface
plugins = []
for plugin_info in all_plugin_info:
def _build_plugin_entry(plugin_info):
plugin_id = plugin_info.get('id')
try:
return _build_plugin_entry_inner(plugin_info, plugin_id)
except Exception:
logger.exception("Error building plugin entry for %s — skipping", plugin_id)
return None
def _build_plugin_entry_inner(plugin_info, plugin_id):
# Re-read manifest from disk to ensure we have the latest metadata
manifest_path = Path(api_v3.plugin_manager.plugins_dir) / plugin_id / "manifest.json"
if manifest_path.exists():
try:
with open(manifest_path, 'r', encoding='utf-8') as f:
fresh_manifest = json.load(f)
# Update plugin_info with fresh manifest data
plugin_info.update(fresh_manifest)
except Exception as e:
# If we can't read the fresh manifest, use the cached one
print(f"Warning: Could not read fresh manifest for {plugin_id}: {e}")
if isinstance(fresh_manifest, dict):
plugin_info.update(fresh_manifest)
else:
logger.debug("Manifest for %s is not a dict (%s) — skipping merge",
plugin_id, type(fresh_manifest).__name__)
except (FileNotFoundError, PermissionError, json.JSONDecodeError) as e:
logger.debug("Could not read fresh manifest for %s: %s", plugin_id, e)
# Get enabled status from config (source of truth)
# Read from config file first, fall back to plugin instance if config doesn't have the key
# Enabled status: config is source of truth, fall back to instance
enabled = None
if api_v3.config_manager:
plugin_config = full_config.get(plugin_id, {})
# Check if 'enabled' key exists in config (even if False)
if 'enabled' in plugin_config:
enabled = bool(plugin_config['enabled'])
plugin_config = full_config.get(plugin_id, {})
if 'enabled' in plugin_config:
enabled = bool(plugin_config['enabled'])
# Fallback to plugin instance if config doesn't have enabled key
# Single get_plugin() call shared for both enabled fallback and Vegas mode
plugin_instance = api_v3.plugin_manager.get_plugin(plugin_id)
if enabled is None:
plugin_instance = api_v3.plugin_manager.get_plugin(plugin_id)
if plugin_instance:
enabled = plugin_instance.enabled
else:
# Default to True if no config key and plugin not loaded (matches BasePlugin default)
enabled = True
enabled = plugin_instance.enabled if plugin_instance else True
# Get verified status from store registry (no GitHub API calls needed)
# Verified from registry (no network call)
store_info = api_v3.plugin_store_manager.get_registry_info(plugin_id)
verified = store_info.get('verified', False) if store_info else False
# Get local git info for installed plugin (actual installed commit)
# Local git info (single subprocess on cache miss, zero on hit)
plugin_path = Path(api_v3.plugin_manager.plugins_dir) / plugin_id
local_git_info = api_v3.plugin_store_manager._get_local_git_info(plugin_path) if plugin_path.exists() else None
# Use local git info if available (actual installed commit), otherwise fall back to manifest/store info
if local_git_info:
last_commit = local_git_info.get('short_sha') or local_git_info.get('sha', '')[:7] if local_git_info.get('sha') else None
sha = local_git_info.get('sha', '')
last_commit = local_git_info.get('short_sha') or (sha[:7] if sha else None)
branch = local_git_info.get('branch')
# Use commit date from git if available
last_updated = local_git_info.get('date_iso') or local_git_info.get('date')
else:
# Fall back to manifest/store info if no local git info
last_updated = plugin_info.get('last_updated')
last_commit = plugin_info.get('last_commit') or plugin_info.get('last_commit_sha')
branch = plugin_info.get('branch')
if store_info:
last_updated = last_updated or store_info.get('last_updated') or store_info.get('last_updated_iso')
last_commit = last_commit or store_info.get('last_commit') or store_info.get('last_commit_sha')
@@ -1836,35 +1832,26 @@ def get_installed_plugins():
if store_info and not last_commit_message:
last_commit_message = store_info.get('last_commit_message')
# Get web_ui_actions from manifest if available
web_ui_actions = plugin_info.get('web_ui_actions', [])
# Get Vegas display mode info from plugin instance
# Vegas mode from instance, overridden by explicit config value
vegas_mode = None
vegas_content_type = None
plugin_instance = api_v3.plugin_manager.get_plugin(plugin_id)
if plugin_instance:
try:
# Try to get the display mode enum
if hasattr(plugin_instance, 'get_vegas_display_mode'):
mode = plugin_instance.get_vegas_display_mode()
vegas_mode = mode.value if hasattr(mode, 'value') else str(mode)
except (AttributeError, TypeError, ValueError) as e:
logger.debug("[%s] Failed to get vegas_display_mode: %s", plugin_id, e)
try:
# Get legacy content type as fallback
if hasattr(plugin_instance, 'get_vegas_content_type'):
vegas_content_type = plugin_instance.get_vegas_content_type()
except (AttributeError, TypeError, ValueError) as e:
logger.debug("[%s] Failed to get vegas_content_type: %s", plugin_id, e)
# Also check plugin config for explicit vegas_mode setting
if api_v3.config_manager:
plugin_cfg = full_config.get(plugin_id, {})
if 'vegas_mode' in plugin_cfg:
vegas_mode = plugin_cfg['vegas_mode']
if 'vegas_mode' in plugin_config:
vegas_mode = plugin_config['vegas_mode']
plugins.append({
return {
'id': plugin_id,
'name': plugin_info.get('name', plugin_id),
'version': plugin_info.get('version', ''),
@@ -1879,10 +1866,15 @@ def get_installed_plugins():
'last_commit': last_commit,
'last_commit_message': last_commit_message,
'branch': branch,
'web_ui_actions': web_ui_actions,
'web_ui_actions': plugin_info.get('web_ui_actions', []),
'vegas_mode': vegas_mode,
'vegas_content_type': vegas_content_type
})
'vegas_content_type': vegas_content_type,
}
from concurrent.futures import ThreadPoolExecutor
with ThreadPoolExecutor(max_workers=8) as executor:
results = list(executor.map(_build_plugin_entry, all_plugin_info))
plugins = [r for r in results if r is not None]
return jsonify({'status': 'success', 'data': {'plugins': plugins}})
except Exception as e: