perf(plugins): dramatically speed up plugin manager tab load time (#333)

* fix(cache): check odds keys before generic live check in get_data_type_from_key

Cache keys like odds_espn_basketball_nba_<id>_live contain both 'odds'
and 'live'. The previous ordering matched the generic 'live' check first,
returning 'sports_live' (30 s TTL) instead of the correct 'odds_live'
(120 s TTL). This caused the ESPN odds API to be hit every 30 s per live
game, frequently triggering the 3-second per-request timeout and returning
no odds data.

Moving the 'odds' check above the generic 'live' block restores the
correct 120-second cache TTL for in-progress game odds.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(display): use single-quoted HTML attributes for JSON hidden inputs

Placing |tojson output (which contains double quotes) inside a
double-quoted HTML attribute broke the attribute — browsers closed
the attribute at the first inner quote, leaving JS with an empty or
truncated value. JSON.parse then failed silently, leaving excluded=[]
so all Vegas scroll plugins appeared checked (included) regardless of
the actual excluded_plugins config.

Switch to single-quoted HTML attributes so the JSON double quotes
are valid inside the attribute value.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* perf(plugins): dramatically speed up plugin manager tab load time

## Problem

The Plugins tab loaded slowly and inconsistently (5–30s depending on
cache state), with a blank spinner for the entire wait. Three root
causes:

1. **N+1 subprocess per installed plugin** — `_get_local_git_info` ran
   4 separate git subprocesses per plugin (rev-parse HEAD, abbrev-ref,
   config --get remote.origin.url, log --format=%cI). With 15 plugins
   that's 60 blocking subprocess spawns before the endpoint returned.

2. **Serial per-plugin loop** — the `/plugins/installed` endpoint
   processed each plugin sequentially: manifest read → git info →
   instance lookup → Vegas mode query, one plugin at a time.

3. **Serial JS loading** — the store search only started after installed
   plugins fully completed, so users waited for both round-trips back
   to back. No UI feedback during the wait.

## Changes

### Backend — src/plugin_system/store_manager.py
- Consolidate 4 git subprocesses → 1: branch read from `.git/HEAD`
  (file I/O, no subprocess), remote URL parsed from `.git/config`
  (file I/O, no subprocess), SHA + commit date fetched together in a
  single `git log -1 --format=%H%n%cI` call
- Existing signature-based cache already eliminates all subprocesses on
  warm hits; this change cuts cold-cache cost from 4 → 1 per plugin

### Backend — web_interface/blueprints/api_v3.py
- Wrap per-plugin work in a `_build_plugin_entry()` helper and execute
  it across a `ThreadPoolExecutor(max_workers=8)` so all plugins are
  processed in parallel instead of sequentially
- Fix double `get_plugin()` call per plugin (was called once for the
  enabled fallback and again for Vegas mode — now one shared call)

### Frontend — web_interface/static/v3/plugins_manager.js
- Fire `searchPluginStore()` and `loadInstalledPlugins()` simultaneously
  instead of waiting for installed to complete before starting the store
- After installed data arrives, call `applyStoreFiltersAndSort(true)` to
  refresh install/update/reinstall badges from already-cached store data
  (instant, no extra network call)

### Frontend — web_interface/templates/v3/partials/plugins.html
- Add responsive skeleton cards to the installed plugins section that
  match real card proportions (removed automatically when data renders)
- Replace the 5 featureless gray boxes in the store skeleton with 10
  structured skeleton cards matching the real card layout

## Measured improvement on Pi 4 (11 installed plugins, ledpi-ticker)

| Scenario | Before | After |
|---|---|---|
| Cold cache (first open) | ~8–15s | **0.9s** |
| Warm cache (git cache hit) | ~1–2s | **55ms** |
| UI feedback during load | blank spinner | skeleton cards |
| Store waits for installed | yes (serial) | no (parallel) |

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(plugins): harden git metadata parsing and plugin entry building

store_manager.py:
- Detect worktree/submodule .git files (gitdir: <path>) and resolve
  to the actual git directory before reading HEAD or config
- Wrap HEAD read_text in try/except OSError/NotADirectoryError so
  atypical repos return None instead of propagating exceptions
- Guard config url line split with '=' presence check to avoid
  IndexError on malformed lines

api_v3.py:
- Wrap _build_plugin_entry body in a try/except via a thin outer
  wrapper so a single plugin's failure doesn't 500 the whole endpoint;
  failed entries return None and are filtered by the existing [r for r
  in results if r is not None] step
- Narrow manifest except clause to FileNotFoundError, PermissionError,
  json.JSONDecodeError instead of bare Exception
- Validate manifest is a dict before calling plugin_info.update() and
  log a debug message when it isn't

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

src/plugin_system/store_manager.py

@@ -1850,58 +1850,72 @@ class PluginStoreManager:
                 return cached[1]
 
         try:
-            sha_result = subprocess.run(
+            # .git may be a file (worktree / submodule) containing "gitdir: <path>".
-                ['git', '-C', str(plugin_path), 'rev-parse', 'HEAD'],
+            # Resolve it to the actual git directory before reading any files.
+            try:
+                if git_dir.is_file():
+                    pointer = git_dir.read_text(encoding='utf-8', errors='replace').strip()
+                    if pointer.startswith('gitdir:'):
+                        resolved = (plugin_path / pointer[len('gitdir:'):].strip()).resolve()
+                        if resolved.is_dir():
+                            git_dir = resolved
+                        else:
+                            return None
+                    else:
+                        return None
+            except (OSError, NotADirectoryError):
+                return None
+
+            # Read branch directly from .git/HEAD (no subprocess).
+            branch = ''
+            try:
+                head_text = (git_dir / 'HEAD').read_text(encoding='utf-8', errors='replace').strip()
+                if head_text.startswith('ref: refs/heads/'):
+                    branch = head_text[len('ref: refs/heads/'):]
+                elif head_text.startswith('ref: '):
+                    branch = head_text[len('ref: '):]
+                # else: detached HEAD — branch stays ''
+            except (OSError, NotADirectoryError):
+                pass
+
+            # Remote URL from .git/config — parse [remote "origin"] url line.
+            remote_url = None
+            try:
+                config_text = (git_dir / 'config').read_text(encoding='utf-8', errors='replace')
+                in_origin = False
+                for line in config_text.splitlines():
+                    stripped = line.strip()
+                    if stripped == '[remote "origin"]':
+                        in_origin = True
+                    elif stripped.startswith('['):
+                        in_origin = False
+                    elif in_origin and stripped.startswith('url') and '=' in stripped:
+                        remote_url = stripped.split('=', 1)[1].strip()
+                        break
+            except (OSError, NotADirectoryError):
+                pass
+
+            # Single subprocess: SHA + commit date in one call.
+            log_result = subprocess.run(
+                ['git', '-C', str(plugin_path), 'log', '-1', '--format=%H%n%cI', 'HEAD'],
                 capture_output=True,
                 text=True,
                 timeout=10,
                 check=True
             )
-            sha = sha_result.stdout.strip()
+            lines = log_result.stdout.strip().splitlines()
-
+            sha = lines[0] if lines else ''
-            branch_result = subprocess.run(
+            commit_date_iso = lines[1] if len(lines) > 1 else ''
-                ['git', '-C', str(plugin_path), 'rev-parse', '--abbrev-ref', 'HEAD'],
-                capture_output=True,
-                text=True,
-                timeout=10,
-                check=True
-            )
-            branch = branch_result.stdout.strip()
-
-            if branch == 'HEAD':
-                branch = ''
-
-            # Get remote URL
-            remote_url_result = subprocess.run(
-                ['git', '-C', str(plugin_path), 'config', '--get', 'remote.origin.url'],
-                capture_output=True,
-                text=True,
-                timeout=10,
-                check=False
-            )
-            remote_url = remote_url_result.stdout.strip() if remote_url_result.returncode == 0 else None
-
-            # Get commit date in ISO format
-            date_result = subprocess.run(
-                ['git', '-C', str(plugin_path), 'log', '-1', '--format=%cI', 'HEAD'],
-                capture_output=True,
-                text=True,
-                timeout=10,
-                check=True
-            )
-            commit_date_iso = date_result.stdout.strip()
 
             result = {
                 'sha': sha,
                 'short_sha': sha[:7] if sha else '',
-                'branch': branch
+                'branch': branch,
             }
-            
+
-            # Add remote URL if available
             if remote_url:
                 result['remote_url'] = remote_url
 
-            # Add commit date if available
             if commit_date_iso:
                 result['date_iso'] = commit_date_iso
                 result['date'] = self._iso_to_date(commit_date_iso)

web_interface/blueprints/api_v3.py

@@ -1772,61 +1772,57 @@ def get_installed_plugins():
         # Load config once before the loop (not per-plugin)
         full_config = api_v3.config_manager.load_config() if api_v3.config_manager else {}
 
-        # Format for the web interface
+        def _build_plugin_entry(plugin_info):
-        plugins = []
-        for plugin_info in all_plugin_info:
             plugin_id = plugin_info.get('id')
+            try:
+                return _build_plugin_entry_inner(plugin_info, plugin_id)
+            except Exception:
+                logger.exception("Error building plugin entry for %s — skipping", plugin_id)
+                return None
 
+        def _build_plugin_entry_inner(plugin_info, plugin_id):
             # Re-read manifest from disk to ensure we have the latest metadata
             manifest_path = Path(api_v3.plugin_manager.plugins_dir) / plugin_id / "manifest.json"
             if manifest_path.exists():
                 try:
                     with open(manifest_path, 'r', encoding='utf-8') as f:
                         fresh_manifest = json.load(f)
-                    # Update plugin_info with fresh manifest data
+                    if isinstance(fresh_manifest, dict):
-                    plugin_info.update(fresh_manifest)
+                        plugin_info.update(fresh_manifest)
-                except Exception as e:
+                    else:
-                    # If we can't read the fresh manifest, use the cached one
+                        logger.debug("Manifest for %s is not a dict (%s) — skipping merge",
-                    print(f"Warning: Could not read fresh manifest for {plugin_id}: {e}")
+                                     plugin_id, type(fresh_manifest).__name__)
+                except (FileNotFoundError, PermissionError, json.JSONDecodeError) as e:
+                    logger.debug("Could not read fresh manifest for %s: %s", plugin_id, e)
 
-            # Get enabled status from config (source of truth)
+            # Enabled status: config is source of truth, fall back to instance
-            # Read from config file first, fall back to plugin instance if config doesn't have the key
             enabled = None
-            if api_v3.config_manager:
+            plugin_config = full_config.get(plugin_id, {})
-                plugin_config = full_config.get(plugin_id, {})
+            if 'enabled' in plugin_config:
-                # Check if 'enabled' key exists in config (even if False)
+                enabled = bool(plugin_config['enabled'])
-                if 'enabled' in plugin_config:
-                    enabled = bool(plugin_config['enabled'])
 
-            # Fallback to plugin instance if config doesn't have enabled key
+            # Single get_plugin() call shared for both enabled fallback and Vegas mode
+            plugin_instance = api_v3.plugin_manager.get_plugin(plugin_id)
             if enabled is None:
-                plugin_instance = api_v3.plugin_manager.get_plugin(plugin_id)
+                enabled = plugin_instance.enabled if plugin_instance else True
-                if plugin_instance:
-                    enabled = plugin_instance.enabled
-                else:
-                    # Default to True if no config key and plugin not loaded (matches BasePlugin default)
-                    enabled = True
 
-            # Get verified status from store registry (no GitHub API calls needed)
+            # Verified from registry (no network call)
             store_info = api_v3.plugin_store_manager.get_registry_info(plugin_id)
             verified = store_info.get('verified', False) if store_info else False
 
-            # Get local git info for installed plugin (actual installed commit)
+            # Local git info (single subprocess on cache miss, zero on hit)
             plugin_path = Path(api_v3.plugin_manager.plugins_dir) / plugin_id
             local_git_info = api_v3.plugin_store_manager._get_local_git_info(plugin_path) if plugin_path.exists() else None
 
-            # Use local git info if available (actual installed commit), otherwise fall back to manifest/store info
             if local_git_info:
-                last_commit = local_git_info.get('short_sha') or local_git_info.get('sha', '')[:7] if local_git_info.get('sha') else None
+                sha = local_git_info.get('sha', '')
+                last_commit = local_git_info.get('short_sha') or (sha[:7] if sha else None)
                 branch = local_git_info.get('branch')
-                # Use commit date from git if available
                 last_updated = local_git_info.get('date_iso') or local_git_info.get('date')
             else:
-                # Fall back to manifest/store info if no local git info
                 last_updated = plugin_info.get('last_updated')
                 last_commit = plugin_info.get('last_commit') or plugin_info.get('last_commit_sha')
                 branch = plugin_info.get('branch')
-
                 if store_info:
                     last_updated = last_updated or store_info.get('last_updated') or store_info.get('last_updated_iso')
                     last_commit = last_commit or store_info.get('last_commit') or store_info.get('last_commit_sha')
@@ -1836,35 +1832,26 @@ def get_installed_plugins():
             if store_info and not last_commit_message:
                 last_commit_message = store_info.get('last_commit_message')
 
-            # Get web_ui_actions from manifest if available
+            # Vegas mode from instance, overridden by explicit config value
-            web_ui_actions = plugin_info.get('web_ui_actions', [])
-
-            # Get Vegas display mode info from plugin instance
             vegas_mode = None
             vegas_content_type = None
-            plugin_instance = api_v3.plugin_manager.get_plugin(plugin_id)
             if plugin_instance:
                 try:
-                    # Try to get the display mode enum
                     if hasattr(plugin_instance, 'get_vegas_display_mode'):
                         mode = plugin_instance.get_vegas_display_mode()
                         vegas_mode = mode.value if hasattr(mode, 'value') else str(mode)
                 except (AttributeError, TypeError, ValueError) as e:
                     logger.debug("[%s] Failed to get vegas_display_mode: %s", plugin_id, e)
                 try:
-                    # Get legacy content type as fallback
                     if hasattr(plugin_instance, 'get_vegas_content_type'):
                         vegas_content_type = plugin_instance.get_vegas_content_type()
                 except (AttributeError, TypeError, ValueError) as e:
                     logger.debug("[%s] Failed to get vegas_content_type: %s", plugin_id, e)
 
-            # Also check plugin config for explicit vegas_mode setting
+            if 'vegas_mode' in plugin_config:
-            if api_v3.config_manager:
+                vegas_mode = plugin_config['vegas_mode']
-                plugin_cfg = full_config.get(plugin_id, {})
-                if 'vegas_mode' in plugin_cfg:
-                    vegas_mode = plugin_cfg['vegas_mode']
 
-            plugins.append({
+            return {
                 'id': plugin_id,
                 'name': plugin_info.get('name', plugin_id),
                 'version': plugin_info.get('version', ''),
@@ -1879,10 +1866,15 @@ def get_installed_plugins():
                 'last_commit': last_commit,
                 'last_commit_message': last_commit_message,
                 'branch': branch,
-                'web_ui_actions': web_ui_actions,
+                'web_ui_actions': plugin_info.get('web_ui_actions', []),
                 'vegas_mode': vegas_mode,
-                'vegas_content_type': vegas_content_type
+                'vegas_content_type': vegas_content_type,
-            })
+            }
+
+        from concurrent.futures import ThreadPoolExecutor
+        with ThreadPoolExecutor(max_workers=8) as executor:
+            results = list(executor.map(_build_plugin_entry, all_plugin_info))
+        plugins = [r for r in results if r is not None]
 
         return jsonify({'status': 'success', 'data': {'plugins': plugins}})
     except Exception as e:

web_interface/static/v3/plugins_manager.js

@@ -1223,13 +1223,21 @@ function initializePlugins() {
     // during a re-swap, fetch fresh data including GitHub commit/version info.
     const isReswapWarm = !!window.pluginManager._reswap && !storeCacheExpired();
     window.pluginManager._reswap = false;
-    // Await the installed-plugins fetch so window.installedPlugins is populated before
+
-    // searchPluginStore renders Installed/Reinstall badges against it.
+    // Fire both requests in parallel so the store doesn't wait for installed plugins.
-    loadInstalledPlugins().catch(err => {
+    // The store renders install/update badges using window.installedPlugins || [] so
-        console.error('[PluginStore] loadInstalledPlugins failed:', err);
+    // it works with an empty list. When installed plugins finish loading we do a
-    }).finally(() => {
+    // lightweight re-render from the already-cached store data to refresh the badges.
-        searchPluginStore(!isReswapWarm);
+    searchPluginStore(!isReswapWarm);
-    });
+    loadInstalledPlugins()
+        .catch(err => console.error('[PluginStore] loadInstalledPlugins failed:', err))
+        .then(() => {
+            // Re-render store from cache to update install/update/reinstall badges now
+            // that window.installedPlugins is populated. No network call — instant.
+            if (typeof applyStoreFiltersAndSort === 'function') {
+                applyStoreFiltersAndSort(true);
+            }
+        });
 
     // Setup search functionality (with guard against duplicate listeners)
     const searchInput = document.getElementById('plugin-search');
@@ -1418,6 +1426,9 @@ function renderInstalledPlugins(plugins) {
         }
     }
 
+    // Remove skeleton cards before rendering real content
+    container.querySelectorAll('.installed-skeleton').forEach(el => el.remove());
+
     if (plugins.length === 0) {
         container.innerHTML = `
             <div class="col-span-full empty-state">

web_interface/templates/v3/partials/plugins.html

@@ -31,7 +31,52 @@
             </div>
             <div id="installed-plugins-content" class="block">
                 <div id="installed-plugins-grid" class="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 xl:grid-cols-4 2xl:grid-cols-5 gap-6">
-                    <!-- Plugins will be loaded here -->
+                    <!-- Skeleton cards shown while installed plugins load -->
+                    <div class="installed-skeleton plugin-card animate-pulse">
+                        <div class="flex items-start justify-between mb-4">
+                            <div class="flex-1 min-w-0 space-y-2">
+                                <div class="h-4 bg-gray-200 rounded w-3/4"></div>
+                                <div class="h-3 bg-gray-200 rounded w-1/2"></div>
+                                <div class="h-3 bg-gray-200 rounded w-2/3"></div>
+                            </div>
+                            <div class="h-6 w-10 bg-gray-200 rounded-full ml-4 flex-shrink-0"></div>
+                        </div>
+                        <div class="space-y-2 mb-4">
+                            <div class="h-3 bg-gray-200 rounded w-full"></div>
+                            <div class="h-3 bg-gray-200 rounded w-5/6"></div>
+                        </div>
+                        <div class="h-8 bg-gray-200 rounded w-full mt-auto"></div>
+                    </div>
+                    <div class="installed-skeleton plugin-card animate-pulse hidden md:block">
+                        <div class="flex items-start justify-between mb-4">
+                            <div class="flex-1 min-w-0 space-y-2">
+                                <div class="h-4 bg-gray-200 rounded w-2/3"></div>
+                                <div class="h-3 bg-gray-200 rounded w-1/3"></div>
+                                <div class="h-3 bg-gray-200 rounded w-1/2"></div>
+                            </div>
+                            <div class="h-6 w-10 bg-gray-200 rounded-full ml-4 flex-shrink-0"></div>
+                        </div>
+                        <div class="space-y-2 mb-4">
+                            <div class="h-3 bg-gray-200 rounded w-full"></div>
+                            <div class="h-3 bg-gray-200 rounded w-4/5"></div>
+                        </div>
+                        <div class="h-8 bg-gray-200 rounded w-full mt-auto"></div>
+                    </div>
+                    <div class="installed-skeleton plugin-card animate-pulse hidden lg:block">
+                        <div class="flex items-start justify-between mb-4">
+                            <div class="flex-1 min-w-0 space-y-2">
+                                <div class="h-4 bg-gray-200 rounded w-4/5"></div>
+                                <div class="h-3 bg-gray-200 rounded w-2/5"></div>
+                                <div class="h-3 bg-gray-200 rounded w-3/5"></div>
+                            </div>
+                            <div class="h-6 w-10 bg-gray-200 rounded-full ml-4 flex-shrink-0"></div>
+                        </div>
+                        <div class="space-y-2 mb-4">
+                            <div class="h-3 bg-gray-200 rounded w-full"></div>
+                            <div class="h-3 bg-gray-200 rounded w-3/4"></div>
+                        </div>
+                        <div class="h-8 bg-gray-200 rounded w-full mt-auto"></div>
+                    </div>
                 </div>
             </div>
         </div>
@@ -203,14 +248,30 @@
             </div>
 
             <div id="plugin-store-grid" class="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 xl:grid-cols-4 2xl:grid-cols-5 gap-6">
-                <!-- Loading skeleton -->
+                <!-- Loading skeleton — hidden by showStoreLoading(false) when data arrives -->
                 <div class="store-loading col-span-full">
-                    <div class="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 xl:grid-cols-4 2xl:grid-cols-5 gap-4">
+                    <div class="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 xl:grid-cols-4 2xl:grid-cols-5 gap-6">
-                        <div class="bg-gray-200 rounded-lg p-4 h-48 animate-pulse"></div>
+                        {% for _ in range(10) %}
-                        <div class="bg-gray-200 rounded-lg p-4 h-48 animate-pulse"></div>
+                        <div class="plugin-card animate-pulse">
-                        <div class="bg-gray-200 rounded-lg p-4 h-48 animate-pulse"></div>
+                            <div class="flex items-start justify-between mb-4">
-                        <div class="bg-gray-200 rounded-lg p-4 h-48 animate-pulse"></div>
+                                <div class="flex-1 min-w-0 space-y-2">
-                        <div class="bg-gray-200 rounded-lg p-4 h-48 animate-pulse"></div>
+                                    <div class="h-4 bg-gray-200 rounded w-3/4"></div>
+                                    <div class="h-3 bg-gray-200 rounded w-1/2"></div>
+                                </div>
+                                <div class="h-5 w-14 bg-gray-200 rounded-full ml-3 flex-shrink-0"></div>
+                            </div>
+                            <div class="space-y-2 mb-4">
+                                <div class="h-3 bg-gray-200 rounded w-full"></div>
+                                <div class="h-3 bg-gray-200 rounded w-5/6"></div>
+                                <div class="h-3 bg-gray-200 rounded w-4/6"></div>
+                            </div>
+                            <div class="flex gap-2 mt-auto">
+                                <div class="h-3 bg-gray-200 rounded w-12"></div>
+                                <div class="h-3 bg-gray-200 rounded w-16"></div>
+                            </div>
+                            <div class="h-8 bg-gray-200 rounded w-full mt-3"></div>
+                        </div>
+                        {% endfor %}
                     </div>
                 </div>
             </div>