Files
LEDMatrix/src/plugin_system/state_reconciliation.py
Chuck 2c2fca2219 fix(web): use HTMX for Plugin Manager tab loading (#294)
* fix: auto-repair missing plugins and graceful config fallback

Plugins whose directories are missing (failed update, migration, etc.)
now get automatically reinstalled from the store on startup. The config
endpoint no longer returns a hard 500 when a schema is unavailable —
it falls back to conservative key-name-based masking so the settings
page stays functional.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: handle ledmatrix- prefix in plugin updates and reconciliation

The store registry uses unprefixed IDs (e.g., 'weather') while older
installs used prefixed config keys (e.g., 'ledmatrix-weather'). Both
update_plugin() and auto-repair now try the unprefixed ID as a fallback
when the prefixed one isn't found in the registry.

Also filters system config keys (schedule, display, etc.) from
reconciliation to avoid false positives.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: address code review findings for plugin auto-repair

- Move backup-folder filter from _get_config_state to _get_disk_state
  where the artifact actually lives
- Run startup reconciliation in a background thread so requests aren't
  blocked by plugin reinstallation
- Set _reconciliation_done only after success so failures allow retries
- Replace print() with proper logger in reconciliation
- Wrap load_schema in try/except so exceptions fall through to
  conservative masking instead of 500
- Handle list values in _conservative_mask_config for nested secrets
- Remove duplicate import re

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: add thread-safe locking to PluginManager and fix reconciliation retry

PluginManager thread safety:
- Add RLock protecting plugin_manifests and plugin_directories
- Build scan results locally in _scan_directory_for_plugins, then update
  shared state under lock
- Protect reads in get_plugin_info, get_all_plugin_info,
  get_plugin_directory, get_plugin_display_modes, find_plugin_for_mode
- Protect manifest mutation in reload_plugin
- Prevents races between background reconciliation thread and request
  handlers reading plugin state

Reconciliation retry:
- Clear _reconciliation_started on exception so next request retries
- Check result.reconciliation_successful before marking done
- Reset _reconciliation_started on non-success results to allow retry

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(web): use HTMX for Plugin Manager tab loading instead of custom fetch

The Plugin Manager tab was the only tab using a custom window.loadPluginsTab()
function with plain fetch() instead of HTMX. This caused a race condition where
plugins_manager.js listened for htmx:afterSwap to initialize, but that event
never fired for the custom fetch. Users had to navigate to a plugin config tab
and back to trigger initialization.

Changes:
- Switch plugins tab to hx-get/hx-trigger="revealed" matching all other tabs
- Remove ~560 lines of dead code (script extraction for a partial with no scripts,
  nested retry intervals, inline HTML card rendering fallbacks)
- Add simple loadPluginsDirect() fallback for when HTMX fails to load
- Remove typeof htmx guard on afterSwap listener so it registers unconditionally
- Tighten afterSwap target check to avoid spurious re-init from other tab swaps

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: address CodeRabbit review findings across plugin system

- plugin_manager.py: clear plugin_manifests/plugin_directories before update
  to prevent ghost entries for uninstalled plugins persisting across scans
- state_reconciliation.py: remove 'enabled' key check that skipped legacy
  plugin configs, default to enabled=True matching PluginManager.load_plugin
- app.py: add threading.Lock around reconciliation start guard to prevent
  race condition spawning duplicate threads; add -> None return annotation
- store_manager.py: use resolved registry ID (alt_id) instead of original
  plugin_id when reinstalling during monorepo migration
- base.html: check Response.ok in loadPluginsDirect fallback; trigger
  fallback on tab click when HTMX unavailable; remove active-tab check
  from 5-second timeout so content preloads regardless

Skipped: api_v3.py secret redaction suggestion — the caller at line 2539
already tries schema-based mask_secret_fields() before falling back to
_conservative_mask_config, making the suggested change redundant.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: skip backup dirs in plugin discovery and fix HTMX event syntax

- plugin_manager.py: skip directories containing '.standalone-backup-'
  during discovery scan, matching state_reconciliation.py behavior and
  preventing backup manifests from overwriting live plugin entries
- base.html: fix hx-on::htmx:response-error → hx-on::response-error
  (the :: shorthand already adds the htmx: prefix, so the original
  syntax resolved to htmx:htmx:response-error making the handler dead)

Skipped findings:
- web-ui-info in _SYSTEM_CONFIG_KEYS: it's a real plugin with manifest.json
  and config entry, not a system key
- store_manager config key migration: valid feature request for handling
  ledmatrix- prefix rename, but new functionality outside this PR scope

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(web): add fetch timeout to loadPluginsDirect fallback

Add AbortController with 10s timeout so a hanging fetch doesn't leave
data-loaded set and block retries. Timer is cleared in both success
and error paths.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-26 15:21:33 -04:00

372 lines
15 KiB
Python

"""
State reconciliation system.
Detects and fixes inconsistencies between:
- Config file state
- Plugin manager state
- Disk state (installed plugins)
- State manager state
"""
from typing import Dict, Any, List, Optional
from dataclasses import dataclass
from enum import Enum
from pathlib import Path
from src.plugin_system.state_manager import PluginStateManager, PluginState, PluginStateStatus
from src.logging_config import get_logger
class InconsistencyType(Enum):
"""Types of state inconsistencies."""
PLUGIN_MISSING_IN_CONFIG = "plugin_missing_in_config"
PLUGIN_MISSING_ON_DISK = "plugin_missing_on_disk"
PLUGIN_ENABLED_MISMATCH = "plugin_enabled_mismatch"
PLUGIN_VERSION_MISMATCH = "plugin_version_mismatch"
PLUGIN_STATE_CORRUPTED = "plugin_state_corrupted"
class FixAction(Enum):
"""Actions that can be taken to fix inconsistencies."""
AUTO_FIX = "auto_fix"
MANUAL_FIX_REQUIRED = "manual_fix_required"
NO_ACTION = "no_action"
@dataclass
class Inconsistency:
"""Represents a state inconsistency."""
plugin_id: str
inconsistency_type: InconsistencyType
description: str
fix_action: FixAction
current_state: Dict[str, Any]
expected_state: Dict[str, Any]
can_auto_fix: bool = False
@dataclass
class ReconciliationResult:
"""Result of state reconciliation."""
inconsistencies_found: List[Inconsistency]
inconsistencies_fixed: List[Inconsistency]
inconsistencies_manual: List[Inconsistency]
reconciliation_successful: bool
message: str
class StateReconciliation:
"""
State reconciliation system.
Compares state from multiple sources and detects/fixes inconsistencies.
"""
def __init__(
self,
state_manager: PluginStateManager,
config_manager,
plugin_manager,
plugins_dir: Path,
store_manager=None
):
"""
Initialize reconciliation system.
Args:
state_manager: PluginStateManager instance
config_manager: ConfigManager instance
plugin_manager: PluginManager instance
plugins_dir: Path to plugins directory
store_manager: Optional PluginStoreManager for auto-repair
"""
self.state_manager = state_manager
self.config_manager = config_manager
self.plugin_manager = plugin_manager
self.plugins_dir = Path(plugins_dir)
self.store_manager = store_manager
self.logger = get_logger(__name__)
def reconcile_state(self) -> ReconciliationResult:
"""
Perform state reconciliation.
Compares state from all sources and fixes safe inconsistencies.
Returns:
ReconciliationResult with findings and fixes
"""
self.logger.info("Starting state reconciliation")
inconsistencies = []
fixed = []
manual_fix_required = []
try:
# Get state from all sources
config_state = self._get_config_state()
disk_state = self._get_disk_state()
manager_state = self._get_manager_state()
state_manager_state = self._get_state_manager_state()
# Find all unique plugin IDs
all_plugin_ids = set()
all_plugin_ids.update(config_state.keys())
all_plugin_ids.update(disk_state.keys())
all_plugin_ids.update(manager_state.keys())
all_plugin_ids.update(state_manager_state.keys())
# Check each plugin for inconsistencies
for plugin_id in all_plugin_ids:
plugin_inconsistencies = self._check_plugin_consistency(
plugin_id,
config_state,
disk_state,
manager_state,
state_manager_state
)
inconsistencies.extend(plugin_inconsistencies)
# Attempt to fix auto-fixable inconsistencies
for inconsistency in inconsistencies:
if inconsistency.can_auto_fix and inconsistency.fix_action == FixAction.AUTO_FIX:
if self._fix_inconsistency(inconsistency):
fixed.append(inconsistency)
else:
manual_fix_required.append(inconsistency)
elif inconsistency.fix_action == FixAction.MANUAL_FIX_REQUIRED:
manual_fix_required.append(inconsistency)
# Build result
success = len(manual_fix_required) == 0
message = (
f"Reconciliation complete: {len(inconsistencies)} inconsistencies found, "
f"{len(fixed)} fixed automatically, {len(manual_fix_required)} require manual attention"
)
return ReconciliationResult(
inconsistencies_found=inconsistencies,
inconsistencies_fixed=fixed,
inconsistencies_manual=manual_fix_required,
reconciliation_successful=success,
message=message
)
except Exception as e:
self.logger.error(f"Error during state reconciliation: {e}", exc_info=True)
return ReconciliationResult(
inconsistencies_found=inconsistencies,
inconsistencies_fixed=fixed,
inconsistencies_manual=manual_fix_required,
reconciliation_successful=False,
message=f"Reconciliation failed: {str(e)}"
)
# Top-level config keys that are NOT plugins
_SYSTEM_CONFIG_KEYS = frozenset({
'web_display_autostart', 'timezone', 'location', 'display',
'plugin_system', 'vegas_scroll_speed', 'vegas_separator_width',
'vegas_target_fps', 'vegas_buffer_ahead', 'vegas_plugin_order',
'vegas_excluded_plugins', 'vegas_scroll_enabled', 'logging',
'dim_schedule', 'network', 'system', 'schedule',
})
def _get_config_state(self) -> Dict[str, Dict[str, Any]]:
"""Get plugin state from config file."""
state = {}
try:
config = self.config_manager.load_config()
for plugin_id, plugin_config in config.items():
if not isinstance(plugin_config, dict):
continue
if plugin_id in self._SYSTEM_CONFIG_KEYS:
continue
state[plugin_id] = {
'enabled': plugin_config.get('enabled', True),
'version': plugin_config.get('version'),
'exists_in_config': True
}
except Exception as e:
self.logger.warning(f"Error reading config state: {e}")
return state
def _get_disk_state(self) -> Dict[str, Dict[str, Any]]:
"""Get plugin state from disk (installed plugins)."""
state = {}
try:
if self.plugins_dir.exists():
for plugin_dir in self.plugins_dir.iterdir():
if plugin_dir.is_dir():
plugin_id = plugin_dir.name
if '.standalone-backup-' in plugin_id:
continue
manifest_path = plugin_dir / "manifest.json"
if manifest_path.exists():
import json
try:
with open(manifest_path, 'r') as f:
manifest = json.load(f)
state[plugin_id] = {
'exists_on_disk': True,
'version': manifest.get('version'),
'name': manifest.get('name')
}
except Exception:
pass
except Exception as e:
self.logger.warning(f"Error reading disk state: {e}")
return state
def _get_manager_state(self) -> Dict[str, Dict[str, Any]]:
"""Get plugin state from plugin manager."""
state = {}
try:
if self.plugin_manager:
# Get discovered plugins
if hasattr(self.plugin_manager, 'plugin_manifests'):
for plugin_id in self.plugin_manager.plugin_manifests.keys():
state[plugin_id] = {
'exists_in_manager': True,
'loaded': plugin_id in getattr(self.plugin_manager, 'plugins', {})
}
except Exception as e:
self.logger.warning(f"Error reading manager state: {e}")
return state
def _get_state_manager_state(self) -> Dict[str, Dict[str, Any]]:
"""Get plugin state from state manager."""
state = {}
try:
all_states = self.state_manager.get_all_states()
for plugin_id, plugin_state in all_states.items():
state[plugin_id] = {
'enabled': plugin_state.enabled,
'status': plugin_state.status.value,
'version': plugin_state.version,
'exists_in_state_manager': True
}
except Exception as e:
self.logger.warning(f"Error reading state manager state: {e}")
return state
def _check_plugin_consistency(
self,
plugin_id: str,
config_state: Dict[str, Dict[str, Any]],
disk_state: Dict[str, Dict[str, Any]],
manager_state: Dict[str, Dict[str, Any]],
state_manager_state: Dict[str, Dict[str, Any]]
) -> List[Inconsistency]:
"""Check consistency for a single plugin."""
inconsistencies = []
config = config_state.get(plugin_id, {})
disk = disk_state.get(plugin_id, {})
manager = manager_state.get(plugin_id, {})
state_mgr = state_manager_state.get(plugin_id, {})
# Check: Plugin exists on disk but not in config
if disk.get('exists_on_disk') and not config.get('exists_in_config'):
inconsistencies.append(Inconsistency(
plugin_id=plugin_id,
inconsistency_type=InconsistencyType.PLUGIN_MISSING_IN_CONFIG,
description=f"Plugin {plugin_id} exists on disk but not in config",
fix_action=FixAction.AUTO_FIX,
current_state={'exists_in_config': False},
expected_state={'exists_in_config': True, 'enabled': False},
can_auto_fix=True
))
# Check: Plugin in config but not on disk
if config.get('exists_in_config') and not disk.get('exists_on_disk'):
can_repair = self.store_manager is not None
inconsistencies.append(Inconsistency(
plugin_id=plugin_id,
inconsistency_type=InconsistencyType.PLUGIN_MISSING_ON_DISK,
description=f"Plugin {plugin_id} in config but not on disk",
fix_action=FixAction.AUTO_FIX if can_repair else FixAction.MANUAL_FIX_REQUIRED,
current_state={'exists_on_disk': False},
expected_state={'exists_on_disk': True},
can_auto_fix=can_repair
))
# Check: Enabled state mismatch
config_enabled = config.get('enabled', False)
state_mgr_enabled = state_mgr.get('enabled')
if state_mgr_enabled is not None and config_enabled != state_mgr_enabled:
inconsistencies.append(Inconsistency(
plugin_id=plugin_id,
inconsistency_type=InconsistencyType.PLUGIN_ENABLED_MISMATCH,
description=f"Plugin {plugin_id} enabled state mismatch: config={config_enabled}, state_manager={state_mgr_enabled}",
fix_action=FixAction.AUTO_FIX,
current_state={'enabled': config_enabled},
expected_state={'enabled': state_mgr_enabled},
can_auto_fix=True
))
return inconsistencies
def _fix_inconsistency(self, inconsistency: Inconsistency) -> bool:
"""Attempt to fix an inconsistency."""
try:
if inconsistency.inconsistency_type == InconsistencyType.PLUGIN_MISSING_IN_CONFIG:
# Add plugin to config with default disabled state
config = self.config_manager.load_config()
config[inconsistency.plugin_id] = {
'enabled': False
}
self.config_manager.save_config(config)
self.logger.info(f"Fixed: Added {inconsistency.plugin_id} to config")
return True
elif inconsistency.inconsistency_type == InconsistencyType.PLUGIN_MISSING_ON_DISK:
return self._auto_repair_missing_plugin(inconsistency.plugin_id)
elif inconsistency.inconsistency_type == InconsistencyType.PLUGIN_ENABLED_MISMATCH:
# Sync enabled state from state manager to config
expected_enabled = inconsistency.expected_state.get('enabled')
config = self.config_manager.load_config()
if inconsistency.plugin_id not in config:
config[inconsistency.plugin_id] = {}
config[inconsistency.plugin_id]['enabled'] = expected_enabled
self.config_manager.save_config(config)
self.logger.info(f"Fixed: Synced enabled state for {inconsistency.plugin_id}")
return True
except Exception as e:
self.logger.error(f"Error fixing inconsistency: {e}", exc_info=True)
return False
return False
def _auto_repair_missing_plugin(self, plugin_id: str) -> bool:
"""Attempt to reinstall a missing plugin from the store."""
if not self.store_manager:
return False
# Try the plugin_id as-is, then without 'ledmatrix-' prefix
candidates = [plugin_id]
if plugin_id.startswith('ledmatrix-'):
candidates.append(plugin_id[len('ledmatrix-'):])
for candidate_id in candidates:
try:
self.logger.info("[AutoRepair] Attempting to reinstall missing plugin: %s", candidate_id)
result = self.store_manager.install_plugin(candidate_id)
if isinstance(result, dict):
success = result.get('success', False)
else:
success = bool(result)
if success:
self.logger.info("[AutoRepair] Successfully reinstalled plugin: %s (config key: %s)", candidate_id, plugin_id)
return True
except Exception as e:
self.logger.error("[AutoRepair] Error reinstalling %s: %s", candidate_id, e, exc_info=True)
self.logger.warning("[AutoRepair] Could not reinstall %s from store", plugin_id)
return False