fix(plugins): prevent root-owned files from blocking plugin updates (#242)

* fix(web): unify operation history tracking for monorepo plugin operations

The operation history UI was reading from the wrong data source
(operation_queue instead of operation_history), install/update records
lacked version details, toggle operations used a type name that didn't
match UI filters, and the Clear History button was non-functional.

- Switch GET /plugins/operation/history to read from OperationHistory
  audit log with return type hint and targeted exception handling
- Add DELETE /plugins/operation/history endpoint; wire up Clear button
- Add _get_plugin_version helper with specific exception handling
  (FileNotFoundError, PermissionError, json.JSONDecodeError) and
  structured logging with plugin_id/path context
- Record plugin version, branch, and commit details on install/update
- Record install failures in the direct (non-queue) code path
- Replace "toggle" operation type with "enable"/"disable"
- Add normalizeStatus() in JS to map completed→success, error→failed
  so status filter works regardless of server-side convention
- Truncate commit SHAs to 7 chars in details display
- Fix HTML filter options, operation type colors, duplicate JS init

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(plugins): prevent root-owned files from blocking plugin updates

The root ledmatrix service creates __pycache__ and data cache files
owned by root inside plugin directories. The web service (non-root)
cannot delete these when updating or uninstalling plugins, causing
operations to fail with "Permission denied".

Defense in depth with three layers:
- Prevent: PYTHONDONTWRITEBYTECODE=1 in systemd service + run.py
- Fallback: sudoers rules for rm on plugin directories
- Code: _safe_remove_directory() now uses sudo as last resort,
  and all bare shutil.rmtree() calls routed through it

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(security): harden sudo removal with path-validated helper script

Address code review findings:

- Replace raw rm/find sudoers wildcards with a vetted helper script
  (safe_plugin_rm.sh) that resolves symlinks and validates the target
  is a strict child of plugin-repos/ or plugins/ before deletion
- Add allow-list validation in sudo_remove_directory() that checks
  resolved paths against allowed bases before invoking sudo
- Check _safe_remove_directory() return value before shutil.move()
  in the manifest ID rename path
- Move stat import to module level in store_manager.py
- Use stat.S_IRWXU instead of 0o777 in chmod fallback stage
- Add ignore_errors=True to temp dir cleanup in finally block
- Use command -v instead of which in configure_web_sudo.sh

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(security): address code review round 2 — harden paths and error handling

- safe_plugin_rm.sh: use realpath --canonicalize-missing for ALLOWED_BASES
  so the script doesn't fail under set -e when dirs don't exist yet
- safe_plugin_rm.sh: add -- before path in rm -rf to prevent flag injection
- permission_utils.py: use shutil.which('bash') instead of hardcoded /bin/bash
  to match whatever path the sudoers BASH_PATH resolves to
- store_manager.py: check _safe_remove_directory() return before shutil.move()
  in _install_from_monorepo_zip to prevent moving into a non-removed target
- store_manager.py: catch OSError instead of PermissionError in Stage 1 removal
  to handle both EACCES and EPERM error codes
- store_manager.py: hoist sudo_remove_directory import to module level
- configure_web_sudo.sh: harden safe_plugin_rm.sh to root-owned 755 so
  the web user cannot modify the vetted helper script

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(security): validate command paths in sudoers config and use resolved paths

- configure_web_sudo.sh: validate that required commands (systemctl, bash,
  python3) resolve to non-empty paths before generating sudoers entries;
  abort with clear error if any are missing; skip optional commands
  (reboot, poweroff, journalctl) with a warning instead of emitting
  malformed NOPASSWD lines; validate helper script exists on disk
- permission_utils.py: pass the already-resolved path to the subprocess
  call and use it for the post-removal exists() check, eliminating a
  TOCTOU window between Python-side validation and shell-side execution

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Chuck <chuck@example.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

run.py

@@ -4,6 +4,11 @@ import sys
 import os
 import argparse
 
+# Prevent Python from creating __pycache__ directories in plugin dirs.
+# The root service loads plugins via importlib, and root-owned __pycache__
+# files block the web service (non-root) from updating/uninstalling plugins.
+sys.dont_write_bytecode = True
+
 # Add project directory to Python path (needed before importing src modules)
 project_dir = os.path.dirname(os.path.abspath(__file__))
 if project_dir not in sys.path:

scripts/fix_perms/safe_plugin_rm.sh

@@ -0,0 +1,61 @@
+#!/bin/bash
+# safe_plugin_rm.sh — Safely remove a plugin directory after validating
+# that the resolved path is inside an allowed base directory.
+#
+# This script is intended to be called via sudo from the web interface.
+# It prevents path traversal attacks by resolving symlinks and verifying
+# the target is a child of plugin-repos/ or plugins/.
+#
+# Usage: safe_plugin_rm.sh <target_path>
+
+set -euo pipefail
+
+if [ $# -ne 1 ]; then
+    echo "Usage: $0 <target_path>" >&2
+    exit 1
+fi
+
+TARGET="$1"
+
+# Determine the project root (parent of scripts/fix_perms/)
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
+
+# Allowed base directories (resolved, no trailing slash)
+# Use --canonicalize-missing so this works even if the dirs don't exist yet
+ALLOWED_BASES=(
+    "$(realpath --canonicalize-missing "$PROJECT_ROOT/plugin-repos")"
+    "$(realpath --canonicalize-missing "$PROJECT_ROOT/plugins")"
+)
+
+# Resolve the target path (follow symlinks)
+# Use realpath --canonicalize-missing so it works even if the path
+# doesn't fully exist (e.g., partially deleted directory)
+RESOLVED_TARGET="$(realpath --canonicalize-missing "$TARGET")"
+
+# Validate: resolved target must be a strict child of an allowed base
+# (must not BE the base itself — only children are allowed)
+ALLOWED=false
+for BASE in "${ALLOWED_BASES[@]}"; do
+    if [[ "$RESOLVED_TARGET" == "$BASE/"* ]]; then
+        ALLOWED=true
+        break
+    fi
+done
+
+if [ "$ALLOWED" = false ]; then
+    echo "DENIED: $RESOLVED_TARGET is not inside an allowed plugin directory" >&2
+    echo "Allowed bases: ${ALLOWED_BASES[*]}" >&2
+    exit 2
+fi
+
+# Safety check: refuse to delete the base directories themselves
+for BASE in "${ALLOWED_BASES[@]}"; do
+    if [ "$RESOLVED_TARGET" = "$BASE" ]; then
+        echo "DENIED: cannot remove plugin base directory itself: $BASE" >&2
+        exit 2
+    fi
+done
+
+# All checks passed — remove the target
+rm -rf -- "$RESOLVED_TARGET"

scripts/install/configure_web_sudo.sh

@@ -10,9 +10,11 @@ echo "Configuring passwordless sudo access for LED Matrix Web Interface..."
 # Get the current user (should be the user running the web interface)
 WEB_USER=$(whoami)
 PROJECT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(cd "$PROJECT_DIR/../.." && pwd)"
 
 echo "Detected web interface user: $WEB_USER"
 echo "Project directory: $PROJECT_DIR"
+echo "Project root: $PROJECT_ROOT"
 
 # Check if running as root
 if [ "$EUID" -eq 0 ]; then
@@ -21,50 +23,92 @@ if [ "$EUID" -eq 0 ]; then
     exit 1
 fi
 
-# Get the full paths to commands
+# Get the full paths to commands and validate each one
-PYTHON_PATH=$(which python3)
+MISSING_CMDS=()
-SYSTEMCTL_PATH=$(which systemctl)
+
-REBOOT_PATH=$(which reboot)
+PYTHON_PATH=$(command -v python3)   || true
-POWEROFF_PATH=$(which poweroff)
+SYSTEMCTL_PATH=$(command -v systemctl) || true
-BASH_PATH=$(which bash)
+REBOOT_PATH=$(command -v reboot)    || true
-JOURNALCTL_PATH=$(which journalctl)
+POWEROFF_PATH=$(command -v poweroff)  || true
+BASH_PATH=$(command -v bash)        || true
+JOURNALCTL_PATH=$(command -v journalctl) || true
+SAFE_RM_PATH="$PROJECT_ROOT/scripts/fix_perms/safe_plugin_rm.sh"
+
+# Validate required commands (systemctl, bash, python3 are essential)
+for CMD_NAME in SYSTEMCTL_PATH BASH_PATH PYTHON_PATH; do
+    CMD_VAL="${!CMD_NAME}"
+    if [ -z "$CMD_VAL" ]; then
+        MISSING_CMDS+=("$CMD_NAME")
+    fi
+done
+
+if [ ${#MISSING_CMDS[@]} -gt 0 ]; then
+    echo "Error: Required commands not found: ${MISSING_CMDS[*]}" >&2
+    echo "Cannot generate valid sudoers configuration without these." >&2
+    exit 1
+fi
+
+# Validate helper script exists
+if [ ! -f "$SAFE_RM_PATH" ]; then
+    echo "Error: Safe plugin removal helper not found: $SAFE_RM_PATH" >&2
+    exit 1
+fi
 
 echo "Command paths:"
 echo "  Python: $PYTHON_PATH"
 echo "  Systemctl: $SYSTEMCTL_PATH"
-echo "  Reboot: $REBOOT_PATH"
+echo "  Reboot: ${REBOOT_PATH:-(not found, skipping)}"
-echo "  Poweroff: $POWEROFF_PATH"
+echo "  Poweroff: ${POWEROFF_PATH:-(not found, skipping)}"
 echo "  Bash: $BASH_PATH"
-echo "  Journalctl: $JOURNALCTL_PATH"
+echo "  Journalctl: ${JOURNALCTL_PATH:-(not found, skipping)}"
+echo "  Safe plugin rm: $SAFE_RM_PATH"
 
 # Create a temporary sudoers file
 TEMP_SUDOERS="/tmp/ledmatrix_web_sudoers_$$"
 
-cat > "$TEMP_SUDOERS" << EOF
+{
-# LED Matrix Web Interface passwordless sudo configuration
+    echo "# LED Matrix Web Interface passwordless sudo configuration"
-# This allows the web interface user to run specific commands without a password
+    echo "# This allows the web interface user to run specific commands without a password"
+    echo ""
+    echo "# Allow $WEB_USER to run specific commands without a password for the LED Matrix web interface"
 
-# Allow $WEB_USER to run specific commands without a password for the LED Matrix web interface
+    # Optional: reboot/poweroff (non-critical — skip if not found)
-$WEB_USER ALL=(ALL) NOPASSWD: $REBOOT_PATH
+    if [ -n "$REBOOT_PATH" ]; then
-$WEB_USER ALL=(ALL) NOPASSWD: $POWEROFF_PATH
+        echo "$WEB_USER ALL=(ALL) NOPASSWD: $REBOOT_PATH"
-$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH start ledmatrix.service
+    fi
-$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH stop ledmatrix.service
+    if [ -n "$POWEROFF_PATH" ]; then
-$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH restart ledmatrix.service
+        echo "$WEB_USER ALL=(ALL) NOPASSWD: $POWEROFF_PATH"
-$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH enable ledmatrix.service
+    fi
-$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH disable ledmatrix.service
+
-$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH status ledmatrix.service
+    # Required: systemctl
-$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH is-active ledmatrix
+    echo "$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH start ledmatrix.service"
-$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH is-active ledmatrix.service
+    echo "$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH stop ledmatrix.service"
-$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH start ledmatrix-web
+    echo "$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH restart ledmatrix.service"
-$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH stop ledmatrix-web
+    echo "$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH enable ledmatrix.service"
-$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH restart ledmatrix-web
+    echo "$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH disable ledmatrix.service"
-$WEB_USER ALL=(ALL) NOPASSWD: $JOURNALCTL_PATH -u ledmatrix.service *
+    echo "$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH status ledmatrix.service"
-$WEB_USER ALL=(ALL) NOPASSWD: $JOURNALCTL_PATH -u ledmatrix *
+    echo "$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH is-active ledmatrix"
-$WEB_USER ALL=(ALL) NOPASSWD: $JOURNALCTL_PATH -t ledmatrix *
+    echo "$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH is-active ledmatrix.service"
-$WEB_USER ALL=(ALL) NOPASSWD: $PYTHON_PATH $PROJECT_DIR/display_controller.py
+    echo "$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH start ledmatrix-web"
-$WEB_USER ALL=(ALL) NOPASSWD: $BASH_PATH $PROJECT_DIR/start_display.sh
+    echo "$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH stop ledmatrix-web"
-$WEB_USER ALL=(ALL) NOPASSWD: $BASH_PATH $PROJECT_DIR/stop_display.sh
+    echo "$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH restart ledmatrix-web"
-EOF
+
+    # Optional: journalctl (non-critical — skip if not found)
+    if [ -n "$JOURNALCTL_PATH" ]; then
+        echo "$WEB_USER ALL=(ALL) NOPASSWD: $JOURNALCTL_PATH -u ledmatrix.service *"
+        echo "$WEB_USER ALL=(ALL) NOPASSWD: $JOURNALCTL_PATH -u ledmatrix *"
+        echo "$WEB_USER ALL=(ALL) NOPASSWD: $JOURNALCTL_PATH -t ledmatrix *"
+    fi
+
+    # Required: python3, bash
+    echo "$WEB_USER ALL=(ALL) NOPASSWD: $PYTHON_PATH $PROJECT_DIR/display_controller.py"
+    echo "$WEB_USER ALL=(ALL) NOPASSWD: $BASH_PATH $PROJECT_DIR/start_display.sh"
+    echo "$WEB_USER ALL=(ALL) NOPASSWD: $BASH_PATH $PROJECT_DIR/stop_display.sh"
+    echo ""
+    echo "# Allow web user to remove plugin directories via vetted helper script"
+    echo "# The helper validates that the target path resolves inside plugin-repos/ or plugins/"
+    echo "$WEB_USER ALL=(ALL) NOPASSWD: $BASH_PATH $SAFE_RM_PATH *"
+} > "$TEMP_SUDOERS"
 
 echo ""
 echo "Generated sudoers configuration:"
@@ -81,6 +125,7 @@ echo "- View system logs via journalctl"
 echo "- Run display_controller.py directly"
 echo "- Execute start_display.sh and stop_display.sh"
 echo "- Reboot and shutdown the system"
+echo "- Remove plugin directories (for update/uninstall when root-owned files block deletion)"
 echo ""
 
 # Ask for confirmation
@@ -94,6 +139,15 @@ fi
 
 # Apply the configuration using visudo
 echo "Applying sudoers configuration..."
+# Harden the helper script: root-owned, not writable by web user
+echo "Hardening safe_plugin_rm.sh ownership..."
+if ! sudo chown root:root "$SAFE_RM_PATH"; then
+    echo "Warning: Could not set ownership on $SAFE_RM_PATH"
+fi
+if ! sudo chmod 755 "$SAFE_RM_PATH"; then
+    echo "Warning: Could not set permissions on $SAFE_RM_PATH"
+fi
+
 if sudo cp "$TEMP_SUDOERS" /etc/sudoers.d/ledmatrix_web; then
     echo "Configuration applied successfully!"
     echo ""

src/common/permission_utils.py

@@ -8,6 +8,8 @@ files that need to be accessible by both root service and web user.
 
 import os
 import logging
+import shutil as _shutil
+import subprocess
 from pathlib import Path
 from typing import Optional
 
@@ -192,9 +194,96 @@ def get_plugin_dir_mode() -> int:
 def get_cache_dir_mode() -> int:
     """
     Return permission mode for cache directories.
-    
+
     Returns:
         Permission mode: 0o2775 (rwxrwxr-x + sticky bit) for group-writable cache directories
     """
     return 0o2775  # rwxrwsr-x (setgid + group writable)
 
+
+def sudo_remove_directory(path: Path, allowed_bases: Optional[list] = None) -> bool:
+    """
+    Remove a directory using sudo as a last resort.
+
+    Used when normal removal fails due to root-owned files (e.g., __pycache__
+    directories created by the root ledmatrix service). Delegates to the
+    safe_plugin_rm.sh helper which validates the path is inside allowed
+    plugin directories.
+
+    Before invoking sudo, this function also validates that the resolved
+    path is a descendant of at least one allowed base directory.
+
+    Args:
+        path: Directory path to remove
+        allowed_bases: List of allowed parent directories. If None, defaults
+            to plugin-repos/ and plugins/ under the project root.
+
+    Returns:
+        True if removal succeeded, False otherwise
+    """
+    # Determine project root (permission_utils.py is at src/common/)
+    project_root = Path(__file__).resolve().parent.parent.parent
+
+    if allowed_bases is None:
+        allowed_bases = [
+            project_root / "plugin-repos",
+            project_root / "plugins",
+        ]
+
+    # Resolve the target path to prevent symlink/traversal tricks
+    try:
+        resolved = path.resolve()
+    except (OSError, ValueError) as e:
+        logger.error(f"Cannot resolve path {path}: {e}")
+        return False
+
+    # Validate the resolved path is a strict child of an allowed base
+    is_allowed = False
+    for base in allowed_bases:
+        try:
+            base_resolved = base.resolve()
+            if resolved != base_resolved and resolved.is_relative_to(base_resolved):
+                is_allowed = True
+                break
+        except (OSError, ValueError):
+            continue
+
+    if not is_allowed:
+        logger.error(
+            f"sudo_remove_directory DENIED: {resolved} is not inside "
+            f"allowed bases {[str(b) for b in allowed_bases]}"
+        )
+        return False
+
+    # Use the safe_plugin_rm.sh helper which does its own validation
+    helper_script = project_root / "scripts" / "fix_perms" / "safe_plugin_rm.sh"
+    if not helper_script.exists():
+        logger.error(f"Safe removal helper not found: {helper_script}")
+        return False
+
+    bash_path = _shutil.which('bash') or '/bin/bash'
+
+    try:
+        result = subprocess.run(
+            ['sudo', '-n', bash_path, str(helper_script), str(resolved)],
+            capture_output=True,
+            text=True,
+            timeout=30
+        )
+        if result.returncode == 0 and not resolved.exists():
+            logger.info(f"Successfully removed {path} via sudo helper")
+            return True
+        else:
+            stderr = result.stderr.strip()
+            logger.error(f"sudo helper failed for {path}: {stderr}")
+            return False
+    except subprocess.TimeoutExpired:
+        logger.error(f"sudo helper timed out for {path}")
+        return False
+    except FileNotFoundError:
+        logger.error("sudo command not found on system")
+        return False
+    except Exception as e:
+        logger.error(f"Unexpected error during sudo helper for {path}: {e}")
+        return False
+

src/plugin_system/store_manager.py

@@ -7,6 +7,7 @@ from both the official registry and custom GitHub repositories.
 
 import os
 import json
+import stat
 import subprocess
 import shutil
 import zipfile
@@ -18,6 +19,8 @@ from pathlib import Path
 from typing import List, Dict, Optional, Any
 import logging
 
+from src.common.permission_utils import sudo_remove_directory
+
 try:
     import jsonschema
     from jsonschema import Draft7Validator, ValidationError
@@ -814,7 +817,7 @@ class PluginStoreManager:
             manifest_path = plugin_path / "manifest.json"
             if not manifest_path.exists():
                 self.logger.error(f"No manifest.json found in plugin: {plugin_id}")
-                shutil.rmtree(plugin_path, ignore_errors=True)
+                self._safe_remove_directory(plugin_path)
                 return False
 
             try:
@@ -825,7 +828,7 @@ class PluginStoreManager:
                 manifest_plugin_id = manifest.get('id')
                 if not manifest_plugin_id:
                     self.logger.error(f"Plugin manifest missing 'id' field")
-                    shutil.rmtree(plugin_path, ignore_errors=True)
+                    self._safe_remove_directory(plugin_path)
                     return False
                 
                 # If manifest ID doesn't match directory name, rename directory to match manifest
@@ -837,7 +840,9 @@ class PluginStoreManager:
                     correct_path = self.plugins_dir / manifest_plugin_id
                     if correct_path.exists():
                         self.logger.warning(f"Target directory {manifest_plugin_id} already exists, removing it")
-                        shutil.rmtree(correct_path)
+                        if not self._safe_remove_directory(correct_path):
+                            self.logger.error(f"Failed to remove existing directory {correct_path}, cannot rename plugin")
+                            return False
                     shutil.move(str(plugin_path), str(correct_path))
                     plugin_path = correct_path
                     manifest_path = plugin_path / "manifest.json"
@@ -865,7 +870,7 @@ class PluginStoreManager:
 
                 if missing:
                     self.logger.error(f"Plugin manifest missing required fields for {plugin_id}: {', '.join(missing)}")
-                    shutil.rmtree(plugin_path, ignore_errors=True)
+                    self._safe_remove_directory(plugin_path)
                     return False
 
                 if 'entry_point' not in manifest:
@@ -879,7 +884,7 @@ class PluginStoreManager:
 
             except Exception as manifest_error:
                 self.logger.error(f"Failed to read/validate manifest for {plugin_id}: {manifest_error}")
-                shutil.rmtree(plugin_path, ignore_errors=True)
+                self._safe_remove_directory(plugin_path)
                 return False
 
             if not self._install_dependencies(plugin_path):
@@ -892,7 +897,7 @@ class PluginStoreManager:
         except Exception as e:
             self.logger.error(f"Error installing plugin {plugin_id}: {e}", exc_info=True)
             if plugin_path.exists():
-                shutil.rmtree(plugin_path, ignore_errors=True)
+                self._safe_remove_directory(plugin_path)
             return False
     
     def install_from_url(self, repo_url: str, plugin_id: str = None, plugin_path: str = None, branch: Optional[str] = None) -> Dict[str, Any]:
@@ -1053,8 +1058,8 @@ class PluginStoreManager:
         finally:
             # Cleanup temp directory if it still exists
             if temp_dir and temp_dir.exists():
-                shutil.rmtree(temp_dir)
+                shutil.rmtree(temp_dir, ignore_errors=True)
-    
+
     def _detect_class_name(self, manager_file: Path) -> Optional[str]:
         """
         Attempt to auto-detect the plugin class name from the manager file.
@@ -1110,7 +1115,7 @@ class PluginStoreManager:
                 last_error = e
                 self.logger.debug(f"Git clone failed for branch {try_branch}: {e}")
                 if target_path.exists():
-                    shutil.rmtree(target_path)
+                    self._safe_remove_directory(target_path)
 
         # Try default branch (Git's configured default) as last resort
         try:
@@ -1127,7 +1132,7 @@ class PluginStoreManager:
         except (subprocess.CalledProcessError, subprocess.TimeoutExpired, FileNotFoundError) as e:
             last_error = e
             if target_path.exists():
-                shutil.rmtree(target_path)
+                self._safe_remove_directory(target_path)
 
         self.logger.error(f"Git clone failed for all attempted branches: {last_error}")
         return None
@@ -1157,7 +1162,7 @@ class PluginStoreManager:
             self.logger.info(f"API-based install failed for {plugin_subpath}, falling back to ZIP download")
             # Ensure no partial files remain before ZIP fallback
             if target_path.exists():
-                shutil.rmtree(target_path, ignore_errors=True)
+                self._safe_remove_directory(target_path)
 
         # Fallback: download full ZIP and extract subdirectory
         return self._install_from_monorepo_zip(download_url, plugin_subpath, target_path)
@@ -1277,7 +1282,7 @@ class PluginStoreManager:
                         f"Path traversal detected: {entry['path']!r} resolves outside target directory"
                     )
                     if target_path.exists():
-                        shutil.rmtree(target_path, ignore_errors=True)
+                        self._safe_remove_directory(target_path)
                     return False
 
                 # Create parent directories
@@ -1290,7 +1295,7 @@ class PluginStoreManager:
                     self.logger.error(f"Failed to download {entry['path']}: HTTP {file_response.status_code}")
                     # Clean up partial download
                     if target_path.exists():
-                        shutil.rmtree(target_path, ignore_errors=True)
+                        self._safe_remove_directory(target_path)
                     return False
 
                 dest_file.write_bytes(file_response.content)
@@ -1302,7 +1307,7 @@ class PluginStoreManager:
             self.logger.debug(f"API-based monorepo install failed: {e}")
             # Clean up partial download
             if target_path.exists():
-                shutil.rmtree(target_path, ignore_errors=True)
+                self._safe_remove_directory(target_path)
             return False
 
     def _install_from_monorepo_zip(self, download_url: str, plugin_subpath: str, target_path: Path) -> bool:
@@ -1363,7 +1368,9 @@ class PluginStoreManager:
                 ensure_directory_permissions(target_path.parent, get_plugin_dir_mode())
                 # Ensure target doesn't exist to prevent shutil.move nesting
                 if target_path.exists():
-                    shutil.rmtree(target_path, ignore_errors=True)
+                    if not self._safe_remove_directory(target_path):
+                        self.logger.error(f"Cannot remove existing target {target_path} for monorepo install")
+                        return False
                 shutil.move(str(source_plugin_dir), str(target_path))
 
             return True
@@ -1577,70 +1584,58 @@ class PluginStoreManager:
     
     def _safe_remove_directory(self, path: Path) -> bool:
         """
-        Safely remove a directory, handling permission errors for __pycache__ directories.
+        Safely remove a directory, handling permission errors for root-owned files.
-        
+
-        This function attempts to remove a directory and handles permission errors
+        Attempts removal in three stages:
-        gracefully, especially for __pycache__ directories that may have been created
+        1. Normal shutil.rmtree()
-        by Python with different permissions.
+        2. Fix permissions via os.chmod() then retry (works for same-owner files)
-        
+        3. Use sudo rm -rf as last resort (works for root-owned __pycache__, etc.)
+
         Args:
             path: Path to directory to remove
-            
+
         Returns:
             True if directory was removed successfully, False otherwise
         """
         if not path.exists():
             return True  # Already removed
-        
+
+        # Stage 1: Try normal removal
         try:
-            # First, try normal removal
             shutil.rmtree(path)
             return True
-        except PermissionError as e:
+        except OSError:
-            # Handle permission errors, especially for __pycache__ directories
+            self.logger.warning(f"Permission error removing {path}, attempting chmod fix...")
-            self.logger.warning(f"Permission error removing {path}: {e}. Attempting to fix permissions...")
+
-            
+        # Stage 2: Try chmod + retry (works when we own the files)
-            try:
+        try:
-                # Try to fix permissions on __pycache__ directories recursively
+            for root, _dirs, files in os.walk(path):
-                import stat
+                root_path = Path(root)
-                for root, dirs, files in os.walk(path):
+                try:
-                    root_path = Path(root)
+                    os.chmod(root_path, stat.S_IRWXU)
+                except (OSError, PermissionError):
+                    pass
+                for file in files:
                     try:
-                        # Make directory writable
+                        os.chmod(root_path / file, stat.S_IRWXU)
-                        os.chmod(root_path, stat.S_IRWXU | stat.S_IRWXG | stat.S_IRWXO)
                     except (OSError, PermissionError):
                         pass
-                    
+            shutil.rmtree(path)
-                    # Fix file permissions
+            self.logger.info(f"Removed {path} after fixing permissions")
-                    for file in files:
+            return True
-                        file_path = root_path / file
+        except (PermissionError, OSError):
-                        try:
+            self.logger.warning(f"chmod fix failed for {path}, attempting sudo removal...")
-                            os.chmod(file_path, stat.S_IRWXU | stat.S_IRWXG | stat.S_IRWXO)
+
-                        except (OSError, PermissionError):
+        # Stage 3: Use sudo rm -rf (for root-owned __pycache__, data/.cache, etc.)
-                            pass
+        if sudo_remove_directory(path):
-                
+            return True
-                # Try removal again after fixing permissions
+
-                shutil.rmtree(path)
+        # Final check — maybe partial removal got everything
-                self.logger.info(f"Successfully removed {path} after fixing permissions")
+        if not path.exists():
-                return True
+            return True
-            except Exception as e2:
+
-                self.logger.error(f"Failed to remove {path} even after fixing permissions: {e2}")
+        self.logger.error(f"All removal strategies failed for {path}")
-                # Last resort: try with ignore_errors
+        return False
-                try:
-                    shutil.rmtree(path, ignore_errors=True)
-                    # Check if it actually got removed
-                    if not path.exists():
-                        self.logger.warning(f"Removed {path} with ignore_errors=True (some files may remain)")
-                        return True
-                    else:
-                        self.logger.error(f"Could not remove {path} even with ignore_errors")
-                        return False
-                except Exception as e3:
-                    self.logger.error(f"Final removal attempt failed for {path}: {e3}")
-                    return False
-        except Exception as e:
-            self.logger.error(f"Unexpected error removing {path}: {e}")
-            return False
     
     def _find_plugin_path(self, plugin_id: str) -> Optional[Path]:
         """

systemd/ledmatrix.service

@@ -6,6 +6,7 @@ After=network.target
 Type=simple
 User=root
 WorkingDirectory=__PROJECT_ROOT_DIR__
+Environment=PYTHONDONTWRITEBYTECODE=1
 ExecStart=/usr/bin/python3 __PROJECT_ROOT_DIR__/run.py
 Restart=on-failure
 RestartSec=10