fix(plugins): prevent root-owned files from blocking plugin updates (#242)

* fix(web): unify operation history tracking for monorepo plugin operations The operation history UI was reading from the wrong data source (operation_queue instead of operation_history), install/update records lacked version details, toggle operations used a type name that didn't match UI filters, and the Clear History button was non-functional. - Switch GET /plugins/operation/history to read from OperationHistory audit log with return type hint and targeted exception handling - Add DELETE /plugins/operation/history endpoint; wire up Clear button - Add _get_plugin_version helper with specific exception handling (FileNotFoundError, PermissionError, json.JSONDecodeError) and structured logging with plugin_id/path context - Record plugin version, branch, and commit details on install/update - Record install failures in the direct (non-queue) code path - Replace "toggle" operation type with "enable"/"disable" - Add normalizeStatus() in JS to map completed→success, error→failed so status filter works regardless of server-side convention - Truncate commit SHAs to 7 chars in details display - Fix HTML filter options, operation type colors, duplicate JS init Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(plugins): prevent root-owned files from blocking plugin updates The root ledmatrix service creates __pycache__ and data cache files owned by root inside plugin directories. The web service (non-root) cannot delete these when updating or uninstalling plugins, causing operations to fail with "Permission denied". Defense in depth with three layers: - Prevent: PYTHONDONTWRITEBYTECODE=1 in systemd service + run.py - Fallback: sudoers rules for rm on plugin directories - Code: _safe_remove_directory() now uses sudo as last resort, and all bare shutil.rmtree() calls routed through it Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(security): harden sudo removal with path-validated helper script Address code review findings: - Replace raw rm/find sudoers wildcards with a vetted helper script (safe_plugin_rm.sh) that resolves symlinks and validates the target is a strict child of plugin-repos/ or plugins/ before deletion - Add allow-list validation in sudo_remove_directory() that checks resolved paths against allowed bases before invoking sudo - Check _safe_remove_directory() return value before shutil.move() in the manifest ID rename path - Move stat import to module level in store_manager.py - Use stat.S_IRWXU instead of 0o777 in chmod fallback stage - Add ignore_errors=True to temp dir cleanup in finally block - Use command -v instead of which in configure_web_sudo.sh Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(security): address code review round 2 — harden paths and error handling - safe_plugin_rm.sh: use realpath --canonicalize-missing for ALLOWED_BASES so the script doesn't fail under set -e when dirs don't exist yet - safe_plugin_rm.sh: add -- before path in rm -rf to prevent flag injection - permission_utils.py: use shutil.which('bash') instead of hardcoded /bin/bash to match whatever path the sudoers BASH_PATH resolves to - store_manager.py: check _safe_remove_directory() return before shutil.move() in _install_from_monorepo_zip to prevent moving into a non-removed target - store_manager.py: catch OSError instead of PermissionError in Stage 1 removal to handle both EACCES and EPERM error codes - store_manager.py: hoist sudo_remove_directory import to module level - configure_web_sudo.sh: harden safe_plugin_rm.sh to root-owned 755 so the web user cannot modify the vetted helper script Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(security): validate command paths in sudoers config and use resolved paths - configure_web_sudo.sh: validate that required commands (systemctl, bash, python3) resolve to non-empty paths before generating sudoers entries; abort with clear error if any are missing; skip optional commands (reboot, poweroff, journalctl) with a warning instead of emitting malformed NOPASSWD lines; validate helper script exists on disk - permission_utils.py: pass the already-resolved path to the subprocess call and use it for the post-removal exists() check, eliminating a TOCTOU window between Python-side validation and shell-side execution Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Chuck <chuck@example.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-10 13:02:59 +00:00 · 2026-02-12 19:28:05 -05:00
parent 9a72adbde1
commit 158e07c82b
6 changed files with 306 additions and 101 deletions
--- a/run.py
+++ b/run.py
@@ -4,6 +4,11 @@ import sys
 import os
 import argparse

+# Prevent Python from creating __pycache__ directories in plugin dirs.
+# The root service loads plugins via importlib, and root-owned __pycache__
+# files block the web service (non-root) from updating/uninstalling plugins.
+sys.dont_write_bytecode = True
+
 # Add project directory to Python path (needed before importing src modules)
 project_dir = os.path.dirname(os.path.abspath(__file__))
 if project_dir not in sys.path:
--- a/scripts/fix_perms/safe_plugin_rm.sh
+++ b/scripts/fix_perms/safe_plugin_rm.sh
@@ -0,0 +1,61 @@
+#!/bin/bash
+# safe_plugin_rm.sh — Safely remove a plugin directory after validating
+# that the resolved path is inside an allowed base directory.
+#
+# This script is intended to be called via sudo from the web interface.
+# It prevents path traversal attacks by resolving symlinks and verifying
+# the target is a child of plugin-repos/ or plugins/.
+#
+# Usage: safe_plugin_rm.sh <target_path>
+
+set -euo pipefail
+
+if [ $# -ne 1 ]; then
+    echo "Usage: $0 <target_path>" >&2
+    exit 1
+fi
+
+TARGET="$1"
+
+# Determine the project root (parent of scripts/fix_perms/)
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
+
+# Allowed base directories (resolved, no trailing slash)
+# Use --canonicalize-missing so this works even if the dirs don't exist yet
+ALLOWED_BASES=(
+    "$(realpath --canonicalize-missing "$PROJECT_ROOT/plugin-repos")"
+    "$(realpath --canonicalize-missing "$PROJECT_ROOT/plugins")"
+)
+
+# Resolve the target path (follow symlinks)
+# Use realpath --canonicalize-missing so it works even if the path
+# doesn't fully exist (e.g., partially deleted directory)
+RESOLVED_TARGET="$(realpath --canonicalize-missing "$TARGET")"
+
+# Validate: resolved target must be a strict child of an allowed base
+# (must not BE the base itself — only children are allowed)
+ALLOWED=false
+for BASE in "${ALLOWED_BASES[@]}"; do
+    if [[ "$RESOLVED_TARGET" == "$BASE/"* ]]; then
+        ALLOWED=true
+        break
+    fi
+done
+
+if [ "$ALLOWED" = false ]; then
+    echo "DENIED: $RESOLVED_TARGET is not inside an allowed plugin directory" >&2
+    echo "Allowed bases: ${ALLOWED_BASES[*]}" >&2
+    exit 2
+fi
+
+# Safety check: refuse to delete the base directories themselves
+for BASE in "${ALLOWED_BASES[@]}"; do
+    if [ "$RESOLVED_TARGET" = "$BASE" ]; then
+        echo "DENIED: cannot remove plugin base directory itself: $BASE" >&2
+        exit 2
+    fi
+done
+
+# All checks passed — remove the target
+rm -rf -- "$RESOLVED_TARGET"
--- a/scripts/install/configure_web_sudo.sh
+++ b/scripts/install/configure_web_sudo.sh
@@ -10,9 +10,11 @@ echo "Configuring passwordless sudo access for LED Matrix Web Interface..."
 # Get the current user (should be the user running the web interface)
 WEB_USER=$(whoami)
 PROJECT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(cd "$PROJECT_DIR/../.." && pwd)"

 echo "Detected web interface user: $WEB_USER"
 echo "Project directory: $PROJECT_DIR"
+echo "Project root: $PROJECT_ROOT"

 # Check if running as root
 if [ "$EUID" -eq 0 ]; then
@@ -21,50 +23,92 @@ if [ "$EUID" -eq 0 ]; then
    exit 1
 fi

-# Get the full paths to commands
-PYTHON_PATH=$(which python3)
-SYSTEMCTL_PATH=$(which systemctl)
-REBOOT_PATH=$(which reboot)
-POWEROFF_PATH=$(which poweroff)
-BASH_PATH=$(which bash)
-JOURNALCTL_PATH=$(which journalctl)
+# Get the full paths to commands and validate each one
+MISSING_CMDS=()
+
+PYTHON_PATH=$(command -v python3)   || true
+SYSTEMCTL_PATH=$(command -v systemctl) || true
+REBOOT_PATH=$(command -v reboot)    || true
+POWEROFF_PATH=$(command -v poweroff)  || true
+BASH_PATH=$(command -v bash)        || true
+JOURNALCTL_PATH=$(command -v journalctl) || true
+SAFE_RM_PATH="$PROJECT_ROOT/scripts/fix_perms/safe_plugin_rm.sh"
+
+# Validate required commands (systemctl, bash, python3 are essential)
+for CMD_NAME in SYSTEMCTL_PATH BASH_PATH PYTHON_PATH; do
+    CMD_VAL="${!CMD_NAME}"
+    if [ -z "$CMD_VAL" ]; then
+        MISSING_CMDS+=("$CMD_NAME")
+    fi
+done
+
+if [ ${#MISSING_CMDS[@]} -gt 0 ]; then
+    echo "Error: Required commands not found: ${MISSING_CMDS[*]}" >&2
+    echo "Cannot generate valid sudoers configuration without these." >&2
+    exit 1
+fi
+
+# Validate helper script exists
+if [ ! -f "$SAFE_RM_PATH" ]; then
+    echo "Error: Safe plugin removal helper not found: $SAFE_RM_PATH" >&2
+    exit 1
+fi

 echo "Command paths:"
 echo "  Python: $PYTHON_PATH"
 echo "  Systemctl: $SYSTEMCTL_PATH"
-echo "  Reboot: $REBOOT_PATH"
-echo "  Poweroff: $POWEROFF_PATH"
+echo "  Reboot: ${REBOOT_PATH:-(not found, skipping)}"
+echo "  Poweroff: ${POWEROFF_PATH:-(not found, skipping)}"
 echo "  Bash: $BASH_PATH"
-echo "  Journalctl: $JOURNALCTL_PATH"
+echo "  Journalctl: ${JOURNALCTL_PATH:-(not found, skipping)}"
+echo "  Safe plugin rm: $SAFE_RM_PATH"

 # Create a temporary sudoers file
 TEMP_SUDOERS="/tmp/ledmatrix_web_sudoers_$$"

-cat > "$TEMP_SUDOERS" << EOF
-# LED Matrix Web Interface passwordless sudo configuration
-# This allows the web interface user to run specific commands without a password
+{
+    echo "# LED Matrix Web Interface passwordless sudo configuration"
+    echo "# This allows the web interface user to run specific commands without a password"
+    echo ""
+    echo "# Allow $WEB_USER to run specific commands without a password for the LED Matrix web interface"

-# Allow $WEB_USER to run specific commands without a password for the LED Matrix web interface
-$WEB_USER ALL=(ALL) NOPASSWD: $REBOOT_PATH
-$WEB_USER ALL=(ALL) NOPASSWD: $POWEROFF_PATH
-$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH start ledmatrix.service
-$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH stop ledmatrix.service
-$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH restart ledmatrix.service
-$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH enable ledmatrix.service
-$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH disable ledmatrix.service
-$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH status ledmatrix.service
-$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH is-active ledmatrix
-$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH is-active ledmatrix.service
-$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH start ledmatrix-web
-$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH stop ledmatrix-web
-$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH restart ledmatrix-web
-$WEB_USER ALL=(ALL) NOPASSWD: $JOURNALCTL_PATH -u ledmatrix.service *
-$WEB_USER ALL=(ALL) NOPASSWD: $JOURNALCTL_PATH -u ledmatrix *
-$WEB_USER ALL=(ALL) NOPASSWD: $JOURNALCTL_PATH -t ledmatrix *
-$WEB_USER ALL=(ALL) NOPASSWD: $PYTHON_PATH $PROJECT_DIR/display_controller.py
-$WEB_USER ALL=(ALL) NOPASSWD: $BASH_PATH $PROJECT_DIR/start_display.sh
-$WEB_USER ALL=(ALL) NOPASSWD: $BASH_PATH $PROJECT_DIR/stop_display.sh
-EOF
+    # Optional: reboot/poweroff (non-critical — skip if not found)
+    if [ -n "$REBOOT_PATH" ]; then
+        echo "$WEB_USER ALL=(ALL) NOPASSWD: $REBOOT_PATH"
+    fi
+    if [ -n "$POWEROFF_PATH" ]; then
+        echo "$WEB_USER ALL=(ALL) NOPASSWD: $POWEROFF_PATH"
+    fi
+
+    # Required: systemctl
+    echo "$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH start ledmatrix.service"
+    echo "$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH stop ledmatrix.service"
+    echo "$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH restart ledmatrix.service"
+    echo "$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH enable ledmatrix.service"
+    echo "$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH disable ledmatrix.service"
+    echo "$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH status ledmatrix.service"
+    echo "$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH is-active ledmatrix"
+    echo "$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH is-active ledmatrix.service"
+    echo "$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH start ledmatrix-web"
+    echo "$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH stop ledmatrix-web"
+    echo "$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH restart ledmatrix-web"
+
+    # Optional: journalctl (non-critical — skip if not found)
+    if [ -n "$JOURNALCTL_PATH" ]; then
+        echo "$WEB_USER ALL=(ALL) NOPASSWD: $JOURNALCTL_PATH -u ledmatrix.service *"
+        echo "$WEB_USER ALL=(ALL) NOPASSWD: $JOURNALCTL_PATH -u ledmatrix *"
+        echo "$WEB_USER ALL=(ALL) NOPASSWD: $JOURNALCTL_PATH -t ledmatrix *"
+    fi
+
+    # Required: python3, bash
+    echo "$WEB_USER ALL=(ALL) NOPASSWD: $PYTHON_PATH $PROJECT_DIR/display_controller.py"
+    echo "$WEB_USER ALL=(ALL) NOPASSWD: $BASH_PATH $PROJECT_DIR/start_display.sh"
+    echo "$WEB_USER ALL=(ALL) NOPASSWD: $BASH_PATH $PROJECT_DIR/stop_display.sh"
+    echo ""
+    echo "# Allow web user to remove plugin directories via vetted helper script"
+    echo "# The helper validates that the target path resolves inside plugin-repos/ or plugins/"
+    echo "$WEB_USER ALL=(ALL) NOPASSWD: $BASH_PATH $SAFE_RM_PATH *"
+} > "$TEMP_SUDOERS"

 echo ""
 echo "Generated sudoers configuration:"
@@ -81,6 +125,7 @@ echo "- View system logs via journalctl"
 echo "- Run display_controller.py directly"
 echo "- Execute start_display.sh and stop_display.sh"
 echo "- Reboot and shutdown the system"
+echo "- Remove plugin directories (for update/uninstall when root-owned files block deletion)"
 echo ""

 # Ask for confirmation
@@ -94,6 +139,15 @@ fi

 # Apply the configuration using visudo
 echo "Applying sudoers configuration..."
+# Harden the helper script: root-owned, not writable by web user
+echo "Hardening safe_plugin_rm.sh ownership..."
+if ! sudo chown root:root "$SAFE_RM_PATH"; then
+    echo "Warning: Could not set ownership on $SAFE_RM_PATH"
+fi
+if ! sudo chmod 755 "$SAFE_RM_PATH"; then
+    echo "Warning: Could not set permissions on $SAFE_RM_PATH"
+fi
+
 if sudo cp "$TEMP_SUDOERS" /etc/sudoers.d/ledmatrix_web; then
    echo "Configuration applied successfully!"
    echo ""
--- a/src/common/permission_utils.py
+++ b/src/common/permission_utils.py
@@ -8,6 +8,8 @@ files that need to be accessible by both root service and web user.

 import os
 import logging
+import shutil as _shutil
+import subprocess
 from pathlib import Path
 from typing import Optional

@@ -198,3 +200,90 @@ def get_cache_dir_mode() -> int:
    """
    return 0o2775  # rwxrwsr-x (setgid + group writable)

+
+def sudo_remove_directory(path: Path, allowed_bases: Optional[list] = None) -> bool:
+    """
+    Remove a directory using sudo as a last resort.
+
+    Used when normal removal fails due to root-owned files (e.g., __pycache__
+    directories created by the root ledmatrix service). Delegates to the
+    safe_plugin_rm.sh helper which validates the path is inside allowed
+    plugin directories.
+
+    Before invoking sudo, this function also validates that the resolved
+    path is a descendant of at least one allowed base directory.
+
+    Args:
+        path: Directory path to remove
+        allowed_bases: List of allowed parent directories. If None, defaults
+            to plugin-repos/ and plugins/ under the project root.
+
+    Returns:
+        True if removal succeeded, False otherwise
+    """
+    # Determine project root (permission_utils.py is at src/common/)
+    project_root = Path(__file__).resolve().parent.parent.parent
+
+    if allowed_bases is None:
+        allowed_bases = [
+            project_root / "plugin-repos",
+            project_root / "plugins",
+        ]
+
+    # Resolve the target path to prevent symlink/traversal tricks
+    try:
+        resolved = path.resolve()
+    except (OSError, ValueError) as e:
+        logger.error(f"Cannot resolve path {path}: {e}")
+        return False
+
+    # Validate the resolved path is a strict child of an allowed base
+    is_allowed = False
+    for base in allowed_bases:
+        try:
+            base_resolved = base.resolve()
+            if resolved != base_resolved and resolved.is_relative_to(base_resolved):
+                is_allowed = True
+                break
+        except (OSError, ValueError):
+            continue
+
+    if not is_allowed:
+        logger.error(
+            f"sudo_remove_directory DENIED: {resolved} is not inside "
+            f"allowed bases {[str(b) for b in allowed_bases]}"
+        )
+        return False
+
+    # Use the safe_plugin_rm.sh helper which does its own validation
+    helper_script = project_root / "scripts" / "fix_perms" / "safe_plugin_rm.sh"
+    if not helper_script.exists():
+        logger.error(f"Safe removal helper not found: {helper_script}")
+        return False
+
+    bash_path = _shutil.which('bash') or '/bin/bash'
+
+    try:
+        result = subprocess.run(
+            ['sudo', '-n', bash_path, str(helper_script), str(resolved)],
+            capture_output=True,
+            text=True,
+            timeout=30
+        )
+        if result.returncode == 0 and not resolved.exists():
+            logger.info(f"Successfully removed {path} via sudo helper")
+            return True
+        else:
+            stderr = result.stderr.strip()
+            logger.error(f"sudo helper failed for {path}: {stderr}")
+            return False
+    except subprocess.TimeoutExpired:
+        logger.error(f"sudo helper timed out for {path}")
+        return False
+    except FileNotFoundError:
+        logger.error("sudo command not found on system")
+        return False
+    except Exception as e:
+        logger.error(f"Unexpected error during sudo helper for {path}: {e}")
+        return False
+
--- a/src/plugin_system/store_manager.py
+++ b/src/plugin_system/store_manager.py
@@ -7,6 +7,7 @@ from both the official registry and custom GitHub repositories.

 import os
 import json
+import stat
 import subprocess
 import shutil
 import zipfile
@@ -18,6 +19,8 @@ from pathlib import Path
 from typing import List, Dict, Optional, Any
 import logging

+from src.common.permission_utils import sudo_remove_directory
+
 try:
    import jsonschema
    from jsonschema import Draft7Validator, ValidationError
@@ -814,7 +817,7 @@ class PluginStoreManager:
            manifest_path = plugin_path / "manifest.json"
            if not manifest_path.exists():
                self.logger.error(f"No manifest.json found in plugin: {plugin_id}")
-                shutil.rmtree(plugin_path, ignore_errors=True)
+                self._safe_remove_directory(plugin_path)
                return False

            try:
@@ -825,7 +828,7 @@ class PluginStoreManager:
                manifest_plugin_id = manifest.get('id')
                if not manifest_plugin_id:
                    self.logger.error(f"Plugin manifest missing 'id' field")
-                    shutil.rmtree(plugin_path, ignore_errors=True)
+                    self._safe_remove_directory(plugin_path)
                    return False
                
                # If manifest ID doesn't match directory name, rename directory to match manifest
@@ -837,7 +840,9 @@ class PluginStoreManager:
                    correct_path = self.plugins_dir / manifest_plugin_id
                    if correct_path.exists():
                        self.logger.warning(f"Target directory {manifest_plugin_id} already exists, removing it")
-                        shutil.rmtree(correct_path)
+                        if not self._safe_remove_directory(correct_path):
+                            self.logger.error(f"Failed to remove existing directory {correct_path}, cannot rename plugin")
+                            return False
                    shutil.move(str(plugin_path), str(correct_path))
                    plugin_path = correct_path
                    manifest_path = plugin_path / "manifest.json"
@@ -865,7 +870,7 @@ class PluginStoreManager:

                if missing:
                    self.logger.error(f"Plugin manifest missing required fields for {plugin_id}: {', '.join(missing)}")
-                    shutil.rmtree(plugin_path, ignore_errors=True)
+                    self._safe_remove_directory(plugin_path)
                    return False

                if 'entry_point' not in manifest:
@@ -879,7 +884,7 @@ class PluginStoreManager:

            except Exception as manifest_error:
                self.logger.error(f"Failed to read/validate manifest for {plugin_id}: {manifest_error}")
-                shutil.rmtree(plugin_path, ignore_errors=True)
+                self._safe_remove_directory(plugin_path)
                return False

            if not self._install_dependencies(plugin_path):
@@ -892,7 +897,7 @@ class PluginStoreManager:
        except Exception as e:
            self.logger.error(f"Error installing plugin {plugin_id}: {e}", exc_info=True)
            if plugin_path.exists():
-                shutil.rmtree(plugin_path, ignore_errors=True)
+                self._safe_remove_directory(plugin_path)
            return False
    
    def install_from_url(self, repo_url: str, plugin_id: str = None, plugin_path: str = None, branch: Optional[str] = None) -> Dict[str, Any]:
@@ -1053,7 +1058,7 @@ class PluginStoreManager:
        finally:
            # Cleanup temp directory if it still exists
            if temp_dir and temp_dir.exists():
-                shutil.rmtree(temp_dir)
+                shutil.rmtree(temp_dir, ignore_errors=True)

    def _detect_class_name(self, manager_file: Path) -> Optional[str]:
        """
@@ -1110,7 +1115,7 @@ class PluginStoreManager:
                last_error = e
                self.logger.debug(f"Git clone failed for branch {try_branch}: {e}")
                if target_path.exists():
-                    shutil.rmtree(target_path)
+                    self._safe_remove_directory(target_path)

        # Try default branch (Git's configured default) as last resort
        try:
@@ -1127,7 +1132,7 @@ class PluginStoreManager:
        except (subprocess.CalledProcessError, subprocess.TimeoutExpired, FileNotFoundError) as e:
            last_error = e
            if target_path.exists():
-                shutil.rmtree(target_path)
+                self._safe_remove_directory(target_path)

        self.logger.error(f"Git clone failed for all attempted branches: {last_error}")
        return None
@@ -1157,7 +1162,7 @@ class PluginStoreManager:
            self.logger.info(f"API-based install failed for {plugin_subpath}, falling back to ZIP download")
            # Ensure no partial files remain before ZIP fallback
            if target_path.exists():
-                shutil.rmtree(target_path, ignore_errors=True)
+                self._safe_remove_directory(target_path)

        # Fallback: download full ZIP and extract subdirectory
        return self._install_from_monorepo_zip(download_url, plugin_subpath, target_path)
@@ -1277,7 +1282,7 @@ class PluginStoreManager:
                        f"Path traversal detected: {entry['path']!r} resolves outside target directory"
                    )
                    if target_path.exists():
-                        shutil.rmtree(target_path, ignore_errors=True)
+                        self._safe_remove_directory(target_path)
                    return False

                # Create parent directories
@@ -1290,7 +1295,7 @@ class PluginStoreManager:
                    self.logger.error(f"Failed to download {entry['path']}: HTTP {file_response.status_code}")
                    # Clean up partial download
                    if target_path.exists():
-                        shutil.rmtree(target_path, ignore_errors=True)
+                        self._safe_remove_directory(target_path)
                    return False

                dest_file.write_bytes(file_response.content)
@@ -1302,7 +1307,7 @@ class PluginStoreManager:
            self.logger.debug(f"API-based monorepo install failed: {e}")
            # Clean up partial download
            if target_path.exists():
-                shutil.rmtree(target_path, ignore_errors=True)
+                self._safe_remove_directory(target_path)
            return False

    def _install_from_monorepo_zip(self, download_url: str, plugin_subpath: str, target_path: Path) -> bool:
@@ -1363,7 +1368,9 @@ class PluginStoreManager:
                ensure_directory_permissions(target_path.parent, get_plugin_dir_mode())
                # Ensure target doesn't exist to prevent shutil.move nesting
                if target_path.exists():
-                    shutil.rmtree(target_path, ignore_errors=True)
+                    if not self._safe_remove_directory(target_path):
+                        self.logger.error(f"Cannot remove existing target {target_path} for monorepo install")
+                        return False
                shutil.move(str(source_plugin_dir), str(target_path))

            return True
@@ -1577,11 +1584,12 @@ class PluginStoreManager:
    
    def _safe_remove_directory(self, path: Path) -> bool:
        """
-        Safely remove a directory, handling permission errors for __pycache__ directories.
+        Safely remove a directory, handling permission errors for root-owned files.

-        This function attempts to remove a directory and handles permission errors
-        gracefully, especially for __pycache__ directories that may have been created
-        by Python with different permissions.
+        Attempts removal in three stages:
+        1. Normal shutil.rmtree()
+        2. Fix permissions via os.chmod() then retry (works for same-owner files)
+        3. Use sudo rm -rf as last resort (works for root-owned __pycache__, etc.)

        Args:
            path: Path to directory to remove
@@ -1592,55 +1600,42 @@ class PluginStoreManager:
        if not path.exists():
            return True  # Already removed

+        # Stage 1: Try normal removal
        try:
-            # First, try normal removal
            shutil.rmtree(path)
            return True
-        except PermissionError as e:
-            # Handle permission errors, especially for __pycache__ directories
-            self.logger.warning(f"Permission error removing {path}: {e}. Attempting to fix permissions...")
+        except OSError:
+            self.logger.warning(f"Permission error removing {path}, attempting chmod fix...")

-            try:
-                # Try to fix permissions on __pycache__ directories recursively
-                import stat
-                for root, dirs, files in os.walk(path):
-                    root_path = Path(root)
+        # Stage 2: Try chmod + retry (works when we own the files)
+        try:
+            for root, _dirs, files in os.walk(path):
+                root_path = Path(root)
+                try:
+                    os.chmod(root_path, stat.S_IRWXU)
+                except (OSError, PermissionError):
+                    pass
+                for file in files:
                    try:
-                        # Make directory writable
-                        os.chmod(root_path, stat.S_IRWXU | stat.S_IRWXG | stat.S_IRWXO)
+                        os.chmod(root_path / file, stat.S_IRWXU)
                    except (OSError, PermissionError):
                        pass
+            shutil.rmtree(path)
+            self.logger.info(f"Removed {path} after fixing permissions")
+            return True
+        except (PermissionError, OSError):
+            self.logger.warning(f"chmod fix failed for {path}, attempting sudo removal...")

-                    # Fix file permissions
-                    for file in files:
-                        file_path = root_path / file
-                        try:
-                            os.chmod(file_path, stat.S_IRWXU | stat.S_IRWXG | stat.S_IRWXO)
-                        except (OSError, PermissionError):
-                            pass
+        # Stage 3: Use sudo rm -rf (for root-owned __pycache__, data/.cache, etc.)
+        if sudo_remove_directory(path):
+            return True

-                # Try removal again after fixing permissions
-                shutil.rmtree(path)
-                self.logger.info(f"Successfully removed {path} after fixing permissions")
-                return True
-            except Exception as e2:
-                self.logger.error(f"Failed to remove {path} even after fixing permissions: {e2}")
-                # Last resort: try with ignore_errors
-                try:
-                    shutil.rmtree(path, ignore_errors=True)
-                    # Check if it actually got removed
-                    if not path.exists():
-                        self.logger.warning(f"Removed {path} with ignore_errors=True (some files may remain)")
-                        return True
-                    else:
-                        self.logger.error(f"Could not remove {path} even with ignore_errors")
-                        return False
-                except Exception as e3:
-                    self.logger.error(f"Final removal attempt failed for {path}: {e3}")
-                    return False
-        except Exception as e:
-            self.logger.error(f"Unexpected error removing {path}: {e}")
-            return False
+        # Final check — maybe partial removal got everything
+        if not path.exists():
+            return True
+
+        self.logger.error(f"All removal strategies failed for {path}")
+        return False
    
    def _find_plugin_path(self, plugin_id: str) -> Optional[Path]:
        """
--- a/systemd/ledmatrix.service
+++ b/systemd/ledmatrix.service
@@ -6,6 +6,7 @@ After=network.target
 Type=simple
 User=root
 WorkingDirectory=__PROJECT_ROOT_DIR__
+Environment=PYTHONDONTWRITEBYTECODE=1
 ExecStart=/usr/bin/python3 __PROJECT_ROOT_DIR__/run.py
 Restart=on-failure
 RestartSec=10