From c3465c35f5204d62ab92b854cf1ba99eab88b7d2 Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Thu, 17 Sep 2026 08:58:03 -0400 Subject: [PATCH] docs(deps): describe how plugin dependencies are really installed The guides said the web service runs as root, that installs pick --user from os.geteuid(), and quoted a warning and a PluginManager._install_plugin_dependencies() method that don't exist. The web unit runs as the installing user; store installs go through install_requirements_file() and sudo safe_pip_install.sh (root), with a user-level fallback that says so, and load-time installs run in the display service's own (root) interpreter. Manual paths now use the configured plugins directory (plugin-repos/ by default) instead of plugins/, which store installs no longer use, and install_plugin_dependencies.sh is described as scanning that directory. Co-Authored-By: Claude Opus 5 --- docs/PLUGIN_DEPENDENCY_GUIDE.md | 294 ++++++++-------------- docs/PLUGIN_DEPENDENCY_TROUBLESHOOTING.md | 159 ++++++------ 2 files changed, 187 insertions(+), 266 deletions(-) diff --git a/docs/PLUGIN_DEPENDENCY_GUIDE.md b/docs/PLUGIN_DEPENDENCY_GUIDE.md index 1388a6fc..5396b0c6 100644 --- a/docs/PLUGIN_DEPENDENCY_GUIDE.md +++ b/docs/PLUGIN_DEPENDENCY_GUIDE.md @@ -2,234 +2,160 @@ ## Overview -The LEDMatrix system has smart dependency installation that adapts based on who is running it. This guide explains how it works and potential pitfalls. +A plugin lists its Python packages in its `requirements.txt`. LEDMatrix +installs them for you when a plugin is installed, updated or loaded. This +guide explains where they end up and what to do when a plugin can't import a +package. -## How It Works +The rule to remember: **packages must be importable by `ledmatrix.service`, +which runs as root.** Anything installed only into another user's +`~/.local/` is invisible to it. -### Execution Context Detection +## Who Runs What -The plugin manager checks if it's running as root: -```python -running_as_root = os.geteuid() == 0 +| Service | Runs as | Set by | +|---------|---------|--------| +| `ledmatrix.service` (display) | `root` | `systemd/ledmatrix.service` | +| `ledmatrix-web.service` (web UI) | the user who ran the installer (e.g. `ledpi`) | `User=__USER__` in `systemd/ledmatrix-web.service`, filled in by `scripts/install/install_service.sh` | + +## How Dependencies Get Installed + +### 1. Installing or updating a plugin from the web UI + +The web interface is not root, so it installs through a narrow sudo helper: + +1. `PluginStoreManager._install_dependencies()` + (`src/plugin_system/store_manager.py`) calls + `install_requirements_file()` (`src/common/permission_utils.py`). +2. That runs `sudo -n bash scripts/fix_perms/safe_pip_install.sh /requirements.txt`. + The helper checks the path is the project's own `requirements.txt` or a + `requirements.txt` under `plugin-repos/` or `plugins/`, then runs + `python3 -m pip install --break-system-packages --ignore-installed -r ...` + **as root**, so the display service can import the packages. +3. The sudoers rule that allows this is written by the installer + (`first_time_install.sh`) or by `scripts/install/configure_web_sudo.sh`. + +If sudo refuses (the rule isn't installed), `install_requirements_file()` +falls back to installing with the web process's own interpreter, as the web +user, and prefixes the pip output with a note like: + +``` +[Root install unavailable (...); installed for the current process's user only. +Packages may not be visible to ledmatrix.service if it runs as a different +user — run scripts/install/configure_web_sudo.sh to fix this.] ``` -Based on this, it chooses the appropriate installation method: +Fix it by running `./scripts/install/configure_web_sudo.sh` as the web +user (not with `sudo`; it asks for your password itself), then +reinstall the plugin (or use the manual install below). -| Running As | Installation Method | Location | Accessible To | -|------------|-------------------|----------|---------------| -| **root** (systemd service) | System-wide (`--break-system-packages`) | `/usr/local/lib/python3.X/dist-packages/` | All users | -| **ledpi** or other user | User-specific (`--user`) | `~/.local/lib/python3.X/site-packages/` | Only that user | +The **Reinstall Plugin Deps** button on the web UI's Tools tab goes +through the same helper for every installed plugin. + +### 2. Loading a plugin + +When a plugin loads, `PluginLoader.install_dependencies()` +(`src/plugin_system/plugin_loader.py`) checks its `requirements.txt`. If the +requirements are already satisfied it does nothing; otherwise it runs +`python3 -m pip install --break-system-packages -r requirements.txt` with the +interpreter of the process doing the loading (retrying with +`--ignore-installed` when a system package without a pip RECORD file is in +the way). + +In `ledmatrix.service` that process is root, so restarting the display +service installs anything missing system-wide: + +```bash +sudo systemctl restart ledmatrix +``` + +If you run `python3 run.py` by hand as a normal user instead, pip cannot +write to the system site-packages and installs into your `~/.local/`. That +works for your manual run but not for the service. ## Common Scenarios -### ✅ Scenario 1: Normal Production Use (Recommended) +### Installing plugins from the web UI (recommended) -**What:** Services running via systemd +Use the **Plugin Manager** tab. Dependencies are installed as root through +the sudo helper and the display service can use them. + +### Running the display manually for debugging ```bash -sudo systemctl start ledmatrix -sudo systemctl start ledmatrix-web +cd ~/LEDMatrix +sudo python3 run.py # same user as the service ``` -- **Runs as:** root (configured in .service files) -- **Installs to:** System-wide -- **Result:** ✅ Works perfectly, all dependencies accessible +Running as your own user works for plugins whose packages are already +installed system-wide, but any *missing* package lands in `~/.local/`. -### ✅ Scenario 2: Web Interface Plugin Installation +### A plugin works when run manually but fails in the service -**What:** Installing/enabling plugins via web interface at `http://pi-ip:5000` +Its packages were installed for your user only. Install them as root (see +below) and restart the service. -- **Web service runs as:** root (ledmatrix-web.service) -- **Installs to:** System-wide -- **Result:** ✅ Works perfectly, systemd service can access them +## Manual Installation -### ✅ Scenario 3: Manual Testing as ledpi (Read-only) - -**What:** Running display manually as ledpi to test/debug +### All plugins ```bash -# As ledpi user -cd /home/ledpi/LEDMatrix -python3 run.py -``` - -- **Runs as:** ledpi -- **Can import:** ✅ System-wide packages (installed by root) -- **Result:** ✅ Works! Can use existing plugins with root-installed dependencies - -### ⚠️ Scenario 4: Manual Plugin Installation as ledpi (Problematic) - -**What:** Enabling a NEW plugin and running manually as ledpi - -```bash -# As ledpi user -cd /home/ledpi/LEDMatrix -# Edit config to enable new plugin -nano config/config.json -# Run display - will try to install new plugin dependencies -python3 run.py -``` - -**What Happens:** -1. Plugin manager runs as `ledpi` -2. Installs dependencies with `--user` flag -3. Dependencies go to `~/.local/lib/python3.X/site-packages/` -4. ⚠️ **Warning logged:** "Installing plugin dependencies for current user (not root)" - -**Problem:** -- When systemd service restarts (as root), it **can't see** `~/.local/` packages -- Plugin will fail to load for the systemd service - -**Solution:** -After testing, restart the service to install dependencies system-wide: -```bash +sudo ~/LEDMatrix/scripts/install_plugin_dependencies.sh sudo systemctl restart ledmatrix ``` -## Best Practices +The script installs every `requirements.txt` found in the plugins directory +configured by `plugin_system.plugins_directory` in `config/config.json` +(default `plugin-repos/`). Run it with `sudo` so the packages are installed +system-wide. -### For Production/Normal Use +### One plugin -1. **Always use the web interface** to install/enable plugins -2. **Or restart the systemd service** after config changes: - ```bash - sudo systemctl restart ledmatrix - ``` - -### For Development/Testing - -1. **Read existing plugins:** Safe to run as `ledpi` - can import system packages -2. **Test new plugins:** Use sudo or restart service to install dependencies: - ```bash - # Option 1: Run as root - sudo python3 run.py - - # Option 2: Install deps manually - sudo pip3 install --break-system-packages -r plugins/my-plugin/requirements.txt - python3 run.py - - # Option 3: Let service install them - sudo systemctl restart ledmatrix - ``` - -## Warning Messages - -### If you see this warning: -``` -Installing plugin dependencies for current user (not root). -These will NOT be accessible to the systemd service. -For production use, install plugins via the web interface or restart the ledmatrix service. -``` - -**What it means:** -- You're running as a non-root user -- Dependencies were installed to your user directory only -- The systemd service won't be able to use this plugin - -**What to do:** ```bash -# Restart the service to install dependencies system-wide +cd ~/LEDMatrix/plugin-repos/PLUGIN-NAME # or your configured plugins directory +sudo python3 -m pip install --break-system-packages --no-cache-dir -r requirements.txt sudo systemctl restart ledmatrix ``` +`--no-cache-dir` avoids errors about `/root/.cache/pip` not being writable. + ## Troubleshooting -### Plugin works when I run manually but fails in systemd service - -**Cause:** Dependencies installed to user directory (`~/.local/`) instead of system-wide - -**Fix:** -```bash -# Check where package is installed -pip3 list -v | grep - -# If it shows ~/.local/, reinstall system-wide: -sudo pip3 install --break-system-packages - -# Or just restart the service: -sudo systemctl restart ledmatrix -``` - ### Permission denied when installing dependencies -**If you see errors like:** ``` ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/root/.local' WARNING: The directory '/root/.cache/pip' or its parent directory is not owned or is not writable ``` -**Quick Fix - Use the Helper Script:** -```bash -sudo /home/ledpi/LEDMatrix/scripts/install_plugin_dependencies.sh -sudo systemctl restart ledmatrix -``` +Use one of the manual installs above (they pass `--no-cache-dir`). -**Manual Fix:** -```bash -# Install dependencies with --no-cache-dir to avoid cache permission issues -cd /home/ledpi/LEDMatrix/plugins/PLUGIN-NAME -sudo pip3 install --break-system-packages --no-cache-dir -r requirements.txt -sudo systemctl restart ledmatrix -``` - -**For more detailed troubleshooting, see:** [Plugin Dependency Troubleshooting Guide](PLUGIN_DEPENDENCY_TROUBLESHOOTING.md) - -## Architecture Summary - -``` -┌─────────────────────────────────────────────────────────────┐ -│ LEDMatrix Services │ -├─────────────────────────────────────────────────────────────┤ -│ │ -│ ledmatrix.service (User=root) │ -│ ledmatrix-web.service (User=root) │ -│ ├── Install dependencies system-wide │ -│ └── Accessible to all users │ -│ │ -├─────────────────────────────────────────────────────────────┤ -│ │ -│ Manual execution as ledpi │ -│ ├── Can READ system-wide packages ✅ │ -│ ├── WRITES go to ~/.local/ ⚠️ │ -│ └── Not accessible to root service │ -│ │ -└─────────────────────────────────────────────────────────────┘ -``` - -## Recommendations - -1. **For end users:** Always use the web interface for plugin management -2. **For developers:** Be aware of the user context when testing -3. **For plugin authors:** Test with `sudo systemctl restart ledmatrix` to ensure dependencies install correctly -4. **For CI/CD:** Always run installation as root or use the service - -## Helper Scripts - -### Install Plugin Dependencies Script - -Located at: `scripts/install_plugin_dependencies.sh` - -This script automatically finds and installs dependencies for all plugins: +### Checking where a package is installed ```bash -# Run as root (recommended for production) -sudo /home/ledpi/LEDMatrix/scripts/install_plugin_dependencies.sh +# How the service sees it +sudo python3 -c "import package_name; print(package_name.__file__)" -# Make executable if needed -chmod +x /home/ledpi/LEDMatrix/scripts/install_plugin_dependencies.sh +# A path under /home//.local/ means it was installed for that user only +python3 -m pip show -f package_name ``` -Features: -- Auto-detects all plugins with requirements.txt -- Uses correct installation method (system-wide vs user) -- Bypasses pip cache to avoid permission issues -- Provides detailed logging and error messages +For more, see the [Plugin Dependency Troubleshooting Guide](PLUGIN_DEPENDENCY_TROUBLESHOOTING.md). + +## For Plugin Authors + +1. Keep `requirements.txt` minimal and pin only what you need. +2. Test that it installs the way the Pi will install it: + ```bash + sudo python3 -m pip install --break-system-packages --no-cache-dir -r requirements.txt + ``` +3. Note any `apt` packages your plugin needs in its README. ## Files to Reference -- Service configs: `ledmatrix.service`, `ledmatrix-web.service` -- Plugin manager: `src/plugin_system/plugin_manager.py` -- Installation script: `first_time_install.sh` -- Dependency installer: `scripts/install_plugin_dependencies.sh` -- Troubleshooting guide: `PLUGIN_DEPENDENCY_TROUBLESHOOTING.md` - +- Service units: `systemd/ledmatrix.service`, `systemd/ledmatrix-web.service` +- Store installs: `src/plugin_system/store_manager.py` (`_install_dependencies`) +- Root install helper: `src/common/permission_utils.py` (`install_requirements_file`), `scripts/fix_perms/safe_pip_install.sh` +- Load-time installs: `src/plugin_system/plugin_loader.py` (`install_dependencies`) +- Sudo rules: `scripts/install/configure_web_sudo.sh` +- Manual installer: `scripts/install_plugin_dependencies.sh` diff --git a/docs/PLUGIN_DEPENDENCY_TROUBLESHOOTING.md b/docs/PLUGIN_DEPENDENCY_TROUBLESHOOTING.md index 1f33cb88..a62b9644 100644 --- a/docs/PLUGIN_DEPENDENCY_TROUBLESHOOTING.md +++ b/docs/PLUGIN_DEPENDENCY_TROUBLESHOOTING.md @@ -1,6 +1,7 @@ # Plugin Dependency Installation Troubleshooting -This guide helps resolve issues with automatic plugin dependency installation in the LEDMatrix system. +This guide helps resolve problems installing a plugin's Python packages. For +how installation works, see the [Plugin Dependency Guide](PLUGIN_DEPENDENCY_GUIDE.md). ## Common Error Symptoms @@ -10,109 +11,118 @@ ERROR: Could not install packages due to an OSError: [Errno 13] Permission denie WARNING: The directory '/root/.cache/pip' or its parent directory is not owned or is not writable ``` -### Context Mismatch +### Installed for the wrong user +The pip output shown after a web-UI install starts with: ``` -WARNING: Installing plugin dependencies for current user (not root). -These will NOT be accessible to the systemd service. +[Root install unavailable (...); installed for the current process's user only. +Packages may not be visible to ledmatrix.service if it runs as a different +user — run scripts/install/configure_web_sudo.sh to fix this.] ``` +### Plugin fails to load with `ModuleNotFoundError` +The display service can't see a package the plugin needs. + ## Root Cause -Plugin dependencies must be installed in a context accessible to the LEDMatrix systemd service, which runs as root. Permission errors typically occur when: +Plugin packages must be importable by `ledmatrix.service`, which runs as +root. The web interface (`ledmatrix-web.service`) runs as the user who +installed LEDMatrix, so it installs through a sudo helper +(`scripts/fix_perms/safe_pip_install.sh`). Problems usually come from: -1. The pip cache directory has incorrect permissions -2. The process tries to install to user directories without proper permissions -3. Environment variables (like HOME) are not set correctly for the service context +1. The sudoers rule for that helper missing, so the web UI installed the + packages for its own user only +2. Running `python3 run.py` by hand as a normal user, which installs missing + packages into `~/.local/` +3. pip's cache directory not being writable for root ## Solutions -### Solution 1: Use the Manual Installation Script (Recommended) - -We provide a helper script that handles dependency installation correctly: +### Solution 1: Restore the sudo rule, then reinstall ```bash -# Run as root to install system-wide (for production) -sudo /home/ledpi/LEDMatrix/scripts/install_plugin_dependencies.sh +cd ~/LEDMatrix +./scripts/install/configure_web_sudo.sh # as the web user, not with sudo +``` -# After installation, restart the service +Then reinstall the plugin from the **Plugin Manager** tab, or click +**Reinstall Plugin Deps** on the **Tools** tab. + +### Solution 2: Install every plugin's dependencies from the terminal + +```bash +sudo ~/LEDMatrix/scripts/install_plugin_dependencies.sh sudo systemctl restart ledmatrix ``` -This script: -- Detects all plugins with requirements.txt files -- Installs dependencies with correct permissions -- Uses `--no-cache-dir` to avoid cache permission issues -- Provides detailed logging for troubleshooting +The script finds each `requirements.txt` in the plugins directory set by +`plugin_system.plugins_directory` in `config/config.json` (default +`plugin-repos/`), installs with `--no-cache-dir`, and reports what it found. -### Solution 2: Manual Installation per Plugin - -If you need to install dependencies for a specific plugin: +### Solution 3: Install one plugin's dependencies ```bash -# Navigate to the plugin directory -cd /home/ledpi/LEDMatrix/plugins/PLUGIN-NAME +# Your configured plugins directory; plugin-repos/ by default +cd ~/LEDMatrix/plugin-repos/PLUGIN-NAME -# Install as root (system-wide) -sudo pip3 install --break-system-packages --no-cache-dir -r requirements.txt +sudo python3 -m pip install --break-system-packages --no-cache-dir -r requirements.txt -# Restart the service sudo systemctl restart ledmatrix ``` -### Solution 3: Fix Cache Directory Permissions +### Solution 4: Let the display service install them -If you specifically have cache permission issues: +When a plugin loads, the display service installs any missing requirements +itself, as root: + +```bash +sudo systemctl restart ledmatrix +sudo journalctl -u ledmatrix -f # watch for "Installing dependencies for plugin ..." +``` + +### Solution 5: Fix pip cache permissions ```bash # Option A: Skip the cache (recommended) -sudo pip3 install --no-cache-dir --break-system-packages -r requirements.txt +sudo python3 -m pip install --no-cache-dir --break-system-packages -r requirements.txt -# Option B: Fix cache permissions (if needed) +# Option B: Fix cache permissions sudo mkdir -p /root/.cache/pip sudo chown -R root:root /root/.cache sudo chmod -R 755 /root/.cache ``` -### Solution 4: Install via Web Interface - -The web interface handles dependency installation correctly in the service context: - -1. Access the web interface (`http://ledpi:5000` or `http://your-pi-ip:5000`) -2. Open the **Plugin Manager** tab (use the **Plugin Store** section to - find the plugin, or **Install from GitHub**) -3. Install the plugin through the web UI -4. The system automatically handles dependency installation in the - service context (which has the right permissions) - ## Prevention ### For Plugin Developers -When creating plugins with dependencies: - 1. **Keep requirements minimal**: Only include essential packages -2. **Test installation**: Verify your requirements.txt works with: +2. **Test installation** the way the Pi does it: ```bash - sudo pip3 install --break-system-packages --no-cache-dir -r requirements.txt + sudo python3 -m pip install --break-system-packages --no-cache-dir -r requirements.txt ``` 3. **Document dependencies**: Note any system packages needed (via apt) ### For Users -1. **Use web interface**: Install plugins via the web UI when possible -2. **Install as root**: When using SSH/terminal, use sudo for plugin installations -3. **Restart service**: After manual installations, restart the ledmatrix service +1. **Use the web interface** to install plugins +2. **Use sudo** for installs from SSH/terminal +3. **Restart the service** after manual installations ## Technical Details -### How Dependency Installation Works +### Where installs happen -The `PluginManager._install_plugin_dependencies()` method: - -1. Detects if running as root using `os.geteuid() == 0` -2. If root: Uses system-wide installation with `--break-system-packages --no-cache-dir` -3. If not root: Uses user installation with `--user --break-system-packages --no-cache-dir` -4. The `--no-cache-dir` flag prevents cache-related permission issues +- **Web UI install/update:** `PluginStoreManager._install_dependencies()` + → `install_requirements_file()` in `src/common/permission_utils.py`, which + runs `sudo -n bash scripts/fix_perms/safe_pip_install.sh `. + The helper only accepts the project's `requirements.txt` or one under + `plugin-repos/` or `plugins/`, and runs + `pip install --break-system-packages --ignore-installed` as root. If sudo + refuses, it falls back to a pip install as the web user and says so. +- **Plugin load:** `PluginLoader.install_dependencies()` in + `src/plugin_system/plugin_loader.py` skips satisfied requirements and + otherwise runs `pip install --break-system-packages` with the loading + process's interpreter — root in `ledmatrix.service`. ### Why `--break-system-packages`? @@ -120,26 +130,20 @@ Debian 12+ (Bookworm) and Raspberry Pi OS based on it implement PEP 668, which p ### Service Context -The ledmatrix.service runs as: -- **User**: root -- **WorkingDirectory**: /home/ledpi/LEDMatrix -- **Python**: /usr/bin/python3 +- `ledmatrix.service` runs as **root** with `/usr/bin/python3` +- `ledmatrix-web.service` runs as **the installing user** -Dependencies must be installed in root's Python environment or system-wide to be accessible. +Dependencies must be installed system-wide (as root) to be visible to the +display service. ## Checking Installation -Verify dependencies are installed correctly: - ```bash # Check as root (how the service sees it) -sudo python3 -c "import package_name" +sudo python3 -c "import package_name; print(package_name.__file__)" -# List installed packages -pip3 list - -# Check specific package -pip3 show package_name +# A path under /home//.local/ means a user-only install +python3 -m pip show -f package_name ``` ## Getting Help @@ -151,19 +155,11 @@ If you continue to experience issues: sudo journalctl -u ledmatrix -f ``` -2. Check pip logs (created by manual script): +2. Verify the plugin manifest and requirements (default plugins directory + shown): ```bash - cat /tmp/pip_install_*.log - ``` - -3. Verify plugin manifest is correct: - ```bash - cat /home/ledpi/LEDMatrix/plugins/PLUGIN-NAME/manifest.json - ``` - -4. Check plugin requirements: - ```bash - cat /home/ledpi/LEDMatrix/plugins/PLUGIN-NAME/requirements.txt + cat ~/LEDMatrix/plugin-repos/PLUGIN-NAME/manifest.json + cat ~/LEDMatrix/plugin-repos/PLUGIN-NAME/requirements.txt ``` ## Related Documentation @@ -171,4 +167,3 @@ If you continue to experience issues: - [Plugin Dependency Guide](PLUGIN_DEPENDENCY_GUIDE.md) - [Plugin Development Guide](PLUGIN_DEVELOPMENT_GUIDE.md) - [Troubleshooting](TROUBLESHOOTING.md) -