LEDMatrix/docs/TROUBLESHOOTING.md

Troubleshooting Guide

Quick Diagnosis Steps

Run these checks first to quickly identify common issues:

1. Check Service Status

# Check all LEDMatrix services
sudo systemctl status ledmatrix
sudo systemctl status ledmatrix-web
sudo systemctl status ledmatrix-wifi-monitor

# Check AP mode services (if using WiFi)
sudo systemctl status hostapd
sudo systemctl status dnsmasq

Note: Look for active (running) status and check for error messages in the output.

2. View Service Logs

IMPORTANT: The web service logs to syslog, NOT stdout. Use journalctl to view logs:

# View all recent logs
sudo journalctl -u ledmatrix -n 50
sudo journalctl -u ledmatrix-web -n 50

# Follow logs in real-time
sudo journalctl -u ledmatrix -f

# View logs from last hour
sudo journalctl -u ledmatrix-web --since "1 hour ago"

# Filter for errors only
sudo journalctl -u ledmatrix -p err

3. Run Diagnostic Scripts

# Web interface diagnostics
bash scripts/diagnose_web_interface.sh

# WiFi setup verification
./scripts/verify_wifi_setup.sh

# Captive portal troubleshooting
./scripts/troubleshoot_captive_portal.sh

Weather is provided by the ledmatrix-weather plugin (installed via the Plugin Store). To troubleshoot weather, check that plugin's tab in the web UI for its API key and recent error messages, then watch the Logs tab.

4. Check Configuration

# Verify web interface autostart
cat config/config.json | grep web_display_autostart

# Check plugin enabled status
cat config/config.json | grep -A 2 "plugin-id"

# Verify API keys present
ls -l config/config_secrets.json

5. Test Manual Startup

# Test web interface manually
python3 web_interface/start.py

# If it works manually but not as a service, check systemd service file

---

Common Issues by Category

Installation & Build Issues

"This version of Raspberry Pi OS is not supported"

LEDMatrix installs on Raspberry Pi OS Lite Trixie (Debian 13, Python 3.13) or Bookworm (Debian 12, Python 3.11). The installer checks /etc/os-release before it changes anything and stops on anything else.

Check what you have:

grep -E '^(PRETTY_NAME|VERSION_ID)=' /etc/os-release
python3 --version

Solutions:

• VERSION_ID="11" (Bullseye) or older: flash a new card with Raspberry Pi Imager, choosing Raspberry Pi OS Lite (64-bit). Trixie is recommended; Bookworm (Legacy) also works. An in-place upgrade from Bullseye is not supported by Raspberry Pi and is not worth the risk.
• "Desktop environment detected": use the Lite image, not the desktop one.
• "python3 is Python 3.x; LEDMatrix needs Python 3.11 or newer": something has replaced the system python3. Point it back at the OS's own Python (/usr/bin/python3 should be 3.11 on Bookworm, 3.13 on Trixie).
• sudo bash scripts/check_system_compatibility.sh runs the same checks without installing anything.

"This Pi manages its network with dhcpcd, not NetworkManager"

A warning, not an error: the install carries on and the display works. But choosing a WiFi network from the web page and the LEDMatrix-Setup hotspot both need NetworkManager, the default on Bookworm and Trixie. It appears when dhcpcd was selected in raspi-config. Switch back with a keyboard and screen attached (or over Ethernet), since the WiFi connection drops briefly:

sudo raspi-config   # Advanced Options -> Network Config -> NetworkManager
sudo reboot

Step 6 fails: "Failed building wheel for rgbmatrix"

Symptoms:

note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for rgbmatrix
Failed to build rgbmatrix
✗ Failed to install rpi-rgb-led-matrix Python package

Cause:

Almost always the kernel's out-of-memory killer, not missing build tools. The rpi-rgb-led-matrix library compiles roughly 45 C++ translation units, two of them Cython-generated — a single cc1plus on those can peak near 800MB. The build system defaults to running several of those at once, which exceeds RAM on 512MB and 1GB boards. Because the OOM killer writes nothing to pip's output, the failure looks like a toolchain problem, and sudo apt install -y python-dev-is-python3 cmake build-essential will report everything is already up to date.

How to confirm:

dmesg -T | grep -i "out of memory"     # look for "Killed process ... (cc1plus)"
free -h                                 # total RAM and swap

Fix:

Current versions of the installer handle this automatically: they cap build parallelism based on available RAM and add a temporary swapfile for the build, removing it when the build finishes. If you are on an older checkout, or the temporary swapfile could not be created, either force a serial compile:

sudo ./first_time_install.sh --build-jobs 1

or add permanent swap and re-run the installer, which resumes at Step 6:

sudo apt install -y dphys-swapfile
sudo sed -i 's/^#\?CONF_SWAPSIZE=.*/CONF_SWAPSIZE=2048/' /etc/dphys-swapfile
sudo sed -i 's/^#\?CONF_MAXSWAP=.*/CONF_MAXSWAP=2048/' /etc/dphys-swapfile
sudo dphys-swapfile swapoff && sudo dphys-swapfile setup && sudo dphys-swapfile swapon
sudo ./first_time_install.sh

CONF_MAXSWAP matters: it defaults to 2048 and silently clamps CONF_SWAPSIZE, so setting only CONF_SWAPSIZE to a larger value has no effect.

Related:

• The installer needs roughly 3GB free on the card to place the swapfile. If disk is tight it will say so and skip the swapfile: sudo apt clean first.
• sudo bash scripts/check_system_compatibility.sh reports RAM and disk.
• sudo bash scripts/diagnose_dependencies.sh dumps build-dependency state.

---

Web Interface & Service Issues

Service Not Running/Starting

Symptoms:

• Cannot access web interface at http://your-pi-ip:5000
• systemctl status ledmatrix-web shows inactive (dead)

Solutions:

1. Start the service:

   sudo systemctl start ledmatrix-web
   

2. Enable on boot:

   sudo systemctl enable ledmatrix-web
   

3. Check why it failed:

   sudo journalctl -u ledmatrix-web -n 50
   

web_display_autostart is False

Symptoms:

• Service exists but web interface doesn't start automatically
• Logs show service starting but nothing happens

Solution:

# Edit config.json
nano config/config.json

# Set web_display_autostart to true
{
  "web_display_autostart": true,
  ...
}

# Restart service
sudo systemctl restart ledmatrix-web

Import or Dependency Errors

Symptoms:

• Logs show ModuleNotFoundError or ImportError
• Service fails to start with Python errors

Solutions:

1. Install dependencies as root, so the root display service can import them:

   sudo python3 -m pip install --break-system-packages --no-cache-dir -r requirements.txt
   sudo python3 -m pip install --break-system-packages --no-cache-dir -r web_interface/requirements.txt
   

2. Test imports step-by-step:

   python3 -c "from src.config_manager import ConfigManager; print('OK')"
   python3 -c "from src.plugin_system.plugin_manager import PluginManager; print('OK')"
   python3 -c "from web_interface.app import app; print('OK')"
   

3. Check Python path:

   python3 -c "import sys; print(sys.path)"
   

Port Already in Use

Symptoms:

• Error: Address already in use
• Service fails to bind to port 5000

Solutions:

1. Check what's using the port:

   sudo lsof -i :5000
   

2. Kill the conflicting process:

   sudo kill -9 <PID>
   

3. Or change the port in start.py:

   app.run(host='0.0.0.0', port=5051)
   

Permission Issues

Symptoms:

• Permission denied errors in logs
• Cannot read/write configuration files

Solutions:

PERMISSIONS.md lists the expected owner and mode of every file and directory, and which scripts/fix_perms/ script to run as which user. Don't chown -R the whole project: the two sudo helper scripts in scripts/fix_perms/ must stay owned by root.

# Config files: web user owns both; secrets must stay 640
stat -c '%U:%G %a %n' config/config.json config/config_secrets.json
sudo chmod 644 config/config.json
sudo chmod 640 config/config_secrets.json

# Which user the web interface runs as
sudo systemctl cat ledmatrix-web | grep User

Flask/Blueprint Import Errors

Symptoms:

• ImportError: cannot import name 'app'
• ModuleNotFoundError: No module named 'blueprints'

Solutions:

1. Verify file structure:

   ls -l web_interface/app.py
   ls -ld web_interface/blueprints/api_v3/
   ls -l web_interface/blueprints/pages_v3.py
   

2. Check for __init__.py files:

   ls -l web_interface/__init__.py
   ls -l web_interface/blueprints/__init__.py
   

3. Test import manually:

   cd /home/ledpi/LEDMatrix
   python3 -c "from web_interface.app import app"
   

---

Issue: Updates and the update channel

Symptoms:

• The General tab says "Stable: this device runs code newer than the newest release ... keeps following main"
• Tools shows a version such as v3.8.0 instead of a branch name, or git status over SSH says HEAD detached at v3.8.0
• Update Code says "already up to date" while GitHub's main has newer commits

Explanation: these are the Stable update channel working as intended (auto_update.channel, General → Update Channel). Stable installs the newest release tag, which git checks out without a branch ("detached HEAD"); that is normal and every update path handles it. Stable never installs an older version than the one running, so a device that is ahead of the newest release keeps following main until a release includes its commit, then switches to releases on its own.

Solutions:

1. Want the newest code instead? Set Update Channel to Beta and click Update Code. The device leaves the release for main and pulls it. Or from SSH:

   curl -X POST http://localhost:5000/api/v3/system/update-channel \
        -H 'Content-Type: application/json' -d '{"channel": "beta"}'
   

2. See what the next update will do:

   curl 'http://localhost:5000/api/v3/system/update-channel?fetch=1'
   

3. Local changes after a channel switch: edits that no longer fit the new version are kept in the git stash rather than lost; git stash list shows them as "LEDMatrix autostash before update".
4. A new install is on a release, not main. The one-shot installer checks out the newest release. For the newest code instead, install with LEDMATRIX_CHANNEL=beta:

   curl -fsSL https://raw.githubusercontent.com/ChuckBuilds/LEDMatrix/main/scripts/install/one-shot-install.sh | LEDMATRIX_CHANNEL=beta bash
   

---

Issue: "service settings ... are not applied yet" after an update

Symptoms:

• Update Code's message, or the web interface log, says an update changes service settings that are not applied yet, and to run the installer
• The display logs ledmatrix.service differs from systemd/ledmatrix.service at startup

Explanation: updates install the systemd units a new version changes through the root helper /usr/local/sbin/ledmatrix-refresh-units, which the installer sets up and grants to the web user in /etc/sudoers.d/ledmatrix_web. A device installed before that has neither, so the new unit settings (for example the display's watchdog) wait for a reinstall. The update itself is fine.

Solution: re-run the installer once, as root:

cd ~/LEDMatrix
sudo ./first_time_install.sh
# or, lighter: install the units and helper, then the sudo rules
sudo ./scripts/install/install_service.sh
./scripts/install/configure_web_sudo.sh

Check it worked:

ls -l /usr/local/sbin/ledmatrix-refresh-units   # root root, rwxr-xr-x
sudo -l | grep ledmatrix-refresh-units           # the two rules

A message that the helper refused a unit (refusing to install it) means a template in systemd/ was edited so that it would run as another account or from another folder. The message names the template. Look at what changed with git diff -- systemd/, save any edit you want to keep, then restore only that file, for example git checkout -- systemd/ledmatrix-web.service.

---

WiFi & AP Mode Issues

AP Mode Not Activating

Symptoms:

• WiFi disconnected but AP mode doesn't start
• Cannot find "LEDMatrix-Setup" network

Solutions:

1. Check auto-enable setting:

   cat config/wifi_config.json | grep auto_enable_ap_mode
   # Should show: "auto_enable_ap_mode": true
   

2. Verify WiFi monitor service is running:

   sudo systemctl status ledmatrix-wifi-monitor
   

3. Wait for grace period (90 seconds):

   • AP mode requires 3 consecutive disconnected checks at 30-second intervals
   • Total wait time: 90 seconds after WiFi disconnects

4. Check if Ethernet is connected:

   nmcli device status
   # If Ethernet is connected, AP mode won't activate
   

5. Check required services:

   systemctl is-active NetworkManager   # must say "active"
   sudo systemctl status hostapd
   sudo systemctl status dnsmasq
   

   On a fresh install hostapd shows as masked. That is expected, on Bookworm and Trixie alike: Debian's hostapd package masks the service when it is installed without a configuration, so the hotspot is brought up through NetworkManager instead (look for nmcli hotspot fallback in journalctl -u ledmatrix-wifi-monitor). If NetworkManager is not active, see "This Pi manages its network with dhcpcd" above.

6. Manually enable AP mode:

   # Via API (the WiFi blueprint is mounted under /api/v3)
   curl -X POST http://localhost:5000/api/v3/wifi/ap/enable
   
   # Via Python
   python3 -c "
   from src.wifi_manager import WiFiManager
   wm = WiFiManager()
   wm.enable_ap_mode()
   "
   

Cannot Connect to AP Mode / Connection Refused

Symptoms:

• Can see "LEDMatrix-Setup" network but can't connect to web interface
• Browser shows "Connection Refused" or "Can't connect to server"
• AP mode active but web interface not accessible

Solutions:

1. Verify web server is running:

   sudo systemctl status ledmatrix-web
   # Should be active (running)
   

2. Use correct IP address and port:

   • Correct: http://192.168.4.1:5000
   • NOT: http://192.168.4.1 (port 80 — nothing listens there)

3. Check wlan0 has correct IP:

   ip addr show wlan0
   # Should show: inet 192.168.4.1/24
   

4. Verify hostapd and dnsmasq are running:

   sudo systemctl status hostapd
   sudo systemctl status dnsmasq
   

5. Test from the Pi itself:

   curl http://192.168.4.1:5000
   # Should return HTML
   

DNS Resolution Failures

Symptoms:

• Captive portal doesn't redirect automatically
• DNS lookups fail when connected to AP mode

Solutions:

1. Check dnsmasq status:

   sudo systemctl status dnsmasq
   sudo journalctl -u dnsmasq -n 20
   

2. Verify DNS configuration:

   cat /etc/dnsmasq.conf | grep -v "^#" | grep -v "^$"
   

3. Test DNS resolution:

   nslookup captive.apple.com
   # Should resolve to 192.168.4.1 when in AP mode
   

4. Manual captive portal testing:

   • Try these URLs manually:
     • http://192.168.4.1:5000
     • http://captive.apple.com
     • http://connectivitycheck.gstatic.com/generate_204

Firewall Blocking Port 5000

Symptoms:

• Services running but cannot connect
• Works from Pi but not from other devices

Solutions:

1. Check UFW status:

   sudo ufw status
   

2. Allow port 5000:

   sudo ufw allow 5000/tcp
   

3. Check iptables:

   sudo iptables -L -n
   

4. Temporarily disable firewall to test:

   sudo ufw disable
   # Test if it works, then re-enable and add rule
   sudo ufw enable
   sudo ufw allow 5000/tcp
   

---

Plugin Issues

Plugin Not Enabled

Symptoms:

• Plugin installed but doesn't appear in rotation
• Plugin shows in web interface but is greyed out

Solutions:

1. Enable in configuration:

   {
     "plugin-id": {
       "enabled": true,
       ...
     }
   }
   

   Or toggle the plugin on in the Plugin Manager tab, which writes the same flag.

2. Wait a few seconds. The display service watches config.json and loads a newly enabled plugin without a restart (DisplayController._reconcile_enabled_plugins() in src/display_controller.py). This needs hot reload, which is on unless LEDMATRIX_HOT_RELOAD=false is set.

3. If it still does not appear, check the logs for a config validation error, then restart: sudo systemctl restart ledmatrix

Plugin Not Loading

Symptoms:

• Plugin enabled but not showing
• Errors in logs about plugin

Solutions:

1. Check plugin directory exists:

   ls -ld plugin-repos/plugin-id/
   

2. Verify manifest.json:

   cat plugin-repos/plugin-id/manifest.json
   # Verify all required fields present
   

3. Check dependencies installed. Install them with sudo: the display service runs as root and does not see packages pip put in your user's ~/.local (see PLUGIN_DEPENDENCY_GUIDE.md):

   if [ -f plugin-repos/plugin-id/requirements.txt ]; then
     sudo python3 -m pip install --break-system-packages --no-cache-dir -r plugin-repos/plugin-id/requirements.txt
   fi
   

4. Check logs for plugin errors:

   sudo journalctl -u ledmatrix -f | grep plugin-id
   

5. Load and render the plugin headlessly:

   python3 scripts/check_plugin.py --plugin plugin-id
   

Panel Frozen, or the Display Restarts Every Few Minutes

Symptoms:

• The panel stops changing while systemctl status ledmatrix says active
• The display restarts on its own, a couple of minutes after it froze
• /api/v3/health shows checks.display_loop.status as stalled

The display's render loop checks in with systemd every few seconds (WatchdogSec=120 in ledmatrix.service) and writes a heartbeat to /run/ledmatrix/display-heartbeat.json. When the loop gets stuck -- almost always inside one plugin's display() -- the check-ins stop, and after two minutes systemd kills and restarts the display. The kill dumps every thread's stack into the log, so it says which plugin was stuck.

Solutions:

1. Find the stuck plugin. Look for the watchdog kill and the stack dump after it. The render loop is the thread whose stack runs through display_controller.py in run (usually the Current thread block); the first plugin-repos/... file in it is the plugin:

   sudo journalctl -u ledmatrix --since "1 hour ago" | grep -A40 "Watchdog timeout"
   

2. Check the heartbeat by hand. Its age should stay under about ten seconds while the display runs:

   cat /run/ledmatrix/display-heartbeat.json
   curl -s http://localhost:5000/api/v3/health | python3 -m json.tool | grep -A3 display_loop
   

   not_reported means the display writes no heartbeat: it has not drawn its first frame yet, or it runs an older version.

3. Disable the plugin in the web UI and report it to its author with the stack dump. Restarts that repeat back off from 10 seconds to two minutes apart, so a plugin that hangs on every start does not restart the display hundreds of times an hour.

4. Is the watchdog installed? Updates install new unit settings once the installer has set up ledmatrix-refresh-units; installs from before that keep their old unit until the installer is re-run (a startup warning says the unit differs from its template):

   systemctl show -p WatchdogUSec ledmatrix   # 2min once running; 0 = not installed
   sudo ./scripts/install/install_service.sh
   

   WatchdogUSec reads 15min for the first minutes after a start: that is the start-up allowance, narrowed to two minutes once the first frame is on the panel.

5. A plugin that legitimately blocks longer than two minutes (it should not; display() runs on the render thread) can be given more time with a drop-in, sudo systemctl edit ledmatrix:

   [Service]
   WatchdogSec=300
   

   WatchdogSec=0 turns the watchdog off.

Stale Cache Data

Symptoms:

• Plugin shows old data
• Data doesn't update even after restarting
• Clearing cache in web interface doesn't help

Solutions:

1. Manual cache clearing:

   The cache does not live in the project directory. The cache manager uses the first writable location among /var/cache/ledmatrix, ~/.ledmatrix_cache, /opt/ledmatrix/cache, and $TMPDIR/ledmatrix_cache. The easiest option is the helper script:

   # Clear the cache with the helper script
   sudo python3 scripts/utils/clear_cache.py --clear-all
   
   # Or remove files manually from the cache dir in use, e.g.:
   sudo rm -rf /var/cache/ledmatrix/*
   
   # Restart display
   sudo systemctl restart ledmatrix
   

2. Check cache permissions. Expected: root:ledmatrix, drwxrwsr-x. setup_cache.sh restores that layout (see PERMISSIONS.md):

   ls -ld /var/cache/ledmatrix
   sudo bash scripts/install/setup_cache.sh
   

---

Weather Plugin Specific Issues

Missing or Invalid API Key

Symptoms:

• "No Weather Data" message on display
• Logs show API authentication errors

Solutions:

1. Get OpenWeatherMap API key:

   • Sign up at https://openweathermap.org/api
   • Free tier: 1,000 calls/day, 60 calls/minute
   • Copy your API key

2. Add to config_secrets.json (recommended):

   {
     "openweathermap_api_key": "your-api-key-here"
   }
   

3. Or add to config.json:

   {
     "ledmatrix-weather": {
       "enabled": true,
       "openweathermap_api_key": "your-api-key-here",
       ...
     }
   }
   

4. Secure the API key file:

   chmod 640 config/config_secrets.json
   

5. Restart display:

   sudo systemctl restart ledmatrix
   

API Rate Limits Exceeded

Symptoms:

• Weather works initially then stops
• Logs show HTTP 429 errors (Too Many Requests)
• Error message: "Rate limit exceeded"

Solutions:

1. Increase update interval:

   {
     "ledmatrix-weather": {
       "update_interval": 300,
       ...
     }
   }
   

   Note: Minimum recommended: 300 seconds (5 minutes)

   How often the core calls the plugin's update() comes from the plugin itself first: its get_update_interval() if it has one, then update_interval in its manifest.json. The update_interval in config.json is used by the scheduler only when the manifest sets none. Many plugins also read their own config update_interval and skip the API call inside update() until it has elapsed, which is what makes the setting above effective; check the plugin's settings form or config_schema.json for the option it actually honours.

2. Check current rate limit usage:

   • OpenWeatherMap free tier: 1,000 calls/day, 60 calls/minute
   • With 300s interval: 288 calls/day (well within limits)

3. Monitor API calls:

   sudo journalctl -u ledmatrix -f | grep "openweathermap"
   

Invalid Location Configuration

Symptoms:

• "No Weather Data" message
• Logs show location not found errors

Solutions:

1. Use correct location format:

   {
     "ledmatrix-weather": {
       "city": "Tampa",
       "state": "FL",
       "country": "US"
     }
   }
   

2. Use ISO country codes:

   • US = United States
   • GB = United Kingdom
   • CA = Canada
   • etc.

3. Test API call manually:

   API_KEY="your-key-here"
   curl "http://api.openweathermap.org/data/2.5/weather?q=Tampa,FL,US&appid=${API_KEY}"
   

Network Connectivity to OpenWeatherMap

Symptoms:

• Other internet features work
• Weather specifically fails
• Connection timeout errors

Solutions:

1. Test connectivity:

   ping api.openweathermap.org
   

2. Test DNS resolution:

   nslookup api.openweathermap.org
   

3. Test API endpoint:

   curl -I https://api.openweathermap.org
   # Should return HTTP 200 or 301
   

4. Check firewall:

   # Ensure HTTPS (443) is allowed for outbound connections
   sudo ufw status
   

---

Diagnostic Commands Reference

Service Commands

# Check status
sudo systemctl status ledmatrix
sudo systemctl status ledmatrix-web
sudo systemctl status ledmatrix-wifi-monitor

# Start service
sudo systemctl start <service-name>

# Stop service
sudo systemctl stop <service-name>

# Restart service
sudo systemctl restart <service-name>

# Enable on boot
sudo systemctl enable <service-name>

# Disable on boot
sudo systemctl disable <service-name>

# View service file
sudo systemctl cat <service-name>

# Reload systemd after editing service files
sudo systemctl daemon-reload

Log Viewing Commands

# View recent logs (last 50 lines)
sudo journalctl -u ledmatrix -n 50

# Follow logs in real-time
sudo journalctl -u ledmatrix -f

# View logs from specific time
sudo journalctl -u ledmatrix --since "1 hour ago"
sudo journalctl -u ledmatrix --since "2024-01-01 10:00:00"

# View logs until specific time
sudo journalctl -u ledmatrix --until "2024-01-01 12:00:00"

# Filter by priority (errors only)
sudo journalctl -u ledmatrix -p err

# Filter by priority (warnings and errors)
sudo journalctl -u ledmatrix -p warning

# Search logs for specific text
sudo journalctl -u ledmatrix | grep "error"
sudo journalctl -u ledmatrix | grep -i "plugin"

# View logs for multiple services
sudo journalctl -u ledmatrix -u ledmatrix-web -n 50

# Export logs to file
sudo journalctl -u ledmatrix > ledmatrix.log

Network Testing Commands

# Test connectivity
ping -c 4 8.8.8.8
ping -c 4 api.openweathermap.org

# Test DNS resolution
nslookup api.openweathermap.org
dig api.openweathermap.org

# Test HTTP endpoint
curl -I http://your-pi-ip:5000
curl http://192.168.4.1:5000

# Check listening ports
sudo lsof -i :5000
sudo netstat -tuln | grep 5000

# Check network interfaces
ip addr show
nmcli device status

File/Directory Verification

# Check file exists
ls -l config/config.json
ls -l plugin-repos/plugin-id/manifest.json

# Check directory structure
ls -la web_interface/
ls -la plugin-repos/

# Check file permissions
ls -l config/config_secrets.json

# Check file contents
cat config/config.json | jq .
cat config/wifi_config.json | grep auto_enable

Python Import Testing

# Test core imports
python3 -c "from src.config_manager import ConfigManager; print('OK')"
python3 -c "from src.plugin_system.plugin_manager import PluginManager; print('OK')"
python3 -c "from src.display_manager import DisplayManager; print('OK')"

# Test web interface imports
python3 -c "from web_interface.app import app; print('OK')"
python3 -c "from web_interface.blueprints.api_v3 import api_v3; print('OK')"

# Test WiFi manager
python3 -c "from src.wifi_manager import WiFiManager; print('OK')"

# Test plugin import
python3 -c "
import sys
sys.path.insert(0, 'plugin-repos/plugin-id')
from manager import PluginClass
print('Plugin imports OK')
"

---

Reinstalling Service Files

If a systemd service file is corrupted or missing, do NOT hand-write one. The real unit files live in the repo's systemd/ directory (ledmatrix.service, ledmatrix-web.service, ledmatrix-wifi-monitor.service) and contain a __PROJECT_ROOT_DIR__ placeholder that the install scripts substitute with your actual checkout path:

# Reinstall the display service unit
sudo ./scripts/install/install_service.sh

# Reinstall the web interface service unit
sudo ./scripts/install/install_web_service.sh

Note that ledmatrix-web.service runs as root via scripts/utils/start_web_conditionally.py — root is needed for system operations (service control, WiFi management), and the wrapper honors the web_display_autostart config flag before actually starting the web server.

---

Complete Diagnostic Script

Run this script for comprehensive diagnostics:

#!/bin/bash

echo "=== LEDMatrix Diagnostic Report ==="
echo ""

echo "1. Service Status:"
systemctl status ledmatrix --no-pager -n 5
systemctl status ledmatrix-web --no-pager -n 5
echo ""

echo "2. Recent Logs:"
journalctl -u ledmatrix -n 20 --no-pager
echo ""

echo "3. Configuration:"
cat config/config.json | grep -E "(web_display_autostart|enabled)"
echo ""

echo "4. Network Status:"
ip addr show | grep -E "(wlan|eth|inet )"
curl -s http://localhost:5000 > /dev/null && echo "Web interface: OK" || echo "Web interface: FAILED"
echo ""

echo "5. File Structure:"
ls -la web_interface/ | head -10
ls -la plugin-repos/ | head -10
echo ""

echo "6. Python Imports:"
python3 -c "from src.config_manager import ConfigManager" && echo "ConfigManager: OK" || echo "ConfigManager: FAILED"
python3 -c "from web_interface.app import app" && echo "Web app: OK" || echo "Web app: FAILED"
echo ""

echo "=== End Diagnostic Report ==="

---

Success Indicators

A properly functioning system should show:

1. Services Running:

   ● ledmatrix.service - active (running)
   ● ledmatrix-web.service - active (running)
   

2. Web Interface Accessible:

   • Navigate to http://your-pi-ip:5000
   • Page loads successfully
   • Display preview visible

3. Logs Show Normal Operation:

   INFO: Web interface started on port 5000
   INFO: Loaded X plugins
   INFO: Display rotation active
   

4. Process Listening on Port:

   $ sudo lsof -i :5000
   COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
   python3  1234 ledpi  3u   IPv4  12345      0t0  TCP *:5000 (LISTEN)
   

5. Plugins Loading:

   • Logs show plugin initialization
   • Plugins appear in web interface
   • Display cycles through enabled plugins

---

Emergency Recovery

If the system is completely broken:

1. Git Rollback

# View recent commits
git log --oneline -10

# Rollback to previous commit
git reset --hard HEAD~1

# Or rollback to specific commit
git reset --hard <commit-hash>

# On the Stable update channel HEAD is a release tag, not a branch:
# go back to an earlier release instead (the next update moves forward again)
git tag --list 'v*' --sort=-v:refname | head
git checkout --detach v3.7.0

# Restart all services
sudo systemctl restart ledmatrix
sudo systemctl restart ledmatrix-web

2. Fresh Service Installation

# Reinstall WiFi monitor
sudo ./scripts/install/install_wifi_monitor.sh

# Recreate service files (substitutes __PROJECT_ROOT_DIR__ in systemd/ units)
sudo ./scripts/install/install_service.sh
sudo ./scripts/install/install_web_service.sh

# Restart
sudo systemctl restart ledmatrix ledmatrix-web

3. Full System Reboot

# As a last resort
sudo reboot

---

Related Documentation

• WEB_INTERFACE_GUIDE.md - Web interface usage
• WIFI_NETWORK_SETUP.md - WiFi configuration
• PLUGIN_STORE_GUIDE.md - Plugin installation
• REST_API_REFERENCE.md - API documentation