Nine CodeRabbit findings, five in code. **Health state (the one that matters).** The non-dict guard did not cover a dict missing fields the callers index directly, which is the shape actually seen in the wild: a record carrying only circuit_state produced `plugin clock-simple operation failed: 'circuit_state'` about fifty times a minute with the panel frozen. The record is now completed against the defaults per field rather than trusted or discarded wholesale. Per field matters: a first pass rejected any incomplete record outright, which reset a tripped breaker and real failure counts to healthy because one optional field was absent -- an existing test caught it. Values of the wrong type (a counter persisted as a string, an unknown circuit_state) fall back individually, valid neighbours survive, and newer fields the schema has grown since (degraded, degraded_reason) are carried through untouched. **Cache ceiling.** MemoryCache.set() accepted entries without bound between cleanup sweeps, which run every 300s by default, so a burst could take the cache far past max_size -- the unbounded growth the limit exists to stop. Eviction now runs under the same lock on every write, sharing one helper with the periodic sweep so the two cannot drift. **Installer, cgroups.** Only cgroup_enable=memory was checked, so a board carrying that without cgroup_memory=1 reported success and got no change, leaving MemoryMax= inert. Each parameter is now checked and appended independently; verified against all four combinations, single line preserved. **Installer, journald.** Persistence was inferred from /var/log/journal being non-empty, which proves neither Storage=persistent nor a size cap -- the directory survives a switch back to volatile. The effective configuration is read instead (systemd-analyze cat-config, falling back to the conf files), and an explicitly configured SystemMaxUse is preserved rather than overwritten. Verified across volatile, persistent-without-cap, persistent-with-user-cap, cap-without-storage, and commented-only configs. **Dependency extras.** _extras_are_satisfied stopped at one level, so a gated dependency that itself requests an extra (requests[socks]) passed on the base distribution's version while the extra's own dependency was missing, and pip was skipped. It now recurses, with a visited (distribution, extras) set so a cycle terminates. Docs: both kernel command-line paths documented (the installer falls back to /boot/cmdline.txt), daemon-reload and restart added after the systemd override example, memory exhaustion added to the SSH summary with its power-cycle-only recovery, and a language on the fenced block for MD040. Tests: five for the health-state repair including the exact wild shape and that record_failure/record_success no longer raise against it, and one for the cache ceiling. Both mutation-checked. Full suite 2927 passed, with the one pre-existing tmpfs failure that also fails on main. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01STMbQE4YctTacQXfbYqKuW
6.8 KiB
SSH Unavailable After Installation - Troubleshooting Guide
Why SSH Becomes Unavailable
After running first_time_install.sh, SSH may become unavailable for the following reasons:
1. WiFi Monitor Service Enables AP Mode
Primary Cause: The WiFi monitor service (ledmatrix-wifi-monitor) automatically enables Access Point (AP) mode when it detects that the Raspberry Pi is not connected to WiFi. When AP mode is active:
- The Pi creates its own WiFi network: LEDMatrix-Setup (password:
ledmatrix123) - The Pi's WiFi interface (
wlan0) switches from client mode to AP mode - This disconnects the Pi from your original WiFi network
- SSH becomes unavailable because the Pi is no longer on your network
2. Network Configuration Changes
The installation script:
- Installs and configures
hostapd(Access Point daemon) - Installs and configures
dnsmasq(DHCP server for AP mode) - These services can interfere with normal WiFi client mode
3. The Board Ran Out of Memory
On a 512MB or 1GB board, memory exhaustion stops sshd being able to fork a
session process. The connection is accepted and then closed immediately, before
any banner:
kex_exchange_identification: Connection closed by remote host
The giveaway is that the board is otherwise healthy — ping is clean and the web UI still responds — but nothing that needs to start a new process works, and the panel is usually dark. Only a power cycle clears it. See LOW_MEMORY_BOARDS.md.
4. Reboot After Installation
If the script reboots the Pi (which it recommends), network services may restart in a different state, potentially triggering AP mode.
How to Regain SSH Access
Option 1: Connect to AP Mode (Recommended for Initial Setup)
-
Find the AP Network:
- Look for a WiFi network named LEDMatrix-Setup on your phone/computer
- Default password:
ledmatrix123
-
Connect to the AP:
- Connect your device to the LEDMatrix-Setup network
- The Pi will have IP address:
192.168.4.1
-
SSH via AP Mode:
ssh devpi@192.168.4.1 -
Disable AP Mode and Reconnect to WiFi: Once connected via SSH:
# Check WiFi status nmcli device status # Disable AP mode manually sudo systemctl stop hostapd sudo systemctl stop dnsmasq # Connect to your WiFi network (replace with your SSID and password) sudo nmcli device wifi connect "YourWiFiSSID" password "YourPassword" # Or use the web interface at http://192.168.4.1:5000 # Navigate to WiFi tab and connect to your network
Option 2: Disable WiFi Monitor Service Temporarily
If you have physical access to the Pi or can connect via AP mode:
# Stop the WiFi monitor service
sudo systemctl stop ledmatrix-wifi-monitor
# Disable it from starting on boot (optional)
sudo systemctl disable ledmatrix-wifi-monitor
# Stop AP mode services
sudo systemctl stop hostapd
sudo systemctl stop dnsmasq
# Reconnect to your WiFi network
sudo nmcli device wifi connect "YourWiFiSSID" password "YourPassword"
Option 3: Use Ethernet Connection
If your Pi is connected via Ethernet:
- SSH should remain available via Ethernet even if WiFi is in AP mode
- Connect via:
ssh devpi@<pi-ip-address>
Option 4: Physical Access
If you have physical access to the Pi:
- Connect a keyboard and monitor
- Log in locally
- Follow Option 2 to disable AP mode and reconnect to WiFi
Preventing SSH Loss in the Future
Method 1: Configure WiFi Before Installation
Before running first_time_install.sh, ensure WiFi is properly configured and connected:
# Check WiFi status
nmcli device status
# If not connected, connect to WiFi
sudo nmcli device wifi connect "YourWiFiSSID" password "YourPassword"
# Verify connection
ping -c 3 8.8.8.8
Method 2: Disable WiFi Monitor Service
If you don't need the WiFi setup feature:
# After installation, disable the WiFi monitor service
sudo systemctl stop ledmatrix-wifi-monitor
sudo systemctl disable ledmatrix-wifi-monitor
Method 3: Configure WiFi Monitor to Not Auto-Enable AP
Edit the WiFi monitor configuration to prevent automatic AP mode:
# Edit the WiFi config (if it exists)
nano /home/devpi/LEDMatrix/config/wifi_config.json
# Or modify the WiFi monitor daemon behavior
# (requires code changes to wifi_monitor_daemon.py)
Verification Steps
After regaining SSH access, verify your installation:
cd /home/devpi/LEDMatrix
./scripts/verify_installation.sh
This script will check:
- Systemd services status
- Python dependencies
- Configuration files
- File permissions
- Web interface availability
- Network connectivity
Quick Reference Commands
# Check WiFi status
nmcli device status
nmcli device wifi list
# Check AP mode status
sudo systemctl status hostapd
sudo systemctl status dnsmasq
# Check WiFi monitor service
sudo systemctl status ledmatrix-wifi-monitor
# View WiFi monitor logs
sudo journalctl -u ledmatrix-wifi-monitor -f
# Connect to WiFi
sudo nmcli device wifi connect "SSID" password "password"
# Disable AP mode
sudo systemctl stop hostapd dnsmasq
# Restart network services
sudo systemctl restart NetworkManager
Web Interface Access
Even if SSH is unavailable, you can access the web interface:
- Via AP Mode: Connect to LEDMatrix-Setup network and visit
http://192.168.4.1:5000 - Via WiFi: If WiFi is connected, visit
http://<pi-ip-address>:5000 - Via Ethernet: Visit
http://<pi-ip-address>:5000
The web interface allows you to:
- Configure WiFi connections
- Enable/disable AP mode
- Check service status
- View logs
- Manage the LED Matrix display
Summary
SSH becomes unavailable because — two unrelated causes, and they need different responses:
AP mode (most common):
- WiFi monitor service enables AP mode when WiFi disconnects
- AP mode switches WiFi from client to access point mode
- Pi loses connection to your original network
Memory exhaustion (low-memory boards):
- The board runs out of memory, so
sshdcannot fork a session process - The connection is accepted and closed before any banner
- Ping still answers and the web UI still responds, so it looks healthy
- The panel is usually dark and the service cannot restart
- Only a power cycle clears this — there is no remote recovery, because every remote route needs a new process
- Prevention and tuning: LOW_MEMORY_BOARDS.md
To regain SSH:
- Connect to LEDMatrix-Setup AP network (password:
ledmatrix123) - SSH to
192.168.4.1 - Disable AP mode and reconnect to your WiFi network
- Or disable the WiFi monitor service if not needed
To prevent future issues:
- Ensure WiFi is connected before installation
- Or disable WiFi monitor service if you don't need AP mode feature