Files
LEDMatrix/docs/SSH_UNAVAILABLE_AFTER_INSTALL.md
T
ChuckBuildsandClaude Opus 5 34a7414275 fix: address review findings on the low-memory work
Nine CodeRabbit findings, five in code.

**Health state (the one that matters).** The non-dict guard did not cover a
dict missing fields the callers index directly, which is the shape actually
seen in the wild: a record carrying only circuit_state produced
`plugin clock-simple operation failed: 'circuit_state'` about fifty times a
minute with the panel frozen. The record is now completed against the
defaults per field rather than trusted or discarded wholesale. Per field
matters: a first pass rejected any incomplete record outright, which reset a
tripped breaker and real failure counts to healthy because one optional
field was absent -- an existing test caught it. Values of the wrong type
(a counter persisted as a string, an unknown circuit_state) fall back
individually, valid neighbours survive, and newer fields the schema has
grown since (degraded, degraded_reason) are carried through untouched.

**Cache ceiling.** MemoryCache.set() accepted entries without bound between
cleanup sweeps, which run every 300s by default, so a burst could take the
cache far past max_size -- the unbounded growth the limit exists to stop.
Eviction now runs under the same lock on every write, sharing one helper
with the periodic sweep so the two cannot drift.

**Installer, cgroups.** Only cgroup_enable=memory was checked, so a board
carrying that without cgroup_memory=1 reported success and got no change,
leaving MemoryMax= inert. Each parameter is now checked and appended
independently; verified against all four combinations, single line preserved.

**Installer, journald.** Persistence was inferred from /var/log/journal being
non-empty, which proves neither Storage=persistent nor a size cap -- the
directory survives a switch back to volatile. The effective configuration is
read instead (systemd-analyze cat-config, falling back to the conf files),
and an explicitly configured SystemMaxUse is preserved rather than
overwritten. Verified across volatile, persistent-without-cap,
persistent-with-user-cap, cap-without-storage, and commented-only configs.

**Dependency extras.** _extras_are_satisfied stopped at one level, so a
gated dependency that itself requests an extra (requests[socks]) passed on
the base distribution's version while the extra's own dependency was
missing, and pip was skipped. It now recurses, with a visited
(distribution, extras) set so a cycle terminates.

Docs: both kernel command-line paths documented (the installer falls back to
/boot/cmdline.txt), daemon-reload and restart added after the systemd
override example, memory exhaustion added to the SSH summary with its
power-cycle-only recovery, and a language on the fenced block for MD040.

Tests: five for the health-state repair including the exact wild shape and
that record_failure/record_success no longer raise against it, and one for
the cache ceiling. Both mutation-checked. Full suite 2927 passed, with the
one pre-existing tmpfs failure that also fails on main.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01STMbQE4YctTacQXfbYqKuW
2026-08-19 09:28:02 -04:00

6.8 KiB

SSH Unavailable After Installation - Troubleshooting Guide

Why SSH Becomes Unavailable

After running first_time_install.sh, SSH may become unavailable for the following reasons:

1. WiFi Monitor Service Enables AP Mode

Primary Cause: The WiFi monitor service (ledmatrix-wifi-monitor) automatically enables Access Point (AP) mode when it detects that the Raspberry Pi is not connected to WiFi. When AP mode is active:

  • The Pi creates its own WiFi network: LEDMatrix-Setup (password: ledmatrix123)
  • The Pi's WiFi interface (wlan0) switches from client mode to AP mode
  • This disconnects the Pi from your original WiFi network
  • SSH becomes unavailable because the Pi is no longer on your network

2. Network Configuration Changes

The installation script:

  • Installs and configures hostapd (Access Point daemon)
  • Installs and configures dnsmasq (DHCP server for AP mode)
  • These services can interfere with normal WiFi client mode

3. The Board Ran Out of Memory

On a 512MB or 1GB board, memory exhaustion stops sshd being able to fork a session process. The connection is accepted and then closed immediately, before any banner:

kex_exchange_identification: Connection closed by remote host

The giveaway is that the board is otherwise healthy — ping is clean and the web UI still responds — but nothing that needs to start a new process works, and the panel is usually dark. Only a power cycle clears it. See LOW_MEMORY_BOARDS.md.

4. Reboot After Installation

If the script reboots the Pi (which it recommends), network services may restart in a different state, potentially triggering AP mode.

How to Regain SSH Access

  1. Find the AP Network:

    • Look for a WiFi network named LEDMatrix-Setup on your phone/computer
    • Default password: ledmatrix123
  2. Connect to the AP:

    • Connect your device to the LEDMatrix-Setup network
    • The Pi will have IP address: 192.168.4.1
  3. SSH via AP Mode:

    ssh devpi@192.168.4.1
    
  4. Disable AP Mode and Reconnect to WiFi: Once connected via SSH:

    # Check WiFi status
    nmcli device status
    
    # Disable AP mode manually
    sudo systemctl stop hostapd
    sudo systemctl stop dnsmasq
    
    # Connect to your WiFi network (replace with your SSID and password)
    sudo nmcli device wifi connect "YourWiFiSSID" password "YourPassword"
    
    # Or use the web interface at http://192.168.4.1:5000
    # Navigate to WiFi tab and connect to your network
    

Option 2: Disable WiFi Monitor Service Temporarily

If you have physical access to the Pi or can connect via AP mode:

# Stop the WiFi monitor service
sudo systemctl stop ledmatrix-wifi-monitor

# Disable it from starting on boot (optional)
sudo systemctl disable ledmatrix-wifi-monitor

# Stop AP mode services
sudo systemctl stop hostapd
sudo systemctl stop dnsmasq

# Reconnect to your WiFi network
sudo nmcli device wifi connect "YourWiFiSSID" password "YourPassword"

Option 3: Use Ethernet Connection

If your Pi is connected via Ethernet:

  • SSH should remain available via Ethernet even if WiFi is in AP mode
  • Connect via: ssh devpi@<pi-ip-address>

Option 4: Physical Access

If you have physical access to the Pi:

  1. Connect a keyboard and monitor
  2. Log in locally
  3. Follow Option 2 to disable AP mode and reconnect to WiFi

Preventing SSH Loss in the Future

Method 1: Configure WiFi Before Installation

Before running first_time_install.sh, ensure WiFi is properly configured and connected:

# Check WiFi status
nmcli device status

# If not connected, connect to WiFi
sudo nmcli device wifi connect "YourWiFiSSID" password "YourPassword"

# Verify connection
ping -c 3 8.8.8.8

Method 2: Disable WiFi Monitor Service

If you don't need the WiFi setup feature:

# After installation, disable the WiFi monitor service
sudo systemctl stop ledmatrix-wifi-monitor
sudo systemctl disable ledmatrix-wifi-monitor

Method 3: Configure WiFi Monitor to Not Auto-Enable AP

Edit the WiFi monitor configuration to prevent automatic AP mode:

# Edit the WiFi config (if it exists)
nano /home/devpi/LEDMatrix/config/wifi_config.json

# Or modify the WiFi monitor daemon behavior
# (requires code changes to wifi_monitor_daemon.py)

Verification Steps

After regaining SSH access, verify your installation:

cd /home/devpi/LEDMatrix
./scripts/verify_installation.sh

This script will check:

  • Systemd services status
  • Python dependencies
  • Configuration files
  • File permissions
  • Web interface availability
  • Network connectivity

Quick Reference Commands

# Check WiFi status
nmcli device status
nmcli device wifi list

# Check AP mode status
sudo systemctl status hostapd
sudo systemctl status dnsmasq

# Check WiFi monitor service
sudo systemctl status ledmatrix-wifi-monitor

# View WiFi monitor logs
sudo journalctl -u ledmatrix-wifi-monitor -f

# Connect to WiFi
sudo nmcli device wifi connect "SSID" password "password"

# Disable AP mode
sudo systemctl stop hostapd dnsmasq

# Restart network services
sudo systemctl restart NetworkManager

Web Interface Access

Even if SSH is unavailable, you can access the web interface:

  1. Via AP Mode: Connect to LEDMatrix-Setup network and visit http://192.168.4.1:5000
  2. Via WiFi: If WiFi is connected, visit http://<pi-ip-address>:5000
  3. Via Ethernet: Visit http://<pi-ip-address>:5000

The web interface allows you to:

  • Configure WiFi connections
  • Enable/disable AP mode
  • Check service status
  • View logs
  • Manage the LED Matrix display

Summary

SSH becomes unavailable because — two unrelated causes, and they need different responses:

AP mode (most common):

  • WiFi monitor service enables AP mode when WiFi disconnects
  • AP mode switches WiFi from client to access point mode
  • Pi loses connection to your original network

Memory exhaustion (low-memory boards):

  • The board runs out of memory, so sshd cannot fork a session process
  • The connection is accepted and closed before any banner
  • Ping still answers and the web UI still responds, so it looks healthy
  • The panel is usually dark and the service cannot restart
  • Only a power cycle clears this — there is no remote recovery, because every remote route needs a new process
  • Prevention and tuning: LOW_MEMORY_BOARDS.md

To regain SSH:

  1. Connect to LEDMatrix-Setup AP network (password: ledmatrix123)
  2. SSH to 192.168.4.1
  3. Disable AP mode and reconnect to your WiFi network
  4. Or disable the WiFi monitor service if not needed

To prevent future issues:

  • Ensure WiFi is connected before installation
  • Or disable WiFi monitor service if you don't need AP mode feature