# Advanced Features Guide

This guide covers advanced LEDMatrix features for users and developers, including Vegas scroll mode, on-demand display, cache management, background services, and permission management.

---

## 1. Vegas Scroll Mode

### Overview

Vegas scroll mode displays content from multiple plugins in a continuous horizontal scroll, similar to news tickers seen in Las Vegas casinos. Plugins contribute content segments that flow across the display in a seamless ticker-style presentation.

### How a Plugin Takes Part

Each plugin has a *Vegas participation*:

**`scroll` (the default):**
- The plugin's content scrolls by with everyone else's
- Best for news-ticker style content: scores, headlines, prices, the time

**`pause`:**
- The scroll stops when the plugin's turn comes round
- The plugin draws the whole panel for its display duration, then the
  scroll resumes
- Best for content that needs to be read in full, or alerts

**`exclude`:**
- The plugin is left out of Vegas mode

A plugin declares its default; set `vegas_participation` in a plugin's
config to override it (see [Per-Plugin Configuration](#per-plugin-configuration)).
Older documentation also describes a *fixed segment* mode; Vegas never
implemented one, and it has always behaved exactly like `scroll`.

### Configuration

Enable Vegas mode in `config/config.json`:

```json
{
  "display": {
    "vegas_scroll": {
      "enabled": true,
      "scroll_speed": 50,
      "separator_width": 32,
      "plugin_order": ["clock", "weather", "sports"],
      "excluded_plugins": ["debug_plugin"],
      "target_fps": 125,
      "buffer_ahead": 2
    }
  }
}
```

Vegas mode can also be configured entirely from the web UI — the
**Display** tab has a Vegas Scroll Mode section (enable toggle, scroll
speed, separator width, dynamic duration, and more), so hand-editing
JSON is optional.

**Configuration Options:**

| Setting | Default | Description |
|---------|---------|-------------|
| `enabled` | `false` | Enable Vegas scroll mode |
| `scroll_speed` | `50` | Pixels per second scroll speed |
| `separator_width` | `32` | Width between plugin segments (pixels) |
| `plugin_order` | `[]` | Plugin display order (empty = auto) |
| `excluded_plugins` | `[]` | Plugins to exclude from Vegas mode |
| `target_fps` | `125` | Target frame rate |
| `buffer_ahead` | `2` | Number of plugins buffered ahead |

This table is a subset — `display.vegas_scroll` supports 30 keys in
total. See the full list in
[CONFIG_REFERENCE.md](CONFIG_REFERENCE.md#displayvegas_scroll--continuous-scroll-mode).

### Live Content in the Ticker

By default, live content **preempts** Vegas mode: while any plugin reports
live priority, the display controller refuses to run the ticker and shows
that plugin's full-screen display instead. You get a big readable scoreboard,
but the marquee stops entirely for the duration of the game.

Set `live_in_ticker` to keep the ticker running and let live content take
**extra turns inside it** instead:

```json
"vegas_scroll": {
  "live_in_ticker": true,
  "live_weight": 3,
  "favorite_live_weight": 5
}
```

#### Why weights exist

The rotation is otherwise a strict round robin — every plugin appears exactly
once per cycle. With a dozen plugins enabled, a live score comes round once a
lap and can be minutes old by the time you see it. A weight of *N* gives a
plugin *N* slots per cycle.

The slots are placed by **Smooth Weighted Round-Robin**, the same scheduler
the sports plugins use internally to rotate their own games. The important
property is that repeats are *spread through the cycle* rather than clumped:
three appearances in a row followed by a long silence would be worse than not
boosting at all.

Twelve plugins, with a favorite's baseball game and an ordinary live hockey
game (`live_weight: 3`, `favorite_live_weight: 5`):

```
baseball > hockey > weather  > clock  > baseball
stocks   > news   > flights  > baseball > hockey
calendar > f1     > music    > baseball > tides
birds    > hockey > baseball
```

18 slots for 12 plugins. Baseball appears 5 times, hockey 3, everything else
once, and no plugin ever appears twice in a row — **including across the seam**
where the cycle loops back on itself. Smooth Weighted Round-Robin schedules the
heaviest item first and usually last as well, so the strip would otherwise show
it twice running at exactly the one join a within-cycle check cannot see. The
trailing repeat is moved into the widest remaining gap. Where a double is
unavoidable — a plugin holding most of the slots has to neighbour itself — the
schedule is left as it is.

#### Where the weight comes from

For each plugin in the rotation, in order:

1. **The plugin's own answer.** If it implements
   `get_vegas_priority_weight()` and returns a number, that wins. This is the
   only route for favorite-team awareness — the core can see *that* a game is
   live, but not *whose*, so a scoreboard has to say so itself.
2. **The core's default.** When the plugin returns `None` (the base-class
   default), a plugin where both `has_live_priority()` and `has_live_content()`
   are true gets `live_weight`.
3. **Everything else** gets 1.

Because of step 2, **existing plugins need no changes** — any scoreboard with
`live_priority` enabled already gets extra turns. Step 1 is opt-in, for
plugins that want to distinguish a favorite's game from any other live game.

Weights are clamped to 1–10. A weight of 1 is no boost; a weight below 1 would
drop the plugin from the rotation entirely, which is never what is meant.

#### Things worth knowing

- **Weights are per plugin, not per game.** A scoreboard showing four live
  games still occupies one slot at a time, rotating its own games within that
  slot using its own `favorite_live_boost`. This controls how often the
  *plugin* comes round.
- **The ticker is zero-sum.** Giving baseball 5 slots does not make the cycle
  faster; it makes the cycle *longer* and everything else proportionally
  rarer. If you want live scores sooner in wall-clock terms, pair this with a
  smaller `plugins_per_cycle`.
- **Frequency is not freshness.** Each appearance redraws from the plugin's
  current data (`refresh_updated_plugins()` drops cached content when a
  plugin's data changes), but how current that data is depends on the
  plugin's own `live_update_interval`. Showing a stale score five times a lap
  is no better than showing it once.
- **Everything still appears.** A boost never starves another plugin out of
  the cycle; low-weight plugins keep their single slot.

### Per-Plugin Configuration

Override Vegas behavior for specific plugins:

```json
{
  "my_plugin": {
    "enabled": true,
    "vegas_participation": "pause",
    "display_duration": 10
  }
}
```

**Per-Plugin Options:**

| Setting | Values | Description |
|---------|--------|-------------|
| `vegas_participation` | `scroll`, `pause`, `exclude` | How this plugin takes part: its content scrolls by, the scroll pauses for its turn and shows it full screen, or it is left out. Unset uses the plugin's own default |
| `display_duration` | seconds | How long a `pause` plugin holds the screen |
| `vegas_width_pct` | 10–100 | Width of this plugin's card, as a percentage of the panel |
| `vegas_overflow` | `rotate`, `truncate` | What to do when its content is wider than its allowance |
| `vegas_max_width_screens` | number of screens | The widest its card may be |

These are core-owned settings (see
[PLUGIN_CONFIG_CORE_PROPERTIES.md](PLUGIN_CONFIG_CORE_PROPERTIES.md)): every
plugin accepts them whether or not its own schema lists them. Set them in
the plugin's section of config.json, in the web UI's **Config Editor**
tab.

Some plugins also offer a `vegas_mode` setting of their own (`scroll`,
`fixed` or `static`). It still works — `static` pauses, the other two scroll
— but `vegas_participation` takes precedence, and `fixed` has never done
anything different from `scroll`. The old `vegas_panel_count` setting never
had an effect and is deprecated (removed in 3.9.0).

### Plugin Integration (Developer Guide)

All of these have defaults in
[`BasePlugin`](../src/plugin_system/base_plugin.py); override only what you
need. The reference is
[PLUGIN_API_REFERENCE.md](PLUGIN_API_REFERENCE.md#vegas-scroll-hooks).

**1. Implement Content Method:**

```python
def get_vegas_content(self):
    # Return a PIL Image, a list of Images, or None.
    # A single image is one block; a list becomes one item per image.
    return [self._render_game(game) for game in self.games]
```

If it returns `None` (the default), Vegas falls back to the plugin's
`scroll_helper` image, then to capturing `display()` output
(`PluginAdapter.get_content()` in
[`src/vegas_mode/plugin_adapter.py`](../src/vegas_mode/plugin_adapter.py)).

**2. Declare how the plugin takes part:**

Most plugins need nothing: the default is `scroll`. A plugin that should
pause the scroll, or stay out of Vegas, says so in `manifest.json`:

```json
{
  "vegas_participation": "pause"
}
```

The user's own `vegas_participation` setting overrides the manifest. When
the answer depends on state, override the method instead:

```python
def get_vegas_participation(self):
    # 'scroll' | 'pause' | 'exclude'
    return 'pause' if self._alert_is_live() else 'scroll'
```

A plugin written for an older core that declares nothing keeps its
behaviour: `get_vegas_display_mode()` returning `VegasDisplayMode.STATIC`
pauses, `get_vegas_content_type()` returning `'none'` excludes, and
everything else scrolls. `get_supported_vegas_modes()`,
`get_vegas_segment_width()` and the SCROLL / FIXED_SEGMENT distinction are
deprecated (removed in 3.9.0): Vegas never read them.

### Content Rendering Guidelines

**Image Dimensions:**
- **Height:** Must match display height (typically 32 pixels)
- **Width:** Any width for `scroll` (recommended 64-512 pixels);
  `get_vegas_render_width()` is the width Vegas would like, and it narrows
  `display_manager` to match while it asks. A `pause` plugin draws the
  whole panel in `display()`.

**Color Mode:**
- Use RGB color mode
- 24-bit color (8 bits per channel)

**Performance Tips:**
1. **Cache rendered images** - Render in `update()`, not in `get_vegas_content()`
2. **Keep images small** - Larger images use more memory
3. **Pre-render on update** - Don't create images on-demand
4. **Reuse images** - Return same image if content unchanged

### Example Integration

Complete example for a weather plugin:

```python
class WeatherPlugin(BasePlugin):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.vegas_image = None

    def update(self):
        """Update data and pre-render Vegas image"""
        # Fetch weather data
        weather_data = self.fetch_weather()

        # Pre-render Vegas image
        self.vegas_image = self._render_vegas_content(weather_data)

    def _render_vegas_content(self, data):
        """Render weather content for Vegas mode"""
        img = Image.new('RGB', (384, 32))
        draw = ImageDraw.Draw(img)

        # Draw temperature
        draw.text((10, 0), f"{data['temp']}°F", fill=(255, 255, 255))

        # Draw condition
        draw.text((100, 0), data['condition'], fill=(200, 200, 200))

        # Draw icon
        icon = Image.open(f"assets/{data['icon']}.png")
        img.paste(icon, (250, 0))

        return img

    def get_vegas_content(self):
        """Return cached Vegas image"""
        return self.vegas_image
```

It scrolls, the default participation, so it declares nothing else.

### System Architecture

Vegas mode consists of four core components working together to provide smooth 125 FPS continuous scrolling:

#### Component Overview

```
┌─────────────────────────────────────────────────────────────┐
│                   VegasModeCoordinator                      │
│  Main orchestrator - manages lifecycle and coordination     │
└───────┬──────────────────┬──────────────────┬──────────────┘
        │                  │                  │
        ▼                  ▼                  ▼
┌───────────────┐  ┌──────────────┐  ┌─────────────────┐
│ PluginAdapter │  │StreamManager │  │ RenderPipeline  │
│               │  │              │  │                 │
│ Converts      │─▶│ Manages      │─▶│ 125 FPS render  │
│ plugin content│  │ content      │  │ Double-buffered │
│ to images     │  │ stream with  │  │ Smooth scroll   │
│               │  │ 1-2 ahead    │  │                 │
└───────────────┘  │ buffering    │  └─────────────────┘
                   └──────────────┘
```

#### 1. VegasModeCoordinator

**Responsibilities:**
- Initialize and coordinate all Vegas mode components
- Manage the high-FPS render loop (target: 125 FPS)
- Handle live priority interruptions
- Process config updates during runtime
- Provide status and control interface

**Key Features:**
- Thread-safe state management
- Config hot-reload support
- Live priority integration
- Interrupt checking for yielding control back to display controller
- Static pause handling (pauses scroll when content fully visible)

**Main Loop:**
1. Check for interrupts (live priority, on-demand, config updates)
2. If static pause active, wait for duration
3. Otherwise, delegate to render pipeline for frame rendering
4. Sleep to maintain target FPS

#### 2. StreamManager

**Responsibilities:**
- Manage plugin content streaming with look-ahead buffering
- Coordinate with PluginAdapter to fetch plugin content
- Handle plugin ordering and exclusions
- Optimize content generation timing

**Buffering Strategy:**
- **Buffer Ahead:** 1-2 panels (configurable)
- **Just-in-Time Generation:** Fetch content only when needed
- **Memory Efficient:** Only keep necessary content in memory

**Content Flow:**
1. Determine which plugins should appear in stream
2. Respect `plugin_order` configuration (or use default order)
3. Exclude plugins in `excluded_plugins` list
4. Request content from each plugin via PluginAdapter
5. Compose into continuous stream with separators

**Key Methods:**
- `get_next_segment()` - Returns the next buffered `ContentSegment` (or `None`)
- `take_next_group(count=None, offscreen_only=False)` - Hands over the next
  slice of the rotation as `(plugin_id, images)` groups
- `get_grouped_content_for_composition()` - Buffered images grouped by plugin
- `mark_plugin_updated(plugin_id)` / `process_updates()` - Refresh one
  plugin's segment in place when its data changes
- `refresh()` - Re-read the plugin list and config
- `advance_cycle()` - Clear the active buffer when a scroll cycle completes

(`src/vegas_mode/stream_manager.py`)

#### 3. PluginAdapter

**Responsibilities:**
- Convert plugin content to scrollable images
- Fetch the content of `scroll` plugins (a `pause` plugin is drawn by
  its own `display()` when the scroll pauses; see StreamManager)
- Manage fallback for plugins without Vegas support
- Cache plugin content for performance

**Plugin Integration:**
1. **Check for Vegas support:**
   - Calls `get_vegas_content()` if available
   - Falls back to `display()` method if not

2. **Participation** is decided by the StreamManager, not here
   (`resolve_vegas_participation()` in
   [`base_plugin.py`](../src/plugin_system/base_plugin.py)): `exclude`
   plugins never reach the adapter, and `pause` plugins are not fetched.

**Fallback Behavior:**
- If plugin doesn't implement Vegas methods:
  - Calls plugin's `display()` method
  - Captures rendered display as static image
  - Scrolls it by as one block
- Ensures all plugins work in Vegas mode without explicit support

#### 4. RenderPipeline

**Responsibilities:**
- High-performance 125 FPS rendering
- Double-buffered composition for smooth scrolling
- Scroll position management
- Frame rate control

**Rendering Process:**
1. **Fetch Stream Content:** Get current stream from StreamManager
2. **Extract Viewport:** Calculate which portion of stream is visible
3. **Compose Frame:** Create frame with visible content
4. **Double Buffer:** Render to off-screen buffer
5. **Display:** Swap buffer to display
6. **Advance:** Update scroll position based on speed and elapsed time

**Performance Optimizations:**
- **Double Buffering:** Eliminates flicker
- **Viewport Extraction:** Only processes visible region
- **Frame Rate Control:** Precise timing to maintain 125 FPS
- **Pre-rendered Content:** Plugins pre-render during update()

**Scroll Speed Calculation:** motion is by elapsed time; `target_fps` paces
the render loop, not the speed.
```python
# frame_based_scrolling: false
scroll_position += scroll_speed * elapsed_time            # scroll_speed in px/s
# frame_based_scrolling: true (the default) -- not stepping, just a clamp
applied = clamp(scroll_speed * scroll_delay, 0.1, 5) / scroll_delay
scroll_position += applied * elapsed_time
```

#### Component Interactions

**Initialization Flow:**
```
1. VegasModeCoordinator created
2. Coordinator creates PluginAdapter
3. Coordinator creates StreamManager (with PluginAdapter)
4. Coordinator creates RenderPipeline (with StreamManager)
5. All components initialized and ready
```

**Render Loop Flow:**
```
1. Coordinator starts render loop
2. Check for interrupts (live priority, on-demand)
3. RenderPipeline.render_frame():
   a. Request current stream from StreamManager
   b. StreamManager uses PluginAdapter to get plugin content
   c. PluginAdapter calls plugin Vegas methods or fallback
   d. Stream content returned to RenderPipeline
   e. RenderPipeline extracts viewport and renders
4. Update scroll position
5. Sleep to maintain target FPS
6. Repeat from step 2
```

**Config Update Flow:**
```
1. Config change detected by Coordinator
2. Set _pending_config_update flag
3. On next render loop iteration:
   a. Pause rendering
   b. Update VegasModeConfig
   c. Notify StreamManager of config change
   d. StreamManager refreshes stream
   e. Resume rendering
```

#### Thread Safety

All components use thread-safe patterns:
- **Coordinator:** Uses `threading.Lock` for state management
- **StreamManager:** Thread-safe content access
- **RenderPipeline:** Atomic frame composition
- **PluginAdapter:** Stateless operations (except caching)

#### Performance Characteristics

**Frame Rate:**
- Target: 125 FPS
- Actual: 100-125 FPS (depends on content complexity)
- Render time budget: ~8ms per frame

**Memory Usage:**
- Stream buffer: ~2-3 panels ahead
- Plugin content: Cached in plugin's `update()` method
- Double buffer: 2x display size

**CPU Usage:**
- Light load: 5-10% (simple content)
- Heavy load: 15-25% (complex content, many plugins)
- Optimized with numpy for pixel operations

### Fallback Behavior

If a plugin doesn't implement Vegas methods:
- System calls the plugin's `display()` method
- Captures the rendered display as a static image
- Scrolls it by as one block

This ensures all plugins work in Vegas mode, even without explicit support.

---

## 2. On-Demand Display

### Overview

On-demand display allows users to manually trigger specific plugins to show immediately on the LED matrix, overriding the normal rotation. This is useful for:
- Quick checks (weather, scores, time)
- Pinning important information
- Testing plugins during development
- Showing specific content to visitors

### Priority Hierarchy

On-demand display has the highest priority:

```
Priority Order (highest to lowest):
1. On-Demand Display (manual trigger)
2. Live Priority (games in progress)
3. Normal Rotation
```

When on-demand expires or is cleared, the display returns to the next highest priority (live priority or normal rotation).

### Web Interface Controls

Each installed plugin has its own tab in the second nav row of the web
UI. Inside the plugin's tab, scroll to **On-Demand Controls**:

- **Run On-Demand** — triggers the plugin immediately, even if it's
  disabled in the rotation
- **Stop On-Demand** — clears on-demand and returns to the normal
  rotation

The display service must be running. The status banner at the top of
the plugin tab shows the active on-demand plugin, mode, and remaining
time when something is active.

### REST API Reference

The API is mounted at `/api/v3` (the `api_v3` blueprint, registered in
`web_interface/app.py`). Full details: [REST_API_REFERENCE.md](REST_API_REFERENCE.md#display-control).

#### Start On-Demand Display

```bash
POST /api/v3/display/on-demand/start

# Body:
{
  "plugin_id": "weather",
  "duration": 30,        # Optional: seconds (0 = indefinite, null = default)
  "pinned": false        # Optional: keep until manually cleared
}

# Examples:
# 30-second preview
curl -X POST http://localhost:5000/api/v3/display/on-demand/start \
  -H "Content-Type: application/json" \
  -d '{"plugin_id": "weather", "duration": 30}'

# Pin indefinitely
curl -X POST http://localhost:5000/api/v3/display/on-demand/start \
  -H "Content-Type: application/json" \
  -d '{"plugin_id": "hockey-scoreboard", "pinned": true}'
```

#### Stop On-Demand Display

```bash
POST /api/v3/display/on-demand/stop

# Body:
{
  "stop_service": false  # Optional: also stop display service
}

# Examples:
# Clear on-demand
curl -X POST http://localhost:5000/api/v3/display/on-demand/stop

# Stop service too
curl -X POST http://localhost:5000/api/v3/display/on-demand/stop \
  -H "Content-Type: application/json" \
  -d '{"stop_service": true}'
```

#### Get On-Demand Status

```bash
GET /api/v3/display/on-demand/status

# Example:
curl http://localhost:5000/api/v3/display/on-demand/status

# Response:
{
  "status": "success",
  "data": {
    "state": {
      "active": true,
      "plugin_id": "weather",
      "mode": "weather",
      "duration": 30,
      "pinned": false,
      "status": "running",
      "last_updated": 1234567890.1
    },
    "service": {"active": true, "returncode": 0, "stdout": "active", "stderr": ""}
  }
}
```

When nothing is running on demand, `data.state` is
`{"active": false, "status": "idle", "last_updated": null}`.

> There is no public Python on-demand API. The display controller's
> on-demand machinery is internal — drive it through the REST endpoints
> above (or the web UI buttons). The API handlers
> (`start_on_demand_display()` / `stop_on_demand_display()` in
> `web_interface/blueprints/api_v3/display.py`) write a request into the cache
> manager under the `display_on_demand_request` key, which
> `DisplayController._poll_on_demand_requests()`
> (`src/display_controller.py`) picks up. A separate
> `display_on_demand_config` key is used by the controller itself
> during activation (`_activate_on_demand()`) to track what's
> currently running, and is cleared by `_clear_on_demand()`.

### Duration Modes

| Duration | Pinned | Behavior |
|----------|--------|----------|
| `None` | `false` | Use plugin's default duration, auto-clear when expires |
| `0` | `false` | Indefinite, clears manually or on error |
| `> 0` | `false` | Timed display, auto-clear after N seconds |
| Any | `true` | Pin until manually cleared (ignores duration) |

### Use Case Examples

**Quick check (30-second preview):**
```bash
curl -X POST http://localhost:5000/api/v3/display/on-demand/start \
  -H "Content-Type: application/json" \
  -d '{"plugin_id": "ledmatrix-weather", "duration": 30}'
```

**Pin important information:**
```bash
curl -X POST http://localhost:5000/api/v3/display/on-demand/start \
  -H "Content-Type: application/json" \
  -d '{"plugin_id": "hockey-scoreboard", "pinned": true}'
# ... later ...
curl -X POST http://localhost:5000/api/v3/display/on-demand/stop
```

**Indefinite display:**
```bash
curl -X POST http://localhost:5000/api/v3/display/on-demand/start \
  -H "Content-Type: application/json" \
  -d '{"plugin_id": "text-display", "duration": 0}'
```

**Testing a plugin during development:** the same call works, or just
click **Run On-Demand** in the plugin's tab.

### Best Practices

**For Users:**
1. Use timed display as default (prevents forgetting to clear)
2. Pin only when necessary
3. Clear when done to return to normal rotation

**For Developers:**
1. Validate plugin ID exists before calling
2. Provide visual feedback in UI (loading state, status updates)
3. Handle concurrent requests gracefully
4. Log on-demand activations for debugging

### Security Considerations

**Authentication:**
- Add authentication to API endpoints
- Restrict on-demand to authorized users

**Rate Limiting:**
- Prevent abuse from rapid requests
- Implement cooldown between activations

**Input Validation:**
- Sanitize plugin IDs
- Validate duration values
- Check plugin exists before activation

---

## 3. On-Demand Cache Management

### Overview

On-demand display uses cache keys (managed by `src/cache_manager.py` —
file-based, not Redis) to coordinate state between the web interface
and the display controller across service restarts. Understanding these
keys helps troubleshoot stuck states.

### Cache Keys

**1. display_on_demand_request** (TTL: 1 hour)
```json
{
  "request_id": "uuid-string",
  "action": "start|stop",
  "plugin_id": "plugin-name",
  "mode": "mode-name",
  "duration": 30.0,
  "pinned": true,
  "timestamp": 1234567890.123
}
```
**Purpose:** Communication from web interface to display controller
**When Set:** API endpoint receives request
**Auto-Cleared:** After processing or 1 hour TTL

**2. display_on_demand_config** (No TTL)
```json
{
  "mode": "mode-name",
  "duration": 30.0,
  "pinned": true
}
```
**Purpose:** Persistent configuration for display controller
**When Set:** Controller processes start request
**Auto-Cleared:** When on-demand stops

**3. display_on_demand_state** (Continuously updated)
```json
{
  "active": true,
  "mode": "mode-name",
  "remaining": 25.5,
  "pinned": true,
  "status": "active|idle|restarting|error"
}
```
**Purpose:** Real-time state for web interface status card
**When Set:** Every display loop iteration
**Auto-Cleared:** Never (continuously updated)

**4. display_on_demand_processed_id** (TTL: 1 hour)
```text
"uuid-string-of-last-processed-request"
```
**Purpose:** Prevents duplicate request processing
**When Set:** After processing request
**Auto-Cleared:** After 1 hour TTL

### When Manual Clearing is Needed

**Scenario 1: Stuck in On-Demand State**
- Symptom: Display stays on one plugin, won't return to rotation
- Clear: `config`, `state`, `request`

**Scenario 2: Mode Switching Issues**
- Symptom: Can't change to different plugin
- Clear: `request`, `processed_id`, `state`

**Scenario 3: On-Demand Not Activating**
- Symptom: Button click does nothing
- Clear: `processed_id`, `request`

**Scenario 4: After Service Crash**
- Symptom: Strange behavior after crash/restart
- Clear: All four keys

### Manual Recovery Procedures

**Via Web Interface (Recommended):**
1. Open the **Cache** tab in the web UI
2. Find the `display_on_demand_*` entries
3. Delete them
4. Restart display: `sudo systemctl restart ledmatrix`

**Via Command Line:**

The cache is stored as JSON files under one of:

- `/var/cache/ledmatrix/` (preferred when the service has permission)
- `~/.ledmatrix_cache/`
- `/opt/ledmatrix/cache/`
- `$TMPDIR/ledmatrix_cache/` (fallback)

```bash
# Find the cache dir actually in use
journalctl -u ledmatrix | grep -i "cache directory" | tail -1

# Clear all on-demand keys (replace path with the one above)
rm /var/cache/ledmatrix/display_on_demand_*

# Restart service
sudo systemctl restart ledmatrix
```

**Via Python:**
```python
from src.cache_manager import CacheManager

cache = CacheManager()
cache.clear_cache('display_on_demand_config')
cache.clear_cache('display_on_demand_state')
cache.clear_cache('display_on_demand_request')
cache.clear_cache('display_on_demand_processed_id')
```

> `CacheManager` also has a `delete(key)` method — a thin wrapper over
> `clear_cache(key)` — so `cache.delete('display_on_demand_config')`
> works equally well.

### Cache Impact on Running Service

**IMPORTANT:** Clearing cache keys does NOT immediately affect the running controller in memory.

**To fully reset:**
1. Stop the service: `sudo systemctl stop ledmatrix`
2. Clear cache keys (web UI Cache tab or `rm` from the cache directory)
3. Clear systemd environment: `sudo systemctl daemon-reload`
4. Start the service: `sudo systemctl start ledmatrix`

### Automatic Cleanup

The display controller automatically handles cleanup:
- **Config key**: Cleared when on-demand stops
- **State key**: Updated every display loop iteration
- **Request key**: Expires after 1 hour TTL (or after processing)
- **Processed ID**: Expires after 1 hour TTL

---

## 4. Background Data Service

### Overview

The Background Data Service enables non-blocking data fetching through background threading. This prevents the main display loop from freezing during slow API requests, maintaining smooth display rotation.

### Benefits

**Performance:**
- Display loop never freezes during API calls
- Immediate response with cached/partial data
- Complete data loads in background

**User Experience:**
- No "frozen" display during data updates
- Smooth transitions between plugins
- Faster perceived load times

**Architecture:**
```
Cache Check → Background Fetch → Partial Data → Completion → Cache
    (0.1s)         (async)            (<1s)         (10-30s)    (cache)
```

### Configuration

Core does not read a `background_service` config block: the service itself
(`src/background_data_service.py`) is a process-wide singleton, and its
worker count is whatever the first caller of `get_background_service()`
passes. The sports scoreboard plugins read their own
`background_service` settings and pass them to it, so the exact keys and
where they sit (top level or per league) are defined by each plugin's
`config_schema.json`. A typical block looks like:

```json
{
  "football-scoreboard": {
    "enabled": true,
    "background_service": {
      "enabled": true,
      "max_workers": 3,
      "request_timeout": 30,
      "max_retries": 3,
      "priority": 2
    }
  }
}
```

**Configuration Options:**

| Setting | Default | Description |
|---------|---------|-------------|
| `enabled` | plugin-defined | Use the background service for this plugin's fetches |
| `max_workers` | `3` | Max concurrent background tasks |
| `request_timeout` | `30` | Timeout per API request (seconds) |
| `max_retries` | `3` | Retry attempts on failure |
| `priority` | `1` | Stored on each request (higher number = higher priority, per `FetchRequest`), but the service runs requests in submission order; it does not reorder by priority |

### Performance Impact

**First Request (Cache Empty):**
- Returns partial data: < 1 second
- Background completes: 10-30 seconds
- Subsequent requests use cache: < 0.1 seconds

**Subsequent Requests (Cache Hit):**
- Returns immediately: < 0.1 seconds
- Background refresh (if stale): async, no blocking

### Plugins using the background service

The background data service is used by all of the sports scoreboard
plugins (football, hockey, baseball/MLB, basketball, soccer, lacrosse,
F1, UFC), the odds ticker, and the leaderboard plugin. Each plugin reads
its own `background_service` block (under its own config namespace); check
that plugin's `config_schema.json` for the keys it accepts.

### Error Handling & Fallback

**Automatic Retry:**
- Exponential backoff (1s, 2s, 4s, 8s, ...)
- Maximum retry attempts configurable
- Logs all retry attempts

**Fallback Behavior:**
- If background service disabled: reverts to synchronous fetching
- If background fetch fails: returns cached data
- If no cache: returns empty/error state

### Testing

```bash
# Check logs for background operations
sudo journalctl -u ledmatrix -f | grep "background"
```

### Monitoring

**View Statistics:**
```python
from src.background_data_service import get_background_service
from src.cache_manager import CacheManager

service = get_background_service(CacheManager())
stats = service.get_statistics()
print(f"Active: {stats['active_requests']}")
print(f"Completed: {stats['completed_requests']}")
print(f"Failed: {stats['failed_requests']}")
```

Other keys: `total_requests`, `cached_hits`, `cache_misses`,
`average_fetch_time`, `completed_requests_count` (results currently held in
memory) — see `BackgroundDataService.get_statistics()` in
[`src/background_data_service.py`](../src/background_data_service.py).

**Enable Debug Logging:**
```python
import logging
logging.getLogger('src.background_data_service').setLevel(logging.DEBUG)
```

---

## 5. Permission Management

Ownership, modes, sudo rules and the repair scripts are listed in
[PERMISSIONS.md](PERMISSIONS.md). This section covers the helpers code uses
to keep files shareable.

### Overview

LEDMatrix uses a dual-user architecture: the display service runs as root (hardware access), while the web interface runs as a non-privileged user. Centralized permission management ensures both can access necessary files.

### Why It Matters

**Problem:**
- Root service creates files with root ownership
- Web user cannot read/write those files
- Results in `PermissionError` exceptions

**Solution:**
- Set group ownership to shared group
- Grant group write permissions
- Use setgid bit for automatic inheritance

### Permission Utilities

```python
from src.common.permission_utils import (
    ensure_directory_permissions,
    ensure_file_permissions,
    get_config_file_mode,
    get_assets_file_mode,
    get_assets_dir_mode,
    get_plugin_file_mode,
    get_cache_dir_mode
)

# Create directory with correct permissions
ensure_directory_permissions(Path("assets/sports"), get_assets_dir_mode())

# Set file permissions after writing
# (get_config_file_mode requires the file path — secrets files get a
# stricter mode than the main config)
config_path = Path("config/config.json")
ensure_file_permissions(config_path, get_config_file_mode(config_path))
```

### When to Use Utilities

**Use permission utilities when:**
1. Creating new directories
2. Writing configuration files
3. Downloading/creating asset files (logos, fonts)
4. Creating plugin files
5. Writing cache files

**Don't use for:**
1. Reading files (permissions don't change)
2. Temporary files in `/tmp`
3. Files in already-managed directories (if parent has setgid)

### Permission Standards

**File Permissions:**

| File Type | Mode | Octal | Description |
|-----------|------|-------|-------------|
| Config (main) | `rw-r--r--` | `0o644` | Owner write, all read |
| Config (secrets) | `rw-r-----` | `0o640` | Owner write, group read |
| Assets | `rw-rw-r--` | `0o664` | Owner/group write, all read |
| Plugins | `rw-rw-r--` | `0o664` | Owner/group write, all read |
| Cache files | `rw-rw----` | `0o660` | Owner/group write, no world access (`_CACHE_FILE_MODE` in `src/cache/disk_cache.py`) |

**Directory Permissions:**

| Directory Type | Mode | Octal | Description |
|----------------|------|-------|-------------|
| All directories | `rwxrwsr-x` | `0o2775` | With setgid bit for inheritance |

**Note:** The `s` in `rwxrwsr-x` is the setgid bit (2000), which makes new files inherit the directory's group ownership.

### Common Patterns

**Pattern 1: Creating Config Directory**
```python
from pathlib import Path
from src.common.permission_utils import ensure_directory_permissions, get_config_dir_mode

config_dir = Path("config/plugins")
ensure_directory_permissions(config_dir, get_config_dir_mode())
```

**Pattern 2: Saving Config File**
```python
from src.common.permission_utils import ensure_file_permissions, get_config_file_mode

config_path = Path("config/config.json")
with open(config_path, 'w') as f:
    json.dump(data, f)
ensure_file_permissions(config_path, get_config_file_mode(config_path))
```

**Pattern 3: Downloading Logo**
```python
from src.common.permission_utils import ensure_directory_permissions, ensure_file_permissions
from src.common.permission_utils import get_assets_dir_mode, get_assets_file_mode

logo_path = Path("assets/sports/nhl/logo.png")
ensure_directory_permissions(logo_path.parent, get_assets_dir_mode())
# ... download and save logo ...
ensure_file_permissions(logo_path, get_assets_file_mode())
```

**Pattern 4: Creating Plugin File**
```python
from src.common.permission_utils import ensure_file_permissions, get_plugin_file_mode

plugin_file = Path("plugins/my-plugin/data.json")
with open(plugin_file, 'w') as f:
    json.dump(data, f)
ensure_file_permissions(plugin_file, get_plugin_file_mode())
```

**Pattern 5: Cache Directory Setup**
```python
from src.common.permission_utils import ensure_directory_permissions, get_cache_dir_mode

cache_dir = Path("cache/plugin-name")
ensure_directory_permissions(cache_dir, get_cache_dir_mode())
```

### Integration with Core Utilities

These core utilities **already handle permissions** - you don't need to call permission utilities when using them:

- **ConfigManager** - Handles config file permissions
- **CacheManager** - Handles cache file permissions
- **LogoHelper** - Handles logo file permissions
- **PluginManager** - Handles plugin file permissions

### Manual Fixes

[PERMISSIONS.md](PERMISSIONS.md) lists who owns what on an installed system,
the expected modes, and which `scripts/fix_perms/` script to run as which
user. In short:

- `fix_assets_permissions.sh`, `fix_cache_permissions.sh` and
  `fix_plugin_permissions.sh` are run with `sudo`.
- `fix_web_permissions.sh` is run as the web interface user, without
  `sudo` (it refuses to run as root and calls `sudo` itself where needed).
  It resets project file ownership for that user, then makes the two
  helper scripts the web user may run as root (`safe_plugin_rm.sh`,
  `safe_pip_install.sh`) root-owned again and restores `config_secrets.json`
  to its owner, the `ledmatrix` group and mode `640`. It does not write
  sudoers rules; `scripts/install/configure_web_sudo.sh` does that.

Do not `chmod` the whole `config/` directory: `config_secrets.json` must stay
`640`.

---

## Related Documentation

- [PLUGIN_DEVELOPMENT_GUIDE.md](PLUGIN_DEVELOPMENT_GUIDE.md) - Creating plugins with Vegas/on-demand support
- [WEB_INTERFACE_GUIDE.md](WEB_INTERFACE_GUIDE.md) - Using on-demand controls in web UI
- [PLUGIN_API_REFERENCE.md](PLUGIN_API_REFERENCE.md) - Complete API documentation
- [DEVELOPMENT.md](DEVELOPMENT.md) - Development environment and testing