diff --git a/.cursor/README.md b/.cursor/README.md deleted file mode 100644 index 0b6d621f..00000000 --- a/.cursor/README.md +++ /dev/null @@ -1,145 +0,0 @@ -# Cursor Helper Files for LEDMatrix Plugin Development - -This directory contains Cursor-specific helper files to assist with plugin development in the LEDMatrix project. - -## Files Overview - -### `.cursorrules` -Comprehensive rules file that Cursor uses to understand plugin development patterns, best practices, and workflows. This file is automatically loaded by Cursor and helps guide AI-assisted development. - -### `plugins_guide.md` -Detailed guide covering: -- Plugin system overview -- Creating new plugins -- Running plugins (emulator and hardware) -- Loading and configuring plugins -- Development workflow -- Testing strategies -- Troubleshooting - -### `plugin_templates/` -Template files for quick plugin creation: -- `manifest.json.template` - Plugin metadata template -- `manager.py.template` - Plugin class template -- `config_schema.json.template` - Configuration schema template -- `README.md.template` - Plugin documentation template -- `requirements.txt.template` - Dependencies template -- `QUICK_START.md` - Quick start guide for using templates - -## Quick Reference - -### Creating a New Plugin - -1. **Using templates** (recommended): -```bash -# See QUICK_START.md in plugin_templates/ -cd plugins -mkdir my-plugin -cd my-plugin -cp ../../.cursor/plugin_templates/*.template . -# Edit files, replacing PLUGIN_ID and other placeholders -``` - -2. **Using dev_plugin_setup.sh**: -```bash -# Link from GitHub -./scripts/dev/dev_plugin_setup.sh link-github my-plugin - -# Link local repo -./scripts/dev/dev_plugin_setup.sh link my-plugin /path/to/repo -``` - -### Running the Display - -```bash -# Emulator mode (development, no hardware required) -python3 run.py --emulator -# (equivalent: EMULATOR=true python3 run.py) - -# Hardware (production, requires the rpi-rgb-led-matrix submodule built) -python3 run.py - -# As a systemd service -sudo systemctl start ledmatrix - -# Dev preview server (renders plugins to a browser without running run.py) -python3 scripts/dev_server.py # then open http://localhost:5001 -``` - -The `-e`/`--emulator` CLI flag is defined in `run.py:19-20` and -sets `os.environ["EMULATOR"] = "true"` before any display imports, -which `src/display_manager.py:2` then reads to switch between the -hardware and emulator backends. - -### Managing Plugins - -```bash -# List plugins -./scripts/dev/dev_plugin_setup.sh list - -# Check status -./scripts/dev/dev_plugin_setup.sh status - -# Update plugin(s) -./scripts/dev/dev_plugin_setup.sh update [plugin-name] - -# Unlink plugin -./scripts/dev/dev_plugin_setup.sh unlink -``` - -## Using These Files with Cursor - -### `.cursorrules` -Cursor automatically reads this file to understand: -- Plugin structure and requirements -- Development workflows -- Best practices -- Common patterns -- API reference - -When asking Cursor to help with plugins, it will use this context to provide better assistance. - -### Plugin Templates -Use templates when creating new plugins: -1. Copy templates from `.cursor/plugin_templates/` -2. Replace placeholders (PLUGIN_ID, PluginClassName, etc.) -3. Customize for your plugin's needs -4. Follow the guide in `plugins_guide.md` - -### Documentation -Refer to `plugins_guide.md` for: -- Detailed explanations -- Troubleshooting steps -- Best practices -- Examples and patterns - -## Plugin Development Workflow - -1. **Plan**: Determine plugin functionality and requirements -2. **Create**: Use templates or dev_plugin_setup.sh to create plugin structure -3. **Develop**: Implement plugin logic following BasePlugin interface -4. **Test**: Test with emulator first, then on hardware -5. **Configure**: Add plugin config to config/config.json -6. **Iterate**: Refine based on testing and feedback - -## Resources - -- **Plugin System**: `src/plugin_system/` -- **Base Plugin**: `src/plugin_system/base_plugin.py` -- **Plugin Manager**: `src/plugin_system/plugin_manager.py` -- **Example Plugins**: see the - [`ledmatrix-plugins`](https://github.com/ChuckBuilds/ledmatrix-plugins) - repo for canonical sources (e.g. `plugins/hockey-scoreboard/`, - `plugins/football-scoreboard/`). Installed plugins land in - `plugin-repos/` (default) or `plugins/` (dev fallback). -- **Architecture Docs**: `docs/PLUGIN_ARCHITECTURE_SPEC.md` -- **Development Setup**: `scripts/dev/dev_plugin_setup.sh` - -## Getting Help - -1. Check `plugins_guide.md` for detailed documentation -2. Review `.cursorrules` for development patterns -3. Look at existing plugins for examples -4. Check logs for error messages -5. Review plugin system code in `src/plugin_system/` - diff --git a/.cursor/plugin_templates/QUICK_START.md b/.cursor/plugin_templates/QUICK_START.md deleted file mode 100644 index 9a9f9591..00000000 --- a/.cursor/plugin_templates/QUICK_START.md +++ /dev/null @@ -1,247 +0,0 @@ -# Quick Start: Creating a New Plugin - -This guide will help you create a new plugin using the templates in `.cursor/plugin_templates/`. - -## Step 1: Create Plugin Directory - -```bash -cd /path/to/LEDMatrix -mkdir -p plugins/my-plugin -cd plugins/my-plugin -``` - -## Step 2: Copy Templates - -```bash -# Copy all template files -cp ../../.cursor/plugin_templates/manifest.json.template ./manifest.json -cp ../../.cursor/plugin_templates/manager.py.template ./manager.py -cp ../../.cursor/plugin_templates/config_schema.json.template ./config_schema.json -cp ../../.cursor/plugin_templates/README.md.template ./README.md -cp ../../.cursor/plugin_templates/requirements.txt.template ./requirements.txt -``` - -## Step 3: Customize Files - -### manifest.json - -Replace placeholders: -- `PLUGIN_ID` → `my-plugin` (lowercase, use hyphens) -- `Plugin Name` → Your plugin's display name -- `PluginClassName` → `MyPlugin` (PascalCase) -- Update description, author, homepage, etc. - -### manager.py - -Replace placeholders: -- `PluginClassName` → `MyPlugin` (must match manifest) -- Implement `_fetch_data()` method -- Implement `_render_content()` method -- Add any custom validation in `validate_config()` - -### config_schema.json - -Customize: -- Update description -- Add/remove configuration properties -- Set default values -- Add validation rules - -### README.md - -Replace placeholders: -- `PLUGIN_ID` → `my-plugin` -- `Plugin Name` → Your plugin's name -- Fill in features, installation, configuration sections - -### requirements.txt - -Add your plugin's dependencies: -```txt -requests>=2.28.0 -pillow>=9.0.0 -``` - -## Step 4: Enable Plugin - -Edit `config/config.json`: - -```json -{ - "my-plugin": { - "enabled": true, - "display_duration": 15 - } -} -``` - -## Step 5: Test Plugin - -### Test with Emulator - -```bash -cd /path/to/LEDMatrix -python run.py --emulator -``` - -### Check Plugin Loading - -Look for logs like: -``` -[INFO] Discovered 1 plugin(s) -[INFO] Loaded plugin: my-plugin v1.0.0 -[INFO] Added plugin mode: my-plugin -``` - -### Test Plugin Display - -The plugin should appear in the display rotation. Check logs for any errors. - -## Step 6: Develop and Iterate - -1. Edit `manager.py` to implement your plugin logic -2. Test with emulator: `python run.py --emulator` -3. Check logs for errors -4. Iterate until working correctly - -## Step 7: Test on Hardware (Optional) - -When ready, test on Raspberry Pi: - -```bash -# Deploy to Pi -rsync -avz plugins/my-plugin/ pi@raspberrypi:/path/to/LEDMatrix/plugins/my-plugin/ - -# Or if using git -ssh pi@raspberrypi "cd /path/to/LEDMatrix/plugins/my-plugin && git pull" - -# Restart service -ssh pi@raspberrypi "sudo systemctl restart ledmatrix" -``` - -## Common Customizations - -### Adding API Integration - -1. Add API key to `config_schema.json`: -```json -{ - "api_key": { - "type": "string", - "description": "API key for service" - } -} -``` - -2. Implement API call in `_fetch_data()`: -```python -import requests - -def _fetch_data(self): - response = requests.get( - "https://api.example.com/data", - headers={"Authorization": f"Bearer {self.api_key}"} - ) - return response.json() -``` - -3. Store API key in `config/config_secrets.json`: -```json -{ - "my-plugin": { - "api_key": "your-secret-key" - } -} -``` - -### Adding Image Rendering - -There is no `draw_image()` helper on `DisplayManager`. To render an -image, paste it directly onto the underlying PIL `Image` -(`display_manager.image`) and then call `update_display()`: - -```python -def _render_content(self): - # Load and paste image onto the display canvas - image = Image.open("assets/logo.png").convert("RGB") - self.display_manager.image.paste(image, (0, 0)) - - # Draw text overlay - self.display_manager.draw_text( - "Text", - x=10, y=20, - color=(255, 255, 255) - ) - - self.display_manager.update_display() -``` - -For transparency, paste with a mask: - -```python -icon = Image.open("assets/icon.png").convert("RGBA") -self.display_manager.image.paste(icon, (5, 5), icon) -``` - - -### Adding Live Priority - -1. Enable in config: -```json -{ - "my-plugin": { - "live_priority": true - } -} -``` - -2. Implement `has_live_content()`: -```python -def has_live_content(self) -> bool: - return self.data and self.data.get("is_live", False) -``` - -3. Override `get_live_modes()` if needed: -```python -def get_live_modes(self) -> list: - return ["my_plugin_live_mode"] -``` - -## Troubleshooting - -### Plugin Not Loading - -- Check `manifest.json` syntax (must be valid JSON) -- Verify `entry_point` file exists -- Ensure `class_name` matches class name in manager.py -- Check for import errors in logs - -### Configuration Errors - -- Validate config against `config_schema.json` -- Check required fields are present -- Verify data types match schema - -### Display Issues - -- Check display dimensions: `display_manager.width`, `display_manager.height` -- Verify coordinates are within bounds -- Ensure `update_display()` is called -- Test with emulator first - -## Next Steps - -- Review existing plugins for patterns: - - `plugins/hockey-scoreboard/` - Sports scoreboard example - - `plugins/ledmatrix-music/` - Real-time data example - - `plugins/ledmatrix-stocks/` - Data display example - -- Read full documentation: - - `.cursor/plugins_guide.md` - Comprehensive guide - - `docs/PLUGIN_ARCHITECTURE_SPEC.md` - Architecture details - - `.cursorrules` - Development rules - -- Check plugin system code: - - `src/plugin_system/base_plugin.py` - Base class - - `src/plugin_system/plugin_manager.py` - Plugin manager - diff --git a/.cursor/plugin_templates/README.md.template b/.cursor/plugin_templates/README.md.template deleted file mode 100644 index ba26fd1b..00000000 --- a/.cursor/plugin_templates/README.md.template +++ /dev/null @@ -1,156 +0,0 @@ -# Plugin Name - -Brief description of what this plugin does. - -## Features - -- Feature 1 -- Feature 2 -- Feature 3 - -## Installation - -1. Link the plugin to your LEDMatrix installation: - -```bash -cd /path/to/LEDMatrix -./scripts/dev/dev_plugin_setup.sh link-github PLUGIN_ID -``` - -Or for local development: - -```bash -./scripts/dev/dev_plugin_setup.sh link PLUGIN_ID /path/to/plugin/repo -``` - -2. Install dependencies: - -```bash -pip install -r plugins/PLUGIN_ID/requirements.txt -``` - -3. Configure the plugin in `config/config.json`: - -```json -{ - "PLUGIN_ID": { - "enabled": true, - "display_duration": 15 - } -} -``` - -**Note:** API keys and other sensitive credentials must be stored in `config/config_secrets.json`, not in `config/config.json`. - -4. Store API keys in `config/config_secrets.json`: - -```json -{ - "PLUGIN_ID": { - "api_key": "your-secret-api-key" - } -} -``` - -## Configuration - -### Required Settings - -- `enabled` (boolean): Enable or disable the plugin -- `api_key` (string): API key for external service (if required) - -### Optional Settings - -- `display_duration` (number): How long to display this plugin (default: 15 seconds) -- `refresh_interval` (integer): How often to refresh data in seconds (default: 60) -- `live_priority` (boolean): Enable live priority takeover (default: false) - -## Display Modes - -This plugin provides the following display modes: - -- `PLUGIN_ID`: Main display mode - -## API Requirements - -This plugin requires: - -- **API Name**: Description of API requirements - - URL: https://api.example.com - - Rate Limit: X requests per minute - - Authentication: API key required - -## Development - -### Running Tests - -```bash -cd plugins/PLUGIN_ID -python test_PLUGIN_ID.py -``` - -### Testing with Emulator - -```bash -cd /path/to/LEDMatrix -python run.py --emulator -``` - -### Debugging - -Enable debug logging in `config/config.json`: - -```json -{ - "logging": { - "level": "DEBUG" - } -} -``` - -Check logs: - -```bash -# On Raspberry Pi (if running as service) -journalctl -u ledmatrix -f - -# Direct execution -python run.py -``` - -## Troubleshooting - -### Plugin Not Loading - -1. Check that `manifest.json` exists and is valid -2. Verify `entry_point` file exists -3. Check that `class_name` matches the class in manager.py -4. Review logs for import errors - -### Configuration Errors - -1. Validate config against `config_schema.json` -2. Check required fields are present -3. Verify data types match schema - -### API Errors - -1. Verify API key is correct -2. Check API rate limits -3. Review network connectivity -4. Check API service status - -## License - -[License information] - -## Author - -Your Name - -## Links - -- GitHub: https://github.com/username/ledmatrix-PLUGIN_ID -- Documentation: [Link to docs] -- Issues: https://github.com/username/ledmatrix-PLUGIN_ID/issues - diff --git a/.cursor/plugin_templates/config_schema.json.template b/.cursor/plugin_templates/config_schema.json.template deleted file mode 100644 index 8d512038..00000000 --- a/.cursor/plugin_templates/config_schema.json.template +++ /dev/null @@ -1,44 +0,0 @@ -{ - "$schema": "http://json-schema.org/draft-07/schema#", - "type": "object", - "title": "Plugin Configuration Schema", - "description": "Configuration schema for Plugin Name", - "properties": { - "enabled": { - "type": "boolean", - "default": true, - "description": "Enable or disable this plugin" - }, - "display_duration": { - "type": "number", - "default": 15, - "minimum": 1, - "maximum": 300, - "description": "How long to display this plugin in seconds" - }, - "live_priority": { - "type": "boolean", - "default": false, - "description": "Enable live priority takeover when plugin has live content" - }, - "refresh_interval": { - "type": "integer", - "default": 60, - "minimum": 1, - "description": "How often to refresh data in seconds" - }, - "api_key": { - "type": "string", - "description": "API key for external service (store in config_secrets.json)", - "default": "" - }, - "custom_setting": { - "type": "string", - "description": "Example custom setting - replace with your plugin's settings", - "default": "default_value" - } - }, - "required": ["enabled"], - "additionalProperties": false -} - diff --git a/.cursor/plugin_templates/manager.py.template b/.cursor/plugin_templates/manager.py.template deleted file mode 100644 index 6afe3e9b..00000000 --- a/.cursor/plugin_templates/manager.py.template +++ /dev/null @@ -1,226 +0,0 @@ -""" -Plugin Name - -Brief description of what this plugin does. - -API Version: 1.0.0 -""" - -from src.plugin_system.base_plugin import BasePlugin -from PIL import Image -from typing import Dict, Any, Optional -import logging -import time - - -class PluginClassName(BasePlugin): - """ - Plugin class that inherits from BasePlugin. - - This plugin demonstrates the basic structure and common patterns - for LEDMatrix plugins. - """ - - def __init__( - self, - plugin_id: str, - config: Dict[str, Any], - display_manager, - cache_manager, - plugin_manager, - ): - """Initialize the plugin.""" - super().__init__(plugin_id, config, display_manager, cache_manager, plugin_manager) - - # Initialize plugin-specific data - self.data = None - self.last_update_time = None - - # Load configuration values - self.api_key = config.get("api_key", "") - self.refresh_interval = config.get("refresh_interval", 60) - - self.logger.info(f"Plugin {plugin_id} initialized") - - def update(self) -> None: - """ - Fetch/update data for this plugin. - - This method is called periodically based on update_interval - specified in the manifest. Use cache_manager to avoid - excessive API calls. - """ - cache_key = f"{self.plugin_id}_data" - - # Check cache first - cached = self.cache_manager.get(cache_key, max_age=self.refresh_interval) - if cached: - self.data = cached - self.logger.debug("Using cached data") - return - - try: - # Fetch new data - self.data = self._fetch_data() - - # Cache the data - self.cache_manager.set(cache_key, self.data, ttl=self.refresh_interval) - self.last_update_time = time.time() - - self.logger.info("Data updated successfully") - - except Exception as e: - self.logger.error(f"Failed to update data: {e}") - # Use cached data if available, even if expired - # Use a very large max_age (1 year) to effectively bypass expiration for fallback - expired_cached = self.cache_manager.get(cache_key, max_age=31536000) - if expired_cached: - self.data = expired_cached - self.logger.warning("Using expired cache due to update failure") - - def display(self, force_clear: bool = False) -> None: - """ - Render this plugin's display. - - Args: - force_clear: If True, clear display before rendering - """ - if force_clear: - self.display_manager.clear() - - # Check if we have data to display - if not self.data: - self._display_error("No data available") - return - - try: - # Render plugin content - self._render_content() - - # Update the display - self.display_manager.update_display() - - except Exception as e: - self.logger.error(f"Display error: {e}") - self._display_error("Display error") - - def _fetch_data(self) -> Dict[str, Any]: - """ - Fetch data from external source. - - Returns: - Dictionary containing fetched data - """ - # TODO: Implement data fetching logic - # Example: - # import requests - # response = requests.get("https://api.example.com/data", - # headers={"Authorization": f"Bearer {self.api_key}"}) - # return response.json() - - # Placeholder - return { - "message": "Hello, World!", - "timestamp": time.time() - } - - def _render_content(self) -> None: - """Render the plugin content on the display.""" - # Get display dimensions - width = self.display_manager.width - height = self.display_manager.height - - # Example: Draw text - text = self.data.get("message", "No data") - x = 5 - y = height // 2 - - self.display_manager.draw_text( - text, - x=x, - y=y, - color=(255, 255, 255) # White - ) - - # Example: Draw image - # if hasattr(self, 'logo_image'): - # self.display_manager.draw_image( - # self.logo_image, - # x=0, - # y=0 - # ) - - def _display_error(self, message: str) -> None: - """Display an error message.""" - self.display_manager.clear() - width = self.display_manager.width - height = self.display_manager.height - - self.display_manager.draw_text( - message, - x=5, - y=height // 2, - color=(255, 0, 0) # Red - ) - self.display_manager.update_display() - - def validate_config(self) -> bool: - """ - Validate plugin configuration. - - Returns: - True if config is valid, False otherwise - """ - # Call parent validation first - if not super().validate_config(): - return False - - # Add custom validation - # Example: Check for required API key - # if self.config.get("require_api_key", True): - # if not self.api_key: - # self.logger.error("API key is required but not provided") - # return False - - return True - - def has_live_content(self) -> bool: - """ - Check if plugin has live content to display. - - Override this method to enable live priority features. - - Returns: - True if plugin has live content, False otherwise - """ - # Example: Check if there's live data - # return self.data and self.data.get("is_live", False) - return False - - def get_info(self) -> Dict[str, Any]: - """ - Return plugin info for display in web UI. - - Returns: - Dictionary with plugin information - """ - info = super().get_info() - - # Add plugin-specific info - info.update({ - "data_available": self.data is not None, - "last_update": self.last_update_time, - # Add more info as needed - }) - - return info - - def cleanup(self) -> None: - """Cleanup resources when plugin is unloaded.""" - # Clean up any resources (threads, connections, etc.) - # Example: - # if hasattr(self, 'api_client'): - # self.api_client.close() - - super().cleanup() - diff --git a/.cursor/plugin_templates/manifest.json.template b/.cursor/plugin_templates/manifest.json.template deleted file mode 100644 index ade5fa58..00000000 --- a/.cursor/plugin_templates/manifest.json.template +++ /dev/null @@ -1,55 +0,0 @@ -{ - "id": "PLUGIN_ID", - "name": "Plugin Name", - "version": "1.0.0", - "author": "Your Name", - "description": "Brief description of what this plugin does", - "homepage": "https://github.com/username/ledmatrix-PLUGIN_ID", - "entry_point": "manager.py", - "class_name": "PluginClassName", - "category": "custom", - "tags": ["custom", "example"], - "icon": "fas fa-icon-name", - "compatible_versions": [">=2.0.0"], - "min_ledmatrix_version": "2.0.0", - "max_ledmatrix_version": "3.0.0", - "requires": { - "python": ">=3.9", - "display_size": { - "min_width": 64, - "min_height": 32 - } - }, - "config_schema": "config_schema.json", - "assets": { - "logos": "Optional: Description of asset requirements" - }, - "update_interval": 60, - "default_duration": 15, - "display_modes": [ - "PLUGIN_ID" - ], - "api_requirements": [ - { - "name": "API Name", - "required": false, - "description": "Description of API requirements", - "url": "https://api.example.com", - "rate_limit": "Rate limit information" - } - ], - "download_url_template": "https://github.com/username/ledmatrix-PLUGIN_ID/archive/refs/tags/v{version}.zip", - "versions": [ - { - "released": "2025-01-01", - "version": "1.0.0", - "ledmatrix_min_version": "2.0.0" - } - ], - "last_updated": "2025-01-01", - "stars": 0, - "downloads": 0, - "verified": false, - "screenshot": "" -} - diff --git a/.cursor/plugin_templates/requirements.txt.template b/.cursor/plugin_templates/requirements.txt.template deleted file mode 100644 index a631c4f0..00000000 --- a/.cursor/plugin_templates/requirements.txt.template +++ /dev/null @@ -1,13 +0,0 @@ -# Plugin Dependencies -# Add your plugin's Python dependencies here - -# Example dependencies (uncomment and modify as needed): -# requests>=2.28.0 -# pillow>=9.0.0 -# python-dateutil>=2.8.0 - -# Note: Core LEDMatrix dependencies are already available: -# - PIL/Pillow (for image handling) -# - Core plugin system classes -# - Display manager, cache manager, config manager - diff --git a/.cursor/plugin_templates/test_manager.py.template b/.cursor/plugin_templates/test_manager.py.template deleted file mode 100644 index 0cff3425..00000000 --- a/.cursor/plugin_templates/test_manager.py.template +++ /dev/null @@ -1,136 +0,0 @@ -""" -Test file for Plugin Name plugin. - -This file provides example unit tests for your plugin. -Run tests with: python -m pytest test_manager.py -Or: python test_manager.py -""" - -import unittest -import sys -from pathlib import Path - -# Add project root to path -PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent -if str(PROJECT_ROOT) not in sys.path: - sys.path.insert(0, str(PROJECT_ROOT)) - -from src.plugin_system.testing import PluginTestCase -from manager import PluginClassName - - -class TestPluginClassName(PluginTestCase): - """Test cases for PluginClassName plugin.""" - - def setUp(self): - """Set up test fixtures.""" - super().setUp() - - # Update plugin_id to match the plugin being tested - self.plugin_id = 'PLUGIN_ID' - - # Create plugin instance - self.plugin = self.create_plugin_instance( - PluginClassName, - plugin_id='PLUGIN_ID', - config=self.get_mock_config() - ) - - def test_plugin_initialization(self): - """Test that plugin initializes correctly.""" - self.assert_plugin_initialized(self.plugin) - self.assertTrue(self.plugin.enabled) - - def test_config_validation(self): - """Test configuration validation.""" - # Valid config should pass - self.assertTrue(self.plugin.validate_config()) - - # Test with invalid config if applicable - # invalid_config = self.get_mock_config(enabled='not-a-boolean') - # invalid_plugin = self.create_plugin_instance( - # PluginClassName, - # config=invalid_config - # ) - # self.assertFalse(invalid_plugin.validate_config()) - - def test_update_method(self): - """Test the update() method.""" - # Reset mocks - self.cache_manager.reset() - - # Call update - self.plugin.update() - - # Assertions - # Example: Check that cache was used - # self.assert_cache_get('PLUGIN_ID_data') - - # Example: Check that data was fetched and cached - # self.assert_cache_set('PLUGIN_ID_data') - - def test_display_method(self): - """Test the display() method.""" - # Ensure plugin has data (call update first if needed) - # self.plugin.update() - - # Call display - self.plugin.display(force_clear=True) - - # Assertions - self.assert_display_cleared() - self.assert_display_updated() - - # Example: Check that text was drawn - # self.assert_text_drawn("Expected Text") - - # Example: Check that image was drawn - # self.assert_image_drawn() - - def test_display_without_data(self): - """Test display() behavior when no data is available.""" - # Clear any cached data - self.cache_manager.reset() - - # Call display - self.plugin.display() - - # Should handle gracefully (no exceptions) - # May show error message or fallback content - self.assert_display_updated() - - def test_get_display_duration(self): - """Test display duration configuration.""" - duration = self.plugin.get_display_duration() - self.assertIsInstance(duration, (int, float)) - self.assertGreater(duration, 0) - - # Test with custom duration - custom_config = self.get_mock_config(display_duration=30.0) - custom_plugin = self.create_plugin_instance( - PluginClassName, - config=custom_config - ) - self.assertEqual(custom_plugin.get_display_duration(), 30.0) - - def test_enable_disable(self): - """Test plugin enable/disable functionality.""" - self.assertTrue(self.plugin.enabled) - - self.plugin.on_disable() - self.assertFalse(self.plugin.enabled) - - self.plugin.on_enable() - self.assertTrue(self.plugin.enabled) - - def test_config_change(self): - """Test configuration change handling.""" - new_config = self.get_mock_config(display_duration=20.0) - self.plugin.on_config_change(new_config) - - self.assertEqual(self.plugin.config.get('display_duration'), 20.0) - - -if __name__ == '__main__': - unittest.main() - diff --git a/.cursor/plugins_guide.md b/.cursor/plugins_guide.md deleted file mode 100644 index 62ba63b6..00000000 --- a/.cursor/plugins_guide.md +++ /dev/null @@ -1,751 +0,0 @@ -# LEDMatrix Plugin Development Guide - -This guide provides comprehensive instructions for creating, running, and loading plugins in the LEDMatrix project. - -## Table of Contents - -1. [Plugin System Overview](#plugin-system-overview) -2. [Creating a New Plugin](#creating-a-new-plugin) -3. [Running Plugins](#running-plugins) -4. [Loading Plugins](#loading-plugins) -5. [Plugin Development Workflow](#plugin-development-workflow) -6. [Testing Plugins](#testing-plugins) -7. [Troubleshooting](#troubleshooting) - ---- - -## Plugin System Overview - -The LEDMatrix project uses a plugin-based architecture where all display functionality (except core calendar) is implemented as plugins. Plugins are dynamically loaded from the `plugins/` directory and integrated into the display rotation. - -### Plugin Architecture - -``` -LEDMatrix Core -├── Plugin Manager (discovers, loads, manages plugins) -├── Display Manager (handles LED matrix rendering) -├── Cache Manager (data persistence) -├── Config Manager (configuration management) -└── Plugins/ (plugin directory) - ├── plugin-1/ - ├── plugin-2/ - └── ... -``` - -### Plugin Lifecycle - -1. **Discovery**: PluginManager scans `plugins/` for directories with `manifest.json` -2. **Loading**: Plugin module is imported and class is instantiated -3. **Configuration**: Plugin config is loaded from `config/config.json` -4. **Validation**: `validate_config()` is called to verify configuration -5. **Registration**: Plugin is added to available display modes -6. **Execution**: `update()` is called periodically, `display()` is called during rotation - ---- - -## Creating a New Plugin - -### Method 1: Using dev_plugin_setup.sh (Recommended) - -This method is best for plugins stored in separate Git repositories. - -#### From GitHub Repository - -```bash -# Link a plugin from GitHub (auto-detects URL) -./scripts/dev/dev_plugin_setup.sh link-github - -# Example: Link hockey-scoreboard plugin -./scripts/dev/dev_plugin_setup.sh link-github hockey-scoreboard - -# With custom URL -./scripts/dev/dev_plugin_setup.sh link-github https://github.com/user/repo.git -``` - -The script will: -- Clone the repository to `~/.ledmatrix-dev-plugins/` (or configured directory) -- Create a symlink in `plugins//` pointing to the cloned repo -- Validate the plugin structure - -#### From Local Repository - -```bash -# Link a local plugin repository -./scripts/dev/dev_plugin_setup.sh link - -# Example: Link a local plugin -./scripts/dev/dev_plugin_setup.sh link my-plugin ../ledmatrix-my-plugin -``` - -### Method 2: Manual Plugin Creation - -1. **Create Plugin Directory** - -```bash -mkdir -p plugins/my-plugin -cd plugins/my-plugin -``` - -2. **Create manifest.json** - -```json -{ - "id": "my-plugin", - "name": "My Plugin", - "version": "1.0.0", - "author": "Your Name", - "description": "Description of what this plugin does", - "entry_point": "manager.py", - "class_name": "MyPlugin", - "category": "custom", - "tags": ["custom", "example"], - "display_modes": ["my_plugin"], - "update_interval": 60, - "default_duration": 15, - "requires": { - "python": ">=3.9" - }, - "config_schema": "config_schema.json" -} -``` - -3. **Create manager.py** - -```python -from src.plugin_system.base_plugin import BasePlugin -from PIL import Image -import logging - -class MyPlugin(BasePlugin): - """My custom plugin implementation.""" - - def update(self): - """Fetch/update data for this plugin.""" - # Fetch data from API, files, etc. - # Use self.cache_manager for caching - cache_key = f"{self.plugin_id}_data" - cached = self.cache_manager.get(cache_key, max_age=3600) - if cached: - self.data = cached - return - - # Fetch new data - self.data = self._fetch_data() - self.cache_manager.set(cache_key, self.data) - - def display(self, force_clear=False): - """Render this plugin's display.""" - if force_clear: - self.display_manager.clear() - - # Render content using display_manager - self.display_manager.draw_text( - "Hello, World!", - x=10, y=15, - color=(255, 255, 255) - ) - - self.display_manager.update_display() - - def _fetch_data(self): - """Fetch data from external source.""" - # Implement your data fetching logic - return {"message": "Hello, World!"} - - def validate_config(self): - """Validate plugin configuration.""" - # Check required config fields - if not super().validate_config(): - return False - - # Add custom validation - required_fields = ['api_key'] # Example - for field in required_fields: - if field not in self.config: - self.logger.error(f"Missing required field: {field}") - return False - - return True -``` - -4. **Create config_schema.json** - -```json -{ - "type": "object", - "properties": { - "enabled": { - "type": "boolean", - "default": true, - "description": "Enable or disable this plugin" - }, - "display_duration": { - "type": "number", - "default": 15, - "minimum": 1, - "description": "How long to display this plugin (seconds)" - }, - "api_key": { - "type": "string", - "description": "API key for external service" - } - }, - "required": ["enabled"] -} -``` - -5. **Create requirements.txt** (if needed) - -``` -requests>=2.28.0 -pillow>=9.0.0 -``` - -6. **Create README.md** - -Document your plugin's functionality, configuration options, and usage. - ---- - -## Running Plugins - -### Development Mode (Emulator) - -Run the LEDMatrix system with emulator for plugin testing: - -```bash -# Using run.py -python run.py --emulator - -# Using emulator script -./run_emulator.sh -``` - -The emulator will: -- Load all enabled plugins -- Display plugin content in a window (simulating LED matrix) -- Show logs for plugin loading and execution -- Allow testing without Raspberry Pi hardware - -### Production Mode (Raspberry Pi) - -Run on actual Raspberry Pi hardware: - -```bash -# Direct execution -python run.py - -# As systemd service -sudo systemctl start ledmatrix -sudo systemctl status ledmatrix -sudo journalctl -u ledmatrix -f # View logs -``` - -### Plugin-Specific Testing - -Test individual plugin loading: - -```python -# test_my_plugin.py -from src.plugin_system.plugin_manager import PluginManager -from src.config_manager import ConfigManager -from src.display_manager import DisplayManager -from src.cache_manager import CacheManager - -# Initialize managers -config_manager = ConfigManager() -config = config_manager.load_config() -display_manager = DisplayManager(config) -cache_manager = CacheManager() - -# Initialize plugin manager -plugin_manager = PluginManager( - plugins_dir="plugins", - config_manager=config_manager, - display_manager=display_manager, - cache_manager=cache_manager -) - -# Discover and load plugin -plugins = plugin_manager.discover_plugins() -print(f"Discovered plugins: {plugins}") - -if "my-plugin" in plugins: - if plugin_manager.load_plugin("my-plugin"): - plugin = plugin_manager.get_plugin("my-plugin") - plugin.update() - plugin.display() - print("Plugin loaded and displayed successfully!") - else: - print("Failed to load plugin") -``` - ---- - -## Loading Plugins - -### Enabling Plugins - -Plugins are enabled/disabled in `config/config.json`: - -```json -{ - "my-plugin": { - "enabled": true, - "display_duration": 15, - "api_key": "your-api-key-here" - } -} -``` - -### Plugin Configuration Structure - -Each plugin has its own section in `config/config.json`: - -```json -{ - "": { - "enabled": true, // Enable/disable plugin - "display_duration": 15, // Display duration in seconds - "live_priority": false, // Enable live priority takeover - "high_performance_transitions": false, // Use 120 FPS transitions - "transition": { // Transition configuration - "type": "redraw", // Transition type - "speed": 2, // Transition speed - "enabled": true // Enable transitions - }, - // ... plugin-specific configuration - } -} -``` - -### Secrets Management - -Store sensitive data (API keys, tokens) in `config/config_secrets.json` -under the same plugin id you use in `config/config.json`: - -```json -{ - "my-plugin": { - "api_key": "secret-api-key-here" - } -} -``` - -At load time, the config manager deep-merges `config_secrets.json` into -the main config (verified at `src/config_manager.py:162-172`). So in -your plugin's code: - -```python -class MyPlugin(BasePlugin): - def __init__(self, plugin_id, config, display_manager, cache_manager, plugin_manager): - super().__init__(plugin_id, config, display_manager, cache_manager, plugin_manager) - self.api_key = config.get("api_key") # already merged from secrets -``` - -There is no separate `config_secrets` reference field — just put the -secret value under the same plugin namespace and read it from the -merged config. - -### Plugin Discovery - -Plugins are automatically discovered when: -- Directory exists in `plugins/` -- Directory contains `manifest.json` -- Manifest has required fields (`id`, `entry_point`, `class_name`) - -Check discovered plugins: - -```bash -# Using dev_plugin_setup.sh -./scripts/dev/dev_plugin_setup.sh list - -# Output shows: -# ✓ plugin-name (symlink) -# → /path/to/repo -# ✓ Git repo is clean (branch: main) -``` - -### Plugin Status - -Check plugin status and git information: - -```bash -./scripts/dev/dev_plugin_setup.sh status - -# Output shows: -# ✓ plugin-name -# Path: /path/to/repo -# Branch: main -# Remote: https://github.com/user/repo.git -# Status: Clean and up to date -``` - ---- - -## Plugin Development Workflow - -### 1. Initial Setup - -```bash -# Create or clone plugin repository -git clone https://github.com/user/ledmatrix-my-plugin.git -cd ledmatrix-my-plugin - -# Link to LEDMatrix project -cd /path/to/LEDMatrix -./scripts/dev/dev_plugin_setup.sh link my-plugin ../ledmatrix-my-plugin -``` - -### 2. Development Cycle - -1. **Edit plugin code** in linked repository -2. **Test with the dev preview server**: - `python3 scripts/dev_server.py` (then open `http://localhost:5001`). - Or run the full display in emulator mode with - `python3 run.py --emulator` (or equivalently - `EMULATOR=true python3 run.py`). The `-e`/`--emulator` CLI flag is - defined in `run.py:19-20` and sets the same `EMULATOR` environment - variable internally. -3. **Check logs** for errors or warnings -4. **Update configuration** in `config/config.json` if needed -5. **Iterate** until plugin works correctly - -### 3. Testing on Hardware - -```bash -# Deploy to Raspberry Pi -rsync -avz plugins/my-plugin/ ledpi@your-pi-ip:/path/to/LEDMatrix/plugins/my-plugin/ - -# Or if using git, pull on Pi -ssh ledpi@your-pi-ip "cd /path/to/LEDMatrix/plugins/my-plugin && git pull" - -# Restart service -ssh ledpi@your-pi-ip "sudo systemctl restart ledmatrix" -``` - -### 4. Updating Plugins - -```bash -# Update single plugin from git -./scripts/dev/dev_plugin_setup.sh update my-plugin - -# Update all linked plugins -./scripts/dev/dev_plugin_setup.sh update -``` - -### 5. Unlinking Plugins - -```bash -# Remove symlink (preserves repository) -./scripts/dev/dev_plugin_setup.sh unlink my-plugin -``` - ---- - -## Testing Plugins - -### Unit Testing - -Create test files in plugin directory: - -```python -# plugins/my-plugin/test_my_plugin.py -import unittest -from unittest.mock import Mock, MagicMock -from manager import MyPlugin - -class TestMyPlugin(unittest.TestCase): - def setUp(self): - self.config = {"enabled": True} - self.display_manager = Mock() - self.cache_manager = Mock() - self.plugin_manager = Mock() - - self.plugin = MyPlugin( - plugin_id="my-plugin", - config=self.config, - display_manager=self.display_manager, - cache_manager=self.cache_manager, - plugin_manager=self.plugin_manager - ) - - def test_plugin_initialization(self): - self.assertEqual(self.plugin.plugin_id, "my-plugin") - self.assertTrue(self.plugin.enabled) - - def test_config_validation(self): - self.assertTrue(self.plugin.validate_config()) - - def test_update(self): - self.cache_manager.get.return_value = None - self.plugin.update() - # Assert data was fetched and cached - - def test_display(self): - self.plugin.display() - self.display_manager.draw_text.assert_called() - self.display_manager.update_display.assert_called() - -if __name__ == '__main__': - unittest.main() -``` - -Run tests: - -```bash -cd plugins/my-plugin -python -m pytest test_my_plugin.py -# or -python test_my_plugin.py -``` - -### Integration Testing - -Test plugin with actual managers: - -```python -# test_plugin_integration.py -from src.plugin_system.plugin_manager import PluginManager -from src.config_manager import ConfigManager -from src.display_manager import DisplayManager -from src.cache_manager import CacheManager - -def test_plugin_loading(): - config_manager = ConfigManager() - config = config_manager.load_config() - display_manager = DisplayManager(config) - cache_manager = CacheManager() - - plugin_manager = PluginManager( - plugins_dir="plugins", - config_manager=config_manager, - display_manager=display_manager, - cache_manager=cache_manager - ) - - plugins = plugin_manager.discover_plugins() - assert "my-plugin" in plugins - - assert plugin_manager.load_plugin("my-plugin") - plugin = plugin_manager.get_plugin("my-plugin") - assert plugin is not None - assert plugin.enabled - - plugin.update() - plugin.display() -``` - -### Emulator Testing - -Test plugin rendering visually: - -```bash -# Run with emulator -python run.py --emulator - -# Plugin should appear in display rotation -# Check logs for plugin loading and execution -``` - -### Hardware Testing - -1. Deploy plugin to Raspberry Pi -2. Enable in `config/config.json` -3. Restart LEDMatrix service -4. Observe LED matrix display -5. Check logs: `journalctl -u ledmatrix -f` - ---- - -## Troubleshooting - -### Plugin Not Loading - -**Symptoms**: Plugin doesn't appear in available modes, no logs about plugin - -**Solutions**: -1. Check plugin directory exists: `ls plugins/my-plugin/` -2. Verify `manifest.json` exists and is valid JSON -3. Check manifest has required fields: `id`, `entry_point`, `class_name` -4. Verify entry_point file exists: `ls plugins/my-plugin/manager.py` -5. Check class name matches: `grep "class.*Plugin" plugins/my-plugin/manager.py` -6. Review logs for import errors - -### Plugin Loading but Not Displaying - -**Symptoms**: Plugin loads successfully but doesn't appear in rotation - -**Solutions**: -1. Check plugin is enabled: `config/config.json` has `"enabled": true` -2. Verify display_modes in manifest match config -3. Check plugin is in rotation schedule -4. Review `display()` method for errors -5. Check logs for runtime errors - -### Configuration Errors - -**Symptoms**: Plugin fails to load, validation errors in logs - -**Solutions**: -1. Validate config against `config_schema.json` -2. Check required fields are present -3. Verify data types match schema -4. Check for typos in config keys -5. Review `validate_config()` method - -### Import Errors - -**Symptoms**: ModuleNotFoundError or ImportError in logs - -**Solutions**: -1. Install plugin dependencies: `pip install -r plugins/my-plugin/requirements.txt` -2. Check Python path includes plugin directory -3. Verify relative imports are correct -4. Check for circular import issues -5. Ensure all dependencies are in requirements.txt - -### Display Issues - -**Symptoms**: Plugin renders incorrectly or not at all - -**Solutions**: -1. Check display dimensions: `display_manager.width`, `display_manager.height` -2. Verify coordinates are within display bounds -3. Check color values are valid (0-255) -4. Ensure `update_display()` is called after rendering -5. Test with emulator first to debug rendering - -### Performance Issues - -**Symptoms**: Slow display updates, high CPU usage - -**Solutions**: -1. Use `cache_manager` to avoid excessive API calls -2. Implement background data fetching -3. Optimize rendering code -4. Consider using `high_performance_transitions` -5. Profile plugin code to identify bottlenecks - -### Git/Symlink Issues - -**Symptoms**: Plugin changes not appearing, broken symlinks - -**Solutions**: -1. Check symlink: `ls -la plugins/my-plugin` -2. Verify target exists: `readlink -f plugins/my-plugin` -3. Update plugin: `./scripts/dev/dev_plugin_setup.sh update my-plugin` -4. Re-link plugin if needed: `./scripts/dev/dev_plugin_setup.sh unlink my-plugin && ./scripts/dev/dev_plugin_setup.sh link my-plugin ` -5. Check git status: `cd plugins/my-plugin && git status` - ---- - -## Best Practices - -### Code Organization - -- Keep plugin code in `plugins//` directory -- Use descriptive class and method names -- Follow existing plugin patterns -- Place shared utilities in `src/common/` if reusable - -### Configuration - -- Always use `config_schema.json` for validation -- Store secrets in `config_secrets.json` -- Provide sensible defaults -- Document all configuration options in README - -### Error Handling - -- Use plugin logger for all logging -- Handle API failures gracefully -- Provide fallback displays when data unavailable -- Cache data to avoid excessive requests - -### Performance - -- Cache API responses appropriately -- Use background data fetching for long operations -- Optimize rendering for Pi's limited resources -- Test performance on actual hardware - -### Testing - -- Write unit tests for core logic -- Test with emulator before hardware -- Test on Raspberry Pi before deploying -- Test with other plugins enabled - -### Documentation - -- Document plugin functionality in README -- Include configuration examples -- Document API requirements and rate limits -- Provide usage examples - ---- - -## Resources - -- **Plugin System Documentation**: `docs/PLUGIN_ARCHITECTURE_SPEC.md` -- **Base Plugin Class**: `src/plugin_system/base_plugin.py` -- **Plugin Manager**: `src/plugin_system/plugin_manager.py` -- **Example Plugins**: - - `plugins/hockey-scoreboard/` - Sports scoreboard example - - `plugins/football-scoreboard/` - Complex multi-league example - - `plugins/ledmatrix-music/` - Real-time data example -- **Development Setup**: `dev_plugin_setup.sh` -- **Example Config**: `dev_plugins.json.example` - ---- - -## Quick Reference - -### Common Commands - -```bash -# Link plugin from GitHub -./scripts/dev/dev_plugin_setup.sh link-github - -# Link local plugin -./scripts/dev/dev_plugin_setup.sh link - -# List all plugins -./scripts/dev/dev_plugin_setup.sh list - -# Check plugin status -./scripts/dev/dev_plugin_setup.sh status - -# Update plugin(s) -./scripts/dev/dev_plugin_setup.sh update [name] - -# Unlink plugin -./scripts/dev/dev_plugin_setup.sh unlink - -# Run with emulator -python run.py --emulator - -# Run on Pi -python run.py -``` - -### Plugin File Structure - -``` -plugins/my-plugin/ -├── manifest.json # Required: Plugin metadata -├── manager.py # Required: Plugin class -├── config_schema.json # Required: Config validation -├── requirements.txt # Optional: Dependencies -├── README.md # Optional: Documentation -└── ... # Plugin-specific files -``` - -### Required Manifest Fields - -- `id`: Plugin identifier -- `entry_point`: Python file (usually "manager.py") -- `class_name`: Plugin class name -- `display_modes`: Array of mode names - diff --git a/.cursor/rules/coding-standards.mdc b/.cursor/rules/coding-standards.mdc deleted file mode 100644 index 5abeb94e..00000000 --- a/.cursor/rules/coding-standards.mdc +++ /dev/null @@ -1,38 +0,0 @@ ---- -globs: *.py ---- - -# Python Coding Standards - -## Code Quality Principles -- **Simplicity First**: Prefer clear, readable code over clever optimizations -- **Explicit over Implicit**: Make intentions clear through naming and structure -- **Fail Fast**: Validate inputs and handle errors early -- **Documentation**: Use docstrings for classes and complex functions - -## Naming Conventions -- **Classes**: PascalCase (e.g., `NHLRecentManager`) -- **Functions/Variables**: snake_case (e.g., `fetch_game_data`) -- **Constants**: UPPER_SNAKE_CASE (e.g., `ESPN_NHL_SCOREBOARD_URL`) -- **Private methods**: Leading underscore (e.g., `_fetch_data`) - -## Error Handling -- **Logging**: Use structured logging with context (e.g., `[NHL Recent]`) -- **Exceptions**: Catch specific exceptions, not bare `except:` -- **User-friendly messages**: Explain what went wrong and potential solutions -- **Graceful degradation**: Continue operation when non-critical features fail - -## Manager Pattern -All sports managers should follow this structure: -```python -class BaseManager: - def __init__(self, config, display_manager, cache_manager) - def update(self) # Fetch and process data - def display(self, force_clear=False) # Render to display -``` - -## Configuration Management -- **Type hints**: Use for function parameters and return values -- **Configuration validation**: Check required fields on initialization -- **Default values**: Provide sensible defaults in code, not config -- **Environment awareness**: Handle different deployment contexts \ No newline at end of file diff --git a/.cursor/rules/configuration-management.mdc b/.cursor/rules/configuration-management.mdc deleted file mode 100644 index 37f68300..00000000 --- a/.cursor/rules/configuration-management.mdc +++ /dev/null @@ -1,42 +0,0 @@ ---- -globs: config/*.json,src/*.py ---- - -# Configuration Management - -## Configuration Structure -- **Main config**: [config/config.json](mdc:config/config.json) - Primary configuration -- **Secrets**: [config/config_secrets.json](mdc:config/config_secrets.json) - API keys and sensitive data -- **Templates**: [config/config.template.json](mdc:config/config.template.json) - Default values - -## Configuration Principles -- **Validation**: Check required fields and data types on startup -- **Defaults**: Provide sensible defaults in code, not just config -- **Environment awareness**: Handle development vs production differences -- **Security**: Never commit secrets to version control - -## Manager Configuration Pattern -```python -def __init__(self, config, display_manager, cache_manager): - self.mode_config = config.get("sport_scoreboard", {}) - self.favorite_teams = self.mode_config.get("favorite_teams", []) - self.show_favorite_only = self.mode_config.get("show_favorite_teams_only", False) -``` - -## Required Configuration Sections -- **Display settings**: Update intervals, display durations -- **API settings**: Timeouts, retry logic, rate limiting -- **Background service**: Threading, caching, priority settings -- **Team preferences**: Favorite teams, filtering options - -## Configuration Validation -- **Type checking**: Ensure numeric values are numbers, lists are lists -- **Range validation**: Check that intervals are reasonable -- **Dependency checking**: Verify required services are available -- **Fallback values**: Provide defaults when config is missing or invalid - -## Best Practices -- **Documentation**: Comment complex configuration options -- **Examples**: Provide working examples in templates -- **Migration**: Handle configuration changes between versions -- **Testing**: Validate configuration in test environments \ No newline at end of file diff --git a/.cursor/rules/error-handling-logging.mdc b/.cursor/rules/error-handling-logging.mdc deleted file mode 100644 index 0bf404ee..00000000 --- a/.cursor/rules/error-handling-logging.mdc +++ /dev/null @@ -1,50 +0,0 @@ ---- -globs: src/*.py ---- - -# Error Handling and Logging - -## Logging Standards -- **Structured prefixes**: Use consistent tags like `[NHL Recent]`, `[NFL Live]` -- **Context information**: Include relevant details (team names, game status, dates) -- **Appropriate levels**: - - `info`: Normal operations and status updates - - `debug`: Detailed information for troubleshooting - - `warning`: Non-critical issues that should be noted - - `error`: Problems that need attention - -## Error Handling Patterns -```python -try: - data = self._fetch_data() - if not data or 'events' not in data: - self.logger.warning("[Manager] No events found in API response") - return -except requests.exceptions.RequestException as e: - self.logger.error(f"[Manager] API error: {e}") - return None -``` - -## User-Friendly Messages -- **Explain the situation**: "No games available during off-season" -- **Provide context**: "NHL season typically runs October-June" -- **Suggest solutions**: "Check back when season starts" -- **Distinguish issues**: API problems vs no data vs filtering results - -## Graceful Degradation -- **Fallback content**: Show alternative games when favorites unavailable -- **Cached data**: Use cached data when API fails -- **Service continuity**: Continue operation when non-critical features fail -- **Clear communication**: Explain what's happening to users - -## Debugging Support -- **Comprehensive logging**: Log API responses, filtering results, display updates -- **State tracking**: Log current state and transitions -- **Performance monitoring**: Track timing and resource usage -- **Error context**: Include stack traces for debugging - -## Off-Season Awareness -- **Seasonal messaging**: Different messages for different times of year -- **Helpful context**: Explain why no games are available -- **Future planning**: Mention when season starts -- **Realistic expectations**: Set appropriate expectations during off-season \ No newline at end of file diff --git a/.cursor/rules/git-workflow.mdc b/.cursor/rules/git-workflow.mdc deleted file mode 100644 index c28c603a..00000000 --- a/.cursor/rules/git-workflow.mdc +++ /dev/null @@ -1,51 +0,0 @@ ---- -alwaysApply: true ---- - -# Git Workflow and Branching - -## Branch Naming Conventions -- **Features**: `feature/description-of-feature` (e.g., `feature/weather-forecast-improvements`) -- **Bug fixes**: `fix/description-of-bug` (e.g., `fix/nhl-manager-improvements`) -- **Hotfixes**: `hotfix/critical-issue-description` -- **Refactoring**: `refactor/description-of-refactor` - -## Commit Message Format -``` -type(scope): description - -[optional body] - -[optional footer] -``` - -**Types**: feat, fix, docs, style, refactor, test, chore -**Examples**: -- `feat(nhl): Add enhanced logging for data visibility` -- `fix(display): Resolve rendering performance issue` -- `docs(api): Update ESPN API integration guide` - -## Pull Request Guidelines -- **Self-review**: Review your own PR before requesting review -- **Testing**: Test thoroughly on Raspberry Pi hardware -- **Documentation**: Update relevant documentation if needed -- **Clean history**: Squash commits if necessary for clean history - -## Code Review Checklist -- **Code Quality**: Proper error handling, logging, type hints -- **Architecture**: Follows project patterns, doesn't break existing functionality -- **Performance**: No negative impact on display performance -- **Testing**: Works on Raspberry Pi hardware -- **Documentation**: Comments added for complex logic - -## Merge Strategies -- **Squash and Merge**: Preferred for feature branches and bug fixes -- **Merge Commit**: For complex features with multiple logical commits -- **Rebase and Merge**: For simple, single-commit changes - -## Best Practices -- **Keep branches small and focused** -- **Commit frequently with meaningful messages** -- **Update branch regularly with main** -- **Test changes incrementally** -- **Delete feature branches after merge** \ No newline at end of file diff --git a/.cursor/rules/github-branches-rule.mdc b/.cursor/rules/github-branches-rule.mdc deleted file mode 100644 index 87b1ea5f..00000000 --- a/.cursor/rules/github-branches-rule.mdc +++ /dev/null @@ -1,213 +0,0 @@ ---- -description: GitHub branching and pull request best practices for LEDMatrix project -globs: ["**/*.py", "**/*.md", "**/*.json", "**/*.sh"] -alwaysApply: true ---- - -# GitHub Branching and Pull Request Guidelines - -## Branch Naming Conventions - -### Feature Branches -- **Format**: `feature/description-of-feature` -- **Examples**: - - `feature/weather-forecast-improvements` - - `feature/stock-api-integration` - - `feature/nba-live-scores` - -### Bug Fix Branches -- **Format**: `fix/description-of-bug` -- **Examples**: - - `fix/leaderboard-scrolling-performance` - - `fix/weather-api-timeout` - - `fix/display-rendering-issue` - -### Hotfix Branches -- **Format**: `hotfix/critical-issue-description` -- **Examples**: - - `hotfix/display-crash-fix` - - `hotfix/api-rate-limit-fix` - -### Refactoring Branches -- **Format**: `refactor/description-of-refactor` -- **Examples**: - - `refactor/sports-manager-architecture` - - `refactor/cache-management-system` - -## Branch Management Rules - -### Main Branch Protection -- **`main`** branch is protected and requires PR reviews -- Never commit directly to `main` -- All changes must go through pull requests - -### Branch Lifecycle -1. **Create** branch from `main` when starting work -2. **Keep** branch up-to-date with `main` regularly -3. **Test** thoroughly before creating PR -4. **Delete** branch after successful merge - -### Branch Updates -```bash -# Before starting new work -git checkout main -git pull origin main - -# Create new branch -git checkout -b feature/your-feature-name - -# Keep branch updated during development -git checkout main -git pull origin main -git checkout feature/your-feature-name -git merge main -``` - -## Pull Request Guidelines - -### PR Title Format -- **Feature**: `feat: Add weather forecast improvements` -- **Fix**: `fix: Resolve leaderboard scrolling performance issue` -- **Refactor**: `refactor: Improve sports manager architecture` -- **Docs**: `docs: Update API integration guide` -- **Test**: `test: Add unit tests for weather manager` - -### PR Description Template -```markdown -## Description -Brief description of changes and motivation. - -## Type of Change -- [ ] Bug fix (non-breaking change) -- [ ] New feature (non-breaking change) -- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) -- [ ] Documentation update -- [ ] Performance improvement -- [ ] Refactoring - -## Testing -- [ ] Tested on Raspberry Pi hardware -- [ ] Verified display rendering works correctly -- [ ] Checked API integration functionality -- [ ] Tested error handling scenarios - -## Screenshots/Videos -(If applicable, add screenshots or videos of the changes) - -## Checklist -- [ ] Code follows project style guidelines -- [ ] Self-review completed -- [ ] Comments added for complex logic -- [ ] No hardcoded values or API keys -- [ ] Error handling implemented -- [ ] Logging added where appropriate -``` - -### PR Review Requirements - -#### For Reviewers -- **Code Quality**: Check for proper error handling, logging, and type hints -- **Architecture**: Ensure changes follow project patterns and don't break existing functionality -- **Performance**: Verify changes don't negatively impact display performance -- **Testing**: Confirm changes work on Raspberry Pi hardware -- **Documentation**: Check if documentation needs updates - -#### For Authors -- **Self-Review**: Review your own PR before requesting review -- **Testing**: Test thoroughly on Pi hardware before submitting -- **Documentation**: Update relevant documentation if needed -- **Clean History**: Squash commits if necessary for clean history - -## Commit Message Guidelines - -### Format -``` -type(scope): description - -[optional body] - -[optional footer] -``` - -### Types -- **feat**: New feature -- **fix**: Bug fix -- **docs**: Documentation changes -- **style**: Code style changes (formatting, etc.) -- **refactor**: Code refactoring -- **test**: Adding or updating tests -- **chore**: Maintenance tasks - -### Examples -``` -feat(weather): Add hourly forecast display -fix(nba): Resolve live score update issue -docs(api): Update ESPN API integration guide -refactor(sports): Improve base class architecture -``` - -## Merge Strategies - -### Squash and Merge (Preferred) -- Use for feature branches and bug fixes -- Creates clean, linear history -- Combines all commits into single commit - -### Merge Commit -- Use for complex features with multiple logical commits -- Preserves commit history -- Use when commit messages are meaningful - -### Rebase and Merge -- Use sparingly for simple, single-commit changes -- Creates linear history without merge commits - -## Release Management - -### Version Tags -- Use semantic versioning: `v1.2.3` -- Tag releases on `main` branch -- Create release notes with technical details - -### Release Branches -- **Format**: `release/v1.2.3` -- Use for release preparation -- Include version bumps and final testing - -## Emergency Procedures - -### Hotfix Process -1. Create `hotfix/` branch from `main` -2. Make minimal fix -3. Test thoroughly -4. Create PR with expedited review -5. Merge to `main` and tag release -6. Cherry-pick to other branches if needed - -### Rollback Process -1. Identify last known good commit -2. Create revert PR if possible -3. Use `git revert` for clean rollback -4. Tag rollback release -5. Document issue and resolution - -## Best Practices - -### Before Creating PR -- [ ] Run all tests locally -- [ ] Test on Raspberry Pi hardware -- [ ] Check for linting errors -- [ ] Update documentation if needed -- [ ] Ensure commit messages are clear - -### During Development -- [ ] Keep branches small and focused -- [ ] Commit frequently with meaningful messages -- [ ] Update branch regularly with main -- [ ] Test changes incrementally - -### After PR Approval -- [ ] Delete feature branch after merge -- [ ] Update local main branch -- [ ] Verify changes work in production -- [ ] Update any related documentation diff --git a/.cursor/rules/project-structure.mdc b/.cursor/rules/project-structure.mdc deleted file mode 100644 index c9878a27..00000000 --- a/.cursor/rules/project-structure.mdc +++ /dev/null @@ -1,23 +0,0 @@ ---- -alwaysApply: true ---- - -# LEDMatrix Project Structure - -## Core Architecture -- **Main entry point**: [run.py](mdc:run.py) - Primary application launcher -- **Configuration**: [config/config.json](mdc:config/config.json) - Main configuration file -- **Display management**: [src/display_controller.py](mdc:src/display_controller.py) - Core display logic -- **Web interface**: [web_interface_v2.py](mdc:web_interface_v2.py) - Modern web UI - -## Source Code Organization -- **Managers**: [src/](mdc:src/) - All sports/weather/stock managers -- **Assets**: [assets/](mdc:assets/) - Logos, fonts, and static resources -- **Tests**: [test/](mdc:test/) - Unit and integration tests -- **Documentation**: [LEDMatrix.wiki/](mdc:LEDMatrix.wiki/) - Comprehensive guides - -## Key Design Principles -- **Single Responsibility**: Each manager handles one sport/domain -- **Consistent Patterns**: All managers follow similar structure -- **Configuration-Driven**: Behavior controlled via [config/config.json](mdc:config/config.json) -- **Raspberry Pi Focus**: Optimized for Pi hardware, not Windows development \ No newline at end of file diff --git a/.cursor/rules/raspberry-pi-development.mdc b/.cursor/rules/raspberry-pi-development.mdc deleted file mode 100644 index 829b7591..00000000 --- a/.cursor/rules/raspberry-pi-development.mdc +++ /dev/null @@ -1,41 +0,0 @@ ---- -alwaysApply: true ---- - -# Raspberry Pi Development Guidelines - -## Hardware Constraints -- **Pi-only execution**: Code must run on Raspberry Pi, not Windows development machine -- **LED matrix library**: Uses [rpi-rgb-led-matrix-master/](mdc:rpi-rgb-led-matrix-master/) for hardware control -- **Memory limitations**: Optimize for Pi's limited RAM -- **Performance**: Consider Pi's CPU capabilities in design - -## Development Workflow -- **Local development**: Write and test code on Windows -- **Pi deployment**: Deploy and test on actual Pi hardware -- **SSH access**: Use SSH for Pi-based testing and debugging -- **Service management**: Use systemd services for production deployment - -## Testing Strategy -- **Unit tests**: Test logic without hardware dependencies -- **Integration tests**: Test with mock display managers -- **Hardware tests**: Validate on actual Pi with LED matrix -- **Performance tests**: Monitor memory and CPU usage - -## Deployment Considerations -- **Service files**: [ledmatrix.service](mdc:ledmatrix.service), [ledmatrix-web.service](mdc:ledmatrix-web.service) -- **Installation scripts**: [first_time_install.sh](mdc:first_time_install.sh), [install_service.sh](mdc:install_service.sh) -- **Dependencies**: [requirements.txt](mdc:requirements.txt) for Pi environment -- **Permissions**: Handle file permissions for Pi user - -## Performance Optimization -- **Caching**: Use [src/cache_manager.py](mdc:src/cache_manager.py) for data persistence -- **Background services**: Non-blocking data fetching -- **Memory management**: Clean up resources regularly -- **Display optimization**: Minimize unnecessary redraws - -## Debugging on Pi -- **Logging**: Comprehensive logging for remote debugging -- **Error reporting**: Clear error messages for troubleshooting -- **Status monitoring**: Health checks and status reporting -- **Remote access**: Web interface for configuration and monitoring \ No newline at end of file diff --git a/.cursor/rules/sports-managers.mdc b/.cursor/rules/sports-managers.mdc deleted file mode 100644 index fe6c6fc8..00000000 --- a/.cursor/rules/sports-managers.mdc +++ /dev/null @@ -1,42 +0,0 @@ ---- -globs: src/*_managers.py ---- - -# Sports Manager Development - -## Manager Architecture -All sports managers inherit from base classes and follow consistent patterns: -- **Base classes**: [src/nhl_managers.py](mdc:src/nhl_managers.py), [src/nfl_managers.py](mdc:src/nfl_managers.py) -- **Common functionality**: Data fetching, caching, display rendering -- **Configuration-driven**: Behavior controlled via config sections - -## Required Methods -```python -def __init__(self, config, display_manager, cache_manager) -def update(self) # Fetch fresh data -def display(self, force_clear=False) # Render current data -``` - -## Data Flow Pattern -1. **Fetch**: Get data from API (with caching) -2. **Process**: Extract relevant game information -3. **Filter**: Apply favorite team preferences -4. **Display**: Render to LED matrix - -## Logging Standards -- **Structured prefixes**: `[NHL Recent]`, `[NFL Live]`, etc. -- **Context information**: Include team names, game status, dates -- **Debug levels**: Use appropriate log levels (info, debug, warning, error) -- **User-friendly messages**: Explain what's happening and why - -## Error Handling -- **API failures**: Log and continue with cached data if available -- **No data scenarios**: Distinguish between API issues vs no games available -- **Off-season awareness**: Provide helpful context during non-active periods -- **Fallback behavior**: Show alternative content when preferred content unavailable - -## Configuration Integration -- **Required settings**: Validate on initialization -- **Optional settings**: Provide sensible defaults -- **Background service**: Use for non-blocking data fetching -- **Caching strategy**: Implement intelligent cache management \ No newline at end of file diff --git a/.cursor/rules/testing-standards.mdc b/.cursor/rules/testing-standards.mdc deleted file mode 100644 index 274b76ef..00000000 --- a/.cursor/rules/testing-standards.mdc +++ /dev/null @@ -1,51 +0,0 @@ ---- -globs: test/*.py,src/*.py ---- - -# Testing Standards - -## Test Organization -- **Test directory**: [test/](mdc:test/) - All test files -- **Unit tests**: Test individual components in isolation -- **Integration tests**: Test component interactions -- **Hardware tests**: Validate on Raspberry Pi with actual LED matrix - -## Testing Principles -- **Test behavior, not implementation**: Focus on what the code does, not how -- **Mock external dependencies**: Use mocks for APIs, display managers, cache -- **Test edge cases**: Empty data, API failures, configuration errors -- **Pi-specific testing**: Validate hardware integration - -## Test Structure -```python -def test_manager_initialization(): - """Test that manager initializes with valid config""" - config = {"sport_scoreboard": {"enabled": True}} - manager = ManagerClass(config, mock_display, mock_cache) - assert manager.enabled == True - -def test_api_failure_handling(): - """Test graceful handling of API failures""" - # Test that system continues when API fails - # Verify fallback to cached data - # Check appropriate error logging -``` - -## Mock Patterns -- **Display Manager**: Mock for testing without hardware -- **Cache Manager**: Mock for testing data persistence -- **API responses**: Mock for consistent test data -- **Configuration**: Use test-specific configs - -## Test Categories -- **Unit tests**: Individual manager methods -- **Integration tests**: Manager interactions with services -- **Configuration tests**: Validate config loading and validation -- **Error handling tests**: API failures, invalid data, edge cases - -## Testing Best Practices -- **Descriptive names**: Test names should explain what they test -- **Single responsibility**: Each test should verify one thing -- **Independent tests**: Tests should not depend on each other -- **Clean setup/teardown**: Reset state between tests -- **Pi compatibility**: Ensure tests work in Pi environment \ No newline at end of file diff --git a/.cursorignore b/.cursorignore deleted file mode 100644 index 6f9f00ff..00000000 --- a/.cursorignore +++ /dev/null @@ -1 +0,0 @@ -# Add directories or file patterns to ignore during indexing (e.g. foo/ or *.csv) diff --git a/.cursorrules b/.cursorrules deleted file mode 100644 index 93b923fb..00000000 --- a/.cursorrules +++ /dev/null @@ -1,364 +0,0 @@ -# LEDMatrix Plugin Development Rules - -## Plugin System Overview - -The LEDMatrix project uses a plugin-based architecture. All display -functionality (except core calendar) is implemented as plugins that are -dynamically loaded from the directory configured by -`plugin_system.plugins_directory` in `config.json` — the default is -`plugin-repos/` (per `config/config.template.json:130`). - -> **Fallback note (scoped):** `PluginManager.discover_plugins()` -> (`src/plugin_system/plugin_manager.py:154`) only scans the -> configured directory — there is no fallback to `plugins/` in the -> main discovery path. A fallback to `plugins/` does exist in two -> narrower places: -> - `store_manager.py:1700-1718` — store operations (install/update/ -> uninstall) check `plugins/` if the plugin isn't found in the -> configured directory, so plugin-store flows work even when your -> dev symlinks live in `plugins/`. -> - `schema_manager.py:70-80` — `get_schema_path()` probes both -> `plugins/` and `plugin-repos/` for `config_schema.json` so the -> web UI form generation finds the schema regardless of where the -> plugin lives. -> -> The dev workflow in `scripts/dev/dev_plugin_setup.sh` creates -> symlinks under `plugins/`, which is why the store and schema -> fallbacks exist. For day-to-day development, set -> `plugin_system.plugins_directory` to `plugins` so the main -> discovery path picks up your symlinks. - -## Plugin Structure - -### Required Files -- **manifest.json**: Plugin metadata, entry point, class name, dependencies -- **manager.py**: Main plugin class (must inherit from `BasePlugin`) -- **config_schema.json**: JSON schema for plugin configuration validation -- **requirements.txt**: Python dependencies (if any) -- **README.md**: Plugin documentation - -### Plugin Class Requirements -- Must inherit from `src.plugin_system.base_plugin.BasePlugin` -- Must implement `update()` method for data fetching -- Must implement `display()` method for rendering -- Should implement `validate_config()` for configuration validation -- Optional: Override `has_live_content()` for live priority features - -## Plugin Development Workflow - -### 1. Creating a New Plugin - -**Option A: Use dev_plugin_setup.sh (Recommended)** -```bash -# Link from GitHub -./scripts/dev/dev_plugin_setup.sh link-github - -# Link local repository -./scripts/dev/dev_plugin_setup.sh link -``` - -**Option B: Manual Setup** -1. Create directory in `plugin-repos//` (or `plugins//` - if you're using the dev fallback location) -2. Add `manifest.json` with required fields -3. Create `manager.py` with plugin class -4. Add `config_schema.json` for configuration -5. Enable plugin in `config/config.json` under `"": {"enabled": true}` - -### 2. Plugin Configuration - -Plugins are configured in `config/config.json`: -```json -{ - "": { - "enabled": true, - "display_duration": 15, - "live_priority": false, - "high_performance_transitions": false, - "transition": { - "type": "redraw", - "speed": 2, - "enabled": true - }, - // ... plugin-specific config - } -} -``` - -### 3. Testing Plugins - -**On Development Machine:** -- Run the dev preview server: `python3 scripts/dev_server.py` (then - open `http://localhost:5001`) — renders plugins in the browser - without running the full display loop -- Or run the full display in emulator mode: - `python3 run.py --emulator` (or equivalently - `EMULATOR=true python3 run.py`, or `./scripts/dev/run_emulator.sh`). - The `-e`/`--emulator` CLI flag is defined in `run.py:19-20`. -- Test plugin loading: Check logs for plugin discovery and loading -- Validate configuration: Ensure config matches `config_schema.json` - -**On Raspberry Pi:** -- Deploy and test on actual hardware -- Monitor logs: `journalctl -u ledmatrix -f` (if running as service) -- Check plugin status in web interface - -### 4. Plugin Development Best Practices - -**Code Organization:** -- Keep plugin code in `plugin-repos//` (or its dev-time - symlink in `plugins//`) -- Use shared assets from `assets/` directory when possible -- Follow existing plugin patterns — canonical sources live in the - [`ledmatrix-plugins`](https://github.com/ChuckBuilds/ledmatrix-plugins) - repo (`plugins/hockey-scoreboard/`, `plugins/football-scoreboard/`, - `plugins/clock-simple/`, etc.) -- Place shared utilities in `src/common/` if reusable across plugins - -**Configuration Management:** -- Use `config_schema.json` for validation -- Store secrets in `config/config_secrets.json` under the same plugin - id namespace as the main config — they're deep-merged into the main - config at load time (`src/config_manager.py:162-172`), so plugin - code reads them directly from `config.get(...)` like any other key -- There is no separate `config_secrets` reference field -- Validate all required fields in `validate_config()` - -**Error Handling:** -- Use plugin's logger: `self.logger.info/error/warning()` -- Handle API failures gracefully -- Cache data to avoid excessive API calls -- Provide fallback displays when data unavailable - -**Performance:** -- Use `cache_manager` for API response caching -- Implement background data fetching if needed -- Use `high_performance_transitions` for smoother animations -- Optimize rendering for Pi's limited resources - -**Display Rendering:** -- Use `display_manager` for all drawing operations -- Support different display sizes (check `display_manager.width/height`) -- Use `apply_transition()` for smooth transitions between displays -- Clear display before rendering: `display_manager.clear()` -- Always call `display_manager.update_display()` after rendering - -## Plugin API Reference - -### BasePlugin Class -Located in: `src/plugin_system/base_plugin.py` - -**Required Methods:** -- `update()`: Fetch/update data (called based on `update_interval` in manifest) -- `display(force_clear=False)`: Render plugin content - -**Optional Methods:** -- `validate_config()`: Validate plugin configuration -- `has_live_content()`: Return True if plugin has live/urgent content -- `get_live_modes()`: Return list of modes for live priority -- `cleanup()`: Clean up resources on unload -- `on_config_change(new_config)`: Handle config updates -- `on_enable()`: Called when plugin enabled -- `on_disable()`: Called when plugin disabled - -**Available Properties:** -- `self.plugin_id`: Plugin identifier -- `self.config`: Plugin configuration dict -- `self.display_manager`: Display manager instance -- `self.cache_manager`: Cache manager instance -- `self.plugin_manager`: Plugin manager reference -- `self.logger`: Plugin-specific logger -- `self.enabled`: Boolean enabled status -- `self.transition_manager`: Transition system (if available) - -### Display Manager -Located in: `src/display_manager.py` - -**Key Methods:** -- `clear()`: Clear the display -- `draw_text(text, x, y, color, font, small_font, centered)`: Draw text -- `update_display()`: Push the buffer to the physical display -- `draw_weather_icon(condition, x, y, size)`: Draw a weather icon -- `width`, `height`: Display dimensions - -**Image rendering**: there is no `draw_image()` helper. Paste directly -onto the underlying PIL Image: -```python -self.display_manager.image.paste(pil_image, (x, y)) -self.display_manager.update_display() -``` -For transparency, paste with a mask: `image.paste(rgba, (x, y), rgba)`. - -### Cache Manager -Located in: `src/cache_manager.py` - -**Key Methods:** -- `get(key, max_age=300)`: Get cached value (returns None if missing/stale) -- `set(key, value, ttl=None)`: Cache a value -- `delete(key)` / `clear_cache(key=None)`: Remove a single cache entry, - or (for `clear_cache` with no argument) every cached entry. `delete` - is an alias for `clear_cache(key)`. -- `get_cached_data_with_strategy(key, data_type)`: Cache get with - data-type-aware TTL strategy -- `get_background_cached_data(key, sport_key)`: Cache get for the - background-fetch service path - -## Plugin Manifest Schema - -Required fields in `manifest.json`: -- `id`: Unique plugin identifier (matches directory name) -- `name`: Human-readable plugin name -- `version`: Semantic version (e.g., "1.0.0") -- `entry_point`: Python file (usually "manager.py") -- `class_name`: Plugin class name (must match class in entry_point) -- `display_modes`: Array of mode names this plugin provides - -Common optional fields: -- `description`: Plugin description -- `author`: Plugin author -- `homepage`: Plugin homepage URL -- `category`: Plugin category (e.g., "sports", "weather") -- `tags`: Array of tags -- `update_interval`: Seconds between update() calls (default: 60) -- `default_duration`: Default display duration (default: 15) -- `requires`: Python version, display size requirements -- `config_schema`: Path to config schema file -- `api_requirements`: API dependencies and rate limits - -## Plugin Loading Process - -1. **Discovery**: PluginManager scans `plugins/` directory for directories containing `manifest.json` -2. **Validation**: Validates manifest structure and required fields -3. **Loading**: Imports plugin module and instantiates plugin class -4. **Configuration**: Loads plugin config from `config/config.json` -5. **Validation**: Calls `validate_config()` on plugin instance -6. **Registration**: Adds plugin to available modes and stores instance -7. **Enablement**: Calls `on_enable()` if plugin is enabled - -## Common Plugin Patterns - -### Sports Scoreboard Plugin -- Use `background_data_service.py` pattern for API fetching -- Implement live/recent/upcoming game modes -- Use `scoreboard_renderer.py` for consistent rendering -- Support team filtering and game filtering -- Use shared sports logos from `assets/sports/` - -### Data Display Plugin -- Fetch data in `update()` method -- Cache API responses using `cache_manager` -- Render in `display()` method -- Handle API errors gracefully -- Provide configuration for refresh intervals - -### Real-time Content Plugin -- Implement `has_live_content()` for live priority -- Use `get_live_modes()` to specify which modes are live -- Set `live_priority: true` in config to enable live takeover -- Update data frequently when live content exists - -## Debugging Plugins - -**Check Plugin Loading:** -- Review logs for plugin discovery messages -- Verify manifest.json syntax is valid JSON -- Check that class_name matches actual class name -- Ensure entry_point file exists and is importable - -**Check Plugin Execution:** -- Add logging statements in `update()` and `display()` -- Use `self.logger` for plugin-specific logging -- Check cache_manager for cached data -- Verify display_manager is rendering correctly - -**Common Issues:** -- Import errors: Check Python path and dependencies -- Config errors: Validate against config_schema.json -- Display issues: Check display dimensions and coordinate calculations -- Performance: Monitor CPU/memory usage on Pi - -## Plugin Testing - -**Unit Tests:** -- Test plugin class instantiation -- Test `update()` data fetching logic -- Test `display()` rendering logic -- Test `validate_config()` with various configs -- Mock `display_manager` and `cache_manager` for testing - -**Integration Tests:** -- Test plugin loading via PluginManager -- Test plugin with actual config -- Test plugin with emulator display -- Test plugin with cache_manager - -**Hardware Tests:** -- Test on Raspberry Pi with LED matrix -- Verify display rendering on actual hardware -- Test performance under load -- Test with other plugins enabled - -## File Organization - -``` -plugins/ - / - manifest.json # Plugin metadata - manager.py # Main plugin class - config_schema.json # Config validation schema - requirements.txt # Python dependencies - README.md # Plugin documentation - # Plugin-specific files - data_manager.py - renderer.py - etc. -``` - -## Git Workflow for Plugins - -**Plugin Development:** -- Plugins are typically separate repositories -- Use `dev_plugin_setup.sh` to link plugins for development -- Symlinks are used to connect plugin repos to `plugins/` directory -- Plugin repos follow naming: `ledmatrix-` - -**Branching:** -- Develop plugins in feature branches -- Follow project branching conventions -- Test plugins before merging to main - -**Automatic Version Bumping:** -- **Automatic Version Management**: Version bumping is handled automatically via the pre-push git hook - no manual version bumping is required for normal development workflows -- **GitHub as Source of Truth**: Plugin store always fetches latest versions from GitHub (releases/tags/manifest/commit) -- **Pre-Push Hook**: Automatically bumps patch version and creates git tags when pushing code changes - - The hook is self-contained (no external dependencies) and works on any dev machine - - Installation: Copy the hook from LEDMatrix repo to your plugin repo: - ```bash - # From your plugin repository directory - cp /path/to/LEDMatrix/scripts/git-hooks/pre-push-plugin-version .git/hooks/pre-push - chmod +x .git/hooks/pre-push - ``` - - Or use the installer script from the main LEDMatrix repo (one-time setup) - - The hook automatically: - 1. Bumps the patch version (x.y.Z) in manifest.json when code changes are detected - 2. Creates a git tag (v{version}) for the new version - 3. Stages manifest.json for commit - - Skip auto-tagging: Set `SKIP_TAG=1` environment variable before pushing -- **Manual Version Bumping (Edge Cases Only)**: Manual version bumps are only needed in rare circumstances: - - CI/CD pipelines that bypass git hooks - - Forked repositories without the pre-push hook installed - - Major/minor version bumps (hook only handles patch versions) - - When skipping auto-tagging but still needing a version bump - - For manual bumps, use the standalone script: `scripts/bump_plugin_version.py` -- **Registry**: The plugin registry (plugins.json) stores only metadata (name, description, repo URL) - no versions -- **Version Priority**: Plugin store checks versions in this order: GitHub Releases → GitHub Tags → Manifest from branch → Git commit hash - -## Resources - -- Plugin System Docs: `docs/PLUGIN_ARCHITECTURE_SPEC.md` -- Plugin Examples: `plugins/hockey-scoreboard/`, `plugins/football-scoreboard/` -- Base Plugin: `src/plugin_system/base_plugin.py` -- Plugin Manager: `src/plugin_system/plugin_manager.py` -- Development Setup: `dev_plugin_setup.sh` -- Example Config: `dev_plugins.json.example` - diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 0000c47f..7f91e726 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -4,6 +4,9 @@ on: pull_request: push: branches: [main] + # Manual runs against any branch — useful when a PR's automatic run + # needs a re-run or didn't get created. + workflow_dispatch: # Both jobs only check out the repo and run pytest. permissions: @@ -13,6 +16,12 @@ jobs: plugin-safety: name: Plugin safety harness + unit tests runs-on: ubuntu-latest + env: + # The bundled fixture plugin gives the harness at least one real plugin + # to render, and REQUIRE_PLUGINS turns "discovered zero plugins" into a + # hard failure instead of a silent all-skip green run. + LEDMATRIX_PLUGINS_DIR: test/fixtures/plugins + LEDMATRIX_REQUIRE_PLUGINS: "1" steps: - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 with: @@ -29,12 +38,9 @@ jobs: pip install -r requirements.txt -r requirements-test.txt pip install RGBMatrixEmulator - - name: Run harness + visual rendering tests + - name: Run plugin safety harness run: | - pytest --no-cov \ - test/plugins/test_harness.py \ - test/plugins/test_visual_rendering.py \ - test/plugins/test_plugin_matrix.py + pytest --no-cov test/plugins/ unit-tests: name: Core unit tests @@ -55,29 +61,15 @@ jobs: pip install -r requirements.txt -r requirements-test.txt pip install RGBMatrixEmulator - # Safety net for the shared sports/scroll/style infrastructure. These - # suites existed but were not enrolled in CI, so a refactor of - # src/base_classes or src/common could regress them silently. Enrolled - # explicitly (not `pytest test/`) so known hardware-only suites don't - # break CI; grow this list as more suites are made headless. + # Run the ENTIRE test tree (except test/plugins, which the + # plugin-safety job owns). New test files are enrolled automatically; + # excluding anything requires a visible, commented --ignore here. + # Coverage is measured and enforced only in this step — pytest.ini + # deliberately carries no coverage flags so local runs stay fast. - name: Run core unit suites run: | - pytest --no-cov \ - test/test_skin_system.py \ - test/test_font_manager.py \ - test/test_data_sources.py \ - test/test_api_extractors.py \ - test/test_scroll_helper.py \ - test/test_scroll_helper_continuous.py \ - test/test_adaptive_layout.py \ - test/test_loader_compat_warning.py \ - test/test_sports_base_characterization.py \ - test/test_element_style.py \ - test/test_sports_core_promotions.py \ - test/test_sports_modes_promotions.py \ - test/test_sports_capabilities.py \ - test/test_sports_scroll.py \ - test/test_version_consistency.py \ - test/test_plugin_compatibility_gate.py \ - test/test_install_preserves_existing.py \ - test/test_core_owned_config_keys.py + pytest -m "not hardware" test/ \ + --ignore=test/plugins \ + --cov=src --cov=web_interface \ + --cov-report=term \ + --cov-fail-under=52 diff --git a/CLAUDE.md b/CLAUDE.md index 496c4cce..b7e027c4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,12 +6,16 @@ - `config/config.json` — User plugin configuration (persists across plugin reinstalls) - `plugin-repos/` — **Default** plugin install directory used by the Plugin Store, set by `plugin_system.plugins_directory` in - `config.json` (default per `config/config.template.json:130`). + `config.json` (default per `config/config.template.json:167`). Not gitignored. - `plugins/` — Legacy/dev plugin location. Gitignored (`plugins/*`). Used by `scripts/dev/dev_plugin_setup.sh` for symlinks. The plugin - loader falls back to it when something isn't found in `plugin-repos/` - (`src/plugin_system/schema_manager.py:77`). + loader does NOT fall back to it — `PluginManager.discover_plugins()` + (`src/plugin_system/plugin_manager.py`) scans only the configured + directory. Fallbacks exist in two narrower places: store operations + (`StoreManager._find_plugin_path()` in `store_manager.py`) and schema + lookup (`SchemaManager.get_schema_path()` in `schema_manager.py`, + which probes `plugins/` *before* `plugin-repos/`). ## Plugin System - Plugins inherit from `BasePlugin` in `src/plugin_system/base_plugin.py` @@ -20,6 +24,16 @@ - Plugin instantiation args: `plugin_id, config, display_manager, cache_manager, plugin_manager` - Config schemas use JSON Schema Draft-7 - Display dimensions: always read dynamically from `self.display_manager.matrix.width/height` +- Secrets: namespaced by plugin id in `config/config_secrets.json`, declared + via `"x-secret": true` in the plugin's config schema, and deep-merged into + the plugin's config dict at load time — plugins read them with plain + `config.get(...)`, never a separate accessor + +## Dev Workflow +- Link a plugin for development: `./scripts/dev/dev_plugin_setup.sh link-github ` (or `link `); symlinks land in `plugins/` — set `plugin_system.plugins_directory` to `plugins` so discovery picks them up +- Browser preview without the display loop: `python3 scripts/dev_server.py` → http://localhost:5001 +- Full display in emulator mode: `python3 run.py -e` (or `EMULATOR=true python3 run.py`) +- Validate one plugin headlessly: `python3 scripts/check_plugin.py --plugin ` ## Plugin Store Architecture - Official plugins live in the `ledmatrix-plugins` monorepo (not individual repos) @@ -33,7 +47,7 @@ ## Skin System (visual overlays for sports scoreboards) - Skins live in `skins//` (skin.json + skin.py), NOT in plugin dirs — plugin reinstall deletes plugin dirs -- Core: `src/skin_system/` (ScoreboardSkin, SkinContext, runtime); hook: `SportsCore._render_game()` in `src/base_classes/sports.py` +- Core: `src/skin_system/` (ScoreboardSkin, SkinContext, runtime); hook: `SportsCore._render_game()` in `src/base_classes/sports/core.py` - Skins render onto `ctx.canvas` only; fallback to built-in renderer on `False`/exception (3 strikes disables for session) - View-model guaranteed keys are frozen (see `test/test_skin_system.py::TestViewModelContract`) — renaming keys in `_extract_game_details_common` or sport extractors breaks published skins - Validate skins headlessly: `python scripts/validate_skin.py --skin `; docs: `docs/SKIN_SYSTEM.md`, `docs/CREATING_SKINS.md` @@ -42,4 +56,7 @@ ## Common Pitfalls - paho-mqtt 2.x needs `callback_api_version=mqtt.CallbackAPIVersion.VERSION1` for v1 compat - BasePlugin uses `get_logger()` from `src.logging_config`, not standard `logging.getLogger()` +- `DisplayManager` has no `draw_image()` — paste onto the PIL image directly: + `self.display_manager.image.paste(img, (x, y))` then `update_display()` + (use a mask for transparency: `image.paste(rgba, (x, y), rgba)`) - When modifying a plugin in the monorepo, you MUST bump `version` in its `manifest.json` and run `python update_registry.py` — otherwise users won't receive the update diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 613b51e1..a450585c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -40,7 +40,7 @@ improvements, and code changes. ## Running the tests ```bash -pip install -r requirements.txt +pip install -r requirements.txt -r requirements-test.txt pytest ``` @@ -57,9 +57,13 @@ integration tests. `docs/`. 3. **Keep PRs focused.** One conceptual change per PR. If you find adjacent bugs while working, fix them in a separate PR. -4. **Follow the existing code style.** Python code uses standard - `black`/`ruff` conventions; HTML/JS in `web_interface/` follows the - patterns already in `templates/v3/` and `static/v3/`. +4. **Follow the existing code style.** The pre-commit hooks run + `flake8` (E9, F63, F7, F82 plus bugbear `B` checks), `mypy` on + `src/`, `bandit`, and `gitleaks` — install the CLI with + `python -m pip install pre-commit`, then run + `pre-commit install` so they run on every commit; HTML/JS in + `web_interface/` follows the patterns already in `templates/v3/` + and `static/v3/`. 5. **Update documentation** alongside code changes. If you add a config key, document it in the relevant `*.md` file (or, for plugins, in `config_schema.json` so the form is auto-generated). diff --git a/README.md b/README.md index 0d50c38e..2d61b614 100644 --- a/README.md +++ b/README.md @@ -50,7 +50,15 @@ I'm trying to be open to constructive criticism and support, as long as it's a r
Core Features -The following plugins are available inside of the LEDMatrix project. These modular, rotating Displays that can be individually enabled or disabled per the user's needs with some configuration around display durations, teams, stocks, weather, timezones, and more. Displays include: +LEDMatrix is a plugin platform: the displays below are plugins installed +from the built-in Plugin Store (web interface → Plugins), where each can be +individually enabled, ordered, and configured — display durations, teams, +stocks, weather, timezones, and more. The core repo ships with just two +bundled plugins (`starlark-apps` and `web-ui-info`); the official plugins +live in the [ledmatrix-plugins](https://github.com/ChuckBuilds/ledmatrix-plugins) +monorepo and install with one click, and third-party plugins can be +installed from their own GitHub repositories. Displays available in the +store include: ### Time and Weather - Real-time clock display (2x 64x32 Displays 4mm Pixel Pitch) @@ -153,9 +161,11 @@ The system supports live, recent, and upcoming game information for multiple spo ### LED Matrix Panels (2x in a horizontal chain is recommended) - [Adafruit 64×32](https://www.adafruit.com/product/2278) – designed for 128×32 but works with dynamic scaling on many displays (pixel pitch is user preference) +**Warning: Lately the Waveshare Panels have had different variations - only some are compatible with this project. I hope to identify what is different to fix it but so far there is a decent chance you get a mis-matched set of panels if you don't buy them all at once! ** - [Waveshare 64×32](https://amzn.to/3Kw55jK) - Does not require E addressable pad -- [Waveshare 96×48](https://amzn.to/4bydNcv) – higher resolution, requires soldering the **E addressable pad** on the [Adafruit RGB Bonnet](https://www.adafruit.com/product/3211) to “8” **OR** toggling the DIP switch on the Adafruit Triple LED Matrix Bonnet *(no soldering required!)* - > Amazon Affiliate Link – ChuckBuilds receives a small commission on purchases +- [Waveshare 96×48](https://amzn.to/4bydNcv) – higher resolution, requires soldering the **E addressable pad** on the [Adafruit RGB Bonnet](https://www.adafruit.com/product/3211) to “8” **OR** toggling the DIP switch on the Adafruit Triple LED Matrix Bonnet *(no soldering required!)* +- There are some Panels on Aliexpress that have worked fine for me, shop around! I think Adafruit is probably the "safest" but they do have some limitation on resolution and layout. + > Amazon Affiliate Links – ChuckBuilds receives a small commission on purchases ### Power Supply - [5V 4A DC Power Supply](https://www.adafruit.com/product/658) (good for 2 -3 displays, depending on brightness and pixel density, you'll need higher amperage for more) @@ -357,6 +367,12 @@ sudo bash ./first_time_install.sh This single script installs services, dependencies, configures permissions and sudoers, and validates the setup. +It finishes by asking whether to reboot. If you run it non-interactively — piped, over a script, or with `-y` — there is no one to ask, so **it reboots immediately without prompting**. Pass `--no-reboot-prompt` to install without rebooting: + +```bash +sudo bash ./first_time_install.sh -y --no-reboot-prompt +``` +
@@ -372,6 +388,10 @@ This single script installs services, dependencies, configures permissions and s ### Initial Setup +For a complete list of every key in `config.json` and +`config_secrets.json`, see +[docs/CONFIG_REFERENCE.md](docs/CONFIG_REFERENCE.md). + For most settings I recommend using the web interface: Edit the project via the web interface at http://[IP ADDRESS or HOSTNAME]:5000 or http://ledpi:5000 . @@ -417,7 +437,7 @@ I recommend using the web-ui "Quick Actions" to control the Display. ## Plugins
-LEDMatrix uses a plugin-based architecture where all display functionality (except the core calendar) is implemented as plugins. All managers that were previously built into the core system are now available as plugins through the Plugin Store. +LEDMatrix uses a plugin-based architecture where all display functionality is implemented as plugins. All managers that were previously built into the core system are now available as plugins through the Plugin Store. ### Plugin Store See the [Plugin Store documentation](https://github.com/ChuckBuilds/ledmatrix-plugins) for detailed installation instructions. @@ -582,6 +602,14 @@ These settings are typically only needed for non-standard panels or custom confi - Leave empty unless you need custom mapping - See rpi-rgb-led-matrix documentation for full options +- **`orientation`** (string, default: "normal") + - Rotates the rendered image to match how the panel is physically mounted + - Set to `"180"` (or use the "Upside Down" option in the web UI's Display + settings) if the panel is mounted upside down — useful for optimizing + where the Raspberry Pi and wiring sit relative to the mounting location + - Applied independently of `pixel_mapper_config` (appended as a trailing + `Rotate:180` mapper), so custom mapper configs keep working alongside it + - **`row_address_type`** (integer, default: 0) - How rows are addressed on the panel - Most panels use 0 (direct addressing) @@ -610,12 +638,7 @@ These settings control runtime behavior and GPIO timing: ### Display Durations (`display.display_durations`) -Controls how long each display module stays visible in seconds before switching to the next one. - -- **`calendar`** (integer, default: 30) - - Duration in seconds for the calendar display - - Increase for more time to read dates/events - - Decrease to cycle through other displays faster +Controls how long each installed plugin stays visible in seconds before switching to the next one, keyed by plugin id. - **Plugin-specific durations** - Each plugin can have its own duration setting diff --git a/assets/README.md b/assets/README.md new file mode 100644 index 00000000..e88d7f0d --- /dev/null +++ b/assets/README.md @@ -0,0 +1,22 @@ +# assets/ + +Static assets bundled with LEDMatrix. **Do not delete these directories** — +several look unused from core code alone but are resolved at runtime by +installed store plugins. + +| Directory | Used by | +|---|---| +| `fonts/` | Core (`FontManager`, `DisplayManager`) and most plugins | +| `sports/` | Core logo tooling (`src/logo_downloader.py`) and the sports scoreboard plugins; team logos are downloaded here on demand | +| `stocks/` | `ledmatrix-stocks` plugin (`crypto_icons/`, `ticker_icons/`) | +| `weather/` | `ledmatrix-weather` plugin (weather icons) | +| `news_logos/` | `news` plugin | +| `broadcast_logos/` | `news` and `odds-ticker` plugins | +| `static_images/` | Legacy examples referenced in the `static-image` plugin's docs; the plugin itself stores uploads under `assets/plugins//uploads/` | +| `plugins/` | Per-plugin uploaded files (`assets/plugins//uploads/`), served by the web interface | + +Plugins resolve these paths relative to the LEDMatrix install directory, so +the directories are part of the de-facto plugin API even where no file in +this repo references them. New plugins should bundle their own assets or +use the per-plugin upload directory instead of adding top-level +directories here. diff --git a/assets/sports/mlb_logos/mlb.png b/assets/sports/mlb_logos/mlb.png deleted file mode 100644 index 7196811c..00000000 Binary files a/assets/sports/mlb_logos/mlb.png and /dev/null differ diff --git a/assets/sports/nba_logos/nba.png b/assets/sports/nba_logos/nba.png deleted file mode 100644 index 6738f818..00000000 Binary files a/assets/sports/nba_logos/nba.png and /dev/null differ diff --git a/assets/sports/ncaa_logos/COR.png b/assets/sports/ncaa_logos/COR.png new file mode 100644 index 00000000..d39027fa Binary files /dev/null and b/assets/sports/ncaa_logos/COR.png differ diff --git a/assets/sports/nfl_logos/nfl.png b/assets/sports/nfl_logos/nfl.png deleted file mode 100644 index 49f71d91..00000000 Binary files a/assets/sports/nfl_logos/nfl.png and /dev/null differ diff --git a/assets/sports/nhl_logos/nhl.png b/assets/sports/nhl_logos/nhl.png deleted file mode 100644 index eaebd594..00000000 Binary files a/assets/sports/nhl_logos/nhl.png and /dev/null differ diff --git a/config/config.template.json b/config/config.template.json index f6330567..95ea0863 100644 --- a/config/config.template.json +++ b/config/config.template.json @@ -110,7 +110,12 @@ "inverse_colors": false, "show_refresh_rate": false, "led_rgb_sequence": "RGB", - "limit_refresh_rate_hz": 100 + "limit_refresh_rate_hz": 100, + "pixel_mapper_config": "", + "orientation": "normal", + "row_address_type": 0, + "multiplexing": 0, + "panel_type": "" }, "runtime": { "gpio_slowdown": 3, @@ -125,6 +130,9 @@ "plugin_rotation_order": [], "use_short_date_format": true, "vegas_scroll": { + "live_in_ticker": false, + "live_weight": 3, + "favorite_live_weight": 5, "enabled": false, "scroll_speed": 50, "separator_width": 32, @@ -145,11 +153,13 @@ "min_plugin_width": 8, "lead_in_width": 0, "plugins_per_cycle": 6, - "max_plugin_width_ratio": 3.0, + "max_plugin_width_ratio": 0.0, "overflow_mode": "rotate", "dynamic_duration_enabled": true, "min_cycle_duration": 60, - "max_cycle_duration": 240 + "max_cycle_duration": 240, + "frame_based_scrolling": true, + "scroll_delay": 0.02 } }, "sync": { @@ -160,7 +170,8 @@ "plugin_system": { "plugins_directory": "plugin-repos", "auto_discover": true, - "auto_load_enabled": true + "auto_load_enabled": true, + "development_mode": false }, "web-ui-info": { "enabled": true, diff --git a/config/config_secrets.template.json b/config/config_secrets.template.json index 61909876..cbf60336 100644 --- a/config/config_secrets.template.json +++ b/config/config_secrets.template.json @@ -1,9 +1,5 @@ { - "youtube": { - "api_key": "YOUR_YOUTUBE_API_KEY", - "channel_id": "YOUR_YOUTUBE_CHANNEL_ID" - }, "github": { "api_token": "YOUR_GITHUB_PERSONAL_ACCESS_TOKEN" } -} \ No newline at end of file +} diff --git a/docs/ADVANCED_FEATURES.md b/docs/ADVANCED_FEATURES.md index ec94df6c..d620fef7 100644 --- a/docs/ADVANCED_FEATURES.md +++ b/docs/ADVANCED_FEATURES.md @@ -47,6 +47,11 @@ Enable Vegas mode in `config/config.json`: } ``` +Vegas mode can also be configured entirely from the web UI — the +**Display** tab has a Vegas Scroll Mode section (enable toggle, scroll +speed, separator width, dynamic duration, and more), so hand-editing +JSON is optional. + **Configuration Options:** | Setting | Default | Description | @@ -57,7 +62,99 @@ Enable Vegas mode in `config/config.json`: | `plugin_order` | `[]` | Plugin display order (empty = auto) | | `excluded_plugins` | `[]` | Plugins to exclude from Vegas mode | | `target_fps` | `125` | Target frame rate | -| `buffer_ahead` | `2` | Number of panels to render ahead | +| `buffer_ahead` | `2` | Number of plugins buffered ahead | + +This table is a subset — `display.vegas_scroll` supports 30 keys in +total. See the full list in +[CONFIG_REFERENCE.md](CONFIG_REFERENCE.md#displayvegas_scroll--continuous-scroll-mode). + +### Live Content in the Ticker + +By default, live content **preempts** Vegas mode: while any plugin reports +live priority, the display controller refuses to run the ticker and shows +that plugin's full-screen display instead. You get a big readable scoreboard, +but the marquee stops entirely for the duration of the game. + +Set `live_in_ticker` to keep the ticker running and let live content take +**extra turns inside it** instead: + +```json +"vegas_scroll": { + "live_in_ticker": true, + "live_weight": 3, + "favorite_live_weight": 5 +} +``` + +#### Why weights exist + +The rotation is otherwise a strict round robin — every plugin appears exactly +once per cycle. With a dozen plugins enabled, a live score comes round once a +lap and can be minutes old by the time you see it. A weight of *N* gives a +plugin *N* slots per cycle. + +The slots are placed by **Smooth Weighted Round-Robin**, the same scheduler +the sports plugins use internally to rotate their own games. The important +property is that repeats are *spread through the cycle* rather than clumped: +three appearances in a row followed by a long silence would be worse than not +boosting at all. + +Twelve plugins, with a favorite's baseball game and an ordinary live hockey +game (`live_weight: 3`, `favorite_live_weight: 5`): + +``` +baseball > hockey > weather > clock > baseball +stocks > news > flights > baseball > hockey +calendar > f1 > music > baseball > tides +birds > hockey > baseball +``` + +18 slots for 12 plugins. Baseball appears 5 times, hockey 3, everything else +once, and no plugin ever appears twice in a row — **including across the seam** +where the cycle loops back on itself. Smooth Weighted Round-Robin schedules the +heaviest item first and usually last as well, so the strip would otherwise show +it twice running at exactly the one join a within-cycle check cannot see. The +trailing repeat is moved into the widest remaining gap. Where a double is +unavoidable — a plugin holding most of the slots has to neighbour itself — the +schedule is left as it is. + +#### Where the weight comes from + +For each plugin in the rotation, in order: + +1. **The plugin's own answer.** If it implements + `get_vegas_priority_weight()` and returns a number, that wins. This is the + only route for favorite-team awareness — the core can see *that* a game is + live, but not *whose*, so a scoreboard has to say so itself. +2. **The core's default.** When the plugin returns `None` (the base-class + default), a plugin where both `has_live_priority()` and `has_live_content()` + are true gets `live_weight`. +3. **Everything else** gets 1. + +Because of step 2, **existing plugins need no changes** — any scoreboard with +`live_priority` enabled already gets extra turns. Step 1 is opt-in, for +plugins that want to distinguish a favorite's game from any other live game. + +Weights are clamped to 1–10. A weight of 1 is no boost; a weight below 1 would +drop the plugin from the rotation entirely, which is never what is meant. + +#### Things worth knowing + +- **Weights are per plugin, not per game.** A scoreboard showing four live + games still occupies one slot at a time, rotating its own games within that + slot using its own `favorite_live_boost`. This controls how often the + *plugin* comes round. +- **The ticker is zero-sum.** Giving baseball 5 slots does not make the cycle + faster; it makes the cycle *longer* and everything else proportionally + rarer. If you want live scores sooner in wall-clock terms, pair this with a + smaller `plugins_per_cycle`. +- **Frequency is not freshness.** Each appearance redraws from the plugin's + current data (`refresh_updated_plugins()` drops cached content when a + plugin's data changes), but how current that data is depends on the + plugin's own `live_update_interval`. Showing a stale score five times a lap + is no better than showing it once. +- **Everything still appears.** A boost never starves another plugin out of + the cycle; low-weight plugins keep their single slot. ### Per-Plugin Configuration @@ -79,9 +176,13 @@ Override Vegas behavior for specific plugins: | Setting | Values | Description | |---------|--------|-------------| | `vegas_mode` | `scroll`, `fixed`, `static` | Display mode for this plugin | -| `vegas_panel_count` | `1-10` | Width in panels (1 panel = display width) | +| `vegas_panel_count` | any positive integer | Width in panels (1 panel = display width) | | `display_duration` | seconds | Pause duration for STATIC mode | +Plugins may also set `vegas_overflow` and `vegas_max_width_screens` in +their config section to control how oversized content is handled (see +`PluginManager` in `src/plugin_system/plugin_manager.py`). + ### Plugin Integration (Developer Guide) **1. Implement Content Method:** @@ -451,7 +552,7 @@ time when something is active. ### REST API Reference -The API is mounted at `/api/v3` (`web_interface/app.py:144`). +The API is mounted at `/api/v3` (`web_interface/app.py:199`). #### Start On-Demand Display @@ -518,13 +619,15 @@ curl http://localhost:5000/api/v3/display/on-demand/status > There is no public Python on-demand API. The display controller's > on-demand machinery is internal — drive it through the REST endpoints -> above (or the web UI buttons), which write a request into the cache -> manager under the `display_on_demand_request` key -> (`web_interface/blueprints/api_v3.py:1622,1687`) that the controller -> polls at `src/display_controller.py:921`. A separate +> above (or the web UI buttons). The API handlers +> (`start_on_demand_display()` / `stop_on_demand_display()` in +> `web_interface/blueprints/api_v3.py`) write a request into the cache +> manager under the `display_on_demand_request` key, which +> `DisplayController._poll_on_demand_requests()` +> (`src/display_controller.py`) picks up. A separate > `display_on_demand_config` key is used by the controller itself -> during activation to track what's currently running (written at -> `display_controller.py:1195`, cleared at `:1221`). +> during activation (`_activate_on_demand()`) to track what's +> currently running, and is cleared by `_clear_on_demand()`. ### Duration Modes @@ -646,13 +749,13 @@ keys helps troubleshoot stuck states. **When Set:** Every display loop iteration **Auto-Cleared:** Never (continuously updated) -**4. display_on_demand_processed_id** (TTL: 5 minutes) -``` +**4. display_on_demand_processed_id** (TTL: 1 hour) +```text "uuid-string-of-last-processed-request" ``` **Purpose:** Prevents duplicate request processing **When Set:** After processing request -**Auto-Cleared:** After 5 minutes TTL +**Auto-Cleared:** After 1 hour TTL ### When Manual Clearing is Needed @@ -685,9 +788,9 @@ keys helps troubleshoot stuck states. The cache is stored as JSON files under one of: - `/var/cache/ledmatrix/` (preferred when the service has permission) -- `~/.cache/ledmatrix/` +- `~/.ledmatrix_cache/` - `/opt/ledmatrix/cache/` -- `/tmp/ledmatrix-cache/` (fallback) +- `$TMPDIR/ledmatrix_cache/` (fallback) ```bash # Find the cache dir actually in use @@ -711,8 +814,9 @@ cache.clear_cache('display_on_demand_request') cache.clear_cache('display_on_demand_processed_id') ``` -> The actual public method is `clear_cache(key=None)` — there is no -> `delete()` method on `CacheManager`. +> `CacheManager` also has a `delete(key)` method — a thin wrapper over +> `clear_cache(key)` — so `cache.delete('display_on_demand_config')` +> works equally well. ### Cache Impact on Running Service @@ -730,7 +834,7 @@ The display controller automatically handles cleanup: - **Config key**: Cleared when on-demand stops - **State key**: Updated every display loop iteration - **Request key**: Expires after 1 hour TTL (or after processing) -- **Processed ID**: Expires after 5 minutes TTL +- **Processed ID**: Expires after 1 hour TTL --- @@ -821,9 +925,6 @@ same shape as the example above. ### Testing ```bash -# Run background service test -python test_background_service.py - # Check logs for background operations sudo journalctl -u ledmatrix -f | grep "background" ``` @@ -832,9 +933,10 @@ sudo journalctl -u ledmatrix -f | grep "background" **View Statistics:** ```python -from src.background_data_service import BackgroundDataService +from src.background_data_service import get_background_service +from src.cache_manager import CacheManager -service = BackgroundDataService() +service = get_background_service(CacheManager()) stats = service.get_statistics() print(f"Active tasks: {stats['active_tasks']}") print(f"Completed: {stats['completed']}") @@ -875,6 +977,7 @@ from src.common.permission_utils import ( ensure_file_permissions, get_config_file_mode, get_assets_file_mode, + get_assets_dir_mode, get_plugin_file_mode, get_cache_dir_mode ) @@ -883,7 +986,10 @@ from src.common.permission_utils import ( ensure_directory_permissions(Path("assets/sports"), get_assets_dir_mode()) # Set file permissions after writing -ensure_file_permissions(Path("config/config.json"), get_config_file_mode()) +# (get_config_file_mode requires the file path — secrets files get a +# stricter mode than the main config) +config_path = Path("config/config.json") +ensure_file_permissions(config_path, get_config_file_mode(config_path)) ``` ### When to Use Utilities @@ -938,7 +1044,7 @@ from src.common.permission_utils import ensure_file_permissions, get_config_file config_path = Path("config/config.json") with open(config_path, 'w') as f: json.dump(data, f) -ensure_file_permissions(config_path, get_config_file_mode()) +ensure_file_permissions(config_path, get_config_file_mode(config_path)) ``` **Pattern 3: Downloading Logo** @@ -984,8 +1090,11 @@ These core utilities **already handle permissions** - you don't need to call per If you encounter permission issues: ```bash -# Fix all permissions at once -sudo ./scripts/fix_permissions.sh +# Targeted permission fixes (see scripts/fix_perms/README.md) +sudo ./scripts/fix_perms/fix_assets_permissions.sh # assets/ tree (logos, fonts) +sudo ./scripts/fix_perms/fix_cache_permissions.sh # all cache directories +sudo ./scripts/fix_perms/fix_plugin_permissions.sh # plugin directories +sudo ./scripts/fix_perms/fix_web_permissions.sh # web interface files # Fix specific directory sudo chown -R ledpi:ledpi /home/ledpi/LEDMatrix/config @@ -1017,7 +1126,7 @@ stat -c "%a %n" config/config.json ## Related Documentation -- [PLUGIN_DEVELOPMENT.md](PLUGIN_DEVELOPMENT.md) - Creating plugins with Vegas/on-demand support +- [PLUGIN_DEVELOPMENT_GUIDE.md](PLUGIN_DEVELOPMENT_GUIDE.md) - Creating plugins with Vegas/on-demand support - [WEB_INTERFACE_GUIDE.md](WEB_INTERFACE_GUIDE.md) - Using on-demand controls in web UI - [PLUGIN_API_REFERENCE.md](PLUGIN_API_REFERENCE.md) - Complete API documentation - [DEVELOPMENT.md](DEVELOPMENT.md) - Development environment and testing diff --git a/docs/CONFIG_REFERENCE.md b/docs/CONFIG_REFERENCE.md new file mode 100644 index 00000000..4af3c0fb --- /dev/null +++ b/docs/CONFIG_REFERENCE.md @@ -0,0 +1,177 @@ +# Configuration Reference + +Every key in `config/config.json`, what it does, its default, and where the +code reads it. The file is created from `config/config.template.json` on +first run, and `ConfigManager._migrate_config()` merges any template keys +added by later releases into your existing config (your values are never +overwritten). Secrets live in `config/config_secrets.json` and are merged +into the config at load time. + +Most settings are editable from the web interface; this page documents the +underlying keys for people editing `config.json` directly or writing +tooling against it. + +## Top level + +| Key | Type / default | Meaning | Read by | +|---|---|---|---| +| `web_display_autostart` | bool, `true` | Whether the web interface service starts with the system | `scripts/utils/start_web_conditionally.py` | +| `timezone` | string, `"America/New_York"` | IANA timezone for schedules and displays | `ConfigManager.get_timezone()` | +| `target_fps` | int, `100` | Frame-rate ceiling for plugin rendering | `src/plugin_system/base_plugin.py`, `src/common/sports_scroll.py` | +| `location` | object | `city` / `state` / `country`. Supplies the **default** for a plugin's own `location_city` / `location_state` / `location_country` setting, so weather, radar and friends follow this device without being configured twice. A value saved on the plugin itself still overrides it. | `SchemaManager.apply_device_location()`, then plugins via merged config | + +## `schedule` — display on/off hours + +| Key | Type / default | Meaning | +|---|---|---| +| `enabled` | bool, `false` | Master switch for scheduled display on/off | +| `mode` | `"global"` or `"per-day"`, template uses `"per-day"` | Whether one time range applies to all days or each day has its own | +| `start_time` / `end_time` | `"HH:MM"`, `07:00`–`23:00` | Global-mode on/off times | +| `days..{enabled,start_time,end_time}` | per-day objects | Per-day-mode overrides | + +Read by `DisplayController` (`src/display_controller.py`, `_check_schedule` +around line 603). Managed in the web UI under Schedule. + +## `dim_schedule` — scheduled brightness dimming + +Same shape as `schedule`, plus: + +| Key | Type / default | Meaning | +|---|---|---| +| `dim_brightness` | int, `30` | Brightness percentage applied while the dim window is active | + +Read by `DisplayController` (`src/display_controller.py` around line 770; +saved via `POST /api/v3/config/dim-schedule`). The display returns to +`display.hardware.brightness` outside the window. + +## `display.hardware` — matrix panel hardware + +All keys map to the corresponding `rpi-rgb-led-matrix` options and are read +in `DisplayManager` (`src/display_manager.py`, ~lines 270–295). + +| Key | Type / default | +|---|---| +| `rows` / `cols` | int, `32` / `64` | +| `chain_length` | int, `2` | +| `parallel` | int, `1` | +| `brightness` | int, `90` | +| `hardware_mapping` | string, `"adafruit-hat"` (code default `"adafruit-hat-pwm"`) | +| `scan_mode` | int, `0` | +| `pwm_bits` | int, `9` (code default 10) | +| `pwm_dither_bits` | int, `1` | +| `pwm_lsb_nanoseconds` | int, `130` (code default 150) | +| `disable_hardware_pulsing` | bool, `false` | +| `inverse_colors` | bool, `false` | +| `show_refresh_rate` | bool, `false` | +| `led_rgb_sequence` | string, `"RGB"` | +| `limit_refresh_rate_hz` | int, `100` (code default 90) | +| `pixel_mapper_config` | string, `""` — e.g. `"U-mapper"` / `"Rotate:90"` | +| `orientation` | string, `"normal"` — `"180"` rotates the rendered image 180° for panels physically mounted upside down (e.g. to move the Pi/wiring to a more convenient side); composed onto `pixel_mapper_config` as a trailing `Rotate:180` mapper, so it stays independent of any custom `pixel_mapper_config` value | +| `row_address_type` | int, `0` — non-standard panel row addressing | +| `multiplexing` | int, `0` — panel multiplexing scheme | +| `panel_type` | string, `""` — set to `"FM6126A"` or `"FM6127"` for panels needing init | + +Where "code default" differs from the template value, the code default only +applies if the key is missing entirely from your config. + +## `display.runtime` + +| Key | Type / default | Meaning | +|---|---|---| +| `gpio_slowdown` | int, `3` | GPIO timing slowdown for faster Pis | +| `rp1_rio` | int, `0` | RP1 RIO mode on Pi 5 (applied only if the installed matrix library supports it) | + +## `display.double_sided` + +Drives `_LogicalMatrix` in `src/display_manager.py` — renders the same +logical image to multiple chained physical panels. + +| Key | Type / default | Meaning | +|---|---|---| +| `enabled` | bool, `false` | Mirror output across panel copies | +| `copies` | int, `2` | Number of physical copies in the chain | +| `axis` | `"horizontal"`, default | Axis along which panels are chained | + +## `display` — other keys + +| Key | Type / default | Meaning | Read by | +|---|---|---|---| +| `display_durations` | object, `{}` | Per-plugin display duration in seconds, keyed by plugin id (e.g. `"clock": 15`) | `src/display_controller.py:1030` | +| `plugin_rotation_order` | array, `[]` | Explicit rotation order of plugin ids; empty = all enabled plugins in discovery order | `src/display_controller.py:2894` | +| `use_short_date_format` | bool, `true` | Compact date rendering in sports scoreboards | `src/base_classes/sports/core.py` | +| `dynamic_duration.max_duration_seconds` | int, optional | Cap for plugins that request dynamic display time | `src/display_controller.py:405` | + +## `display.vegas_scroll` — continuous scroll mode + +Read by `src/vegas_mode/config.py` (`VegasScrollConfig.from_config`). See +[ADVANCED_FEATURES.md](ADVANCED_FEATURES.md) for behavior details, including +[live content in the ticker](ADVANCED_FEATURES.md#live-content-in-the-ticker). + +| Key | Type / default | +|---|---| +| `enabled` | bool, `false` | +| `scroll_speed` | int, `50` (px/s) | +| `separator_width` | int, `32` | +| `plugin_order` | array, `[]` | +| `excluded_plugins` | array, `[]` | +| `target_fps` | int, `125` | +| `buffer_ahead` | int, `2` | +| `intra_plugin_gap` | int, `8` | +| `render_width_pct` | int, `100` | +| `min_content_separation` | int, `24` | +| `min_cut_gap` | int, `6` | +| `continuous_scroll` | bool, `true` | +| `smooth_scroll` | bool, `true` | +| `extend_threshold_screens` | float, `2.0` | +| `auto_trim` | bool, `true` | +| `trim_threshold` | int, `10` | +| `content_padding` | int, `8` | +| `min_plugin_width` | int, `8` | +| `lead_in_width` | int, `0` | +| `plugins_per_cycle` | int, `6` | +| `max_plugin_width_ratio` | float, `0.0` | +| `overflow_mode` | string, `"rotate"` | +| `dynamic_duration_enabled` | bool, `true` | +| `min_cycle_duration` | int, `60` | +| `max_cycle_duration` | int, `240` | +| `frame_based_scrolling` | bool, `true` — frame-count-based scroll stepping | +| `scroll_delay` | float, `0.02` — seconds between scroll updates (~50 FPS) | +| `live_in_ticker` | bool, `false` — keep scrolling during live games instead of handing the display to a full-screen scoreboard | +| `live_weight` | int, `3` (1–10) — slots per cycle for a plugin with live content | +| `favorite_live_weight` | int, `5` (1–10) — slots per cycle when a plugin reports a favorite team is live | + +## `sync` — multi-display synchronization + +Read by `src/common/sync_manager.py` and `src/display_controller.py`. + +| Key | Type / default | Meaning | +|---|---|---| +| `role` | `"standalone"` (default), `"leader"`, or `"follower"` | This device's role in a synced pair | +| `port` | int, `5765` | TCP port used for sync traffic | +| `follower_position` | `"left"` (default) or `"right"` | Which half of the combined image this follower renders (`src/display_controller.py:522`) | + +## `plugin_system` + +Read by the plugin loader/manager (`src/plugin_system/`). + +| Key | Type / default | Meaning | +|---|---|---| +| `plugins_directory` | string, `"plugin-repos"` | Where the Plugin Store installs plugins | +| `auto_discover` | bool, `true` | Scan the plugins directory at startup | +| `auto_load_enabled` | bool, `true` | Load discovered plugins automatically | +| `development_mode` | bool, `false` | Development conveniences in the web UI (editable under General settings) | + +## Plugin config blocks + +Every installed plugin stores its settings under a top-level key equal to +its plugin id (the template ships one for the bundled `web-ui-info` +plugin). The shape of each block is defined by that plugin's +`config_schema.json`; common keys are `enabled` and `display_duration`. +See [PLUGIN_CONFIG_CORE_PROPERTIES.md](PLUGIN_CONFIG_CORE_PROPERTIES.md). + +## `config/config_secrets.json` + +| Key | Meaning | +|---|---| +| `github.api_token` | Optional GitHub token the Plugin Store uses to avoid API rate limits (`src/plugin_system/store_manager.py:348`) | +| `.*` | Secrets a plugin declares with `"x-secret": true` in its config schema; merged into that plugin's config at load time | diff --git a/docs/DEVELOPER_QUICK_REFERENCE.md b/docs/DEVELOPER_QUICK_REFERENCE.md index 93ca8e16..7eefb0c9 100644 --- a/docs/DEVELOPER_QUICK_REFERENCE.md +++ b/docs/DEVELOPER_QUICK_REFERENCE.md @@ -31,7 +31,7 @@ POST /api/v3/system/action **Base URL**: `http://your-pi-ip:5000/api/v3` -See [API_REFERENCE.md](API_REFERENCE.md) for complete documentation. +See [REST_API_REFERENCE.md](REST_API_REFERENCE.md) for complete documentation. ## Display Manager Quick Methods @@ -190,12 +190,13 @@ def display(self, force_clear=False): ``` LEDMatrix/ -├── plugins/ # Installed plugins +├── plugin-repos/ # Installed plugins (default; plugins/ is only +│ # for dev symlinks via scripts/dev/dev_plugin_setup.sh) ├── config/ │ ├── config.json # Main configuration │ └── config_secrets.json # API keys and secrets ├── docs/ # Documentation -│ ├── API_REFERENCE.md +│ ├── REST_API_REFERENCE.md │ ├── PLUGIN_API_REFERENCE.md │ └── ... └── src/ @@ -207,7 +208,7 @@ LEDMatrix/ ## Quick Links -- [Complete API Reference](API_REFERENCE.md) +- [Complete REST API Reference](REST_API_REFERENCE.md) - [Plugin API Reference](PLUGIN_API_REFERENCE.md) - [Plugin Development Guide](PLUGIN_DEVELOPMENT_GUIDE.md) - [Advanced Patterns](ADVANCED_PLUGIN_DEVELOPMENT.md) diff --git a/docs/EMULATOR_SETUP_GUIDE.md b/docs/EMULATOR_SETUP_GUIDE.md index e30c3063..d5130ccf 100644 --- a/docs/EMULATOR_SETUP_GUIDE.md +++ b/docs/EMULATOR_SETUP_GUIDE.md @@ -69,23 +69,24 @@ default configuration as it ships in the repo: ```json { "pixel_outline": 0, - "pixel_size": 5, + "pixel_size": 16, "pixel_style": "square", "pixel_glow": 6, - "display_adapter": "pygame", + "display_adapter": "browser", + "allow_adapter_fallback": true, "icon_path": null, "emulator_title": null, "suppress_font_warnings": false, - "suppress_adapter_load_errors": false, "browser": { "_comment": "For use with the browser adapter only.", "port": 8888, - "target_fps": 24, + "target_fps": 60, "fps_display": false, "quality": 70, "image_border": true, "debug_text": false, - "image_format": "JPEG" + "image_format": "JPEG", + "open_immediately": false }, "log_level": "info" } @@ -96,13 +97,13 @@ default configuration as it ships in the repo: | Option | Description | Default | Values | |--------|-------------|---------|--------| | `pixel_outline` | Pixel border thickness | 0 | 0-5 | -| `pixel_size` | Size of each pixel | 5 | 1-64 (8–16 is typical for testing) | +| `pixel_size` | Size of each pixel | 16 | 1-64 (8–16 is typical for testing) | | `pixel_style` | Pixel shape | "square" | "square", "circle" | | `pixel_glow` | Glow effect intensity | 6 | 0-20 | -| `display_adapter` | Display backend | "pygame" | "pygame", "browser" | +| `display_adapter` | Display backend | "browser" | "browser", "pygame" | +| `allow_adapter_fallback` | Fall back to another adapter if the configured one fails to load | true | true/false | | `emulator_title` | Window title | null | Any string | | `suppress_font_warnings` | Hide font warnings | false | true/false | -| `suppress_adapter_load_errors` | Hide adapter errors | false | true/false | ### 3. Browser Adapter Configuration @@ -111,18 +112,32 @@ When using the browser adapter, additional options are available: | Option | Description | Default | |--------|-------------|---------| | `port` | Web server port | 8888 | -| `target_fps` | Target frames per second | 24 | +| `target_fps` | Target frames per second | 60 | | `fps_display` | Show FPS counter | false | | `quality` | Image compression quality | 70 | | `image_border` | Show image border | true | | `debug_text` | Show debug information | false | | `image_format` | Image format | "JPEG" | +| `open_immediately` | Open the browser page automatically on start | false | ## Running the Emulator -### 1. Set Environment Variable +### 1. Use the `-e` Flag (Recommended) -Enable emulator mode by setting the `EMULATOR` environment variable: +`run.py` accepts exactly two flags: `-e`/`--emulator` and +`-d`/`--debug`. + +```bash +python3 run.py -e + +# With verbose logging +python3 run.py -e -d +``` + +### 2. Alternative: Set the Environment Variable + +You can also enable emulator mode via the `EMULATOR` environment +variable: **Windows (Command Prompt):** ```cmd @@ -137,15 +152,6 @@ python run.py ``` **Linux/macOS:** -```bash -export EMULATOR=true -python3 run.py -``` - -### 2. Alternative: Direct Python Execution - -You can also run the emulator directly: - ```bash EMULATOR=true python3 run.py ``` @@ -153,7 +159,8 @@ EMULATOR=true python3 run.py ### 3. Verify Emulator Mode When running in emulator mode, you should see: -- A window displaying the LED matrix simulation +- The emulated matrix — a web page at `http://localhost:8888` with the + default browser adapter, or a desktop window with the pygame adapter - Console output indicating emulator mode - No hardware initialization errors @@ -161,7 +168,36 @@ When running in emulator mode, you should see: LEDMatrix supports two display adapters for the emulator: -### 1. Pygame Adapter (Default) +### 1. Browser Adapter (Default) + +The browser adapter runs a web server and displays the matrix as a web +page at `http://localhost:8888`. This is the adapter the shipped +`emulator_config.json` uses. + +**Features:** +- Web-based interface +- Remote access capability +- Mobile-friendly +- Screenshot capture + +**Configuration:** +```json +{ + "display_adapter": "browser", + "browser": { + "port": 8888, + "target_fps": 60, + "quality": 70 + } +} +``` + +**Usage:** +1. Start the emulator (`python3 run.py -e`) +2. Open browser to `http://localhost:8888` +3. View the LED matrix display + +### 2. Pygame Adapter (Alternative) The pygame adapter provides a native desktop window with real-time display. @@ -186,33 +222,6 @@ The pygame adapter provides a native desktop window with real-time display. - `+/-` - Zoom in/out - `R` - Reset zoom -### 2. Browser Adapter - -The browser adapter runs a web server and displays the matrix in a web browser. - -**Features:** -- Web-based interface -- Remote access capability -- Mobile-friendly -- Screenshot capture - -**Configuration:** -```json -{ - "display_adapter": "browser", - "browser": { - "port": 8888, - "target_fps": 24, - "quality": 70 - } -} -``` - -**Usage:** -1. Start the emulator with browser adapter -2. Open browser to `http://localhost:8888` -3. View the LED matrix display - ## Troubleshooting ### Common Issues @@ -274,8 +283,7 @@ Enable debug logging: ```json { "log_level": "debug", - "suppress_font_warnings": false, - "suppress_adapter_load_errors": false + "suppress_font_warnings": false } ``` @@ -299,17 +307,18 @@ Modify the display dimensions in your main config: ### 2. Plugin Development -For plugin development with the emulator: +`run.py` always runs the full rotation — it has no single-plugin flag. +To preview or check one plugin in isolation, use the dev tools: ```bash -# Enable emulator mode -export EMULATOR=true +# Run the full display in emulator mode (optionally with debug logging) +python3 run.py -e -d -# Run with specific plugin -python run.py --plugin my-plugin +# Live single-plugin preview in the browser (port 5001) +python3 scripts/dev_server.py -# Debug mode -python run.py --debug +# Headless render/validation of one plugin +python3 scripts/check_plugin.py --plugin my-plugin ``` ### 3. Performance Tuning @@ -344,11 +353,10 @@ The emulator can work alongside the web interface: ```bash # Terminal 1: Start emulator -export EMULATOR=true -python run.py +python3 run.py -e -# Terminal 2: Start web interface -python web_interface/app.py +# Terminal 2: Start web interface (supported entry point) +python3 web_interface/start.py ``` Access the web interface at `http://localhost:5000` while the emulator runs. @@ -365,13 +373,14 @@ Access the web interface at `http://localhost:5000` while the emulator runs. ### 2. Plugin Testing ```bash -# Test specific plugin -export EMULATOR=true -python run.py --plugin clock-simple +# Test a specific plugin (headless check) +python3 scripts/check_plugin.py --plugin clock-simple -# Test all plugins -export EMULATOR=true -python run.py --test-plugins +# Preview a single plugin live in the browser (port 5001) +python3 scripts/dev_server.py + +# Test the full rotation in the emulator +python3 run.py -e ``` ### 3. Configuration Management @@ -385,9 +394,8 @@ python run.py --test-plugins ### Basic Clock Display ```bash -# Start emulator with clock -export EMULATOR=true -python run.py +# Start emulator with clock enabled in config.json +python3 run.py -e ``` ### Sports Scores @@ -395,16 +403,16 @@ python run.py ```bash # Configure for sports display # Edit config/config.json to enable sports plugins -export EMULATOR=true -python run.py +python3 run.py -e ``` ### Custom Text Display ```bash -# Use text display plugin -export EMULATOR=true -python run.py --plugin text-display --text "Hello World" +# Preview the text display plugin on its own +python3 scripts/check_plugin.py --plugin text-display +# or use the live dev preview server +python3 scripts/dev_server.py ``` ## Support diff --git a/docs/GETTING_STARTED.md b/docs/GETTING_STARTED.md index de6e15b6..d0102422 100644 --- a/docs/GETTING_STARTED.md +++ b/docs/GETTING_STARTED.md @@ -21,18 +21,30 @@ This guide will help you set up your LEDMatrix display for the first time and ge --- -## Quick Start (5 Minutes) +## Quick Start -### 1. First Boot +### 1. Install LEDMatrix -1. Insert the MicroSD card with LEDMatrix installed -2. Connect the LED matrix to your Raspberry Pi -3. Plug in the power supply -4. Wait for the Pi to boot (about 60 seconds) +There is no prebuilt SD card image — you install LEDMatrix onto stock +Raspberry Pi OS Lite yourself: -**Expected Behavior:** +1. Flash Raspberry Pi OS Lite to the MicroSD card (Raspberry Pi Imager) +2. Connect the LED matrix to your Raspberry Pi, insert the card, and + power on +3. SSH into the Pi and run the one-shot installer: + ```bash + curl -fsSL https://raw.githubusercontent.com/ChuckBuilds/LEDMatrix/main/scripts/install/one-shot-install.sh | bash + ``` + or clone the repo and run `sudo ./first_time_install.sh` — see the + [README Installation Steps / Quick Install](../README.md#installation-steps) + for full details + +**Expected Behavior after install:** - LED matrix will light up -- Display will show default plugins (clock, weather, etc.) +- A fresh install ships only the bundled `starlark-apps` and + `web-ui-info` plugins — clock, weather, sports, etc. must be + installed from the Plugin Store (web UI → Plugin Manager) before + anything else displays - Pi creates WiFi network "LEDMatrix-Setup" if not connected ### 2. Connect to WiFi @@ -73,7 +85,7 @@ You should see: 2. Set your matrix configuration: - **Rows**: 32 or 64 (match your hardware) - **Columns**: commonly 64 or 96; the web UI accepts any integer - in the 16–128 range, but 64 and 96 are the values the bundled + in the 1–128 range, but 64 and 96 are the values the bundled panel hardware ships with - **Chain Length**: Number of panels chained horizontally - **Hardware Mapping**: usually `adafruit-hat-pwm` (with the PWM jumper @@ -115,11 +127,16 @@ You can also install community plugins straight from a GitHub URL using the 1. Each installed plugin gets its own tab in the second navigation row 2. Open that plugin's tab to edit its settings (favorite teams, API keys, - update intervals, display duration, etc.) + update intervals, etc.) 3. Click **Save** 4. Restart the display service from **Overview** so the new settings take effect +**Note:** how long each plugin stays on screen is not set in the +plugin's own tab — use the **Rotation** tab's **Screen Durations** +section instead (saved to `display.display_durations` in +`config.json`). + **Example: Weather Plugin** - Set your location (city, state, country) - Add an API key from OpenWeatherMap (free signup) to @@ -208,12 +225,14 @@ The fastest way to verify a plugin works without waiting for the rotation: ### Customize Your Display **Adjust display durations:** -- Each plugin's tab has a **Display Duration (seconds)** field — set how - long that plugin stays on screen each rotation. +- Open the **Rotation** tab and use the **Screen Durations** section to + set how long each plugin stays on screen per rotation (saved to + `display.display_durations`). **Organize plugin order:** -- Use the **Plugin Manager** tab to enable/disable plugins. The display - cycles through enabled plugins in the order they appear. +- The **Rotation** tab also has a drag-and-drop **Rotation Order** list + (saved to `display.plugin_rotation_order`). Enable/disable plugins + from the **Plugin Manager** tab. **Add more plugins:** - Check the **Plugin Store** section of **Plugin Manager** for new plugins. @@ -280,10 +299,14 @@ sudo journalctl -u ledmatrix-web -f │ ├── config_secrets.json # API keys and secrets │ └── wifi_config.json # WiFi settings ├── plugin-repos/ # Installed plugins (default location) -├── cache/ # Cached data └── web_interface/ # Web interface files ``` +> Cached data does not live in the project directory — the cache manager +> uses the first writable location among `/var/cache/ledmatrix`, +> `~/.ledmatrix_cache`, `/opt/ledmatrix/cache`, and +> `$TMPDIR/ledmatrix_cache`. +> > The plugin install location is configurable via > `plugin_system.plugins_directory` in `config.json`. The default is > `plugin-repos/`. Plugin discovery (`PluginManager.discover_plugins()`) @@ -303,11 +326,14 @@ System tabs: - WiFi Network selection and AP-mode setup - Schedule Power and dim schedules - Display Matrix hardware configuration +- Rotation Rotation order (drag-and-drop) and screen durations - Config Editor Raw config.json editor +- Backup & Restore Config backup and restore - Fonts Upload and manage fonts - Logs Real-time log viewing - Cache Cached data inspection and cleanup - Operation History Recent service operations +- Tools System diagnostics, updates, dependencies, maintenance Plugin tabs (second row): - Plugin Manager Browse the Plugin Store, install/enable plugins diff --git a/docs/HOW_TO_RUN_TESTS.md b/docs/HOW_TO_RUN_TESTS.md index 5dde96ec..db1cef12 100644 --- a/docs/HOW_TO_RUN_TESTS.md +++ b/docs/HOW_TO_RUN_TESTS.md @@ -10,10 +10,7 @@ Make sure you have the testing packages installed: ```bash # Install all dependencies including test packages -pip install -r requirements.txt - -# Or install just the test dependencies -pip install pytest pytest-cov pytest-mock +pip install -r requirements.txt -r requirements-test.txt ``` ### 2. Set Environment Variables @@ -253,7 +250,6 @@ test/ ├── test_error_aggregator.py # Error aggregation tests ├── test_schema_manager.py # Schema manager tests ├── test_web_api.py # Web API tests -├── test_nba_*.py # NBA-specific test suites ├── plugins/ # Per-plugin test suites │ ├── test_clock_simple.py │ ├── test_calendar.py @@ -303,7 +299,7 @@ If tests fail due to missing packages: ```bash # Install all dependencies -pip install -r requirements.txt +pip install -r requirements.txt -r requirements-test.txt # Or install specific missing package pip install @@ -335,15 +331,15 @@ pytest --cov=src --cov-report=html ## Continuous Integration -The repo runs -[`.github/workflows/security-audit.yml`](../.github/workflows/security-audit.yml) -(bandit + semgrep) on every push. A pytest CI workflow at -`.github/workflows/tests.yml` is queued to land alongside this -PR ([ChuckBuilds/LEDMatrix#307](https://github.com/ChuckBuilds/LEDMatrix/pull/307)); -the workflow file itself was held back from that PR because the -push token lacked the GitHub `workflow` scope, so it needs to be -committed separately by a maintainer. Once it's in, this section -will be updated to describe what the job runs. +The repo runs the pytest suite via +[`.github/workflows/test.yml`](../.github/workflows/test.yml) on every +push and pull request: a plugin-safety job (harness, visual rendering +and plugin-matrix tests) plus a unit-test job that runs an explicit +allowlist of suites — new test files must be added to that list to run +in CI. Release version consistency is checked by +[`.github/workflows/release-version-check.yml`](../.github/workflows/release-version-check.yml). +Bandit, flake8, mypy and gitleaks run as pre-commit hooks (see +`.pre-commit-config.yaml`), not in CI. ## Best Practices diff --git a/docs/LOW_MEMORY_BOARDS.md b/docs/LOW_MEMORY_BOARDS.md new file mode 100644 index 00000000..186e8a4a --- /dev/null +++ b/docs/LOW_MEMORY_BOARDS.md @@ -0,0 +1,115 @@ +# Running on Low-Memory Boards + +Applies to the Pi Zero 2 W (512 MB), Pi 3 / 3B+ (1 GB), and the 1 GB Pi 4. +If your board has 2 GB or more you can skip this document. + +## The failure this prevents + +The display process is the largest thing on the board. On a 1 GB Pi 3B+ with +around 20 plugins enabled it settles near **600 MB of 905 MB usable**, leaving +under 200 MB of headroom for everything else. + +When that headroom runs out, the board does not crash cleanly. `fork()` starts +failing, and because a new process is needed to do almost anything, the +symptoms look nothing like "out of memory": + +| What you see | Why | +|---|---| +| SSH accepts the connection then closes it instantly, before any banner | `sshd` forks a session per connection; the fork fails | +| The web UI still responds quickly | Already running, serves from existing threads, forks nothing | +| Ping is perfect, 0% loss | Handled entirely in the kernel | +| The panel is dark | The display process was killed and cannot be respawned | +| The clock is wrong after the next boot | `fake-hwclock`'s periodic save is a scheduled job, and it cannot fork either | + +The board looks healthy from the outside and cannot be logged into. Only a +power cycle clears it. If you are here because SSH stopped working, also see +[SSH_UNAVAILABLE_AFTER_INSTALL.md](SSH_UNAVAILABLE_AFTER_INSTALL.md), which +covers the more common cause (AP mode). + +## Check your headroom + +```bash +free -m +ps -eo rss,comm --sort=-rss | head -5 +``` + +If `MemAvailable` is under ~150 MB while the display is running, you are close +to the edge. To watch it over time: + +```bash +watch -n 30 'free -m | head -2' +``` + +Available memory that falls steadily rather than holding flat means you will +reach the wall; it is a question of when. + +## What to do + +**1. Enable the memory cgroup controller.** Without it, the `MemoryMax=85%` in +`systemd/ledmatrix.service` is accepted by systemd and silently ignored, so the +service has no ceiling and a runaway takes the whole board down instead of just +restarting. Raspberry Pi firmware disables this controller by default. + +`first_time_install.sh` does this for you. To check it took effect: + +```bash +grep memory /sys/fs/cgroup/cgroup.controllers +``` + +If that prints nothing, add `cgroup_enable=memory cgroup_memory=1` to the +kernel command line and reboot. Edit whichever file your image uses — +`/boot/firmware/cmdline.txt` on current Raspberry Pi OS, `/boot/cmdline.txt` on +older layouts (the installer checks the first and falls back to the second). +Everything must stay on a single line. + +This changes the failure mode from "the board becomes unreachable" to "the +display service restarts". It is a safety net, not a fix. + +**2. Run fewer plugins.** This is the actual remedy. Every enabled plugin costs +memory permanently — its module, its parsed config, and its cached API +responses. On a 512 MB or 1 GB board, keep the enabled set small and prefer +plugins that poll infrequently. + +**3. Lower the cache ceiling.** The in-memory cache is sized from total RAM +(150 entries at 1 GB and below, up to 1500 at 8 GB). To go lower still: + +```ini +# /etc/systemd/system/ledmatrix.service.d/override.conf +[Service] +Environment=LEDMATRIX_CACHE_MAX_ENTRIES=75 +``` + +Writing the file does not change the running service. Reload systemd and +restart it: + +```bash +sudo systemctl daemon-reload +sudo systemctl restart ledmatrix +``` + +Fewer entries means more API calls, so lower this only while you are actually +short of memory. + +**4. Consider `MemoryHigh`.** `MemoryMax` kills and restarts. `MemoryHigh` +throttles and reclaims instead, which is gentler — but on a board where the +process genuinely wants more than the limit, sustained reclaim can stall the +render loop and show as visible stutter on the panel. Add it only if you prefer +degraded output to a restart: + +```ini +[Service] +MemoryHigh=70% +``` + +## Keep your logs + +These images default to volatile journald storage, so every reboot destroys the +logs — including the ones explaining why the board rebooted. `first_time_install.sh` +enables persistent storage capped at 64 MB. To confirm: + +```bash +journalctl --list-boots +``` + +More than one boot listed means logs are surviving reboots. If only one is +listed, journald is still writing to `/run` (tmpfs). diff --git a/docs/MIGRATION_GUIDE.md b/docs/MIGRATION_GUIDE.md index 422cd064..b64269b9 100644 --- a/docs/MIGRATION_GUIDE.md +++ b/docs/MIGRATION_GUIDE.md @@ -86,7 +86,7 @@ The plugin system has been enhanced but remains backward compatible with existin If you encounter issues during migration: -1. Check the [README.md](README.md) for current installation and usage instructions +1. Check the [project root README](../README.md) for current installation and usage instructions 2. Review script README files: - [`scripts/install/README.md`](../scripts/install/README.md) - Installation scripts documentation - [`scripts/fix_perms/README.md`](../scripts/fix_perms/README.md) - Permission scripts documentation diff --git a/docs/PLUGIN_API_REFERENCE.md b/docs/PLUGIN_API_REFERENCE.md index 751d5609..4045c50b 100644 --- a/docs/PLUGIN_API_REFERENCE.md +++ b/docs/PLUGIN_API_REFERENCE.md @@ -170,6 +170,47 @@ Default returns `False`. List of display modes to show during a live takeover. Default returns the plugin's `display_modes` from its manifest. +#### `get_vegas_priority_weight() -> Optional[int]` + +How many slots per Vegas cycle this plugin should get. Default returns +`None`, which defers to the core. + +The Vegas ticker is otherwise a strict round robin — every plugin appears +exactly once per cycle — so with a dozen plugins enabled a live score can be +minutes stale by the time it comes round. A weight of *N* gives the plugin +*N* slots per cycle, spread evenly through it rather than clumped. + +**You usually do not need this.** When the hook returns `None`, the core +already gives a plugin `vegas_scroll.live_weight` whenever +`has_live_priority()` and `has_live_content()` are both true. Live sports get +extra turns with no code at all. + +Implement it only when the plugin knows something the core cannot. The +motivating case is favorite teams — the core can see *that* a game is live, +but not *whose*: + +```python +def get_vegas_priority_weight(self): + if not (self.has_live_priority() and self.has_live_content()): + return None # let the core decide + vegas = self.global_config.get('display', {}).get('vegas_scroll', {}) + if self._favorite_is_live(): + return vegas.get('favorite_live_weight', 5) + return vegas.get('live_weight', 3) +``` + +The weight is per *plugin*, not per game: a scoreboard showing four live games +still occupies one slot at a time and rotates its own games within it. Values +are clamped to 1–10 by the caller. An exception here is caught and logged, and +the core then falls back to its own live-content check — so a plugin whose +weight calculation is broken still gets `live_weight` for a game that really +is live, rather than being demoted to 1. + +Only consulted when the user has set `vegas_scroll.live_in_ticker`. With the +default (`false`) live content preempts Vegas entirely and there is no ticker +to be weighted within. See +[ADVANCED_FEATURES.md](ADVANCED_FEATURES.md#live-content-in-the-ticker). + ### Vegas scroll hooks Vegas mode shows multiple plugins as a single continuous scroll instead of @@ -201,8 +242,9 @@ the mode selector for this plugin. #### `get_vegas_segment_width() -> Optional[int]` -For `FIXED_SEGMENT` plugins, the width in pixels of the segment they -occupy in the scroll. `None` lets the controller pick a default. +For `FIXED_SEGMENT` plugins, the number of *panels* the segment +occupies in the scroll (pixel width = panels × `single_panel_width`, +from `display.hardware.cols`). `None` uses the default of 1 panel. > The full source for `BasePlugin` lives in > `src/plugin_system/base_plugin.py`. If a method here disagrees with the diff --git a/docs/PLUGIN_ARCHITECTURE_SPEC.md b/docs/PLUGIN_ARCHITECTURE_SPEC.md index fbd45b34..00a2914d 100644 --- a/docs/PLUGIN_ARCHITECTURE_SPEC.md +++ b/docs/PLUGIN_ARCHITECTURE_SPEC.md @@ -8,9 +8,12 @@ > - Code paths reference `web_interface_v2.py`; the current web UI is > `web_interface/app.py` with v3 Blueprint-based templates. > - The example Flask routes use `/api/plugins/*`; the real API -> blueprint is mounted at `/api/v3` (`web_interface/app.py:144`). +> blueprint is mounted at `/api/v3` (`web_interface/app.py:199`). > - The default plugin location is `plugin-repos/` (configurable via > `plugin_system.plugins_directory`), not `./plugins/`. +> - Example imports use `src/plugin_system/base_classes/*_plugin.py`; +> the shipped base classes live in `src/base_classes/` (e.g. +> `src.base_classes.sports.SportsCore`, `src.base_classes.hockey.Hockey`). > - The "Migration Strategy" and "Implementation Roadmap" sections > describe work that has now shipped. > diff --git a/docs/PLUGIN_CONFIG_ARCHITECTURE.md b/docs/PLUGIN_CONFIG_ARCHITECTURE.md index 28fa1ba3..823bb4db 100644 --- a/docs/PLUGIN_CONFIG_ARCHITECTURE.md +++ b/docs/PLUGIN_CONFIG_ARCHITECTURE.md @@ -1,5 +1,11 @@ # Plugin Configuration Tabs - Architecture +> This page covers internals (how the config system works under the +> hood). For designing a plugin's config schema, the canonical guide is +> [PLUGIN_CONFIGURATION_GUIDE.md](PLUGIN_CONFIGURATION_GUIDE.md); for +> the user-facing tabs feature, see +> [PLUGIN_CONFIGURATION_TABS.md](PLUGIN_CONFIGURATION_TABS.md). + ## System Architecture ### Component Overview diff --git a/docs/PLUGIN_CUSTOM_ICONS.md b/docs/PLUGIN_CUSTOM_ICONS.md index da9db63c..79cabc5b 100644 --- a/docs/PLUGIN_CUSTOM_ICONS.md +++ b/docs/PLUGIN_CUSTOM_ICONS.md @@ -296,7 +296,7 @@ Want to change icons programmatically? While not officially supported, you could ## Related Documentation - [Plugin Configuration Tabs](PLUGIN_CONFIGURATION_TABS.md) - Main plugin tabs documentation -- [Plugin Development Guide](plugin_docs/) - How to create plugins +- [Plugin Development Guide](PLUGIN_DEVELOPMENT_GUIDE.md) - How to create plugins - [Font Awesome Icons](https://fontawesome.com/icons) - Browse all available icons - [Emoji Reference](https://unicode.org/emoji/charts/full-emoji-list.html) - All emoji options diff --git a/docs/PLUGIN_DEPENDENCY_TROUBLESHOOTING.md b/docs/PLUGIN_DEPENDENCY_TROUBLESHOOTING.md index 755760c9..1f33cb88 100644 --- a/docs/PLUGIN_DEPENDENCY_TROUBLESHOOTING.md +++ b/docs/PLUGIN_DEPENDENCY_TROUBLESHOOTING.md @@ -169,6 +169,6 @@ If you continue to experience issues: ## Related Documentation - [Plugin Dependency Guide](PLUGIN_DEPENDENCY_GUIDE.md) -- [Plugin Development Guide](docs/plugin_development.md) -- [Troubleshooting Quick Start](TROUBLESHOOTING_QUICK_START.md) +- [Plugin Development Guide](PLUGIN_DEVELOPMENT_GUIDE.md) +- [Troubleshooting](TROUBLESHOOTING.md) diff --git a/docs/PLUGIN_DEVELOPMENT_GUIDE.md b/docs/PLUGIN_DEVELOPMENT_GUIDE.md index 31cafc5e..e2a0c240 100644 --- a/docs/PLUGIN_DEVELOPMENT_GUIDE.md +++ b/docs/PLUGIN_DEVELOPMENT_GUIDE.md @@ -589,11 +589,24 @@ Your plugin must: ### Versioning Best Practices - **Use semantic versioning**: `MAJOR.MINOR.PATCH` (e.g., `1.2.3`) -- **Automatic version bumping**: Use the pre-push git hook for automatic patch version bumps -- **Manual versioning**: Only needed for major/minor bumps or special cases -- **GitHub as source of truth**: Plugin store fetches versions from GitHub releases/tags/manifest +- **GitHub as source of truth**: the plugin store resolves versions in this + order: GitHub Releases → GitHub Tags → manifest from branch → git commit hash +- **Automatic version bumping**: install the self-contained pre-push hook in + your plugin repo and patch versions bump themselves on push (a git tag + `v{version}` is created and `manifest.json` staged automatically): -See the [Git Workflow rules](../.cursorrules) for version management details. + ```bash + # From your plugin repository directory + cp /path/to/LEDMatrix/scripts/git-hooks/pre-push-plugin-version .git/hooks/pre-push + chmod +x .git/hooks/pre-push + ``` + + Set `SKIP_TAG=1` in the environment to skip auto-tagging for one push. +- **Manual versioning**: only needed for major/minor bumps, CI pipelines that + bypass hooks, or forks without the hook — use + `scripts/bump_plugin_version.py`. +- **Registry stores no versions**: `plugins.json` holds only metadata (name, + description, repo URL). ### Submitting to Official Registry @@ -667,5 +680,5 @@ For your plugin to work well in the plugin store: - [Advanced Plugin Development](ADVANCED_PLUGIN_DEVELOPMENT.md) - Advanced patterns and examples - [Plugin Quick Reference](PLUGIN_QUICK_REFERENCE.md) - Quick development reference - [Plugin Configuration Guide](PLUGIN_CONFIGURATION_GUIDE.md) - Configuration setup -- [Plugin Store User Guide](PLUGIN_STORE_USER_GUIDE.md) - Using the plugin store +- [Plugin Store Guide](PLUGIN_STORE_GUIDE.md) - Using the plugin store diff --git a/docs/PLUGIN_QUICK_REFERENCE.md b/docs/PLUGIN_QUICK_REFERENCE.md index 7f8c0bf0..850fbb22 100644 --- a/docs/PLUGIN_QUICK_REFERENCE.md +++ b/docs/PLUGIN_QUICK_REFERENCE.md @@ -14,8 +14,10 @@ and [PLUGIN_DEVELOPMENT_GUIDE.md](PLUGIN_DEVELOPMENT_GUIDE.md). ✅ **GitHub Store**: Discovery from `ledmatrix-plugins` registry plus any GitHub URL ✅ **Plugin Location**: configured by `plugin_system.plugins_directory` - in `config.json` (default `plugin-repos/`; the loader also searches - `plugins/` as a fallback) + in `config.json` (default `plugin-repos/`). Plugin discovery scans + only this directory — there is no loader fallback to `plugins/` + (only Plugin Store operations and schema lookup additionally probe + `plugins/`) ## File Structure @@ -109,7 +111,7 @@ git push -u origin main git tag v1.0.0 git push origin v1.0.0 -# Submit to registry (PR to ChuckBuilds/ledmatrix-plugin-registry) +# Submit to registry (PR to ChuckBuilds/ledmatrix-plugins) ``` ## Using Plugins @@ -120,12 +122,12 @@ git push origin v1.0.0 2. **Install**: Click **Install** in the plugin's row 3. **Configure**: open the plugin's tab in the second nav row 4. **Enable/Disable**: toggle switch in the **Installed Plugins** list -5. **Reorder**: order is set by the position in `display_modes` / - plugin order; rearranging via drag-and-drop is not yet supported +5. **Reorder**: use the drag-and-drop **Rotation Order** list in the + **Rotation** tab (saved to `display.plugin_rotation_order`) ### REST API -The API is mounted at `/api/v3` (`web_interface/app.py:144`). +The API is mounted at `/api/v3` (`web_interface/app.py:199`). ```bash # Install plugin from the registry diff --git a/docs/PLUGIN_REGISTRY_SETUP_GUIDE.md b/docs/PLUGIN_REGISTRY_SETUP_GUIDE.md index 7c1569b8..d1426979 100644 --- a/docs/PLUGIN_REGISTRY_SETUP_GUIDE.md +++ b/docs/PLUGIN_REGISTRY_SETUP_GUIDE.md @@ -323,16 +323,22 @@ curl -X POST http://pi:5000/api/v3/plugins/install-from-url \ ### Regular Updates ```bash -# Update stars/downloads counts -python3 scripts/update_stats.py +# Refresh local clones of all plugin repos +python3 scripts/update_plugin_repos.py -# Validate all plugin entries -python3 scripts/validate_registry.py +# (Re-)create local plugin repo checkouts from the registry +python3 scripts/setup_plugin_repos.py -# Check for plugin updates -python3 scripts/check_updates.py +# Audit installed plugins for manifest/schema problems +python3 scripts/audit_plugins.py + +# Validate a single plugin +python3 scripts/check_plugin.py --plugin ``` +Registry regeneration (`update_registry.py`) lives in the +`ledmatrix-plugins` monorepo, not in this repo. + ## Converting Existing Plugins To convert your existing plugins (hello-world, clock-simple) to this system: @@ -400,7 +406,7 @@ print(f'Found {len(registry[\"plugins\"])} plugins') ## References -- Plugin Store Implementation: See `PLUGIN_STORE_IMPLEMENTATION_SUMMARY.md` -- User Guide: See `PLUGIN_STORE_USER_GUIDE.md` +- Plugin Store Implementation: See `PLUGIN_IMPLEMENTATION_SUMMARY.md` +- User Guide: See `PLUGIN_STORE_GUIDE.md` - Architecture: See `PLUGIN_ARCHITECTURE_SPEC.md` diff --git a/docs/PLUGIN_STORE_GUIDE.md b/docs/PLUGIN_STORE_GUIDE.md index 4482d5b9..421c99de 100644 --- a/docs/PLUGIN_STORE_GUIDE.md +++ b/docs/PLUGIN_STORE_GUIDE.md @@ -481,13 +481,13 @@ A: Yes, if a plugin needs API keys, it can access them like core managers do. A: Most plugins are small (1-5MB). Check individual plugin documentation for specific requirements. **Q: Can I create my own plugin?** -A: Yes! See [PLUGIN_DEVELOPMENT.md](PLUGIN_DEVELOPMENT.md) for instructions. +A: Yes! See [PLUGIN_DEVELOPMENT_GUIDE.md](PLUGIN_DEVELOPMENT_GUIDE.md) for instructions. --- ## Related Documentation -- [PLUGIN_DEVELOPMENT.md](PLUGIN_DEVELOPMENT.md) - Create your own plugins +- [PLUGIN_DEVELOPMENT_GUIDE.md](PLUGIN_DEVELOPMENT_GUIDE.md) - Create your own plugins - [PLUGIN_API_REFERENCE.md](PLUGIN_API_REFERENCE.md) - Plugin API documentation -- [PLUGIN_ARCHITECTURE.md](PLUGIN_ARCHITECTURE.md) - Plugin system architecture +- [PLUGIN_ARCHITECTURE_SPEC.md](PLUGIN_ARCHITECTURE_SPEC.md) - Plugin system architecture (historical) - [REST_API_REFERENCE.md](REST_API_REFERENCE.md) - Complete REST API reference diff --git a/docs/README.md b/docs/README.md index 24345c52..63128da6 100644 --- a/docs/README.md +++ b/docs/README.md @@ -14,6 +14,7 @@ the one-shot installer. The pages here go deeper. 5. [TROUBLESHOOTING.md](TROUBLESHOOTING.md) — common issues and fixes 6. [SSH_UNAVAILABLE_AFTER_INSTALL.md](SSH_UNAVAILABLE_AFTER_INSTALL.md) — recovering SSH after install 7. [CONFIG_DEBUGGING.md](CONFIG_DEBUGGING.md) — diagnosing config problems +8. [LOW_MEMORY_BOARDS.md](LOW_MEMORY_BOARDS.md) — Pi Zero 2 W / 3B+ / 1GB Pi 4 memory limits ## I want to write a plugin @@ -29,15 +30,16 @@ Start here: Going deeper: - [ADVANCED_PLUGIN_DEVELOPMENT.md](ADVANCED_PLUGIN_DEVELOPMENT.md) — advanced patterns -- [PLUGIN_ARCHITECTURE_SPEC.md](PLUGIN_ARCHITECTURE_SPEC.md) — full plugin-system spec +- [PLUGIN_ARCHITECTURE_SPEC.md](PLUGIN_ARCHITECTURE_SPEC.md) — original plugin-system design spec (historical; see its banner for what has drifted) - [PLUGIN_DEPENDENCY_GUIDE.md](PLUGIN_DEPENDENCY_GUIDE.md) / [PLUGIN_DEPENDENCY_TROUBLESHOOTING.md](PLUGIN_DEPENDENCY_TROUBLESHOOTING.md) - [PLUGIN_WEB_UI_ACTIONS.md](PLUGIN_WEB_UI_ACTIONS.md) (+ [example JSON](PLUGIN_WEB_UI_ACTIONS_EXAMPLE.json)) -- [PLUGIN_CUSTOM_ICONS.md](PLUGIN_CUSTOM_ICONS.md) / - [PLUGIN_CUSTOM_ICONS_FEATURE.md](PLUGIN_CUSTOM_ICONS_FEATURE.md) +- [PLUGIN_CUSTOM_ICONS.md](PLUGIN_CUSTOM_ICONS.md) - [PLUGIN_REGISTRY_SETUP_GUIDE.md](PLUGIN_REGISTRY_SETUP_GUIDE.md) (+ [registry template](plugin_registry_template.json)) - [STARLARK_APPS_GUIDE.md](STARLARK_APPS_GUIDE.md) — Starlark-based mini-apps - [widget-guide.md](widget-guide.md) — widget development +- [ADAPTIVE_LAYOUT.md](ADAPTIVE_LAYOUT.md) — render legibly on any panel size (opt-in font/layout scaling) +- [plugin-safety-harness.md](plugin-safety-harness.md) — test a plugin across every screen and matrix size ## Configuring plugins @@ -52,9 +54,12 @@ Going deeper: - [ADVANCED_FEATURES.md](ADVANCED_FEATURES.md) — Vegas scroll, on-demand display, cache management, background services, permissions - [FONT_MANAGER.md](FONT_MANAGER.md) — font system +- [SKIN_SYSTEM.md](SKIN_SYSTEM.md) — skin architecture for sports scoreboards +- [CREATING_SKINS.md](CREATING_SKINS.md) — writing and validating a skin ## Reference +- [CONFIG_REFERENCE.md](CONFIG_REFERENCE.md) — every key in config.json and config_secrets.json - [REST_API_REFERENCE.md](REST_API_REFERENCE.md) — all web-interface HTTP endpoints - [PLUGIN_API_REFERENCE.md](PLUGIN_API_REFERENCE.md) — Python APIs available to plugins - [DEVELOPER_QUICK_REFERENCE.md](DEVELOPER_QUICK_REFERENCE.md) — common dev tasks @@ -66,6 +71,7 @@ Going deeper: - [HOW_TO_RUN_TESTS.md](HOW_TO_RUN_TESTS.md) — running the test suite - [MULTI_ROOT_WORKSPACE_SETUP.md](MULTI_ROOT_WORKSPACE_SETUP.md) — multi-repo workspace - [MIGRATION_GUIDE.md](MIGRATION_GUIDE.md) — breaking changes between releases +- [SPORTS_UNIFICATION.md](SPORTS_UNIFICATION.md) — how the sports scoreboard base classes are organized ## Archive diff --git a/docs/REST_API_REFERENCE.md b/docs/REST_API_REFERENCE.md index 132547b0..f95cb84c 100644 --- a/docs/REST_API_REFERENCE.md +++ b/docs/REST_API_REFERENCE.md @@ -31,9 +31,9 @@ All endpoints return JSON responses with a standard format: - [Plugin-specific endpoints](#plugin-specific-endpoints) - [Starlark Apps](#starlark-apps) -> The API blueprint is mounted at `/api/v3` (`web_interface/app.py:144`). +> The API blueprint is mounted at `/api/v3` (`web_interface/app.py:199`). > SSE stream endpoints (`/api/v3/stream/*`) are defined directly on the -> Flask app at `app.py:607-615`. There are about 92 routes total — see +> Flask app at `app.py:799-809`. There are 94 routes total — see > `web_interface/blueprints/api_v3.py` for the canonical list. --- diff --git a/docs/SKIN_SYSTEM.md b/docs/SKIN_SYSTEM.md index b10dbcff..f7f4861f 100644 --- a/docs/SKIN_SYSTEM.md +++ b/docs/SKIN_SYSTEM.md @@ -33,7 +33,7 @@ crashing) simply restores the built-in look. ## The render funnel Every sports scoreboard (baseball, football, basketball, hockey — anything -built on `src/base_classes/sports.py`) renders through exactly one seam: +built on the `src/base_classes/sports/` package, `core.py`) renders through exactly one seam: `SportsCore._render_game(game, force_clear)`. 1. The mode class's `display()` (live, `SportsUpcoming`, `SportsRecent`) diff --git a/docs/SPORTS_UNIFICATION.md b/docs/SPORTS_UNIFICATION.md index 14b9c199..d783e15b 100644 --- a/docs/SPORTS_UNIFICATION.md +++ b/docs/SPORTS_UNIFICATION.md @@ -27,7 +27,7 @@ These are independent concerns. Conflating them is what produces god classes. | Plugin loads on a core that predates a module | Guarded import with a bundled fallback (`try: from src.X import Y / except ModuleNotFoundError: from y import Y`) | | Plugin loads on a core that predates a *method* | Capability probing — `hasattr(SportsCore, "_detect_stale_games")` — never a version comparison. The loader's compat check is advisory-only (it logs and continues), so probing is the real protection. | | Core changes never break a plugin's rendering | The **view-model contract**: `_extract_game_details_common` returns a dict whose `GUARANTEED_KEYS` are frozen by `test/test_skin_system.py::TestViewModelContract`. Keys may be added, never renamed or removed. | -| A plugin can drop its bundled copy safely | The **sunset rule**: its manifest must floor `ledmatrix_min_version` at the first core release shipping the module (recorded in `CHANGELOG.md`) — *necessary but not sufficient*. Nothing enforces that floor today, so the copy also waits for the B6 gate below. | +| A plugin can drop its bundled copy safely | The **sunset rule**: its manifest must floor `ledmatrix_min_version` at the first core release shipping the module (recorded in `CHANGELOG.md`) — *necessary but not sufficient*. The store enforces that floor on every registry-managed install and on both supported update paths (sideloading via `install_from_url` is not gated), but a floor cannot reach a user who never updates, so the copy also waits for the B6 gate below. | The core API is **additive-only**. A method the plugins call is never removed or given a new required parameter; new behavior arrives as new methods with @@ -214,7 +214,7 @@ one of them cannot break a user on an old core and the other can. | **B3** | Upstream the scroll **orchestration** layer as `src/common/sports_scroll.py`, reading `global_config['target_fps']` natively | ✅ | Content building stays per-sport | | **B4** | Ship 3.2.0 *and* make version reporting trustworthy | ✅ | Released 2026-08-03; tag, release and `src.__version__` agree; compatibility gate merged (#428, #431, #433) | | **B5** | Adoption — guarded core imports, all eight. **Bundled copies stay.** | ✅ | All eight adopted; harness byte-identical; see the B5 retrospective below — four shipped broken and were repaired in plugins #251 | -| **B6** | Sunset — delete the bundled copies | **blocked, deliberately** | 3.2.0 *in users' hands*. Released 2026-08-03; still no adoption data at the 2026-08-26 re-check. The prerequisite regression test is built (#505); only the evidence is missing. See "B6 — the decision as of 2026-08-05" | +| **B6** | Sunset — delete the bundled copies | ✅ | Ran 2026-09-01, all eight. Floors at 3.2.0; the store refuses on all three routes in (#431/#433, #508, #510). See "B6 — what actually happened" | ### B4 — what "ship 3.2.0" actually requires @@ -234,6 +234,20 @@ a floor can be trusted against, and today it is not: compares the plugin's manifest version against the registry's `latest_version` and nothing else. + *Fixed, in two parts.* `install_plugin` gained the gate in #431/#433, which + covers every registry-managed install and, through `_reinstall_with_rollback`, + the update path that re-downloads. + `update_plugin`'s git branch pulls in place and re-downloads nothing, so it + stayed ungated until `_gate_pulled_commit` closed it — checked after the pull + (the registry carries no floor field, so the incoming floor is unknowable + before it) and undone with `git reset --hard` to the pre-pull commit. That + route is rare in practice, since monorepo plugins install as archives; it was + closed because the sunset rule in the plugins repo's + `08-shared-sports-code.md` states as **condition 3** that the core enforces + the floor "at install/update time", and B6 rests on that being true rather + than merely written down. `install_from_url` — sideloading a plugin from a + URL — is still ungated. + So B4 is: tag and release 3.2.0; make the tag, the release, and `__version__` agree, and keep them agreeing; reconsider the `< 2.0.0` skip; migrate manifests from `ledmatrix_min` to `ledmatrix_min_version`; and add the install/update @@ -330,46 +344,69 @@ gate rather than trusting the failure to be noticed. The same suite should exercise the install/update gate, since it is the other half of the guarantee. -### B6 — the decision as of 2026-08-05 +### B6 — what actually happened -**Do not run B6 yet. Do not abandon it either.** The blocker is no longer -technical; it is calendar time, and it is the one thing here that cannot be -worked around by writing more code. +**Ran 2026-09-01, across all eight scoreboards.** Held from 2026-08-05 to +2026-09-01 on the argument below, which is kept because the reasoning applies to +the next module, not because it is still in force. -**Why not yet.** 3.2.0 was published **2026-08-03**. Its predecessor 3.1.0 ran -for nine months. B6's entire safety argument is "cores without -`src.common.sports_scroll` are gone", and two days after release that is not -close to true. There are no release assets to count and no install telemetry, so -we cannot demonstrate otherwise — and that absence of evidence *is* the answer. -Executing B6 now would strand essentially the whole user base on their current -plugin versions. +**The hold, and why it lifted.** The stated gate was evidence of 3.2.0 uptake — +"a few months of it being the default download, or store-side install data". +That evidence never arrived and could not: the core updates by +`git pull --rebase`, so release-asset counts cannot measure uptake, and no +store-side telemetry exists. What changed instead is that the *risk* the gate +protected against was closed directly. The store now refuses a plugin whose +floor exceeds the running core on **all three** routes in: -**What is already done and waiting.** The hard part is built and tested. The -install gate refuses a plugin whose floor exceeds the core's version, and -refuses one whose floor is above 2.0.0 when the core reports an untrustworthy -version — so a v3.1.0-release user (who reports `1.0.0`) keeps a working plugin -instead of receiving one that cannot load. Every adopted plugin has a -`test_core_fallback.py` covering both paths. +| route | gated by | +|---|---| +| `install_plugin` — every path that re-downloads, `_reinstall_with_rollback` included | #431, #433 | +| `update_plugin`'s git branch — pulls in place, re-downloads nothing | #508 | +| `install_from_url` — sideloading | #510 | -**What would unblock it.** Evidence of 3.2.0 uptake — a few months of it being -the default download, or store-side install data if that is ever added. Revisit -then, not on a schedule. +With all three closed a pre-3.2.0 user cannot receive a sunset plugin at all; +they keep the version they already run. The population the hold existed to +protect is protected by refusal rather than by a bundled copy — which is what +the copy was standing in for. -**Re-checked 2026-08-26 — still held.** v3.2.0 remains the latest release and -`src.__version__` is still `3.2.0`; no 3.3.0 has been cut. That is 23 days, not -the few months above, and no store-side install data has appeared. Note also -that the core updates by `git pull --rebase` rather than by downloading a -release, so release-asset counts would not measure uptake even if we had them — -whatever eventually unblocks this has to come from the store side. The -prerequisite test is now built (#505), so when the evidence does arrive the -remaining work is the sunset itself. +**What shipped.** Eight plugins, ~5,800 lines of frozen fallback deleted. Each: +copy removed, guarded import collapsed to a plain one, floor raised to 3.2.0, +`test_core_fallback.py` rewritten as `test_core_scroll.py` asserting the sunset +rather than the fallback. `scripts/check_scroll_adoption.py` gained +`sunset_violations` and a `SUNSET_PLUGINS` set naming all eight, so a +resurrected copy or a returned guard fails CI. -**When it happens, remember:** four plugins declare their floor top-level, where -editing `versions[0]` is a silent no-op, and the floor has three live spellings -(`min_ledmatrix_version`, `requires.min_ledmatrix_version`, -`versions[].ledmatrix_min_version`, plus deprecated `ledmatrix_min`). See +Delivered as plugins #346 (hockey, later folded into #351), #349 (football), +#350 (baseball), #351 (the remaining six). + +**Two things found by doing it, both worth carrying forward:** + +- **Only one fallback held orchestration logic the core lacked.** baseball's + `_configure_scroll_helper` reinterpreted `scroll_speed` as pixels-per-*frame* + when `speed × delay` fell outside the 0.1–5.0 window — measured, 10–20× + faster than configured for a speed between 1.0 and 5.0. Standardised onto the + core's behaviour (honour the documented px/sec, clamp) rather than preserved. + Every other difference across the eight was a docstring, an unreachable + `scroll_helper is None` guard, or an equivalent diagnostic. +- **Two tests had been leaning on the guard without anyone knowing.** + `soccer/test_live_screens.py` stubbed `src` in a way that shadowed the core, + so its guarded import fell back and the test had been exercising the *frozen + copy* rather than the shipping class since B5. Before sunsetting anything else + that carries a guarded core import, grep for tests that stub `src`. + +**The floor-raising traps still apply** to any future sunset: four plugins +declare their floor top-level, where editing `versions[0]` is a silent no-op, +and the floor has three live spellings (`min_ledmatrix_version`, +`requires.min_ledmatrix_version`, `versions[].ledmatrix_min_version`, plus +deprecated `ledmatrix_min`). See `src/plugin_system/compatibility.py:declared_min_version` for the resolution -order any floor-raising tool must reproduce. +order any floor-raising tool must reproduce — and note the name is **inverted** +between the top level and `versions[]`. + +**Still not adopted, deliberately:** `data_sources.py`, `game_renderer.py` and +`base_odds_manager.py`. The standing decision held them until B6 closed; it now +has, so they can be reconsidered — with B5's lesson applied, which is to build +the object and diff rendered output rather than trust a static check. ### B5 retrospective — what the adoption actually cost @@ -430,33 +467,21 @@ a version number CI now asserts (#428), the compatibility gate is in What actually remains, smallest first: -1. **Nothing on the critical path.** B6 is the only remaining phase and it is - waiting on calendar time, not on work. Resist the urge to fill the gap by - adopting more modules — see the decision above. -2. ~~**The stale plugin-test tranche**~~ — **done** (verified 2026-08-26). - The 5 failures across baseball, hockey and basketball are gone; - `scripts/run_plugin_tests.py --all` reports 174 passed, 2 skipped, 0 failed - across the whole fleet. Re-check with that runner rather than by pointing - pytest at a plugin directory: these are standalone scripts, not a pytest - suite, and one of them calls `sys.exit(1)` at import, which collapses a - pytest run into an INTERNALERROR that looks nothing like the real state. -3. **Soak the remaining adoptions on hardware.** Only baseball has been watched - through a live game, and hockey has been loaded on devpi. The other six are - proven by harness, unit tests and pixel comparison — not by a live match. - Out-of-season sports cannot be soaked until their season starts. **This is - now the only open item that needs work rather than calendar time.** -4. ~~**`CLAUDE.md` says four panel sizes**~~ — **done.** It reads "Default - matrix sizes are eight … design for the classic four first". -5. **Then, when the evidence supports it, B6.** Its prerequisite — the - four-case compatibility regression test — is built: core - `test/test_sports_sunset_matrix.py` (#505). It drives all four cells through - the real `PluginManager.load_plugin` and asserts the sunset cell as - `PluginState.ERROR` plus an error naming the missing module, not as a raise. - Two modelling traps it had to work through, worth knowing before trusting - any successor: the copy-removed shape must be an *unguarded* import, or the - failure names the missing `scroll_display_legacy` instead of the core - module; and only the leaf module may be hidden, since a pre-3.2.0 core still - ships `src/common/`. +1. **Soak the adoptions on hardware.** football and hockey have been run on a + live rig through real games; baseball was watched through one earlier. The + rest are proven by harness, unit tests and pixel comparison. Out-of-season + sports cannot be soaked until their season starts. When you do, **check the + rig's `*_display_mode` first** — a board in `switch` mode will happily load a + sunset plugin and tell you nothing about the scroll code the sunset changed. +2. **Cut 3.3.0.** Not required by B6 — its floors are 3.2.0, which is released — + but `calendar` 1.2.3 floors at 3.3.0 for the device-authorization endpoints + that landed after 3.2.0, so it is un-installable until the release exists. +3. **Reconsider the held modules** (`data_sources.py`, `game_renderer.py`, + `base_odds_manager.py`) now that the sunset has closed. `game_renderer.py` is + the largest single duplication left: ~11,500 lines across eight plugins, with + ~36,500 more in the eight `sports.py`. Note that core already ships + `src/base_classes/sports/` (~143KB, promoted in B1/B2) that **no plugin + imports** — check whether it has drifted before treating it as the target. ## How to keep this project healthy diff --git a/docs/SSH_UNAVAILABLE_AFTER_INSTALL.md b/docs/SSH_UNAVAILABLE_AFTER_INSTALL.md index d9830165..bc065eb1 100644 --- a/docs/SSH_UNAVAILABLE_AFTER_INSTALL.md +++ b/docs/SSH_UNAVAILABLE_AFTER_INSTALL.md @@ -20,7 +20,22 @@ The installation script: - Installs and configures `dnsmasq` (DHCP server for AP mode) - These services can interfere with normal WiFi client mode -### 3. Reboot After Installation +### 3. The Board Ran Out of Memory + +On a 512MB or 1GB board, memory exhaustion stops `sshd` being able to fork a +session process. The connection is accepted and then closed immediately, before +any banner: + +```text +kex_exchange_identification: Connection closed by remote host +``` + +The giveaway is that the board is otherwise healthy — ping is clean and the web +UI still responds — but nothing that needs to start a new process works, and +the panel is usually dark. Only a power cycle clears it. See +[LOW_MEMORY_BOARDS.md](LOW_MEMORY_BOARDS.md). + +### 4. Reboot After Installation If the script reboots the Pi (which it recommends), network services may restart in a different state, potentially triggering AP mode. @@ -190,11 +205,23 @@ The web interface allows you to: ## Summary -**SSH becomes unavailable because**: +**SSH becomes unavailable because** — two unrelated causes, and they need +different responses: + +*AP mode (most common):* - WiFi monitor service enables AP mode when WiFi disconnects - AP mode switches WiFi from client to access point mode - Pi loses connection to your original network +*Memory exhaustion (low-memory boards):* +- The board runs out of memory, so `sshd` cannot fork a session process +- The connection is accepted and closed before any banner +- Ping still answers and the web UI still responds, so it looks healthy +- The panel is usually dark and the service cannot restart +- **Only a power cycle clears this** — there is no remote recovery, because + every remote route needs a new process +- Prevention and tuning: [LOW_MEMORY_BOARDS.md](LOW_MEMORY_BOARDS.md) + **To regain SSH**: 1. Connect to **LEDMatrix-Setup** AP network (password: `ledmatrix123`) 2. SSH to `192.168.4.1` diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md index 453b5556..fe3a1a74 100644 --- a/docs/TROUBLESHOOTING.md +++ b/docs/TROUBLESHOOTING.md @@ -330,8 +330,8 @@ sudo systemctl cat ledmatrix-web | grep User 6. **Manually enable AP mode:** ```bash - # Via API - curl -X POST http://localhost:5000/api/wifi/ap/enable + # Via API (the WiFi blueprint is mounted under /api/v3) + curl -X POST http://localhost:5000/api/v3/wifi/ap/enable # Via Python python3 -c " @@ -482,19 +482,19 @@ sudo systemctl cat ledmatrix-web | grep User 1. **Check plugin directory exists:** ```bash - ls -ld plugins/plugin-id/ + ls -ld plugin-repos/plugin-id/ ``` 2. **Verify manifest.json:** ```bash - cat plugins/plugin-id/manifest.json + cat plugin-repos/plugin-id/manifest.json # Verify all required fields present ``` 3. **Check dependencies installed:** ```bash - if [ -f plugins/plugin-id/requirements.txt ]; then - pip3 install --break-system-packages -r plugins/plugin-id/requirements.txt + if [ -f plugin-repos/plugin-id/requirements.txt ]; then + pip3 install --break-system-packages -r plugin-repos/plugin-id/requirements.txt fi ``` @@ -507,7 +507,7 @@ sudo systemctl cat ledmatrix-web | grep User ```bash python3 -c " import sys - sys.path.insert(0, 'plugins/plugin-id') + sys.path.insert(0, 'plugin-repos/plugin-id') from manager import PluginClass print('Plugin imports successfully') " @@ -523,12 +523,18 @@ sudo systemctl cat ledmatrix-web | grep User **Solutions:** 1. **Manual cache clearing:** - ```bash - # Remove plugin-specific cache - rm -rf cache/plugin-id* - # Or remove all cache - rm -rf cache/* + The cache does not live in the project directory. The cache manager + uses the first writable location among `/var/cache/ledmatrix`, + `~/.ledmatrix_cache`, `/opt/ledmatrix/cache`, and + `$TMPDIR/ledmatrix_cache`. The easiest option is the helper script: + + ```bash + # Clear the cache with the helper script + sudo python3 scripts/utils/clear_cache.py + + # Or remove files manually from the cache dir in use, e.g.: + sudo rm -rf /var/cache/ledmatrix/* # Restart display sudo systemctl restart ledmatrix @@ -536,8 +542,8 @@ sudo systemctl cat ledmatrix-web | grep User 2. **Check cache permissions:** ```bash - ls -ld cache/ - sudo chown -R ledpi:ledpi cache/ + ls -ld /var/cache/ledmatrix + sudo ./scripts/fix_perms/fix_cache_permissions.sh ``` --- @@ -772,11 +778,11 @@ nmcli device status ```bash # Check file exists ls -l config/config.json -ls -l plugins/plugin-id/manifest.json +ls -l plugin-repos/plugin-id/manifest.json # Check directory structure ls -la web_interface/ -ls -la plugins/ +ls -la plugin-repos/ # Check file permissions ls -l config/config_secrets.json @@ -804,7 +810,7 @@ python3 -c "from src.wifi_manager import WiFiManager; print('OK')" # Test plugin import python3 -c " import sys -sys.path.insert(0, 'plugins/plugin-id') +sys.path.insert(0, 'plugin-repos/plugin-id') from manager import PluginClass print('Plugin imports OK') " @@ -812,40 +818,29 @@ print('Plugin imports OK') --- -## Service File Template +## Reinstalling Service Files -If your systemd service file is corrupted or missing, use this template: - -```ini -[Unit] -Description=LEDMatrix Web Interface -After=network.target - -[Service] -Type=simple -User=ledpi -Group=ledpi -WorkingDirectory=/home/ledpi/LEDMatrix -Environment="PYTHONUNBUFFERED=1" -ExecStart=/usr/bin/python3 /home/ledpi/LEDMatrix/web_interface/start.py -Restart=on-failure -RestartSec=5s -StandardOutput=journal -StandardError=journal -SyslogIdentifier=ledmatrix-web - -[Install] -WantedBy=multi-user.target -``` - -Save to `/etc/systemd/system/ledmatrix-web.service` and run: +If a systemd service file is corrupted or missing, do NOT hand-write +one. The real unit files live in the repo's `systemd/` directory +(`ledmatrix.service`, `ledmatrix-web.service`, +`ledmatrix-wifi-monitor.service`) and contain a +`__PROJECT_ROOT_DIR__` placeholder that the install scripts substitute +with your actual checkout path: ```bash -sudo systemctl daemon-reload -sudo systemctl enable ledmatrix-web -sudo systemctl start ledmatrix-web +# Reinstall the display service unit +sudo ./scripts/install/install_service.sh + +# Reinstall the web interface service unit +sudo ./scripts/install/install_web_service.sh ``` +Note that `ledmatrix-web.service` runs as root via +`scripts/utils/start_web_conditionally.py` — root is needed for +system operations (service control, WiFi management), and the wrapper +honors the `web_display_autostart` config flag before actually +starting the web server. + --- ## Complete Diagnostic Script @@ -878,7 +873,7 @@ echo "" echo "5. File Structure:" ls -la web_interface/ | head -10 -ls -la plugins/ | head -10 +ls -la plugin-repos/ | head -10 echo "" echo "6. Python Imports:" @@ -954,12 +949,11 @@ sudo systemctl restart ledmatrix-web # Reinstall WiFi monitor sudo ./scripts/install/install_wifi_monitor.sh -# Recreate service files from templates -sudo cp templates/ledmatrix.service /etc/systemd/system/ -sudo cp templates/ledmatrix-web.service /etc/systemd/system/ +# Recreate service files (substitutes __PROJECT_ROOT_DIR__ in systemd/ units) +sudo ./scripts/install/install_service.sh +sudo ./scripts/install/install_web_service.sh -# Reload and restart -sudo systemctl daemon-reload +# Restart sudo systemctl restart ledmatrix ledmatrix-web ``` diff --git a/docs/WEB_INTERFACE_GUIDE.md b/docs/WEB_INTERFACE_GUIDE.md index 09afc494..c57691b9 100644 --- a/docs/WEB_INTERFACE_GUIDE.md +++ b/docs/WEB_INTERFACE_GUIDE.md @@ -39,12 +39,18 @@ present: - **WiFi** — Network selection and AP-mode setup - **Schedule** — Power and dim schedules - **Display** — Matrix hardware configuration (rows, cols, hardware - mapping, GPIO slowdown, brightness, PWM) + mapping, GPIO slowdown, brightness, PWM) and Vegas Scroll Mode + settings +- **Rotation** — drag-and-drop **Rotation Order** list and per-plugin + **Screen Durations** - **Config Editor** — Raw `config.json` editor with validation +- **Backup & Restore** — back up and restore your configuration - **Fonts** — Upload and manage fonts - **Logs** — Real-time log streaming - **Cache** — Cached data inspection and cleanup - **Operation History** — Recent service operations +- **Tools** — system diagnostics, git & updates, Python dependencies, + maintenance, power supply, network radio, services, and plugin health A second nav row holds plugin tabs: @@ -111,6 +117,12 @@ Configure your LED matrix hardware: - Dynamic Duration — global cap for plugins that extend their display time based on content +**Vegas Scroll Mode:** the Display tab also has a full Vegas Scroll +Mode section — enable toggle, scroll speed, separator width, dynamic +duration, and related settings — so you can configure Vegas mode +entirely from the web UI without hand-editing JSON. See +[ADVANCED_FEATURES.md](ADVANCED_FEATURES.md) for what the options do. + Changes require **Restart Display Service** from the Overview tab. ### Plugin Manager Tab @@ -159,9 +171,10 @@ Manage fonts for your display: - See font previews - Check font sizes and styles -**Plugin Font Overrides:** -- Set custom fonts for specific plugins -- Override default font choices +**Font Overrides:** +- Overrides are set per display *element* (e.g. a specific score or + clock text element), not per plugin +- Override default font choices for individual elements - Preview font changes **Delete Fonts:** @@ -183,9 +196,11 @@ View real-time system logs: - Filter by plugin or component **Actions:** +- **Refresh**: Reload the log view - **Clear**: Clear the current view - **Download**: Download logs for offline analysis -- **Pause**: Pause auto-scrolling +- **Auto-scroll** checkbox: toggle automatic scrolling to the latest + entries --- @@ -248,7 +263,8 @@ The web interface uses Server-Sent Events (SSE) for real-time updates: **Performance:** - Minimal bandwidth usage - Server-side rendering for fast load times -- Progressive enhancement - works without JavaScript +- The UI is built on Alpine.js and HTMX, so JavaScript must be enabled + in the browser --- @@ -267,17 +283,6 @@ The interface is fully responsive and works on mobile devices: --- -## Keyboard Shortcuts - -Use keyboard shortcuts for faster navigation: - -- **Tab**: Navigate between form fields -- **Enter**: Submit forms -- **Esc**: Close modals -- **Ctrl+F**: Search in logs - ---- - ## API Access The web interface is built on a REST API that you can access programmatically: @@ -288,7 +293,7 @@ http://your-pi-ip:5000/api/v3 ``` The API blueprint mounts at `/api/v3` (see -`web_interface/app.py:144`). All endpoints below are relative to that +`web_interface/app.py:199`). All endpoints below are relative to that base. **Common Endpoints:** diff --git a/.cursor/plans/implement_audit_fixes_plan.md b/docs/archive/CURSOR_PLUGIN_SCHEMA_AUDIT_PLAN.md similarity index 100% rename from .cursor/plans/implement_audit_fixes_plan.md rename to docs/archive/CURSOR_PLUGIN_SCHEMA_AUDIT_PLAN.md diff --git a/docs/PLUGIN_CUSTOM_ICONS_FEATURE.md b/docs/archive/PLUGIN_CUSTOM_ICONS_FEATURE.md similarity index 100% rename from docs/PLUGIN_CUSTOM_ICONS_FEATURE.md rename to docs/archive/PLUGIN_CUSTOM_ICONS_FEATURE.md diff --git a/docs/widget-guide.md b/docs/widget-guide.md index 7a1ed850..9bc271eb 100644 --- a/docs/widget-guide.md +++ b/docs/widget-guide.md @@ -206,6 +206,47 @@ To use an existing widget in your plugin's `config_schema.json`, simply add the The widget will be automatically rendered when the plugin configuration form is loaded. +## Labelling Enum Options (`x-options.labels`) + +A plain `enum` renders as a dropdown whose option text is the value with +underscores replaced and title case applied — `day_first` becomes "Day First". +That is fine for values that read as their own label, and wrong for values that +do not: `vs` becomes "Vs", and `abbrev` says nothing about the `Sep 19` it +actually produces. + +Supply `x-options.labels` to set the visible text. This is the same convention +the `checkbox-group` widget uses: + +```json +{ + "date_format": { + "type": "string", + "enum": ["abbrev", "numeric", "day_first"], + "default": "abbrev", + "x-options": { + "labels": { + "abbrev": "Sep 19", + "numeric": "9/19", + "day_first": "19 Sep" + } + } + } +} +``` + +Labels are **display only** — the stored value is still the enum value, so +adding them never changes a saved config. The map may be partial: any value +without a label keeps the humanised fallback. Older cores that predate this +support ignore `x-options` and render the fallback for every option, so a +plugin can ship labels without requiring a core upgrade. + +Array-table columns (`x-widget: array-table`) accept the same +`x-options.labels` on a column definition, but their fallback is the **raw +value** rather than the humanised one, because those columns hold values such +as ticker symbols where `aapl` → "Aapl" would be wrong. Rows added in the +browser use the labels too (`array-table.js`), so a column reads the same +before and after a page reload. + ## Marking Fields as Advanced (`x-advanced`) Add `"x-advanced": true` to any top-level, non-object property to move it out diff --git a/first_time_install.sh b/first_time_install.sh old mode 100644 new mode 100755 index d8ef6338..198a8f2e --- a/first_time_install.sh +++ b/first_time_install.sh @@ -821,10 +821,6 @@ if [ ! -f "$PROJECT_ROOT_DIR/config/config_secrets.json" ]; then echo "⚠ Template config/config_secrets.template.json not found; creating a minimal secrets file" cat > "$PROJECT_ROOT_DIR/config/config_secrets.json" <<'EOF' { - "youtube": { - "api_key": "YOUR_YOUTUBE_API_KEY", - "channel_id": "YOUR_YOUTUBE_CHANNEL_ID" - }, "github": { "api_token": "YOUR_GITHUB_PERSONAL_ACCESS_TOKEN" } @@ -1187,25 +1183,26 @@ else cd "$PROJECT_ROOT_DIR" # Try to install dependencies using the smart installer if available + WEB_DEPS_OK=true if [ -f "$PROJECT_ROOT_DIR/scripts/install_dependencies_apt.py" ]; then echo "Using smart dependency installer..." # -u: unbuffered stdout/stderr so output is captured in $LOG_FILE in # real time and in order relative to this script's own echo statements - python3 -u "$PROJECT_ROOT_DIR/scripts/install_dependencies_apt.py" - else - echo "Using pip to install dependencies..." - if [ -f "$PROJECT_ROOT_DIR/requirements_web_v2.txt" ]; then - # --ignore-installed: see the Step 5 web_interface/requirements.txt - # install above — same apt/pip RECORD-file conflict applies here. - python3 -m pip install --break-system-packages --prefer-binary --ignore-installed -r requirements_web_v2.txt - else - echo "⚠ requirements_web_v2.txt not found; skipping web dependency install" + if ! python3 -u "$PROJECT_ROOT_DIR/scripts/install_dependencies_apt.py"; then + WEB_DEPS_OK=false fi + else + echo "Web dependencies already installed from web_interface/requirements.txt in Step 5" fi - # Create marker file to indicate dependencies are installed - touch "$PROJECT_ROOT_DIR/.web_deps_installed" - echo "✓ Web interface dependencies installed" + # Create the marker only when installation actually succeeded, so a + # re-run retries instead of silently skipping missing dependencies. + if [ "$WEB_DEPS_OK" = true ]; then + touch "$PROJECT_ROOT_DIR/.web_deps_installed" + echo "✓ Web interface dependencies installed" + else + echo "⚠ Web interface dependency install reported errors; not creating .web_deps_installed (will retry on next run)" + fi fi echo "" @@ -1422,9 +1419,16 @@ $ACTUAL_USER ALL=(ALL) NOPASSWD: $BASH_PATH $PROJECT_ROOT_DIR/scripts/fix_perms/ EOF if [ -n "$JOURNALCTL_PATH" ]; then cat >> /tmp/ledmatrix_web_sudoers << EOF -$ACTUAL_USER ALL=(ALL) NOPASSWD: $JOURNALCTL_PATH -u ledmatrix.service * -$ACTUAL_USER ALL=(ALL) NOPASSWD: $JOURNALCTL_PATH -u ledmatrix * -$ACTUAL_USER ALL=(ALL) NOPASSWD: $JOURNALCTL_PATH -t ledmatrix * +# NOEXEC, because these rules end in a wildcard and journalctl starts a pager +# when its output is a terminal. From that pager (less) a "!sh" is a root +# shell -- the standard journalctl escalation. The web interface always passes +# --no-pager, so nothing here needs it, but the rule cannot require a flag that +# sits in the middle of the command line. NOEXEC stops the command executing +# another program at all, which closes the hole without depending on wildcard +# matching subtleties. +$ACTUAL_USER ALL=(ALL) NOPASSWD:NOEXEC: $JOURNALCTL_PATH -u ledmatrix.service * +$ACTUAL_USER ALL=(ALL) NOPASSWD:NOEXEC: $JOURNALCTL_PATH -u ledmatrix * +$ACTUAL_USER ALL=(ALL) NOPASSWD:NOEXEC: $JOURNALCTL_PATH -t ledmatrix * EOF fi @@ -1483,27 +1487,54 @@ if [ -f "$PROJECT_ROOT_DIR/config/config.json" ]; then fi # Set proper permissions for secrets file (restrictive: owner rw, group r) -# If service runs as root, set ownership to root so it can read as owner -# Otherwise, use ACTUAL_USER and rely on group membership +# Owned by whoever WRITES the file, which is the web interface. +# +# This used to read the User= of ledmatrix.service — the display service — +# and, finding root, hand the file to root:ledmatrix 640. But the display +# service only ever reads secrets, and root can read any file regardless of +# mode. The account that *writes* them is the web interface: it saves config +# edits and performs backup restores, and it deliberately does not run as root +# (a web server should not). So a root-owned, group-read-only file left the web +# UI unable to write its own secrets, and restoring a backup failed with +# "Permission denied: config_secrets.json" while every other file in the same +# backup restored fine. +# +# Owning by the writer keeps the tighter 640 rather than loosening to +# group-writable, and root still reads it as superuser. if [ -f "$PROJECT_ROOT_DIR/config/config_secrets.json" ]; then - # Check if service runs as root (from service file or template) - SERVICE_USER="root" - if [ -f "/etc/systemd/system/ledmatrix.service" ]; then - SERVICE_USER=$(grep "^User=" /etc/systemd/system/ledmatrix.service | cut -d'=' -f2 || echo "root") - elif [ -f "$PROJECT_ROOT_DIR/systemd/ledmatrix.service" ]; then - SERVICE_USER=$(grep "^User=" "$PROJECT_ROOT_DIR/systemd/ledmatrix.service" | cut -d'=' -f2 || echo "root") + # The web service is the writer; fall back to the display service, then to + # the installing user, so an unusual layout still lands somewhere sensible. + SECRETS_OWNER="" + for unit in "/etc/systemd/system/ledmatrix-web.service" \ + "$PROJECT_ROOT_DIR/systemd/ledmatrix-web.service"; do + if [ -f "$unit" ]; then + SECRETS_OWNER=$(grep -m1 "^User=" "$unit" | cut -d'=' -f2) + [ -n "$SECRETS_OWNER" ] && break + fi + done + if [ -z "$SECRETS_OWNER" ]; then + SECRETS_OWNER="$ACTUAL_USER" fi - - if [ "$SERVICE_USER" = "root" ]; then - # Service runs as root - set ownership to root so it can read as owner - chown "root:$LEDMATRIX_GROUP" "$PROJECT_ROOT_DIR/config/config_secrets.json" || true - echo "✓ Secrets file permissions set (root:ledmatrix for root service)" - else - # Service runs as regular user - use ACTUAL_USER and rely on group membership - chown "$ACTUAL_USER:$LEDMATRIX_GROUP" "$PROJECT_ROOT_DIR/config/config_secrets.json" || true - echo "✓ Secrets file permissions set ($ACTUAL_USER:ledmatrix)" + SECRETS_FILE="$PROJECT_ROOT_DIR/config/config_secrets.json" + # A root-owned file is only correct when the writer really is root. + if ! chown "$SECRETS_OWNER:$LEDMATRIX_GROUP" "$SECRETS_FILE"; then + echo "✗ ERROR: Failed to set ownership on $SECRETS_FILE to $SECRETS_OWNER:$LEDMATRIX_GROUP" >&2 + echo " Try: sudo chown $SECRETS_OWNER:$LEDMATRIX_GROUP $SECRETS_FILE" >&2 + exit 1 fi - chmod 640 "$PROJECT_ROOT_DIR/config/config_secrets.json" + if ! chmod 640 "$SECRETS_FILE"; then + echo "✗ ERROR: Failed to set permissions on $SECRETS_FILE to 640" >&2 + echo " Try: sudo chmod 640 $SECRETS_FILE" >&2 + exit 1 + fi + ACTUAL_OWNERSHIP=$(stat -c '%U:%G' "$SECRETS_FILE" 2>/dev/null || echo "unknown") + ACTUAL_MODE=$(stat -c '%a' "$SECRETS_FILE" 2>/dev/null || echo "unknown") + if [ "$ACTUAL_OWNERSHIP" != "$SECRETS_OWNER:$LEDMATRIX_GROUP" ] || [ "$ACTUAL_MODE" != "640" ]; then + echo "✗ ERROR: $SECRETS_FILE ended up as $ACTUAL_OWNERSHIP mode $ACTUAL_MODE, expected $SECRETS_OWNER:$LEDMATRIX_GROUP mode 640" >&2 + echo " The web interface may be unable to read or write config_secrets.json." >&2 + exit 1 + fi + echo "✓ Secrets file owned by the web service user ($SECRETS_OWNER:$LEDMATRIX_GROUP, mode 640)" fi # Set proper permissions for YTM auth file (readable by all users including root service) @@ -1664,6 +1695,97 @@ else echo "✗ $CMDLINE_FILE not found; skipping isolcpus optimization" fi +# Enable the memory cgroup controller (idempotent). +# The Pi firmware boots with cgroup_disable=memory, so systemd's MemoryMax= is +# accepted and silently ignored — the display service then has no ceiling, and +# a runaway takes the whole board down (sshd can no longer fork, the panel goes +# dark) rather than just restarting the one service. +if [ "$SKIP_PERF" != "1" ] && [ -f "$CMDLINE_FILE" ]; then + # Both parameters are required for the memory controller, and they can get + # separated -- an image, another tool or a half-applied earlier run can + # leave one without the other. Checking only cgroup_enable=memory would + # report success while MemoryMax= silently does nothing, so each is checked + # and appended independently. + cgroup_missing="" + for cgroup_param in cgroup_enable=memory cgroup_memory=1; do + if ! grep -qw "$cgroup_param" "$CMDLINE_FILE"; then + cgroup_missing="$cgroup_missing $cgroup_param" + fi + done + if [ -z "$cgroup_missing" ]; then + echo "cgroup memory parameters already present in $CMDLINE_FILE" + else + echo "Adding${cgroup_missing} to $CMDLINE_FILE..." + cp "$CMDLINE_FILE" "$CMDLINE_FILE.bak" 2>/dev/null || true + # The kernel command line must stay on one line. + sed -i "1 s|\$|${cgroup_missing}|" "$CMDLINE_FILE" + echo " Takes effect after reboot. Verify with:" + echo " grep memory /sys/fs/cgroup/cgroup.controllers" + fi +fi + +# Persist the journal (idempotent). +# These images default to volatile storage: journald keeps everything in /run +# (tmpfs), so every reboot destroys the logs — including the ones that would +# explain why the board rebooted. Capped so an SD card is not worn out by logs. +# A non-empty /var/log/journal does not prove journald is configured the way +# this needs: the directory survives a switch back to volatile storage, and it +# says nothing about whether a size cap is set. Read the effective +# configuration instead, and only write the keys the user has not set +# themselves so an explicit local limit is preserved. +journald_effective() { + # systemd-analyze merges journald.conf with every drop-in; grep is the + # fallback for images that ship without it. + if command -v systemd-analyze >/dev/null 2>&1 && + systemd-analyze cat-config systemd/journald.conf >/dev/null 2>&1; then + systemd-analyze cat-config systemd/journald.conf 2>/dev/null + else + cat /etc/systemd/journald.conf /etc/systemd/journald.conf.d/*.conf 2>/dev/null || true + fi +} +journald_conf="$(journald_effective)" +# Both settings are optional, and stock images ship them commented out. grep +# exits 1 on no match, which pipefail turns fatal under set -e — an absent +# setting must read as empty, not abort the install. +journald_storage="$(printf '%s\n' "$journald_conf" | grep -E '^[[:space:]]*Storage=' | tail -n1 | cut -d= -f2 | tr -d '[:space:]' || true)" +journald_cap="$(printf '%s\n' "$journald_conf" | grep -E '^[[:space:]]*SystemMaxUse=' | tail -n1 | cut -d= -f2 | tr -d '[:space:]' || true)" + +if [ "$journald_storage" = "persistent" ] && [ -n "$journald_cap" ]; then + echo "Persistent journald storage already configured (SystemMaxUse=$journald_cap)" +else + echo "Enabling persistent journald storage..." + mkdir -p /etc/systemd/journald.conf.d + { + echo "# Installed by LEDMatrix first_time_install.sh" + echo "[Journal]" + echo "Storage=persistent" + if [ -n "$journald_cap" ]; then + echo "# SystemMaxUse left to your existing setting ($journald_cap)" + else + # Capped so logs cannot wear out or fill an SD card. + echo "SystemMaxUse=64M" + fi + } > /etc/systemd/journald.conf.d/ledmatrix-persistent.conf + mkdir -p /var/log/journal + systemd-tmpfiles --create --prefix /var/log/journal >/dev/null 2>&1 || true + systemctl restart systemd-journald >/dev/null 2>&1 || true + + # Drop-ins are applied in lexical order, so a locally added file that sorts + # after ledmatrix-persistent.conf (zz-local.conf and friends) still wins. + # Writing the file is not evidence it took effect -- re-read and say so + # plainly rather than reporting success we cannot confirm. + journald_now="$(journald_effective | grep -E '^[[:space:]]*Storage=' | tail -n1 | cut -d= -f2 | tr -d '[:space:]' || true)" + if [ "$journald_now" = "persistent" ]; then + echo " Persistent journald storage active" + else + echo " WARNING: journald storage is still '${journald_now:-unset}' after" + echo " writing /etc/systemd/journald.conf.d/ledmatrix-persistent.conf." + echo " Another drop-in that sorts later is overriding it. Check:" + echo " systemd-analyze cat-config systemd/journald.conf | grep -n Storage=" + echo " Logs will not survive a reboot until that is resolved." + fi +fi + # Ensure dtparam=audio=off in config.txt (idempotent) if [ "$SKIP_PERF" = "1" ]; then : # skipped diff --git a/pytest.ini b/pytest.ini index a13cc6cb..aaffa360 100644 --- a/pytest.ini +++ b/pytest.ini @@ -10,16 +10,13 @@ python_functions = test_* testpaths = test # Output options -# Note: Coverage options require pytest-cov to be installed -# Run: pip install pytest-cov -addopts = +# Coverage is deliberately NOT configured here: a bare local `pytest` should +# be fast and dependency-light. Coverage is measured and enforced in exactly +# one place — the unit-tests job in .github/workflows/test.yml. +addopts = -v --strict-markers --tb=short - --cov=src - --cov-report=term-missing - --cov-report=html - --cov-fail-under=30 # Markers markers = diff --git a/requirements-test.txt b/requirements-test.txt index a11b33a4..60e26951 100644 --- a/requirements-test.txt +++ b/requirements-test.txt @@ -1,9 +1,9 @@ -# Test-only dependencies for the plugin safety harness and pytest suite. +# Test/dev-only dependencies (not needed on a running display). # Install alongside requirements.txt: pip install -r requirements.txt -r requirements-test.txt -# -# pytest, pytest-cov, pytest-mock, and jsonschema are already pinned (with -# major-version caps) in requirements.txt, so they are intentionally NOT -# repeated here — re-pinning pytest to <9 collided with requirements.txt's -# pytest>=9.0.3,<10 and made the two files impossible to install together. -# Only declare what requirements.txt doesn't already provide. +pytest>=9.0.3,<10.0.0 +pytest-cov>=4.1.0,<5.0.0 +pytest-mock>=3.11.0,<4.0.0 freezegun>=1.2,<2 # deterministic time for golden-image tests +psutil>=6.0.0,<8.0.0 # optional at runtime; installed for tests so the + # /system/status endpoint's real path is exercised +mypy>=1.5.0,<2.0.0 # static type checking (also pinned in .pre-commit-config.yaml) diff --git a/requirements.txt b/requirements.txt index 9725d6c3..760dc596 100644 --- a/requirements.txt +++ b/requirements.txt @@ -11,27 +11,22 @@ pytz>=2024.2,<2025.0 # Updated for latest timezone data # HTTP requests requests>=2.33.0,<3.0.0 +urllib3>=2.7.0,<3.0.0 # requests transitive, but imported directly (urllib3.util.retry.Retry); floor is a security floor, not the API floor — 1.26.x carries ~10 CVEs # Google API integration # Font rendering freetype-py>=2.5.1,<3.0.0 -# Spotify integration +# Spotify integration (used by web_interface/blueprints/api_v3.py OAuth endpoints) spotipy>=2.25.2,<3.0.0 # Flask web framework Flask>=3.1.3,<4.0.0 -# Text processing - -# Calendar integration - -# WebSocket support -python-socketio>=5.14.0,<6.0.0 -python-engineio>=4.9.0,<5.0.0 -websockets>=12.0,<14.0 -websocket-client>=1.8.0,<2.0.0 +# WebSocket support: intentionally NOT declared here. Plugins that need +# it (e.g. ledmatrix-music's Socket.IO client) declare it in their own +# requirements.txt, which the plugin store installs. # JSON Schema validation jsonschema>=4.20.0,<5.0.0 @@ -39,11 +34,8 @@ jsonschema>=4.20.0,<5.0.0 # Requirement specifier parsing (plugin dependency satisfaction checks) packaging>=23.0,<27.0 -# Testing dependencies -pytest>=9.0.3,<10.0.0 -pytest-cov>=4.1.0,<5.0.0 -pytest-mock>=3.11.0,<4.0.0 -mypy>=1.5.0,<2.0.0 +# Testing dependencies live in requirements-test.txt: +# pip install -r requirements.txt -r requirements-test.txt # ─────────────────────────────────────────────────────────────────────── # Optional dependencies — the code imports these inside try/except @@ -59,7 +51,9 @@ mypy>=1.5.0,<2.0.0 # psutil — per-plugin resource monitoring in # src/plugin_system/resource_monitor.py. The monitor # silently no-ops when missing (PSUTIL_AVAILABLE = False). -# pip install 'psutil>=5.9.0,<6.0.0' +# Note: web_interface/requirements.txt requires this +# range as a hard dependency — keep the two in sync. +# pip install 'psutil>=6.0.0,<7.0.0' # # Flask-Limiter — request rate limiting in web_interface/app.py # (accidental-abuse protection, not security). The diff --git a/scripts/add_defaults_to_schemas.py b/scripts/add_defaults_to_schemas.py index c0474530..3ef33e94 100755 --- a/scripts/add_defaults_to_schemas.py +++ b/scripts/add_defaults_to_schemas.py @@ -201,7 +201,7 @@ def process_schema_file(schema_path: Path) -> bool: def main(): """Main entry point.""" project_root = Path(__file__).parent.parent - plugins_dir = project_root / 'plugins' + plugins_dir = project_root / 'plugin-repos' if not plugins_dir.exists(): print(f"Error: Plugins directory not found: {plugins_dir}") diff --git a/scripts/analyze_plugin_schemas.py b/scripts/analyze_plugin_schemas.py index d10a308f..e05f20eb 100755 --- a/scripts/analyze_plugin_schemas.py +++ b/scripts/analyze_plugin_schemas.py @@ -193,7 +193,7 @@ def analyze_schema(schema_path: Path) -> Dict[str, Any]: def main(): """Main analysis function.""" project_root = Path(__file__).parent.parent - plugins_dir = project_root / "plugins" + plugins_dir = project_root / "plugin-repos" if not plugins_dir.exists(): print(f"Plugins directory not found: {plugins_dir}") diff --git a/scripts/check_plugin.py b/scripts/check_plugin.py index 997bf6d8..3f111d4d 100644 --- a/scripts/check_plugin.py +++ b/scripts/check_plugin.py @@ -41,6 +41,7 @@ from src.plugin_system.testing.loading import ( # noqa: E402 ) from src.plugin_system.testing.harness import ( # noqa: E402 RenderResult, render_plugin_matrix, compare_to_goldens, write_goldens, + check_empty_claimed, check_scale_up, ) from src.plugin_system.testing.sizes import ( # noqa: E402 @@ -115,6 +116,11 @@ def check_one(plugin_id: str, search_dirs: List[str], sizes, mock_data: Dict, declared = load_manifest(plugin_dir).get("display", {}).get("design_size", {}) design_size = (int(declared.get("width", 128)), int(declared.get("height", 32))) fill_strict = spec.get("fill_check") == "strict" + # A mode that renders nothing without returning False is never skipped by + # the display controller, so it holds a blank panel for its whole duration. + # Warn-only by default: a scroll mode's first frame is legitimately its + # blank scroll-in buffer. + empty_strict = spec.get("empty_check") == "strict" # Every run: the base config, plus one per harness.json "variant" — # a config overlay with its own golden dir (e.g. adaptive layout mode @@ -142,6 +148,7 @@ def check_one(plugin_id: str, search_dirs: List[str], sizes, mock_data: Dict, compare_to_goldens(results, golden_dir) check_scale_up(results, design_size=design_size, strict=fill_strict) + check_empty_claimed(results, strict=empty_strict) # Tag variant runs so the report and PNG dumps stay distinguishable. if variant_name: @@ -178,6 +185,9 @@ def print_report(all_results: Dict[str, List[RenderResult]]) -> bool: # warn-only underfill: big panel left mostly empty ex, ey = r.fill_extent detail += f" (fill warn: extent {ex:.0%}x{ey:.0%})" + if r.empty_claimed and r.empty_ok is None: + detail += (f" (empty warn: drew nothing but display() returned" + f" {r.display_returned!r}, so the mode is not skipped)") else: everything_ok = False if r.error is not None: @@ -191,6 +201,11 @@ def print_report(all_results: Dict[str, List[RenderResult]]) -> bool: ex, ey = r.fill_extent or (0.0, 0.0) status = "FAIL" detail = f" fill: extent {ex:.0%}x{ey:.0%} below required coverage" + elif r.empty_ok is False: + status = "FAIL" + detail = (f" drew nothing but display() returned" + f" {r.display_returned!r}; return False so the" + f" controller skips the mode") else: status, detail = "FAIL", "" print(f" [{status}] {r.size_label:>7} {r.mode}{detail}") diff --git a/scripts/debug/check_imports.py b/scripts/debug/check_imports.py deleted file mode 100644 index a1cefae0..00000000 --- a/scripts/debug/check_imports.py +++ /dev/null @@ -1,28 +0,0 @@ -#!/usr/bin/env python3 -""" -Check what imports are actually in the app.py file on the Pi -""" - -from pathlib import Path - -# Read the app.py file and check the import lines -app_py_path = Path.home() / 'LEDMatrix' / 'web_interface' / 'app.py' - -print(f"🔍 Checking imports in: {app_py_path}") -print(f"📁 File exists: {app_py_path.exists()}") - -if app_py_path.exists(): - with open(app_py_path, 'r') as f: - lines = f.readlines() - - print("\n🔍 Import lines in app.py:") - for i, line in enumerate(lines, 1): - if 'from' in line and 'blueprints' in line and 'import' in line: - print(f" Line {i}: {line.strip()}") - - print("\n🔍 Blueprint registration lines:") - for i, line in enumerate(lines, 1): - if 'register_blueprint' in line: - print(f" Line {i}: {line.strip()}") -else: - print("❌ app.py file not found!") diff --git a/scripts/debug/debug_web_manual.py b/scripts/debug/debug_web_manual.py index 36f64e1a..62620946 100644 --- a/scripts/debug/debug_web_manual.py +++ b/scripts/debug/debug_web_manual.py @@ -13,8 +13,8 @@ def main(): print("🔍 LED Matrix Web Interface Debug Tool") print("=" * 50) - # Change to project root (where this script is located) - project_root = Path(__file__).parent.resolve() + # Change to project root (two levels up from scripts/debug/) + project_root = Path(__file__).parent.parent.parent.resolve() os.chdir(project_root) print(f"📁 Working directory: {os.getcwd()}") diff --git a/scripts/debug/direct_fix_imports.py b/scripts/debug/direct_fix_imports.py deleted file mode 100644 index 79d451a9..00000000 --- a/scripts/debug/direct_fix_imports.py +++ /dev/null @@ -1,58 +0,0 @@ -#!/usr/bin/env python3 -""" -Direct fix for import issues - manually edit the app.py file -""" - -import os -from pathlib import Path - -def fix_imports(): - app_py_path = Path.home() / 'LEDMatrix' / 'web_interface' / 'app.py' - - print(f"🔧 Directly fixing imports in: {app_py_path}") - - # Read the file - with open(app_py_path, 'r') as f: - lines = f.readlines() - - # Find and fix the import lines - fixed = False - for i, line in enumerate(lines, 1): - if 'from blueprints.pages_v3 import' in line: - lines[i-1] = "from web_interface.blueprints.pages_v3 import pages_v3\n" - print(f"✅ Fixed line {i}: from blueprints.pages_v3 import → from web_interface.blueprints.pages_v3 import") - fixed = True - elif 'from blueprints.api_v3 import' in line: - lines[i-1] = "from web_interface.blueprints.api_v3 import api_v3\n" - print(f"✅ Fixed line {i}: from blueprints.api_v3 import → from web_interface.blueprints.api_v3 import") - fixed = True - - if not fixed: - print("❌ No import lines found to fix") - return False - - # Write the fixed file back - with open(app_py_path, 'w') as f: - f.writelines(lines) - - print("✅ File updated successfully") - return True - -def verify_fix(): - print("\n🔍 Verifying the fix...") - os.system("python3 check_imports.py") - -if __name__ == "__main__": - if fix_imports(): - print("\n🧹 Clearing Python cache...") - os.system("find ~/LEDMatrix -type d -name '__pycache__' -exec rm -rf {} + 2>/dev/null || true") - os.system("find ~/LEDMatrix -name '*.pyc' -delete 2>/dev/null || true") - - print("\n✅ Imports fixed and cache cleared!") - verify_fix() - - print("\n🚀 Now try running the web interface:") - print("cd ~/LEDMatrix") - print("python3 web_interface/start.py") - else: - print("\n❌ Fix failed") diff --git a/scripts/download_nba_logos.py b/scripts/download_nba_logos.py index c0bfdb89..99bb5b4b 100644 --- a/scripts/download_nba_logos.py +++ b/scripts/download_nba_logos.py @@ -7,8 +7,8 @@ import os import logging from typing import Tuple -# Add the src directory to Python path so we can import the logo downloader -sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'src')) +# Add the project root to Python path so we can import the logo downloader +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) # Set up logging logging.basicConfig( @@ -28,7 +28,7 @@ def download_nba_logos(force_download: bool = False) -> Tuple[int, int]: Tuple of (downloaded_count, failed_count) """ try: - from logo_downloader import download_all_logos_for_league + from src.logo_downloader import download_all_logos_for_league logger.info("🏀 Starting NBA logo download...") logger.info(f"Target directory: assets/sports/nba_logos/") diff --git a/scripts/download_pixlet.sh b/scripts/download_pixlet.sh index 53a38d22..80caeb74 100755 --- a/scripts/download_pixlet.sh +++ b/scripts/download_pixlet.sh @@ -24,9 +24,29 @@ echo "========================================" # Auto-detect latest version if needed if [ "$PIXLET_VERSION" = "latest" ]; then echo "Detecting latest version..." - PIXLET_VERSION=$(curl -s "https://api.github.com/repos/${REPO}/releases/latest" | grep '"tag_name"' | sed -E 's/.*"([^"]+)".*/\1/') - if [ -z "$PIXLET_VERSION" ]; then - echo "Failed to detect latest version, using fallback" + # When this response arrives on a single line -- as it did on the device + # where Starlark apps were failing -- `grep '"tag_name"'` matches the whole + # document and a greedy `sed 's/.*"([^"]+)".*/\1/'` captures the LAST + # quoted token in it rather than the tag. That resolved to + # "mentions_count", which built a download URL for a release that does not + # exist. (The API is pretty-printed by default, which is why the old + # command looks correct when you try it by hand -- but the formatting is + # not something to depend on.) Match the field itself and take the value + # after it, which is right for either shape. + PIXLET_VERSION=$(curl -fsSL "https://api.github.com/repos/${REPO}/releases/latest" \ + | grep -o '"tag_name"[[:space:]]*:[[:space:]]*"[^"]*"' \ + | head -n1 \ + | sed -E 's/.*:[[:space:]]*"([^"]*)".*/\1/') + + # A wrong-but-non-empty value is what made the old bug silent, so check the + # shape rather than just that something came back. Anchored at both ends: a + # partial match would accept "v0.53garbage" or "0.53" and build a URL for a + # release that cannot exist, which is the failure this check is here to + # stop. Every tronbyt/pixlet release to date is vX.Y.Z; the optional suffix + # leaves room for a future -rc.1 or +build tag. + if ! printf '%s' "$PIXLET_VERSION" \ + | grep -qE '^v[0-9]+\.[0-9]+\.[0-9]+([-+][0-9A-Za-z.-]+)?$'; then + echo "Could not detect the latest version (got: '${PIXLET_VERSION:-}'), using fallback" PIXLET_VERSION="v0.50.2" fi fi @@ -67,8 +87,26 @@ download_binary() { temp_dir=$(mktemp -d -p "$PROJECT_ROOT" -t pixlet_download.XXXXXXXXXX) local temp_file="$temp_dir/$archive_name" - if ! curl -L -o "$temp_file" "$url" 2>/dev/null; then - echo "✗ Failed to download $arch" + # -f so an HTTP error is a failure. Without it curl writes the 404 body + # to the file and exits 0, and the first sign of trouble is tar saying + # "not in gzip format" about what is actually a page of HTML. + if ! curl -fL -o "$temp_file" "$url" 2>/dev/null; then + echo "✗ Failed to download $arch from $url" + rm -rf "$temp_dir" + return 1 + fi + + # Belt and braces: a mirror or proxy can return 200 with an error page. + if ! gzip -t "$temp_file" 2>/dev/null; then + echo "✗ Downloaded file is not a gzip archive: $url" + # These bytes come from whatever answered the request, so strip + # everything non-printable before echoing them: an error page carrying + # terminal escapes would otherwise be able to rewrite this output or + # bury it in a CI log. Printable characters are kept rather than + # hex-encoding the lot, because "" is the diagnostic. + local first_bytes + first_bytes=$(head -c 60 "$temp_file" | tr -cd '[:print:]') + printf ' (first bytes: %s)\n' "$first_bytes" rm -rf "$temp_dir" return 1 fi diff --git a/scripts/fix_perms/README.md b/scripts/fix_perms/README.md index 96f293d2..5616e51a 100644 --- a/scripts/fix_perms/README.md +++ b/scripts/fix_perms/README.md @@ -31,9 +31,6 @@ owned by the `ledmatrix` service user or by `root`. systemd journal access, and the sudoers entries the web interface needs to control the display service. -- **`fix_nhl_cache.sh`** — Targeted fix for NHL plugin cache issues - (clears the NHL cache and restarts the display service). - - **`safe_plugin_rm.sh`** — Validates that a plugin removal path is inside an allowed base directory before deleting it. Used by the web interface (via sudo) when a user clicks **Uninstall** on a plugin — diff --git a/scripts/fix_perms/fix_assets_permissions.sh b/scripts/fix_perms/fix_assets_permissions.sh old mode 100644 new mode 100755 diff --git a/scripts/fix_perms/fix_cache_permissions.sh b/scripts/fix_perms/fix_cache_permissions.sh old mode 100644 new mode 100755 diff --git a/scripts/fix_perms/fix_nhl_cache.sh b/scripts/fix_perms/fix_nhl_cache.sh deleted file mode 100644 index ce84514a..00000000 --- a/scripts/fix_perms/fix_nhl_cache.sh +++ /dev/null @@ -1,21 +0,0 @@ -#!/bin/bash -""" -Script to fix NHL cache issues on Raspberry Pi. -This will clear the NHL cache and restart the display service. -""" - -echo "==========================================" -echo "Fixing NHL Cache Issues" -echo "==========================================" - -# Clear NHL cache -echo "Clearing NHL cache..." -python3 clear_nhl_cache.py - -# Restart the display service to force fresh data fetch -echo "Restarting display service..." -sudo systemctl restart ledmatrix.service - -echo "NHL cache cleared and service restarted!" -echo "NHL managers should now fetch fresh data from ESPN API." -echo "Check the logs to see if NHL games are now being displayed." diff --git a/scripts/fix_perms/fix_plugin_permissions.sh b/scripts/fix_perms/fix_plugin_permissions.sh old mode 100644 new mode 100755 diff --git a/scripts/fix_perms/fix_web_permissions.sh b/scripts/fix_perms/fix_web_permissions.sh old mode 100644 new mode 100755 diff --git a/scripts/generate_report.py b/scripts/generate_report.py deleted file mode 100644 index 753ea35b..00000000 --- a/scripts/generate_report.py +++ /dev/null @@ -1,356 +0,0 @@ -#!/usr/bin/env python3 -""" -Security Report Generator - -Aggregates JSON output from all CI security audit jobs into a single -Markdown report suitable for PR comments and artifact storage. - -Expected artifact layout (from actions/download-artifact@v4): - / - sast-results/ - bandit-results.json - semgrep-results.json - dependency-audit-results/ - pip-audit-results.json - safety-results.json - secrets-scan-results/ - gitleaks-results.json - security-proofs-results/ - security-proofs-results.json - plugin-audit-results/ - plugin-audit-results.json - -Usage: - python scripts/generate_report.py --artifact-dir audit-artifacts/ --output report.md - python scripts/generate_report.py --artifact-dir audit-artifacts/ --output report.md --verbose -""" - -import argparse -import json -import sys -from pathlib import Path -from datetime import datetime, timezone - -PROJECT_ROOT = Path(__file__).resolve().parent.parent - -# Gitleaks matches exactly equal to one of these (not a substring match -- a -# real secret that merely contains one of these words as part of its actual -# value must still be reported) are known template placeholders. -_GITLEAKS_SUPPRESS_EXACT_VALUES = { - "YOUR_YOUTUBE_API_KEY", - "YOUR_YOUTUBE_CHANNEL_ID", - "YOUR_GITHUB_PERSONAL_ACCESS_TOKEN", -} - -# Findings in these files are suppressed regardless of value -- they are -# template/example files that are expected to only ever contain placeholders. -_GITLEAKS_SUPPRESS_PATHS = [ - "config_secrets.template.json", - "config.template.json", -] - - -# ───────────────────────────────────────────────────────────────────────────── -# Helpers -# ───────────────────────────────────────────────────────────────────────────── - -def _load(path: Path) -> tuple[dict | list | None, str | None]: - """Load a JSON artifact file. - - Returns (data, error): error is None on success (data is whatever was - parsed, which may legitimately be an empty list/dict for a clean scan); - otherwise error is a human-readable reason the artifact is unavailable, - distinguishing "missing/malformed artifact" from "valid empty result" so - callers don't silently treat a broken CI job as a clean pass. - """ - if not path.exists(): - return None, f"artifact not found: {path}" - try: - return json.loads(path.read_text(encoding="utf-8")), None - except (json.JSONDecodeError, OSError) as exc: - return None, f"could not read/parse {path}: {exc}" - - -def _md_sanitize_cell(value: object) -> str: - """Escape/normalize a value so scanner-controlled content (a matched - secret, a bandit issue_text, a file path) can't alter the Markdown - table's structure: pipes would add bogus columns, newlines would break - out of the row (or forge a fake header/separator line).""" - text = str(value) - text = text.replace("\\", "\\\\").replace("|", "\\|") - text = text.replace("\r\n", " ").replace("\n", " ").replace("\r", " ") - return text - - -def _md_table_row(*cells: str) -> str: - return "| " + " | ".join(_md_sanitize_cell(c) for c in cells) + " |" - - -# ───────────────────────────────────────────────────────────────────────────── -# Per-tool summarizers -# Returns: (markdown_lines: list[str], critical_count: int, available: bool) -# `available=False` means the artifact was missing or malformed -- distinct -# from a valid scan that simply found nothing -- so the caller can report -# INCOMPLETE instead of silently counting it as a clean pass. -# ───────────────────────────────────────────────────────────────────────────── - -def _summarize_bandit(artifact_dir: Path) -> tuple[list[str], int, bool]: - data, error = _load(artifact_dir / "sast-results" / "bandit-results.json") - if error: - return [f"_bandit results unavailable: {error}_"], 0, False - - results = data.get("results", []) - high = [r for r in results if r.get("issue_severity") == "HIGH"] - medium = [r for r in results if r.get("issue_severity") == "MEDIUM"] - low = [r for r in results if r.get("issue_severity") == "LOW"] - - lines = [ - f"**Bandit**: {len(high)} HIGH · {len(medium)} MEDIUM · {len(low)} LOW" - ] - - if high: - lines += [ - "", - "| Severity | File | Line | Issue |", - "| --- | --- | --- | --- |", - ] - for r in high[:10]: - fname = Path(r.get("filename", "")).name - lines.append(_md_table_row( - "HIGH", f"`{fname}`", - str(r.get("line_number", "?")), - r.get("issue_text", "") - )) - if len(high) > 10: - lines.append(f"_… and {len(high) - 10} more HIGH findings_") - - return lines, len(high), True - - -def _summarize_pip_audit(artifact_dir: Path) -> tuple[list[str], int, bool]: - data, error = _load(artifact_dir / "dependency-audit-results" / "pip-audit-results.json") - if error: - return [f"_pip-audit results unavailable: {error}_"], 0, False - - # pip-audit JSON format: {"dependencies": [{"name": ..., "vulns": [...]}]} - vulns: list[dict] = [] - for dep in data.get("dependencies", []): - for v in dep.get("vulns", []): - vulns.append({"package": dep.get("name", "?"), **v}) - - lines = [f"**pip-audit**: {len(vulns)} vulnerabilities found"] - - if vulns: - lines += ["", "| Package | ID | Fix |", "| --- | --- | --- |"] - for v in vulns[:10]: - fix = v.get("fix_versions", ["none"]) - fix_str = ", ".join(fix) if fix else "none" - lines.append(_md_table_row( - v.get("package", "?"), - v.get("id", "?"), - fix_str, - )) - - # Treat known vulnerabilities as warnings, not critical (they may be unavoidable) - return lines, 0, True - - -def _summarize_gitleaks(artifact_dir: Path) -> tuple[list[str], int, bool]: - data, error = _load(artifact_dir / "secrets-scan-results" / "gitleaks-results.json") - if error: - return [f"_gitleaks results unavailable: {error}_"], 0, False - - if not isinstance(data, list): - data = [] - - real_findings = [] - suppressed = 0 - for finding in data: - secret_val = str(finding.get("Secret", "") or finding.get("Match", "")) - file_name = Path(finding.get("File", "")).name - if (secret_val in _GITLEAKS_SUPPRESS_EXACT_VALUES - or file_name in _GITLEAKS_SUPPRESS_PATHS): - suppressed += 1 - else: - real_findings.append(finding) - - lines = [ - f"**Gitleaks**: {len(real_findings)} finding(s) " - f"({suppressed} suppressed as template placeholders)" - ] - - if real_findings: - lines += ["", "| Rule | File | Line | Description |", "| --- | --- | --- | --- |"] - for f in real_findings[:10]: - fname = Path(f.get("File", "")).name - lines.append(_md_table_row( - f.get("RuleID", "?"), - f"`{fname}`", - str(f.get("StartLine", "?")), - f.get("Description", ""), - )) - - critical = len(real_findings) # any real secret is critical - return lines, critical, True - - -def _summarize_security_proofs(artifact_dir: Path) -> tuple[list[str], int, bool]: - data, error = _load(artifact_dir / "security-proofs-results" / "security-proofs-results.json") - if error: - return [f"_security proofs results unavailable: {error}_"], 0, False - - if not isinstance(data, list): - data = [] - - critical = [r for r in data if r.get("severity") == "CRITICAL"] - warnings = [r for r in data if r.get("severity") == "WARNING"] - passed = [r for r in data if r.get("severity") == "PASS"] - skipped = [r for r in data if r.get("severity") == "SKIP"] - - lines = [ - f"**Security Proofs**: " - f"{len(passed)} PASS · {len(warnings)} WARN · " - f"{len(critical)} CRITICAL · {len(skipped)} SKIP", - "", - ] - - _icon = {"PASS": "✅", "INFO": "ℹ️", "WARNING": "⚠️", # nosec B105 - severity labels, not credentials - "CRITICAL": "🚨", "SKIP": "⏭️"} - for r in data: - icon = _icon.get(r.get("severity", ""), "❓") - lines.append( - f"- {icon} **{r.get('test_id', '?')}**: {r.get('message', '')}" - ) - if r.get("details") and r.get("severity") in ("CRITICAL", "WARNING"): - lines.append(f" - _{r['details']}_") - - return lines, len(critical), True - - -def _summarize_plugin_audit(artifact_dir: Path) -> tuple[list[str], int, bool]: - data, error = _load(artifact_dir / "plugin-audit-results" / "plugin-audit-results.json") - if error: - return [f"_plugin audit results unavailable: {error}_"], 0, False - - summary = data.get("summary", {}) - findings = data.get("findings", []) - critical_findings = [f for f in findings if f.get("severity") == "CRITICAL"] - warning_findings = [f for f in findings if f.get("severity") == "WARNING"] - - lines = [ - f"**Plugin Audit**: {data.get('plugins_scanned', '?')} plugins scanned — " - f"{summary.get('critical', 0)} CRITICAL · {summary.get('warnings', 0)} WARNINGS" - ] - - if critical_findings: - lines += ["", "| Plugin | File | Line | Rule | Message |", - "| --- | --- | --- | --- | --- |"] - for f in critical_findings[:10]: - fname = Path(f.get("file", "")).name - lines.append(_md_table_row( - f.get("plugin_id", "?"), - f"`{fname}`", - str(f.get("line", "?")), - f.get("rule", "?"), - f.get("message", ""), - )) - - if warning_findings and not critical_findings: - lines.append(f"\n_{len(warning_findings)} warning(s) found — see artifact for details_") - - return lines, summary.get("critical", 0), True - - -# ───────────────────────────────────────────────────────────────────────────── -# Main -# ───────────────────────────────────────────────────────────────────────────── - -def main() -> int: - parser = argparse.ArgumentParser( - description="Generate consolidated security audit report", - formatter_class=argparse.RawDescriptionHelpFormatter, - ) - parser.add_argument("--artifact-dir", required=True, - help="Directory containing downloaded CI artifacts") - parser.add_argument("--output", "-o", required=True, - help="Output Markdown file path") - parser.add_argument("--verbose", "-v", action="store_true") - args = parser.parse_args() - - artifact_dir = Path(args.artifact_dir) - timestamp = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M UTC") - - bandit_lines, bandit_crit, bandit_ok = _summarize_bandit(artifact_dir) - pip_audit_lines, pip_audit_crit, pip_audit_ok = _summarize_pip_audit(artifact_dir) - gitleaks_lines, gitleaks_crit, gitleaks_ok = _summarize_gitleaks(artifact_dir) - proofs_lines, proofs_crit, proofs_ok = _summarize_security_proofs(artifact_dir) - plugins_lines, plugins_crit, plugins_ok = _summarize_plugin_audit(artifact_dir) - - unavailable_tools = [ - name for name, ok in [ - ("bandit", bandit_ok), ("pip-audit", pip_audit_ok), - ("gitleaks", gitleaks_ok), ("security-proofs", proofs_ok), - ("plugin-audit", plugins_ok), - ] if not ok - ] - - total_critical = bandit_crit + pip_audit_crit + gitleaks_crit + proofs_crit + plugins_crit - if unavailable_tools: - # A missing/malformed artifact means that tool's checks never - # actually ran -- this must not be reported as a clean PASS just - # because the *artifacts that did load* found nothing. - overall = "INCOMPLETE ⚠️" - elif total_critical > 0: - overall = "ACTION REQUIRED 🚨" - else: - overall = "PASSED ✅" - - def section(title: str, lines: list[str]) -> str: - return f"### {title}\n\n" + "\n".join(lines) + "\n" - - incomplete_note = ( - f"\n_⚠️ Incomplete: results unavailable for {', '.join(unavailable_tools)} " - f"— see the corresponding section(s) below for details_\n" - if unavailable_tools else "" - ) - - report = f"""## 🔒 Security Audit — {overall} - -_Generated: {timestamp}_ -{incomplete_note} -| Critical | High/Warn | Overall | -| :---: | :---: | :---: | -| {'🚨 ' + str(total_critical) if total_critical else '✅ 0'} | ⚠️ see below | {overall} | - ---- - -{section('SAST — Bandit', bandit_lines)} -{section('Dependencies — pip-audit', pip_audit_lines)} -{section('Secrets — Gitleaks', gitleaks_lines)} -{section('LEDMatrix Security Proofs', proofs_lines)} -{section('Plugin Security Audit', plugins_lines)} ---- - -_Total critical findings: **{total_critical}**_ -""" - - output_path = Path(args.output) - output_path.write_text(report, encoding="utf-8") - - if args.verbose: - print(f" Report written to: {output_path}") - print(f" Status: {overall}") - print(f" Critical findings: {total_critical}") - print(f" bandit={bandit_crit} pip-audit={pip_audit_crit} " - f"gitleaks={gitleaks_crit} proofs={proofs_crit} plugins={plugins_crit}") - if unavailable_tools: - print(f" Unavailable: {', '.join(unavailable_tools)}") - - if unavailable_tools: - return 1 - - return 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/scripts/install/configure_web_sudo.sh b/scripts/install/configure_web_sudo.sh index feccc3e7..bbcc14ed 100644 --- a/scripts/install/configure_web_sudo.sh +++ b/scripts/install/configure_web_sudo.sh @@ -100,10 +100,15 @@ TEMP_SUDOERS="/tmp/ledmatrix_web_sudoers_$$" echo "$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH restart ledmatrix-web.service" # Optional: journalctl (non-critical — skip if not found) + # + # NOEXEC, matching first_time_install.sh. These rules end in a wildcard and + # journalctl starts a pager, so without it the caller can reach a shell: + # less runs "!command" as the user the pager belongs to, which here is + # root. NOEXEC stops the granted command executing anything of its own. if [ -n "$JOURNALCTL_PATH" ]; then - echo "$WEB_USER ALL=(ALL) NOPASSWD: $JOURNALCTL_PATH -u ledmatrix.service *" - echo "$WEB_USER ALL=(ALL) NOPASSWD: $JOURNALCTL_PATH -u ledmatrix *" - echo "$WEB_USER ALL=(ALL) NOPASSWD: $JOURNALCTL_PATH -t ledmatrix *" + echo "$WEB_USER ALL=(ALL) NOPASSWD:NOEXEC: $JOURNALCTL_PATH -u ledmatrix.service *" + echo "$WEB_USER ALL=(ALL) NOPASSWD:NOEXEC: $JOURNALCTL_PATH -u ledmatrix *" + echo "$WEB_USER ALL=(ALL) NOPASSWD:NOEXEC: $JOURNALCTL_PATH -t ledmatrix *" fi # Required: python3, bash diff --git a/scripts/install/configure_wifi_permissions.sh b/scripts/install/configure_wifi_permissions.sh index cfba4bc4..37e2c7db 100755 --- a/scripts/install/configure_wifi_permissions.sh +++ b/scripts/install/configure_wifi_permissions.sh @@ -25,9 +25,44 @@ if [ "$EUID" -eq 0 ]; then exit 1 fi +# Resolve command paths against a fixed PATH, and check what we resolved. +# +# Every path found here is written into a sudoers file as a NOPASSWD grant, so +# whoever controls the binary at that path controls root. first_time_install.sh +# re-execs itself with `sudo -E`, which preserves the invoking user's +# environment -- PATH included -- so without pinning it, `which nmcli` can +# resolve to anything on that PATH: a writable directory early in it turns a +# compromise of the low-privilege web user into permanent root. +PATH=/usr/sbin:/usr/bin:/sbin:/bin +export PATH + +# A binary named in a sudoers rule must be root-owned and writable by nobody +# else, or the grant hands root to whoever can rewrite it. +require_trusted_binary() { + local label="$1" path="$2" + if [ ! -x "$path" ]; then + echo "✗ $label: $path is not an executable file" + exit 1 + fi + local owner perms + owner=$(stat -c '%u' "$path") || exit 1 + perms=$(stat -c '%a' "$path") || exit 1 + if [ "$owner" != "0" ]; then + echo "✗ $label: $path is not owned by root (uid $owner); refusing to" + echo " grant it NOPASSWD sudo." + exit 1 + fi + # Group- or world-writable means someone other than root can replace it. + case "$perms" in + *[2367]) echo "✗ $label: $path is writable by group or other ($perms);" + echo " refusing to grant it NOPASSWD sudo." + exit 1 ;; + esac +} + # Get the full paths to commands -NMCLI_PATH=$(which nmcli || echo "/usr/bin/nmcli") -SYSTEMCTL_PATH=$(which systemctl) +NMCLI_PATH=$(command -v nmcli || echo "/usr/bin/nmcli") +SYSTEMCTL_PATH=$(command -v systemctl) echo "Command paths:" echo " nmcli: $NMCLI_PATH" @@ -37,6 +72,18 @@ echo " systemctl: $SYSTEMCTL_PATH" echo "" echo "Step 1: Configuring sudo permissions for nmcli..." SUDOERS_FILE="/etc/sudoers.d/ledmatrix_wifi" +SYSCTL_PATH=$(command -v sysctl || echo /usr/sbin/sysctl) +NFT_PATH=$(command -v nft || echo /usr/sbin/nft) +RFKILL_PATH=$(command -v rfkill || echo /usr/sbin/rfkill) +MKDIR_PATH=$(command -v mkdir || echo /usr/bin/mkdir) + +# Checked before any of them reaches the sudoers file. +require_trusted_binary "nmcli" "$NMCLI_PATH" +require_trusted_binary "systemctl" "$SYSTEMCTL_PATH" +require_trusted_binary "sysctl" "$SYSCTL_PATH" +require_trusted_binary "nft" "$NFT_PATH" +require_trusted_binary "rfkill" "$RFKILL_PATH" +require_trusted_binary "mkdir" "$MKDIR_PATH" # Create a temporary sudoers file using mktemp (handles permissions better) TEMP_SUDOERS=$(mktemp) || { @@ -62,6 +109,36 @@ $WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH start dnsmasq $WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH stop dnsmasq $WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH restart dnsmasq $WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH restart NetworkManager +# The captive portal turns IP forwarding on while the access point is up and +# restores the previous value when it comes down (wifi_manager._setup_iptables_ +# redirect / _teardown_iptables_redirect). Without this rule that sudo call +# needs a password, so forwarding stays off and clients associate to the AP but +# cannot route. It goes unnoticed on a stock Raspberry Pi image, where +# /etc/sudoers.d/010_pi-nopasswd grants the default user blanket NOPASSWD and +# masks every gap in this file -- it only bites once that blanket rule is +# removed. +$WEB_USER ALL=(ALL) NOPASSWD: $SYSCTL_PATH -w net.ipv4.ip_forward=0 +$WEB_USER ALL=(ALL) NOPASSWD: $SYSCTL_PATH -w net.ipv4.ip_forward=1 +# The portal's redirect lives in its own nftables table, created when the AP +# comes up and deleted when it goes down, and the radio has to be unblocked +# before the AP can start at all. Same story as the sysctl rules above: called +# with sudo, never granted here, and invisible on a stock Pi image. +$WEB_USER ALL=(ALL) NOPASSWD: $NFT_PATH add table ip ledmatrix +$WEB_USER ALL=(ALL) NOPASSWD: $NFT_PATH delete table ip ledmatrix +$WEB_USER ALL=(ALL) NOPASSWD: $RFKILL_PATH unblock wifi +# NetworkManager's dnsmasq drop-in directory, exact path. +$WEB_USER ALL=(ALL) NOPASSWD: $MKDIR_PATH -p /etc/NetworkManager/dnsmasq-shared.d +# +# iptables is deliberately NOT granted here. Its rules are built from the live +# interface name and port, so a rule covering them needs a trailing wildcard -- +# and `iptables --modprobe=/path/to/anything` runs that path as root, so +# `NOPASSWD: iptables *` is a root shell for the web user by another name. That +# is a worse outcome than the gap it would close, which today is masked anyway +# by the blanket NOPASSWD rule on stock Pi images. +# +# Closing it safely means a wrapper script that builds the rules itself and +# takes only an interface and a port, granted the way safe_plugin_rm.sh already +# is. That belongs in its own change rather than being smuggled into this one. # Allow copying hostapd and dnsmasq config files into place $WEB_USER ALL=(ALL) NOPASSWD: /usr/bin/cp /tmp/hostapd.conf /etc/hostapd/hostapd.conf diff --git a/scripts/install/install_service.sh b/scripts/install/install_service.sh old mode 100644 new mode 100755 diff --git a/scripts/install/install_web_service.sh b/scripts/install/install_web_service.sh old mode 100644 new mode 100755 diff --git a/scripts/install_dependencies_apt.py b/scripts/install_dependencies_apt.py index a03d0243..44e6adc5 100644 --- a/scripts/install_dependencies_apt.py +++ b/scripts/install_dependencies_apt.py @@ -44,17 +44,12 @@ def install_via_apt(package_name: str) -> Tuple[bool, str]: apt_package_map = { 'flask': 'python3-flask', 'PIL': 'python3-pil', - 'freetype': 'python3-freetype', + 'freetype-py': 'python3-freetype', 'psutil': 'python3-psutil', 'werkzeug': 'python3-werkzeug', 'numpy': 'python3-numpy', 'requests': 'python3-requests', - 'python-dateutil': 'python3-dateutil', - 'pytz': 'python3-tz', - 'geopy': 'python3-geopy', - 'unidecode': 'python3-unidecode', - 'websockets': 'python3-websockets', - 'websocket-client': 'python3-websocket-client' + 'pytz': 'python3-tz' } apt_package = apt_package_map.get(package_name, f'python3-{package_name}') @@ -81,8 +76,8 @@ def install_via_pip(package_name: str) -> Tuple[bool, str]: pip RECORD file, so an uninstall attempt fails with "uninstall-no-record-file" and aborts the whole install. With --ignore-installed, pip lays the new version down in /usr/local where it shadows the apt copy instead of removing - it. This matters when a pip dependency (google-api-python-client pulls a - newer requests) needs to upgrade an apt-managed package. + it. This matters when a pip dependency needs to upgrade an apt-managed + package (e.g. a package that pulls a newer requests). Returns (success, output). """ @@ -101,13 +96,35 @@ def install_via_pip(package_name: str) -> Tuple[bool, str]: # Distribution (pip/apt) names whose importable module name differs. IMPORT_NAME_MAP = { - 'python-dateutil': 'dateutil', - 'websocket-client': 'websocket', + 'freetype-py': 'freetype', +} + +# Minimum versions that must be met for an already-installed package to count +# as satisfied. Debian Bookworm's python3-freetype is 2.3.0, below the +# freetype-py>=2.5.1 pin in requirements.txt, so an import-only check would +# wrongly skip the pip upgrade. +MIN_VERSIONS = { + 'freetype-py': (2, 5, 1), } +def _installed_version_tuple(dist_name: str) -> tuple: + """Return the installed distribution version as an int tuple, or () if unknown.""" + try: + from importlib.metadata import version + parts = [] + for part in version(dist_name).split('.'): + digits = ''.join(ch for ch in part if ch.isdigit()) + if not digits: + break + parts.append(int(digits)) + return tuple(parts) + except Exception: + return () + + def check_package_installed(package_name: str) -> bool: - """Check if a package is already installed.""" + """Check if a package is already installed (and meets any minimum version).""" import_name = IMPORT_NAME_MAP.get(package_name, package_name) # Suppress deprecation warnings when checking if packages are installed # (we're just checking, not using them) @@ -115,9 +132,16 @@ def check_package_installed(package_name: str) -> bool: warnings.filterwarnings('ignore', category=DeprecationWarning) try: __import__(import_name) - return True except ImportError: return False + minimum = MIN_VERSIONS.get(package_name) + if minimum: + installed = _installed_version_tuple(package_name) + if not installed or installed < minimum: + print(f"{package_name} is installed but below the required " + f"{'.'.join(map(str, minimum))}; will upgrade via pip") + return False + return True def print_failure_summary(failed_packages: List[str], failure_details: dict) -> None: @@ -147,17 +171,12 @@ def main(): required_packages = [ 'flask', 'PIL', - 'freetype', + 'freetype-py', 'psutil', 'werkzeug', 'numpy', 'requests', - 'python-dateutil', - 'pytz', - 'geopy', - 'unidecode', - 'websockets', - 'websocket-client' + 'pytz' ] failed_packages = [] @@ -168,8 +187,13 @@ def main(): print(f"{package} is already installed") continue - # Try apt first, then pip + # Try apt first, then pip. An apt install only counts if it also + # satisfies any minimum version (Debian's python3-freetype can be + # older than the freetype-py pin), otherwise fall through to pip. ok, apt_output = install_via_apt(package) + if ok and package in MIN_VERSIONS and not check_package_installed(package): + ok = False + apt_output = f"apt version of {package} is below the required minimum" if not ok: ok, pip_output = install_via_pip(package) if not ok: @@ -177,15 +201,12 @@ def main(): failure_details[package] = pip_output or apt_output # Install packages that don't have apt equivalents + # Packages without apt equivalents. Plugin-specific dependencies + # (timezonefinder, google-api stack, icalevents, socketio, ...) are + # no longer installed here — store plugins declare their own + # requirements.txt, which the plugin store installs. special_packages = [ - 'timezonefinder>=6.5.0,<7.0.0', - 'google-auth-oauthlib>=1.2.0,<2.0.0', - 'google-auth-httplib2>=0.2.0,<1.0.0', - 'google-api-python-client>=2.147.0,<3.0.0', 'spotipy', - 'icalevents', - 'python-socketio>=5.11.0,<6.0.0', - 'python-engineio>=4.9.0,<5.0.0' ] for package in special_packages: diff --git a/scripts/remove_plugin_backups.sh b/scripts/remove_plugin_backups.sh deleted file mode 100755 index ab8146b4..00000000 --- a/scripts/remove_plugin_backups.sh +++ /dev/null @@ -1,117 +0,0 @@ -#!/bin/bash - -# Script to safely remove plugin backup directories -# These were created during the plugin-to-submodule conversion - -set -euo pipefail - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" -PLUGINS_DIR="$PROJECT_ROOT/plugins" - -# Colors -GREEN='\033[0;32m' -YELLOW='\033[1;33m' -RED='\033[0;31m' -NC='\033[0m' - -log_info() { - echo -e "${GREEN}[INFO]${NC} $1" -} - -log_warn() { - echo -e "${YELLOW}[WARN]${NC} $1" -} - -log_error() { - echo -e "${RED}[ERROR]${NC} $1" -} - -# Verify submodules are working -verify_submodules() { - log_info "Verifying submodules are working..." - local issues=0 - - for submod in football-scoreboard hockey-scoreboard ledmatrix-flights \ - ledmatrix-leaderboard ledmatrix-stocks ledmatrix-weather \ - mqtt-notifications; do - if [ ! -d "$PLUGINS_DIR/$submod" ]; then - log_error "Submodule directory missing: $submod" - issues=$((issues + 1)) - elif [ ! -f "$PLUGINS_DIR/$submod/.git" ]; then - log_error "Submodule .git file missing: $submod" - issues=$((issues + 1)) - elif [ ! -f "$PLUGINS_DIR/$submod/manifest.json" ]; then - log_warn "Submodule manifest missing: $submod (may be OK)" - fi - done - - if [ $issues -eq 0 ]; then - log_info "All submodules verified ✓" - return 0 - else - log_error "Found $issues issues with submodules" - return 1 - fi -} - -# Remove backup directories -remove_backups() { - log_info "Removing backup directories..." - - local removed=0 - local total_size=0 - - for backup in "$PLUGINS_DIR"/*.backup*; do - if [ -d "$backup" ]; then - local name=$(basename "$backup") - local size=$(du -sb "$backup" 2>/dev/null | awk '{print $1}') - total_size=$((total_size + size)) - - log_info "Removing: $name" - rm -rf "$backup" - removed=$((removed + 1)) - fi - done - - if [ $removed -gt 0 ]; then - log_info "Removed $removed backup directory(ies)" - log_info "Freed approximately $(numfmt --to=iec-i --suffix=B $total_size 2>/dev/null || echo "$total_size bytes")" - else - log_info "No backup directories found" - fi -} - -# Main -main() { - cd "$PROJECT_ROOT" - - echo "=== Plugin Backup Removal Script ===" - echo - - # Verify submodules first - if ! verify_submodules; then - log_error "Submodule verification failed. Not removing backups." - log_warn "Please fix submodule issues before removing backups." - exit 1 - fi - - echo - log_warn "This will permanently delete backup directories:" - ls -1d "$PLUGINS_DIR"/*.backup* 2>/dev/null | sed 's|.*/| - |' || echo " (none found)" - echo - - read -p "Continue? (y/N): " -n 1 -r - echo - if [[ ! $REPLY =~ ^[Yy]$ ]]; then - log_info "Aborted" - exit 0 - fi - - remove_backups - - log_info "Done!" -} - -main "$@" - diff --git a/scripts/utils/clear_cache.py b/scripts/utils/clear_cache.py index 6d4d7b0d..c490435b 100644 --- a/scripts/utils/clear_cache.py +++ b/scripts/utils/clear_cache.py @@ -8,10 +8,10 @@ import os import sys import argparse -# Add the src directory to the path so we can import our modules -sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'src')) +# Add the project root to the path so we can import our modules +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))) -from cache_manager import CacheManager +from src.cache_manager import CacheManager def list_cache_keys(cache_manager): """List all available cache keys.""" diff --git a/src/background_data_service.py b/src/background_data_service.py index 355c93c4..fa20783c 100644 --- a/src/background_data_service.py +++ b/src/background_data_service.py @@ -14,11 +14,12 @@ Key Features: - Memory-efficient data storage """ +import itertools import time import logging import threading import requests -from typing import Dict, Any, Optional, Callable +from typing import Dict, Any, Optional, Callable, List from dataclasses import dataclass, field from enum import Enum import queue @@ -50,14 +51,33 @@ class FetchRequest: max_retries: int = 3 priority: int = 1 # Higher number = higher priority callback: Optional[Callable] = None + # Callbacks from submitters that JOINED this fetch instead of starting a + # duplicate one. The primary `callback` above belongs to whoever created + # the request; these belong to everyone who asked for the same cache_key + # while it was still in flight. + extra_callbacks: List[Callable] = field(default_factory=list) created_at: float = field(default_factory=time.time) status: FetchStatus = FetchStatus.PENDING + # Set once the worker has decided this response will be cached, while it + # still holds the lock. From that point cancelling is refused: the write + # is already authorised, and abandoning it here would put the payload in + # the cache with the callbacks suppressed -- joiners waiting forever for a + # fetch that did, in fact, succeed. + commit_claimed: bool = False result: Optional[Any] = None error: Optional[str] = None @dataclass class FetchResult: - """Result of a background fetch operation.""" + """Result of a background fetch operation. + + ``data`` survives on the stored result only for requests submitted without + a ``callback``, where polling ``get_result()`` is the sole way to collect + it. When a callback was given, the payload has already been delivered and + the service releases it -- see :meth:`BackgroundDataService._release_payload`. + Either way the data remains in the cache under the request's ``cache_key``, + which is where consumers read it from. + """ request_id: str success: bool data: Optional[Any] = None @@ -66,6 +86,10 @@ class FetchResult: fetch_time: float = 0.0 retry_count: int = 0 completed_at: float = field(default_factory=time.time) # Timestamp when request completed + # The request's final status, recorded so a finished request can still be + # reported accurately. Without it a caller can only be told COMPLETED or + # FAILED, which turns "you cancelled this" into "this errored". + final_status: Optional[FetchStatus] = None class BackgroundDataService: """ @@ -90,6 +114,20 @@ class BackgroundDataService: # Thread management self.executor = ThreadPoolExecutor(max_workers=max_workers, thread_name_prefix="BackgroundData") + # cache_key -> request_id for fetches currently in flight. Submitting + # the same key twice used to start two identical fetches: request_id + # carries a millisecond timestamp, so every submit looked new, and + # active_requests is keyed by it rather than by what is being fetched. + # On a real board the season-schedule key is requested by both the + # Recent and the Upcoming manager, which miss the cache in the same + # millisecond and each download and parse the same payload. + self._inflight_by_cache_key: Dict[str, str] = {} + # request_id was sport_year_milliseconds, which is not unique: two + # submits inside the same millisecond produced the SAME id, so one + # silently replaced the other in active_requests and completed_requests. + # Rare before, but dedupe hands this id back to every joiner as their + # handle for get_result(), so it has to be unique. A counter is enough. + self._request_seq = itertools.count() self.active_requests: Dict[str, FetchRequest] = {} self.completed_requests: Dict[str, FetchResult] = {} self.request_queue = queue.PriorityQueue() @@ -177,7 +215,9 @@ class BackgroundDataService: if cache_key is None: cache_key = self.get_sport_cache_key(sport) - request_id = f"{sport}_{year}_{int(time.time() * 1000)}" + with self._lock: + request_id = (f"{sport}_{year}_{int(time.time() * 1000)}" + f"_{next(self._request_seq)}") # Check cache first cached_data = self.cache_manager.get(cache_key) @@ -191,14 +231,19 @@ class BackgroundDataService: cached=True, fetch_time=0.0 ) + # Filed before the callback runs, as it always was: a callback + # that queries get_result()/is_request_complete() for its own + # request must still find it. Releasing afterwards mutates the + # same object the dict holds. self.completed_requests[request_id] = result - + if callback: try: callback(result) except Exception as e: logger.error(f"Error in callback for request {request_id}: {e}") - + self._release_payload(result) + logger.debug(f"Cache hit for {sport} {year} data") return request_id @@ -218,7 +263,29 @@ class BackgroundDataService: ) with self._lock: + existing_id = self._inflight_by_cache_key.get(cache_key) + existing = self.active_requests.get(existing_id) if existing_id else None + if existing_id and existing is None: + # Stranded index entry: the request it names is gone. Drop it and + # fetch normally. Looking the request up rather than trusting the + # id is what stops a stale entry wedging a key forever. + del self._inflight_by_cache_key[cache_key] + if existing is not None: + # Someone is already fetching this key. Ride along rather than + # duplicating the download, the parse and the resident copy. + if callback: + existing.extra_callbacks.append(callback) + self.stats['deduplicated_requests'] = ( + self.stats.get('deduplicated_requests', 0) + 1 + ) + logger.info( + "Joined in-flight fetch %s for %s (cache_key=%s) instead of " + "starting a duplicate", existing_id, sport, cache_key + ) + return existing_id + self.active_requests[request_id] = request + self._inflight_by_cache_key[cache_key] = request_id self.stats['total_requests'] += 1 self.stats['cache_misses'] += 1 @@ -243,7 +310,31 @@ class BackgroundDataService: try: with self._lock: - request.status = FetchStatus.IN_PROGRESS + # A request cancelled while it sat in the executor queue must + # stay cancelled. Overwriting the status here undid the cancel + # outright: the worker went on to download, cache and call back + # for work the caller had already withdrawn. + if request.status == FetchStatus.CANCELLED: + cancelled_before_start = True + else: + cancelled_before_start = False + request.status = FetchStatus.IN_PROGRESS + if cancelled_before_start: + logger.info( + "Request %s was cancelled before its worker started; " + "skipping the fetch entirely", request.id + ) + # Assign before returning: the finally block stores `result` + # in completed_requests, so building a fresh one here would + # file the untouched placeholder instead of this outcome. + result = FetchResult( + request_id=request.id, + success=False, + error="cancelled", + fetch_time=time.time() - start_time, + retry_count=request.retry_count + ) + return result logger.info(f"Starting background fetch for {request.sport} {request.year}") @@ -269,6 +360,37 @@ class BackgroundDataService: # Log data validation logger.debug(f"Validated {len(events)} events for {request.sport} {request.year}") + # A cancelled request must not commit anything. Cancelling + # releases the cache_key, so a replacement fetch for the same key + # may already be in flight or finished -- writing this response to + # the cache now would overwrite fresher data with the response + # nobody wanted. The worker has no way to abort the HTTP call, so + # this is where the work gets discarded. + with self._lock: + cancelled = request.status == FetchStatus.CANCELLED + if not cancelled: + # Claim the commit in the same critical section that read + # the status, so a cancel cannot slip in between the check + # and the cache write below. The write itself stays outside + # the lock: it serialises a multi-megabyte payload to the + # SD card, and holding the service lock across that would + # stall every submit, status query and cancel behind it. + request.commit_claimed = True + if cancelled: + logger.info( + "Discarding response for cancelled request %s; %s may " + "already belong to a replacement fetch", + request.id, request.cache_key + ) + result = FetchResult( + request_id=request.id, + success=False, + error="cancelled", + fetch_time=time.time() - start_time, + retry_count=request.retry_count + ) + return result + # Cache the data self.cache_manager.set(request.cache_key, data) @@ -294,7 +416,12 @@ class BackgroundDataService: logger.error(f"Failed to fetch {request.sport} {request.year} data: {error_msg}") with self._lock: - request.status = FetchStatus.FAILED + # Don't relabel a cancelled request. The callback gate in the + # finally block only suppresses CANCELLED, so promoting it to + # FAILED here delivered an error callback for a fetch nobody + # was waiting on any more. + if request.status != FetchStatus.CANCELLED: + request.status = FetchStatus.FAILED request.error = error_msg result = FetchResult( @@ -308,9 +435,26 @@ class BackgroundDataService: finally: # Store result and clean up with self._lock: + result.final_status = request.status self.completed_requests[request.id] = result if request.id in self.active_requests: del self.active_requests[request.id] + # Stop accepting joiners and take the callback list in the same + # critical section. A submitter that arrives after this point + # finds no in-flight entry and either hits the cache (written + # above, before the result was built) or starts a fresh fetch -- + # what it must never do is join a fetch whose callbacks have + # already run and then never be called. + if self._inflight_by_cache_key.get(request.cache_key) == request.id: + del self._inflight_by_cache_key[request.cache_key] + # A cancelled request delivers nothing: its joiners were told + # about a fetch that has been abandoned, and a replacement will + # call them via its own request. + if request.status == FetchStatus.CANCELLED: + callbacks = [] + else: + callbacks = ([request.callback] if request.callback else []) + callbacks.extend(request.extra_callbacks) # Update statistics if result.success: @@ -327,14 +471,53 @@ class BackgroundDataService: # Periodic cleanup after storing result self._cleanup_completed_requests() - # Call callback if provided - if request.callback: + # Call every callback: the original submitter's and any that joined + # this fetch. One raising must not stop the others being delivered. + for cb in callbacks: try: - request.callback(result) + cb(result) except Exception as e: logger.error(f"Error in callback for request {request.id}: {e}") - + + # Released AFTER the loop, not inside it. Every callback here holds + # the same FetchResult, so releasing per-delivery handed the first + # one the data and every joiner `result.data is None` -- which is + # not a quiet degradation: they read `result.data.get('events')` and + # raise AttributeError, which this very loop catches and logs, so + # the symptom was one ERROR line and a manager that silently never + # got its schedule. Deduplication is the normal case, not a corner: + # a sport's recent, upcoming and live managers all ride one season + # fetch. + # + # Guarded on `callbacks`, because a request submitted without one + # has no other way to collect its payload than polling get_result(). + # The old per-delivery release got that right by accident: an empty + # list never entered the loop body. + if callbacks: + self._release_payload(result) + request.result = None + return result + + @staticmethod + def _release_payload(result: FetchResult) -> None: + """Drop a delivered payload, keeping the result's status and timings. + + Only called once EVERY callback has been handed the data -- callers + that joined an in-flight fetch share this object, so releasing between + deliveries strips the payload out from under the ones still queued. + Consumers read fetched data back from the cache under ``cache_key``; + the copy carried + here was pinning a parsed season schedule -- 946 games for NCAA + football, roughly a tenth of total RAM on a 1GB Pi -- in memory until + the hourly sweep. + + The cache-hit path matters most: it runs once per update interval per + sport, mints a fresh request_id each time, and a memory-tier miss + re-parses the payload from disk. Those were genuinely separate copies + accumulating toward the 500-entry cap, not shared references. + """ + result.data = None def _make_request_with_retry(self, request: FetchRequest) -> requests.Response: """ @@ -422,6 +605,8 @@ class BackgroundDataService: return self.active_requests[request_id].status elif request_id in self.completed_requests: result = self.completed_requests[request_id] + if result.final_status is not None: + return result.final_status return FetchStatus.COMPLETED if result.success else FetchStatus.FAILED return None @@ -438,8 +623,22 @@ class BackgroundDataService: with self._lock: if request_id in self.active_requests: request = self.active_requests[request_id] + if request.commit_claimed: + # Too late: the worker holds an authorised commit. Report + # the failure rather than half-cancelling a request whose + # data is about to land in the cache. + logger.debug( + "Not cancelling %s: its response is already being " + "committed", request_id + ) + return False request.status = FetchStatus.CANCELLED del self.active_requests[request_id] + # Cancelling is the other way a request leaves active_requests, + # so the in-flight index has to be released here too or the key + # stays pointed at a request that no longer exists. + if self._inflight_by_cache_key.get(request.cache_key) == request_id: + del self._inflight_by_cache_key[request.cache_key] logger.info(f"Cancelled request {request_id}") return True return False diff --git a/src/backup_manager.py b/src/backup_manager.py index c54ce81e..e8d03b18 100644 --- a/src/backup_manager.py +++ b/src/backup_manager.py @@ -16,6 +16,7 @@ import json import logging import os import shutil +import stat import socket import tempfile import zipfile @@ -82,6 +83,10 @@ BUNDLED_FONTS: frozenset[str] = frozenset({ _CONFIG_REL = Path("config/config.json") _SECRETS_REL = Path("config/config_secrets.json") _WIFI_REL = Path("config/wifi_config.json") +# Sits in config/ next to the three above and is pure user state — a +# YouTube Music session that has to be re-authenticated by hand if lost. +# It was omitted from backups, so a restore silently signed the user out. +_YTM_REL = Path("config/ytm_auth.json") _FONTS_REL = Path("assets/fonts") _PLUGIN_UPLOADS_REL = Path("assets/plugins") _STATE_REL = Path("data/plugin_state.json") @@ -303,6 +308,9 @@ def create_backup( if (project_root / _WIFI_REL).exists(): zf.write(project_root / _WIFI_REL, _WIFI_REL.as_posix()) contents.append("wifi") + if (project_root / _YTM_REL).exists(): + zf.write(project_root / _YTM_REL, _YTM_REL.as_posix()) + contents.append("ytm_auth") # User-uploaded fonts. user_fonts = iter_user_fonts(project_root) @@ -348,6 +356,7 @@ def preview_backup_contents(project_root: Path) -> Dict[str, Any]: "has_config": (project_root / _CONFIG_REL).exists(), "has_secrets": (project_root / _SECRETS_REL).exists(), "has_wifi": (project_root / _WIFI_REL).exists(), + "has_ytm_auth": (project_root / _YTM_REL).exists(), "user_fonts": [p.name for p in iter_user_fonts(project_root)], "plugin_uploads": len(iter_plugin_uploads(project_root)), "plugins": list_installed_plugins(project_root), @@ -429,6 +438,8 @@ def validate_backup(zip_path: Path) -> Tuple[bool, str, Dict[str, Any]]: detected.append("secrets") if _WIFI_REL.as_posix() in names: detected.append("wifi") + if _YTM_REL.as_posix() in names: + detected.append("ytm_auth") if any(n.startswith(_FONTS_REL.as_posix() + "/") for n in names): detected.append("fonts") if any( @@ -481,8 +492,61 @@ def _extract_zip_safe(zip_path: Path, dest_dir: Path) -> None: def _copy_file(src: Path, dst: Path) -> None: + """Replace ``dst`` with ``src``, atomically, without needing to own ``dst``. + + ``shutil.copy2`` opens the destination for writing, so it needs write + permission on the *existing file*. Several config files are installed + root-owned and group-readable while the web interface — which is what runs + a restore — deliberately runs as a non-root user. Restoring those failed + with EACCES even though the account could create files in the same + directory perfectly well. + + Writing a temporary file alongside and renaming over the target needs only + directory permission, which the web user has. It is also atomic: a crash + mid-restore can no longer leave a half-written config behind. + + The destination's existing mode is preserved when there is one, so + restoring secrets does not silently widen them to the umask default. + """ dst.parent.mkdir(parents=True, exist_ok=True) - shutil.copy2(src, dst) + + existing_mode: Optional[int] = None + existing_owner: Optional[Tuple[int, int]] = None + if dst.exists(): + try: + info = dst.stat() + existing_mode = stat.S_IMODE(info.st_mode) + existing_owner = (info.st_uid, info.st_gid) + except OSError: + existing_mode = None + existing_owner = None + + fd, tmp_name = tempfile.mkstemp(dir=str(dst.parent), prefix=f".{dst.name}.", suffix=".tmp") + os.close(fd) + tmp_path = Path(tmp_name) + try: + shutil.copyfile(src, tmp_path) + if existing_mode is not None: + os.chmod(tmp_path, existing_mode) + else: + shutil.copymode(src, tmp_path) + if existing_owner is not None: + # Replacing a file creates a new inode owned by whoever is running, + # which would silently move a root-owned config to the web user. + # Carry the previous owner across when the OS permits it — only + # root can hand a file to another user, so this is best-effort and + # a plain restore as the web user simply keeps its own ownership. + try: + os.chown(tmp_path, existing_owner[0], existing_owner[1]) + except (OSError, PermissionError): + pass + os.replace(tmp_path, dst) + except BaseException: + try: + tmp_path.unlink() + except OSError: + pass + raise def restore_backup( @@ -513,7 +577,8 @@ def restore_backup( try: _extract_zip_safe(Path(zip_path), tmp_dir) except (ValueError, zipfile.BadZipFile, OSError) as e: - result.errors.append(f"Failed to extract backup: {e}") + logger.error("[Backup] Failed to extract backup: %s", e, exc_info=True) + result.errors.append("Failed to extract backup") return result # Main config. @@ -522,7 +587,8 @@ def restore_backup( _copy_file(tmp_dir / _CONFIG_REL, project_root / _CONFIG_REL) result.restored.append("config") except OSError as e: - result.errors.append(f"Failed to restore config.json: {e}") + logger.error("[Backup] Failed to restore config.json: %s", e, exc_info=True) + result.errors.append("Failed to restore config.json") elif (tmp_dir / _CONFIG_REL).exists(): result.skipped.append("config") @@ -532,7 +598,10 @@ def restore_backup( _copy_file(tmp_dir / _SECRETS_REL, project_root / _SECRETS_REL) result.restored.append("secrets") except OSError as e: - result.errors.append(f"Failed to restore config_secrets.json: {e}") + logger.error( + "[Backup] Failed to restore config_secrets.json: %s", e, exc_info=True + ) + result.errors.append("Failed to restore config_secrets.json") elif (tmp_dir / _SECRETS_REL).exists(): result.skipped.append("secrets") @@ -542,10 +611,26 @@ def restore_backup( _copy_file(tmp_dir / _WIFI_REL, project_root / _WIFI_REL) result.restored.append("wifi") except OSError as e: - result.errors.append(f"Failed to restore wifi_config.json: {e}") + logger.error( + "[Backup] Failed to restore wifi_config.json: %s", e, exc_info=True + ) + result.errors.append("Failed to restore wifi_config.json") elif (tmp_dir / _WIFI_REL).exists(): result.skipped.append("wifi") + # YouTube Music session. Follows restore_wifi rather than getting its + # own flag: it is device-local auth in the same sense, and a separate + # toggle for one file would be noise in the restore dialog. + if options.restore_wifi and (tmp_dir / _YTM_REL).exists(): + try: + _copy_file(tmp_dir / _YTM_REL, project_root / _YTM_REL) + result.restored.append("ytm_auth") + except OSError as e: + logger.error("[Backup] Failed to restore ytm_auth.json: %s", e, exc_info=True) + result.errors.append("Failed to restore ytm_auth.json") + elif (tmp_dir / _YTM_REL).exists(): + result.skipped.append("ytm_auth") + # User fonts — skip anything that collides with a bundled font. tmp_fonts = tmp_dir / _FONTS_REL if options.restore_fonts and tmp_fonts.exists(): @@ -560,7 +645,10 @@ def restore_backup( _copy_file(font, project_root / _FONTS_REL / font.name) restored_count += 1 except OSError as e: - result.errors.append(f"Failed to restore font {font.name}: {e}") + logger.error( + "[Backup] Failed to restore font %s: %s", font.name, e, exc_info=True + ) + result.errors.append(f"Failed to restore font {font.name}") if restored_count: result.restored.append(f"fonts ({restored_count})") elif tmp_fonts.exists(): @@ -581,7 +669,8 @@ def restore_backup( _copy_file(src, project_root / rel) count += 1 except OSError as e: - result.errors.append(f"Failed to restore {rel}: {e}") + logger.error("[Backup] Failed to restore %s: %s", rel, e, exc_info=True) + result.errors.append(f"Failed to restore {rel}") if count: result.restored.append(f"plugin_uploads ({count})") elif tmp_uploads.exists(): @@ -599,7 +688,8 @@ def restore_backup( if isinstance(p, dict) and p.get("plugin_id") ] except (OSError, json.JSONDecodeError) as e: - result.errors.append(f"Could not read plugins.json: {e}") + logger.error("[Backup] Could not read plugins.json: %s", e, exc_info=True) + result.errors.append("Could not read plugins.json") result.success = not result.errors return result diff --git a/src/base_classes/data_sources.py b/src/base_classes/data_sources.py index 777320ea..065ea577 100644 --- a/src/base_classes/data_sources.py +++ b/src/base_classes/data_sources.py @@ -44,9 +44,16 @@ class DataSource(ABC): """Fetch standings for a sport/league.""" def get_headers(self) -> Dict[str, str]: - """Get headers for API requests.""" + """Get headers for API requests. + + The agent carries the project URL deliberately. Around 2026-08-04 ESPN + began returning 403 for bare custom tokens like 'LEDMatrix/1.0' — and + for browser-style strings — while accepting an agent that identifies + the client and links to it. An Accept header alone does not rescue the + bare form when the request goes out through requests. + """ return { - 'User-Agent': 'LEDMatrix/1.0', + 'User-Agent': 'LEDMatrix/1.0 (+https://github.com/ChuckBuilds/LEDMatrix)', 'Accept': 'application/json' } diff --git a/src/base_classes/sports/core.py b/src/base_classes/sports/core.py index e43c6e45..3291a1bf 100644 --- a/src/base_classes/sports/core.py +++ b/src/base_classes/sports/core.py @@ -383,10 +383,15 @@ class SportsCore(ABC): ctx = skin_runtime.build_context(self, game, size=size) card = skin.render_vegas_card(ctx, dict(game)) if card is not None: + # A successful render clears accumulated strikes, mirroring + # _render_game — transient failures must not add up across + # the session and disable a working skin. + self._skin_failures = 0 return card ctx = skin_runtime.build_context(self, game, size=size) render = getattr(skin, f"render_{self.SKIN_MODE}") if render(ctx, dict(game)): + self._skin_failures = 0 return ctx.canvas except Exception: # Card failures count toward the same 3-strike session disable diff --git a/src/base_classes/sports/modes.py b/src/base_classes/sports/modes.py index 06e73467..8f6e3cce 100644 --- a/src/base_classes/sports/modes.py +++ b/src/base_classes/sports/modes.py @@ -145,8 +145,7 @@ class SportsUpcoming(SportsCore): if (game['home_abbr'] in self.favorite_teams or game['away_abbr'] in self.favorite_teams): favorite_games_found += 1 - if self.show_odds: - self._fetch_odds(game) + # Odds are NOT fetched here -- see after selection below. # Enhanced logging for debugging self.logger.info(f"Found {all_upcoming_games} total upcoming games in data") @@ -190,6 +189,20 @@ class SportsUpcoming(SportsCore): # Limit to the specified number of upcoming games team_games = team_games[:self.upcoming_games_to_show] + # Odds are fetched here, for the games that survived selection, + # rather than in the loop that collects them. That loop walks every + # upcoming game in the schedule window, and for a college league + # the window is enormous -- a live rig logged 946 upcoming games in + # one cycle and displayed 1 of them. The comment up there claimed + # odds were fetched "only for games that will be displayed", but + # the only narrowing it applied was show_favorite_teams_only, which + # is not the default; in the usual case nothing narrowed it at all + # and every game cost a separate ESPN request on a Pi that is also + # driving the panel. + if self.show_odds: + for game in team_games: + self._fetch_odds(game) + # Log changes or periodically should_log = ( current_time - self.last_log_time >= self.log_interval or diff --git a/src/base_odds_manager.py b/src/base_odds_manager.py index 3520ce67..4b455069 100644 --- a/src/base_odds_manager.py +++ b/src/base_odds_manager.py @@ -12,6 +12,8 @@ Follows LEDMatrix configuration management patterns: """ import logging +import time + import requests import json from typing import Dict, Any, Optional, List @@ -42,10 +44,35 @@ class BaseOddsManager: self.config_manager = config_manager self.logger = logging.getLogger(__name__) self.base_url = "https://sports.core.api.espn.com/v2/sports" + + # This path used a bare requests.get, so it identified itself as + # python-requests/x.y -- the one thing ESPN is known to reject. Around + # 2026-08-04 it began 403ing browser strings and bare custom tokens + # alike; what it accepts is a token with a URL that says who is + # calling. Every other ESPN caller in the tree already sends this + # (src/common/api_helper.py, src/base_classes/data_sources.py); the + # odds path was simply missed, and it is the one whose failures cost + # the caller its whole update budget. + # + # Deliberately no retry adapter, unlike api_helper: retries multiply + # request_timeout, which is set to 5s precisely to stay inside that + # budget. One try, then the cooldown below. + self.session = requests.Session() + self.session.headers.update({ + 'User-Agent': 'LEDMatrix/1.0 (+https://github.com/ChuckBuilds/LEDMatrix)', + 'Accept': 'application/json', + }) # Configuration with defaults self.update_interval = 3600 # 1 hour default - self.request_timeout = 30 # 30 seconds default + # Well under the plugin executor's 30s operation budget. At 30s a + # single stalled ESPN request consumed the entire budget and the whole + # update() was killed -- and odds are fetched per live game, inside the + # live update loop, with show_odds defaulting on. Losing one game's + # odds beats losing the update that carries every game's score. + self.request_timeout = 5 + # Set when a request fails; until then, skip the network entirely. + self._skip_network_until = 0.0 self.cache_ttl = 1800 # 30 minutes default # Load configuration if available @@ -73,6 +100,14 @@ class BaseOddsManager: except Exception as e: self.logger.warning(f"Failed to load BaseOddsManager configuration: {e}") + # After a network failure, stop trying for this long and serve cache only. + # A short per-request timeout bounds one stall, but a full Sunday slate is + # ~16 games fetched in a loop, so 16 consecutive timeouts still blow the + # budget. When ESPN is unreachable it is unreachable for all of them, so + # the first failure is enough to know: skip the rest of this pass and try + # again shortly. + _FAILURE_COOLDOWN = 60.0 + def get_odds(self, sport: str | None, league: str | None, event_id: str, update_interval_seconds: int = None) -> Optional[Dict[str, Any]]: """ @@ -101,8 +136,18 @@ class BaseOddsManager: self.logger.info(f"Using cached odds from ESPN for {cache_key}") return cached_data + if time.monotonic() < self._skip_network_until: + # A recent request failed, so ESPN is very likely still unreachable. + # Returning now keeps the caller's update inside its time budget + # instead of paying the timeout again for every remaining game. + self.logger.debug( + "Skipping odds fetch for %s: a recent request failed, holding off " + "for another %.0fs", cache_key, + self._skip_network_until - time.monotonic()) + return None + self.logger.info(f"Cache miss - fetching fresh odds from ESPN for {cache_key}") - + try: # Map league names to ESPN API format league_mapping = { @@ -117,10 +162,12 @@ class BaseOddsManager: url = f"{self.base_url}/{sport}/leagues/{espn_league}/events/{event_id}/competitions/{event_id}/odds" self.logger.info(f"Requesting odds from URL: {url}") - response = requests.get(url, timeout=self.request_timeout) + response = self.session.get(url, timeout=self.request_timeout) response.raise_for_status() raw_data = response.json() + self._skip_network_until = 0.0 # reachable again + self.logger.debug(f"Received raw odds data from ESPN: {json.dumps(raw_data, indent=2)}") odds_data = self._extract_espn_data(raw_data) @@ -140,7 +187,11 @@ class BaseOddsManager: return odds_data except requests.exceptions.RequestException as e: - self.logger.error(f"Error fetching odds from ESPN API for {cache_key}: {e}") + self._skip_network_until = time.monotonic() + self._FAILURE_COOLDOWN + self.logger.error( + "Error fetching odds from ESPN API for %s: %s. Holding off on odds " + "for %.0fs so a slate of games does not pay this timeout each.", + cache_key, e, self._FAILURE_COOLDOWN) except json.JSONDecodeError: self.logger.error(f"Error decoding JSON response from ESPN API for {cache_key}.") @@ -163,19 +214,25 @@ class BaseOddsManager: item = data["items"][0] self.logger.debug(f"First item keys: {list(item.keys())}") - # The ESPN API returns odds data directly in the item, not in a providers array - # Extract the odds data directly from the item + # The ESPN API returns odds data directly in the item, not in a + # providers array. ESPN sends explicit JSON nulls for absent + # sides ("homeTeamOdds": null), so every level uses `or {}` — + # .get's default only applies when the key is missing entirely. + home = item.get("homeTeamOdds") or {} + away = item.get("awayTeamOdds") or {} extracted_data = { "details": item.get("details"), "over_under": item.get("overUnder"), "spread": item.get("spread"), "home_team_odds": { - "money_line": item.get("homeTeamOdds", {}).get("moneyLine"), - "spread_odds": item.get("homeTeamOdds", {}).get("current", {}).get("pointSpread", {}).get("value") + "money_line": home.get("moneyLine"), + "spread_odds": ((home.get("current") or {}) + .get("pointSpread") or {}).get("value") }, "away_team_odds": { - "money_line": item.get("awayTeamOdds", {}).get("moneyLine"), - "spread_odds": item.get("awayTeamOdds", {}).get("current", {}).get("pointSpread", {}).get("value") + "money_line": away.get("moneyLine"), + "spread_odds": ((away.get("current") or {}) + .get("pointSpread") or {}).get("value") } } self.logger.debug(f"Returning extracted odds data: {json.dumps(extracted_data, indent=2)}") @@ -260,9 +317,13 @@ class BaseOddsManager: Returns: Formatted odds summary string """ - if not self.is_odds_available(odds_data): + # Gate only on truly-empty / negative-cached data. is_odds_available + # deliberately ignores money lines (its callers decide whether to + # RENDER an odds widget), but a summary of money-line-only odds is + # still meaningful — the parts loop below handles them. + if not odds_data or odds_data.get('no_odds'): return "No odds available" - + parts = [] # Add spread information diff --git a/src/cache/disk_cache.py b/src/cache/disk_cache.py index 64b72f07..03cfb2f1 100644 --- a/src/cache/disk_cache.py +++ b/src/cache/disk_cache.py @@ -14,6 +14,13 @@ import zlib from typing import Dict, Any, Optional, Protocol from datetime import datetime +# How old an abandoned write's temp file must be before the sweep removes it. +# A real write holds its temp file for milliseconds, so an hour is far beyond +# any in-flight write while still clearing the same day's debris. Deliberately +# not tied to the retention policies: those describe how long data stays +# useful, and a half-written file was never useful. +_ORPHAN_TEMP_MAX_AGE_SECONDS = 3600 + class CacheStrategyProtocol(Protocol): @@ -112,6 +119,22 @@ class DiskCache: record_ts = None now = time.time() + + # An explicit per-entry ttl wins over the caller's max_age. The + # caller that wrote the record knows what its data is; max_age is + # inferred from substrings in the key ("live", "odds", "stock") and + # is only a fallback for records that never said. Until now the ttl + # was stored and ignored, so `set(key, data, ttl=...)` did nothing + # at all -- 48 plugin call sites and 4 in the core were writing a + # number no read path consulted. + effective_max_age = max_age + if isinstance(record, dict): + stored_ttl = record.get('ttl') + if isinstance(stored_ttl, (int, float)) and not isinstance(stored_ttl, bool) \ + and stored_ttl >= 0: + effective_max_age = stored_ttl + max_age = effective_max_age + # max_age=None means "never expires" (mirrors MemoryCache and the # cache_manager docstring). Guard it explicitly — otherwise the # comparison below raises TypeError and the record is treated as a @@ -331,6 +354,23 @@ class DiskCache: """Get the cache directory path.""" return self.cache_dir + @staticmethod + def _is_orphaned_temp(filename: str) -> bool: + """Whether a name is one of set()'s temp files rather than real data. + + Matches only what this class creates: mkstemp with a prefix of + ".." , so ".weather.json.a1b2c3d4". The shape is + checked rather than just the leading dot, because this predicate + deletes things -- a stray dotfile someone left in the cache directory + is not ours to remove, and a completed ".json" never is either. + """ + if not filename.startswith('.') or filename.endswith('.json'): + return False + head, sep, suffix = filename.rpartition('.json.') + # head is the key (non-empty after the leading dot), suffix is + # mkstemp's random component. + return bool(sep) and len(head) > 1 and bool(suffix) + def cleanup_expired_files(self, cache_strategy: CacheStrategyProtocol, retention_policies: Dict[str, int]) -> Dict[str, Any]: """ Clean up expired cache files based on retention policies. @@ -365,11 +405,50 @@ class DiskCache: try: with self._lock: # Get snapshot of files while holding lock briefly - filenames = [f for f in os.listdir(self.cache_dir) if f.endswith('.json')] + entries = os.listdir(self.cache_dir) except OSError as list_error: self.logger.error("Error listing cache directory %s: %s", self.cache_dir, list_error, exc_info=True) stats['errors'] += 1 return stats + + filenames = [f for f in entries if f.endswith('.json')] + + # Sweep temp files abandoned by a write that never finished. set() + # removes its own in a finally, so these are the ones where the + # process died between mkstemp and os.replace -- a SIGKILL, a lost + # restart race, a power cut. Nothing ever collected them: they are + # named "..json.", and the scan above only matches + # names ending in .json, so they accumulated indefinitely. Measured + # on a live rig: 76 files, 1,050 MB, 81% of the whole cache + # directory, the oldest six months old. + stats['orphan_temp_files_deleted'] = 0 + for filename in (f for f in entries if self._is_orphaned_temp(f)): + # Counted as scanned like any other candidate, so files_deleted + # can never exceed files_scanned and the summary line reads + # honestly ("77/8864", not "77/0"). + stats['files_scanned'] += 1 + path = os.path.join(self.cache_dir, filename) + try: + # An in-flight write lives for milliseconds, so anything + # this old is certainly abandoned rather than in progress. + if (current_time - os.path.getmtime(path)) <= _ORPHAN_TEMP_MAX_AGE_SECONDS: + continue + with self._lock: + size = os.path.getsize(path) + os.remove(path) + stats['files_deleted'] += 1 + stats['orphan_temp_files_deleted'] += 1 + stats['space_freed_bytes'] += size + except FileNotFoundError: + continue # another sweep got there first + except OSError as e: + stats['errors'] += 1 + self.logger.warning("Error deleting orphaned temp file %s: %s", filename, e) + + if stats['orphan_temp_files_deleted']: + self.logger.info( + "Removed %d abandoned cache temp file(s)", + stats['orphan_temp_files_deleted']) # Process files outside the lock to avoid blocking get/set operations for filename in filenames: diff --git a/src/cache/memory_cache.py b/src/cache/memory_cache.py index 33c9963b..55ca010d 100644 --- a/src/cache/memory_cache.py +++ b/src/cache/memory_cache.py @@ -4,11 +4,58 @@ Memory Cache Handles in-memory caching with TTL support, size limits, and automatic cleanup. """ +import os import time import threading import logging from typing import Dict, Any, Optional +# Historical fixed ceiling, kept as the fallback when RAM cannot be read. +DEFAULT_MAX_SIZE = 1000 + + +def _total_memory_mb() -> Optional[float]: + """Physical RAM in MB, or None where /proc/meminfo is unavailable.""" + try: + with open('/proc/meminfo', 'r', encoding='utf-8') as fh: + for line in fh: + if line.startswith('MemTotal:'): + return int(line.split()[1]) / 1024 + except (OSError, ValueError, IndexError): + return None + return None + + +def default_max_size() -> int: + """Entry ceiling scaled to this machine's RAM. + + One fixed ceiling cannot serve both a 512 MB Pi Zero 2 W and an 8 GB Pi 5. + Entries here are parsed API payloads that routinely run tens of kilobytes + each, so a thousand of them is a comfortable cache on a large board and a + substantial fraction of total RAM on a small one — where the process + competing for that RAM is also driving the panel. Set + LEDMATRIX_CACHE_MAX_ENTRIES to override. + """ + override = os.environ.get('LEDMATRIX_CACHE_MAX_ENTRIES') + if override: + try: + value = int(override) + if value > 0: + return value + except ValueError: + pass + + total_mb = _total_memory_mb() + if total_mb is None: + return DEFAULT_MAX_SIZE + if total_mb < 1536: # 512 MB and 1 GB boards + return 150 + if total_mb < 3072: # 2 GB + return 400 + if total_mb < 6144: # 4 GB + return 800 + return 1500 # 8 GB and up + class MemoryCache: """Manages in-memory cache with TTL and size limits.""" @@ -57,6 +104,16 @@ class MemoryCache: if timestamp is None: return None + # An explicit per-entry ttl wins over the caller's max_age, matching + # DiskCache. max_age is inferred from substrings in the key and is + # only a fallback for records that did not say what they wanted. + record = self._cache[key] + if isinstance(record, dict): + stored_ttl = record.get('ttl') + if isinstance(stored_ttl, (int, float)) and not isinstance(stored_ttl, bool) \ + and stored_ttl >= 0: + max_age = stored_ttl + # Check expiration if max_age is not None and (now - timestamp) > max_age: # Expired - remove it @@ -77,6 +134,32 @@ class MemoryCache: with self._lock: self._cache[key] = value self._timestamps[key] = time.time() + # Enforce the ceiling here rather than leaving it to the periodic + # cleanup, which only runs every cleanup_interval seconds (300 by + # default). A burst of inserts between two sweeps could otherwise + # take the cache far past _max_size, which is the memory growth this + # limit exists to prevent -- and on a 1GB board that is the + # difference between a bounded cache and an unreachable Pi. + self._evict_over_limit_locked() + + def _evict_over_limit_locked(self) -> int: + """Drop oldest entries until the cache is within _max_size. + + Caller must hold self._lock. Returns the number of entries removed. + """ + excess = len(self._cache) - self._max_size + if excess <= 0: + return 0 + oldest = sorted( + self._timestamps.items(), + key=lambda item: float(item[1]) if isinstance(item[1], (int, float)) else 0.0 + ) + removed = 0 + for key, _ in oldest[:excess]: + self._cache.pop(key, None) + self._timestamps.pop(key, None) + removed += 1 + return removed def clear(self, key: Optional[str] = None) -> None: """ @@ -133,22 +216,8 @@ class MemoryCache: self._timestamps.pop(key, None) removed_count += 1 - # Enforce size limit by removing oldest entries if cache is too large - if len(self._cache) > self._max_size: - # Sort by timestamp (oldest first) - sorted_entries = sorted( - self._timestamps.items(), - key=lambda x: float(x[1]) if isinstance(x[1], (int, float)) else 0 - ) - - # Remove oldest entries until we're under the limit - excess_count = len(self._cache) - self._max_size - for i in range(excess_count): - if i < len(sorted_entries): - key = sorted_entries[i][0] - self._cache.pop(key, None) - self._timestamps.pop(key, None) - removed_count += 1 + # Same ceiling enforcement set() uses, so the two cannot drift. + removed_count += self._evict_over_limit_locked() self._last_cleanup = current_time diff --git a/src/cache_manager.py b/src/cache_manager.py index 5065dcdb..67624de1 100644 --- a/src/cache_manager.py +++ b/src/cache_manager.py @@ -33,24 +33,34 @@ import logging import threading import tempfile from src.exceptions import CacheError -from src.cache.memory_cache import MemoryCache +from src.cache.memory_cache import MemoryCache, default_max_size from src.cache.disk_cache import DiskCache from src.cache.cache_strategy import CacheStrategy from src.cache.cache_metrics import CacheMetrics from src.logging_config import get_logger -class DateTimeEncoder(json.JSONEncoder): - """JSON encoder that serialises ``datetime`` objects as ISO-8601 strings.""" - - def default(self, obj): - """Return ISO-8601 string for datetime; delegate all other types to the base encoder.""" - if isinstance(obj, datetime): - return obj.isoformat() - return super().default(obj) +# Canonical implementation lives in src.cache.disk_cache; re-exported here +# because this module's docstring documents it and external code may import +# it from either path. +from src.cache.disk_cache import DateTimeEncoder # noqa: F401 - deliberate re-export class CacheManager: """Manages caching of API responses to reduce API calls.""" - + + # Which cache directories already have a cleanup thread in this process. + # + # The sweep is directory-scoped work -- it lists a directory and deletes + # from it -- so one per directory is the right number no matter how many + # managers exist. Nothing enforced that before: every instance started its + # own, and because the loop closes over `self`, a discarded manager could + # never be collected and its thread woke to re-scan the same directory + # every 24 hours for the life of the process. Startup validation runs + # twice and built a throwaway manager each time, so a display process + # carried three threads for one cache. + _cleanup_owners: Dict[str, 'CacheManager'] = {} + _cleanup_owners_lock = threading.Lock() + + def __init__(self) -> None: # Initialize logger first self.logger: logging.Logger = get_logger(__name__) @@ -74,7 +84,9 @@ class CacheManager: self.logger.warning("ConfigManager not available, using default cache intervals") # Initialize cache components using composition - self._memory_cache_component = MemoryCache(max_size=1000, cleanup_interval=300.0) + self._memory_cache_component = MemoryCache( + max_size=default_max_size(), cleanup_interval=300.0 + ) self._disk_cache_component = DiskCache(cache_dir=self.cache_dir, logger=self.logger) self._strategy_component = CacheStrategy(config_manager=self.config_manager, logger=self.logger) self._metrics_component = CacheMetrics(logger=self.logger) @@ -598,8 +610,10 @@ class CacheManager: Args: key: Cache key data: Data to cache - ttl: Optional time-to-live in seconds (stored for compatibility but - expiration is still controlled via max_age when reading) + ttl: Time-to-live in seconds for this entry. Takes precedence over + the max_age a reader would otherwise apply, which is inferred + from the key and is only a fallback for entries that did not + say. Omit it to keep that inferred behaviour. """ cache_data = { 'data': data, @@ -720,11 +734,29 @@ class CacheManager: } def start_cleanup_thread(self) -> None: - """Start background thread for periodic disk cache cleanup.""" + """Start background thread for periodic disk cache cleanup. + + At most one thread per cache directory per process: the sweep is + directory-scoped, so a second one only duplicates the scan. + """ if self._cleanup_thread and self._cleanup_thread.is_alive(): self.logger.debug("Cleanup thread already running") return - + + with CacheManager._cleanup_owners_lock: + owner = CacheManager._cleanup_owners.get(self.cache_dir) + if owner is not None and owner is not self: + thread = owner._cleanup_thread + if thread is not None and thread.is_alive(): + self.logger.debug( + "Cleanup thread for %s already owned by another cache " + "manager in this process; not starting a second", + self.cache_dir) + return + # The owner's thread died or was stopped -- take over. + CacheManager._cleanup_owners[self.cache_dir] = self + + def cleanup_loop(): """Background loop that runs cleanup periodically.""" self.logger.info("Disk cache cleanup thread started (interval: %d hours)", @@ -772,10 +804,17 @@ class CacheManager: Signals the thread to stop and waits for it to finish (with timeout). This allows for clean shutdown during testing or application termination. """ + # Release ownership first and unconditionally, so a manager that never + # started a thread (or whose thread already exited) cannot keep the + # directory claimed and block a live manager from sweeping it. + with CacheManager._cleanup_owners_lock: + if CacheManager._cleanup_owners.get(self.cache_dir) is self: + del CacheManager._cleanup_owners[self.cache_dir] + if not self._cleanup_thread or not self._cleanup_thread.is_alive(): self.logger.debug("Cleanup thread not running") return - + self.logger.info("Stopping disk cache cleanup thread...") self._cleanup_stop_event.set() # Signal thread to stop diff --git a/src/common/README.md b/src/common/README.md index cccaa40b..4246ccff 100644 --- a/src/common/README.md +++ b/src/common/README.md @@ -99,11 +99,6 @@ Helpers for ensuring directory permissions and ownership are correct when running as a service (used by `CacheManager` to set up its persistent cache directory). -## CLI Helpers (`cli.py`) - -Shared CLI argument parsing helpers used by `scripts/dev/*` and other -command-line entry points. - ## Best Practices 1. **Use centralized logging**: Import from `src.logging_config` instead of creating loggers directly diff --git a/src/common/api_helper.py b/src/common/api_helper.py index 9d9b076f..d6974b22 100644 --- a/src/common/api_helper.py +++ b/src/common/api_helper.py @@ -56,7 +56,9 @@ class APIHelper: # Default headers self.session.headers.update({ - 'User-Agent': 'LEDMatrix-Common/1.0', + # Identifies the client and links to it: ESPN began 403ing bare + # custom tokens (and browser strings) around 2026-08-04. + 'User-Agent': 'LEDMatrix/1.0 (+https://github.com/ChuckBuilds/LEDMatrix)', 'Accept': 'application/json', 'Accept-Language': 'en-US,en;q=0.9', 'Accept-Encoding': 'gzip, deflate, br', @@ -270,20 +272,35 @@ class APIHelper: def clear_cache(self, pattern: Optional[str] = None) -> None: """ Clear cache data. - + + Uses CacheManager's real surface (clear_cache / delete / + list_cache_files); safely no-ops on managers without it. The old + implementation guarded on a nonexistent ``clear`` method, so it + silently never cleared anything. + Args: - pattern: Optional pattern to match cache keys + pattern: Optional substring to match cache keys; only matching + entries are deleted. """ - if self.cache_manager: - if hasattr(self.cache_manager, 'clear'): - if pattern: - # Clear only keys matching pattern - keys = self.cache_manager.keys() - for key in keys: - if pattern in key: - self.cache_manager.delete(key) - else: - self.cache_manager.clear() + if not self.cache_manager: + return + if pattern: + if (hasattr(self.cache_manager, 'list_cache_files') + and hasattr(self.cache_manager, 'delete')): + for entry in self.cache_manager.list_cache_files(): + key = entry.get('key') if isinstance(entry, dict) else None + if key and pattern in key: + self.cache_manager.delete(key) + else: + self.logger.debug( + "Cache manager lacks list_cache_files/delete; " + "cannot clear by pattern") + elif hasattr(self.cache_manager, 'clear_cache'): + self.cache_manager.clear_cache() + elif hasattr(self.cache_manager, 'clear'): + self.cache_manager.clear() + else: + self.logger.debug("Cache manager exposes no clear method; no-op") def _get_from_cache(self, key: str) -> Optional[Any]: """Get data from cache.""" diff --git a/src/common/cli.py b/src/common/cli.py deleted file mode 100644 index ec33caaa..00000000 --- a/src/common/cli.py +++ /dev/null @@ -1,111 +0,0 @@ -""" -LEDMatrix Common CLI - -Command-line interface for LEDMatrix Common utilities. -""" - -import argparse -import sys -from pathlib import Path - - -def main(): - """Main CLI entry point.""" - parser = argparse.ArgumentParser( - description="LEDMatrix Common Utilities", - prog="ledmatrix-common" - ) - - subparsers = parser.add_subparsers(dest='command', help='Available commands') - - # Test command - test_parser = subparsers.add_parser('test', help='Test common utilities') - test_parser.add_argument('--display-width', type=int, default=128, help='Display width') - test_parser.add_argument('--display-height', type=int, default=64, help='Display height') - - # Validate command - validate_parser = subparsers.add_parser('validate', help='Validate configuration') - validate_parser.add_argument('config_file', help='Configuration file to validate') - - args = parser.parse_args() - - if args.command == 'test': - test_utilities(args.display_width, args.display_height) - elif args.command == 'validate': - validate_config(args.config_file) - else: - parser.print_help() - - -def test_utilities(display_width: int, display_height: int): - """Test common utilities.""" - print(f"Testing LEDMatrix Common utilities with {display_width}x{display_height} display") - - try: - from ledmatrix_common import LogoHelper, TextHelper, DisplayHelper, GameHelper, ConfigHelper - - # Test LogoHelper - print("Testing LogoHelper...") - logo_helper = LogoHelper(display_width, display_height) - print(f"Logo cache stats: {logo_helper.get_cache_stats()}") - - # Test TextHelper - print("Testing TextHelper...") - text_helper = TextHelper() - fonts = text_helper.load_fonts() - print(f"Loaded {len(fonts)} fonts") - - # Test DisplayHelper - print("Testing DisplayHelper...") - display_helper = DisplayHelper(display_width, display_height) - img = display_helper.create_base_image() - print(f"Created {img.size} base image") - - # Test GameHelper - print("Testing GameHelper...") - GameHelper() - print("GameHelper initialized") - - # Test ConfigHelper - print("Testing ConfigHelper...") - ConfigHelper() - print("ConfigHelper initialized") - - print("All tests passed!") - - except ImportError as e: - print(f"Import error: {e}") - sys.exit(1) - except Exception as e: - print(f"Test error: {e}") - sys.exit(1) - - -def validate_config(config_file: str): - """Validate configuration file.""" - config_path = Path(config_file) - - if not config_path.exists(): - print(f"Configuration file not found: {config_file}") - sys.exit(1) - - try: - from ledmatrix_common import ConfigHelper - - config_helper = ConfigHelper() - config = config_helper.load_config(config_path) - - if config: - print(f"Configuration loaded successfully from {config_file}") - print(f"Found {len(config)} top-level keys") - else: - print(f"Failed to load configuration from {config_file}") - sys.exit(1) - - except Exception as e: - print(f"Validation error: {e}") - sys.exit(1) - - -if __name__ == '__main__': - main() diff --git a/src/common/config_helper.py b/src/common/config_helper.py index 79840188..b3e5aeac 100644 --- a/src/common/config_helper.py +++ b/src/common/config_helper.py @@ -5,6 +5,7 @@ Handles configuration management and validation for LED matrix plugins. Extracted from LEDMatrix core to provide reusable functionality for plugins. """ +import copy import json import logging from pathlib import Path @@ -160,17 +161,20 @@ class ConfigHelper: override_config: Configuration to merge in (takes precedence) Returns: - Merged configuration dictionary + Merged configuration dictionary (fully independent of both + inputs — a shallow copy would alias un-overridden nested dicts, + so mutating the result would mutate the caller's base config). """ - merged = base_config.copy() - + merged = copy.deepcopy(base_config) + for key, value in override_config.items(): if key in merged and isinstance(merged[key], dict) and isinstance(value, dict): # Recursively merge nested dictionaries merged[key] = self.merge_configs(merged[key], value) else: - # Override with new value - merged[key] = value + # Override with new value — deep-copied so mutating the + # merged result can't reach back into override_config. + merged[key] = copy.deepcopy(value) return merged diff --git a/src/common/display_helper.py b/src/common/display_helper.py index 13dc3030..c88c9e63 100644 --- a/src/common/display_helper.py +++ b/src/common/display_helper.py @@ -115,17 +115,13 @@ class DisplayHelper: if home_logo and away_logo: self._draw_logos(main_img, home_logo, away_logo) - # Draw status/period text (top center) - if status_text or period_text: - status_display = f"{period_text} {status_text}".strip() - if status_display: - self._draw_centered_text(draw, status_display, - fonts.get('time', fonts.get('status')), - y_position=1) - - # Draw clock if available - if clock: - self._draw_centered_text(draw, clock, fonts.get('time'), y_position=1) + # Draw one combined top line (period/status/clock all share y=1 — + # drawing them separately overprinted each other). + top_line = " ".join(p for p in [period_text, status_text, clock] if p) + if top_line: + self._draw_centered_text(draw, top_line, + fonts.get('time', fonts.get('status')), + y_position=1) # Draw scores (center) score_text = f"{away_score}-{home_score}" @@ -153,26 +149,28 @@ class DisplayHelper: """ Draw a ticker/scrolling text layout. + Renders a single static frame with the text at the left edge; the + caller advances the scroll by re-rendering or shifting. The + scroll_speed parameter is accepted for API compatibility but does + not affect this frame. (Previously the text was drawn starting at + x=display_width — entirely off-canvas — so every frame was blank.) + Args: text: Text to display font: Font to use background_color: Background color text_color: Text color - scroll_speed: Pixels to scroll per frame - + scroll_speed: Accepted for compatibility; unused per-frame + Returns: PIL Image with ticker layout """ img = self.create_base_image(background_color) draw = ImageDraw.Draw(img) - - # Start text off-screen to the right - x_position = self.display_width - - # Draw text - self._draw_text_with_outline(draw, text, (x_position, self.display_height // 2 - 6), + + self._draw_text_with_outline(draw, text, (0, self.display_height // 2 - 6), font, fill=text_color) - + return img def draw_centered_text(self, text: str, font: ImageFont.ImageFont, @@ -214,15 +212,9 @@ class DisplayHelper: Returns: PIL Image with error message """ - img = self.create_base_image((50, 0, 0)) # Dark red background - - # Use default font + # Dark red background, white text font = ImageFont.load_default() - - # Draw centered error message - self._draw_centered_text(message, font, (50, 0, 0), (255, 255, 255)) - - return img + return self.draw_centered_text(message, font, (50, 0, 0), (255, 255, 255)) def draw_no_data_message(self, message: str = "No Data") -> Image.Image: """ @@ -234,11 +226,8 @@ class DisplayHelper: Returns: PIL Image with no data message """ - img = self.create_base_image((0, 0, 0)) font = ImageFont.load_default() - self._draw_centered_text(message, font, (0, 0, 0), (150, 150, 150)) - - return img + return self.draw_centered_text(message, font, (0, 0, 0), (150, 150, 150)) def get_display_dimensions(self) -> Tuple[int, int]: """ diff --git a/src/common/logo_helper.py b/src/common/logo_helper.py index 7d0dc4df..743e7b7b 100644 --- a/src/common/logo_helper.py +++ b/src/common/logo_helper.py @@ -6,6 +6,8 @@ Extracted from LEDMatrix core to provide reusable functionality for plugins. """ import logging +import os +import tempfile from pathlib import Path from typing import Dict, List, Optional, Union @@ -19,6 +21,10 @@ from src.common.permission_utils import ( ) +# Well above any real team logo; bounds what a remote URL can write to disk. +MAX_LOGO_BYTES = 10 * 1024 * 1024 + + class LogoHelper: """ Helper class for logo loading, caching, and resizing. @@ -187,10 +193,17 @@ class LogoHelper: def normalize_abbreviation(self, team_abbr: str) -> str: """ Normalize team abbreviation for consistent filename usage. - + + NOTE: this deliberately differs from + LogoDownloader.normalize_abbreviation (src/logo_downloader.py), + which replaces filesystem-unsafe characters (/ \\ : * ? " < > |) + but does not strip spaces. Plugins call the LogoDownloader + version; changing either implementation changes which logo + filenames resolve on existing installs. + Args: team_abbr: Raw team abbreviation - + Returns: Normalized abbreviation """ @@ -219,7 +232,10 @@ class LogoHelper: return { 'cached_logos': len(self._logo_cache), 'cache_size_limit': self.cache_size, - 'cache_usage_percent': (len(self._logo_cache) / self.cache_size) * 100 + 'cache_usage_percent': ( + (len(self._logo_cache) / self.cache_size) * 100 + if self.cache_size else 0 + ), } def _resize_logo(self, logo: Image.Image, max_width: Optional[int] = None, @@ -251,21 +267,64 @@ class LogoHelper: self._cache_order.append(cache_key) def _download_logo(self, url: str, file_path: Path) -> None: - """Download logo from URL.""" + """Download logo from URL. + + The response size is capped and the saved file is verified as a + decodable image before it is left on disk: a logo URL is remote + input, and without this an oversized or malformed response would + be cached for every later load_logo() call to trip over. + + The body is streamed and counted as it arrives rather than read + through response.content, which buffers the whole thing first — + a server that omits Content-Length and never stops sending would + exhaust memory before any size check could run. Nothing lands at + file_path until the download completes and decodes, so a failed + download cannot leave a truncated logo behind either. + """ # Ensure directory exists with proper permissions ensure_directory_permissions(file_path.parent, get_assets_dir_mode()) - - # Download with timeout - response = self.session.get(url, timeout=30) - response.raise_for_status() - - # Save to file - with open(file_path, 'wb') as f: - f.write(response.content) - + + # A unique temp name, not a fixed ".part": two plugins can + # ask for the same logo at once, and a shared name would let them + # interleave writes into one file, publish the mixture, or delete + # each other's partial. Same directory, so os.replace stays atomic. + fd, tmp_name = tempfile.mkstemp( + dir=str(file_path.parent), prefix=file_path.name + '.', suffix='.part') + tmp_path = Path(tmp_name) + try: + # fdopen outermost so the descriptor mkstemp handed back is + # always adopted and closed, including when the request itself + # raises — load_logo_with_download swallows that, so a leak + # here would accumulate quietly on a URL that keeps failing. + with os.fdopen(fd, 'wb') as f: + with self.session.get(url, timeout=30, stream=True) as response: + response.raise_for_status() + downloaded = 0 + for chunk in response.iter_content(chunk_size=64 * 1024): + if not chunk: + continue + downloaded += len(chunk) + if downloaded > MAX_LOGO_BYTES: + raise ValueError( + f"Logo at {url} exceeds the " + f"{MAX_LOGO_BYTES}-byte limit; not saved") + f.write(chunk) + + # Verify it decodes before it becomes the cached logo. PIL + # raises DecompressionBombError past its own pixel limit; a + # partial or non-image response raises UnidentifiedImageError + # (an OSError subclass). + with Image.open(tmp_path) as probe: + probe.load() + + os.replace(tmp_path, file_path) + except BaseException: + tmp_path.unlink(missing_ok=True) + raise + # Set proper file permissions after saving ensure_file_permissions(file_path, get_assets_file_mode()) - + self.logger.debug(f"Downloaded logo to {file_path}") def _create_placeholder_logo(self, team_abbr: str, diff --git a/src/common/scroll_helper.py b/src/common/scroll_helper.py index 4f2e215e..6e2f4218 100644 --- a/src/common/scroll_helper.py +++ b/src/common/scroll_helper.py @@ -328,7 +328,11 @@ class ScrollHelper: elapsed_time = current_time - (self.scroll_start_time or current_time) # The image already includes display_width padding, so we only need total_scroll_width required_total_distance = self.total_scroll_width - self.logger.info( + # Progress telemetry, emitted every few seconds for the whole of + # every scroll. It says how far along a marquee is, which is what + # you turn debug on to watch and not something an operator needs + # in the journal on a device that scrolls all day. + self.logger.debug( "Scroll progress: elapsed=%.2fs, target=%.2fs, total_scrolled=%.0f/%d px (%.1f%%)", elapsed_time, self.calculated_duration, diff --git a/src/common/sync_manager.py b/src/common/sync_manager.py index d51bbb27..12fddc8f 100644 --- a/src/common/sync_manager.py +++ b/src/common/sync_manager.py @@ -19,6 +19,7 @@ Port default: 5765 (UDP). Open this port on both Pis if ufw is active: import io import json +import math import os import socket import struct @@ -37,6 +38,13 @@ _RAW_MAGIC = b'SYNC_RAW' _RAW_HEADER = struct.Struct(' None: hw = self._hw_config @@ -273,11 +286,10 @@ class DisplaySyncManager: break data.extend(chunk) img = Image.open(io.BytesIO(data)) - _MAX_W, _MAX_H = 100_000, 256 # generous for any real scroll image - if img.width > _MAX_W or img.height > _MAX_H: + if img.width > _MAX_FRAME_W or img.height > _MAX_FRAME_H: self.logger.warning( "Sync: rejected oversized scroll image %dx%d (max %dx%d) from %s", - img.width, img.height, _MAX_W, _MAX_H, addr, + img.width, img.height, _MAX_FRAME_W, _MAX_FRAME_H, addr, ) continue try: @@ -396,7 +408,7 @@ class DisplaySyncManager: data = header + arr.tobytes() if len(data) <= 65000: self._send_sock.sendto(data, (self._peer_ip, self.port)) - elif not getattr(self, '_oversized_frame_warned', False): + elif not self._oversized_frame_warned: self._oversized_frame_warned = True self.logger.warning( "Sync: frame too large for UDP (%d bytes, max 65000) — " @@ -451,43 +463,76 @@ class DisplaySyncManager: ) self.write_status_file() + def _handle_received_frame(self, img: Image.Image, sender_ip: str) -> None: + """Record a decoded leader frame and enter follower mode if needed.""" + with self._frame_lock: + self._latest_frame = img + self._last_leader_frame_time = time.time() + self._leader_ip = sender_ip + + if self._follower_state == FollowerState.STANDALONE: + self._follower_state = FollowerState.FOLLOWER + self.logger.info( + "Sync: leader active at %s — switching to follower mode", + sender_ip, + ) + self.write_status_file() + def _follower_recv_loop(self) -> None: while self._running: try: data, addr = self._recv_sock.recvfrom(65535) sender_ip = addr[0] - if data[:8] == _RAW_MAGIC or len(data) > 512: - # Frame data: prefer magic-tagged raw RGB; fall back to legacy PNG + if data[:8] == _RAW_MAGIC: + # Magic-tagged raw RGB frame — self-describing, no guessing. try: - if data[:8] == _RAW_MAGIC: - w, h = _RAW_HEADER.unpack(data[8:12]) - raw = data[12:] - img = Image.frombuffer( - "RGB", (w, h), raw, "raw", "RGB", 0, 1 - ) - else: - # Fallback: try legacy PNG - img = Image.open(io.BytesIO(data)) - img.load() - with self._frame_lock: - self._latest_frame = img - self._last_leader_frame_time = time.time() - self._leader_ip = sender_ip - - if self._follower_state == FollowerState.STANDALONE: - self._follower_state = FollowerState.FOLLOWER - self.logger.info( - "Sync: leader active at %s — switching to follower mode", - sender_ip, - ) - self.write_status_file() + w, h = _RAW_HEADER.unpack(data[8:12]) + raw = data[12:] + img = Image.frombuffer( + "RGB", (w, h), raw, "raw", "RGB", 0, 1 + ) + self._handle_received_frame(img, sender_ip) except Exception as exc: self.logger.debug("Sync: frame decode error: %s", exc) else: - # Control message + # No magic prefix. Whether the payload parses as JSON + # decides between a control message and a legacy + # (pre-magic) PNG frame — both wire formats are + # self-describing, so no size heuristic is needed. A + # >512-byte control message used to be misrouted into + # image decode and silently dropped. try: msg = json.loads(data.decode("utf-8")) + except (json.JSONDecodeError, UnicodeDecodeError): + # Not JSON — try a legacy PNG frame. + try: + img = Image.open(io.BytesIO(data)) + if img.width > _MAX_FRAME_W or img.height > _MAX_FRAME_H: + # Same cap the TCP image path applies: decode + # is deferred until load(), so check first. + self.logger.debug( + "Sync: rejected oversized legacy frame %dx%d from %s", + img.width, img.height, sender_ip, + ) + continue + img.load() + self._handle_received_frame(img, sender_ip) + except Exception as exc: + self.logger.debug("Sync: frame decode error: %s", exc) + continue + + # It parsed, so it is a control message and never a + # frame. Read and validate its fields under a guard — + # a UDP payload is attacker-shaped, so a non-object + # body makes .get() raise AttributeError and an "sx" + # carrying a non-numeric x raises ValueError/TypeError + # — but dispatch the callback *outside* it. Running + # the callback in here would let a fault in someone + # else's code read as a malformed packet and be + # logged as one. + fire_new_cycle = False + try: t = msg.get("t") if t == "hello_ack": self._leader_ip = sender_ip @@ -501,7 +546,17 @@ class DisplaySyncManager: self.write_status_file() elif t == "sx": # Vegas scroll-position sync — tiny message, renders locally - self._latest_scroll_x = float(msg["x"]) + scroll_x = float(msg["x"]) + if not math.isfinite(scroll_x): + # json.loads accepts the NaN/Infinity literals, + # and float("nan") accepts the strings, so a + # non-finite x reaches here intact. Left alone + # it poisons every offset computed from it — + # NaN comparisons are all false, so the + # follower renders a frame it can never scroll + # back from. Treat it as malformed. + raise ValueError(f"non-finite scroll x: {msg['x']!r}") + self._latest_scroll_x = scroll_x self._last_leader_frame_time = time.time() self._leader_ip = sender_ip if self._follower_state == FollowerState.STANDALONE: @@ -511,19 +566,22 @@ class DisplaySyncManager: sender_ip, ) self.write_status_file() - if self._on_new_cycle: - self._on_new_cycle() # build initial scroll image + fire_new_cycle = True # build initial scroll image elif t == "nc": # Leader started a new scroll cycle — rebuild local image - if self._on_new_cycle: - self._on_new_cycle() - except (json.JSONDecodeError, UnicodeDecodeError, KeyError): - pass + fire_new_cycle = True + except (KeyError, AttributeError, TypeError, ValueError) as exc: + self.logger.debug("Sync: malformed control message: %s", exc) + continue + + if fire_new_cycle and self._on_new_cycle: + self._on_new_cycle() except socket.timeout: continue except Exception as exc: self.logger.debug("Sync follower recv error: %s", exc) + time.sleep(0.1) def _follower_announce_loop(self) -> None: hw = self._hw_config diff --git a/src/config_manager.py b/src/config_manager.py index 3eb888f7..453e73b1 100644 --- a/src/config_manager.py +++ b/src/config_manager.py @@ -106,18 +106,13 @@ class ConfigManager: Returns: SaveResult with status and details """ - # Load current secrets to preserve them - secrets_content = {} - if os.path.exists(self.secrets_path): - try: - with open(self.secrets_path, 'r') as f_secrets: - secrets_content = json.load(f_secrets) - except Exception as e: - self.logger.warning(f"Could not load secrets file {self.secrets_path} during save: {e}") - + # Load current secrets to preserve them (raises if unreadable — see + # _load_secrets_for_save) + secrets_content = self._load_secrets_for_save() + # Strip secrets from main config before saving config_to_write = self._strip_secrets_recursive(new_config_data, secrets_content) - + # Use atomic manager to save atomic_mgr = self._get_atomic_manager() result = atomic_mgr.save_config_atomic( @@ -274,35 +269,86 @@ class ConfigManager: self.logger.error(error_msg, exc_info=True) raise ConfigError(error_msg, config_path=self.config_path) from e + @staticmethod + def _is_parallel_secrets_list(value: Any) -> bool: + """True for the parallel-placeholder list shape emitted by + ``secret_helpers.separate_secrets`` for array-item secrets: a + non-empty list whose elements are ALL dicts (``{}`` marks an item + with no secrets). Any other list-shaped secrets value is a + whole-key secret (e.g. a list of secret scalars).""" + return (isinstance(value, list) and bool(value) + and all(isinstance(item, dict) for item in value)) + def _strip_secrets_recursive(self, data_to_filter: Dict[str, Any], secrets: Dict[str, Any]) -> Dict[str, Any]: """Recursively remove secret keys from a dictionary.""" result = {} for key, value in data_to_filter.items(): - if key in secrets: - if isinstance(value, dict) and isinstance(secrets[key], dict): - # This key is a shared group, recurse - stripped_sub_dict = self._strip_secrets_recursive(value, secrets[key]) - if stripped_sub_dict: # Only add if there's non-secret data left - result[key] = stripped_sub_dict - # Else, it's a secret key at this level, so we skip it - else: + if key not in secrets: # This key is not in secrets, so we keep it result[key] = value + continue + sec = secrets[key] + if isinstance(value, dict) and isinstance(sec, dict): + # This key is a shared group, recurse + stripped_sub_dict = self._strip_secrets_recursive(value, sec) + if stripped_sub_dict: # Only add if there's non-secret data left + result[key] = stripped_sub_dict + elif isinstance(value, list) and self._is_parallel_secrets_list(sec): + # Parallel-list shape from separate_secrets: sec[i] holds the + # secret fields of value[i] ({} = item i has none). Strip each + # item and ALWAYS keep the list — indices must survive so the + # merge-on-load can realign secrets with their items. The + # regular list's length is authoritative: extra secrets + # entries are ignored. + stripped_items = [] + for i, item in enumerate(value): + s_item = sec[i] if i < len(sec) else {} + if isinstance(item, dict) and s_item: + stripped_items.append(self._strip_secrets_recursive(item, s_item)) + else: + stripped_items.append(item) + result[key] = stripped_items + # Else: whole-key secret (scalar, list of secret scalars, or a + # shape mismatch) -> drop the key entirely. Never leak. return result + def _load_secrets_for_save(self) -> Dict[str, Any]: + """Load config_secrets.json for stripping before a save. + + A missing secrets file is fine (nothing to strip). But a file that + EXISTS and cannot be read or parsed means stripping is impossible — + and the in-memory config being saved has secrets deep-merged into it, + so proceeding would write them into config.json in plaintext. That + was the historical behavior; it is now a hard refusal. The save + raises so the caller (and user) fixes the secrets file instead of + silently leaking its contents into the world-readable main config. + """ + if not os.path.exists(self.secrets_path): + return {} + try: + with open(self.secrets_path, 'r') as f_secrets: + return json.load(f_secrets) + # Only the expected read/parse failures — an unexpected implementation + # error should propagate as itself, not masquerade as a secrets-file + # problem. (JSONDecodeError and UnicodeDecodeError are ValueErrors.) + except (OSError, ValueError, RecursionError) as e: + error_msg = ( + f"Refusing to save config: secrets file {self.secrets_path} exists " + f"but could not be loaded ({e}). Saving without it would write " + f"merged secret values into config.json in plaintext. Fix or " + f"remove the secrets file, then retry." + ) + self.logger.error("[Config] %s", error_msg, exc_info=True) + raise ConfigError(error_msg, config_path=self.secrets_path) from e + def save_config(self, new_config_data: Dict[str, Any]) -> None: - """Save configuration to the main JSON file, stripping out secrets.""" - secrets_content = {} - if os.path.exists(self.secrets_path): - try: - with open(self.secrets_path, 'r') as f_secrets: - secrets_content = json.load(f_secrets) - except Exception as e: - self.logger.warning(f"Could not load secrets file {self.secrets_path} during save: {e}") - # Continue without stripping if secrets can't be loaded, or handle as critical error - # For now, we'll proceed cautiously and save the full new_config_data if secrets are unreadable - # to prevent accidental data loss if the secrets file is temporarily corrupt. - # A more robust approach might be to fail the save or use a cached version of secrets. + """Save configuration to the main JSON file, stripping out secrets. + + Raises ConfigError when the secrets file exists but cannot be loaded, + because stripping would be impossible and secrets would leak into + config.json. + """ + secrets_content = self._load_secrets_for_save() config_to_write = self._strip_secrets_recursive(new_config_data, secrets_content) @@ -339,11 +385,39 @@ class ConfigManager: return None def _deep_merge(self, target: Dict[str, Any], source: Dict[str, Any]) -> None: - """Deep merge source dict into target dict.""" + """Deep merge source dict into target dict. + + Sole call site: merging config_secrets.json into the loaded config. + Understands the parallel-list shape separate_secrets emits for + array-item secrets (see _is_parallel_secrets_list): each secrets + list item is merged into the config list item at the same index + ({} placeholders skipped). The config list's length is + authoritative — a user deleting an array item from config.json + must not have it resurrected from a stale secrets entry.""" for key, value in source.items(): if key in target and isinstance(target[key], dict) and isinstance(value, dict): self._deep_merge(target[key], value) + elif (key in target and isinstance(target[key], list) + and self._is_parallel_secrets_list(value)): + tlist = target[key] + for i, s_item in enumerate(value): + if i >= len(tlist): + # Interpolate only config-side data here — nothing + # iterated out of the secrets dict (not even the key + # name) may reach the log. + self.logger.warning( + "A secrets list is longer than the config list it " + "parallels (config has %d item(s)); ignoring the " + "extra entries", len(tlist)) + break + if not s_item: + continue # {} placeholder: item i has no secrets + if isinstance(tlist[i], dict): + self._deep_merge(tlist[i], s_item) + else: + tlist[i] = s_item # shape drift; the secret wins else: + # Scalars AND whole-secret scalar arrays: replace (legacy). target[key] = value def _create_config_from_template(self) -> None: @@ -449,10 +523,6 @@ class ConfigManager: """Get display configuration.""" return self.config.get('display', {}) - def get_clock_config(self) -> Dict[str, Any]: - """Get clock configuration.""" - return self.config.get('clock', {}) - def get_config(self) -> Dict[str, Any]: """Get the full configuration dictionary. diff --git a/src/display_controller.py b/src/display_controller.py index 01877279..bb650698 100644 --- a/src/display_controller.py +++ b/src/display_controller.py @@ -44,6 +44,20 @@ from src.common.sync_manager import DisplaySyncManager, SyncRole # Get logger with consistent configuration logger = get_logger(__name__) +# How long startup will wait for plugins to fetch their first data before +# showing anything. Each plugin's update blocks for up to the executor's 30s +# timeout and they run one after another, so the uncapped total is the sum of +# every slow plugin: 82 seconds on the worst boot measured, with a blank panel +# throughout. Whatever does not finish in time is picked up by the scheduled +# update tick moments later, with the display already running. +_INITIAL_UPDATE_BUDGET_SECONDS = 20.0 + +# The least budget worth starting a plugin with. Below this the plugin is +# deferred instead: granting it a floor would let the pass run past its +# deadline, and granting it the true remainder would record a timeout for a +# slot it never had a chance to use. +_MIN_INITIAL_UPDATE_TIMEOUT_SECONDS = 2.0 + # Vegas mode import (lazy loaded to avoid circular imports) _vegas_mode_imported = False VegasModeCoordinator = None @@ -90,7 +104,8 @@ class DisplayController: # Validate startup configuration try: from src.startup_validator import StartupValidator - validator = StartupValidator(self.config_manager) + validator = StartupValidator(self.config_manager, + cache_manager=self.cache_manager) is_valid, errors, warnings = validator.validate_all() if warnings: @@ -166,6 +181,16 @@ class DisplayController: self.plugin_modes = {} # mode -> plugin_instance mapping for plugin-first dispatch self.mode_to_plugin_id: Dict[str, str] = {} self.plugin_display_modes: Dict[str, List[str]] = {} + # plugin_display_modes is mutated only by _register_loaded_plugin / + # _unregister_plugin on the render thread, but the config-watcher + # thread reads it in _enabled_plugin_not_running. Both mutation sites + # run during reconcile (rare), so this lock never touches the per-frame + # path -- the hot-path reads are same-thread as the writes. + self._plugin_modes_lock = threading.Lock() + # Guards the consume-and-clear of _pending_plugin_reconcile. Only taken + # when a reconcile is actually pending or a config change arrives, both + # rare -- the per-frame path just reads the bool. + self._reconcile_flag_lock = threading.Lock() # Per-plugin config-change callbacks, kept so we can unsubscribe a # plugin when it is disabled live. self._plugin_config_callbacks: Dict[str, Callable] = {} @@ -258,7 +283,8 @@ class DisplayController: # Validate plugins after plugin manager is created try: from src.startup_validator import StartupValidator - validator = StartupValidator(self.config_manager, self.plugin_manager) + validator = StartupValidator(self.config_manager, self.plugin_manager, + cache_manager=self.cache_manager) is_valid, errors, warnings = validator.validate_all() if warnings: @@ -447,8 +473,10 @@ class DisplayController: self._refresh_config_cache(new_config) # If a plugin was enabled/disabled, flag a reconcile for the main # loop to apply (loading/unloading off the watcher thread is unsafe). - if self._enabled_set_changed(old_config, new_config): - self._pending_plugin_reconcile = True + if (self._enabled_set_changed(old_config, new_config) + or self._enabled_plugin_not_running(new_config)): + with self._reconcile_flag_lock: + self._pending_plugin_reconcile = True self.config_service.subscribe(_controller_config_change) @@ -461,7 +489,7 @@ class DisplayController: # Initial data update for plugins (ensures data available on first display) logger.info("Performing initial plugin data update...") update_start = time.time() - self._update_modules() + self._update_modules(deadline=update_start + _INITIAL_UPDATE_BUDGET_SECONDS) logger.info("Initial plugin update completed in %.3f seconds", time.time() - update_start) # Initialize Vegas mode coordinator @@ -817,14 +845,42 @@ class DisplayController: self._cached_target_brightness = normal_brightness # persist for minute-gate return normal_brightness - def _update_modules(self): - """Update all plugin modules.""" + def _update_modules(self, deadline: Optional[float] = None): + """Update all plugin modules. + + Args: + deadline: Wall-clock time after which remaining plugins are left + for the scheduled update tick instead of being waited on. Each + update blocks this thread for up to the executor's timeout, and + they run one after another, so without a bound the total is the + sum of every slow plugin on the system. Measured at startup on + a live rig: 82 seconds, 55 and 26 on the two boots before -- all + of it with nothing on the panel. + """ if not self.plugin_manager: return - + # Update all loaded plugins plugins_dict = getattr(self.plugin_manager, 'loaded_plugins', None) or getattr(self.plugin_manager, 'plugins', {}) + deferred = [] for plugin_id, plugin_instance in plugins_dict.items(): + update_timeout = None + if deadline is not None: + update_timeout = deadline - time.time() + if update_timeout < _MIN_INITIAL_UPDATE_TIMEOUT_SECONDS: + # Too little left to be worth starting. Deferring rather + # than granting a floor keeps the budget a real ceiling -- + # clamping up to a minimum let a plugin that began with a + # sliver left run on past the deadline -- and a plugin + # handed a slot it cannot use would just be recorded as + # having timed out. + # + # Nothing is lost either way: a plugin that has never + # updated is immediately due, so run_scheduled_updates() + # picks it up within seconds, with the display already + # running. + deferred.append(plugin_id) + continue # Check circuit breaker before attempting update if hasattr(self.plugin_manager, 'health_tracker') and self.plugin_manager.health_tracker: if self.plugin_manager.health_tracker.should_skip_plugin(plugin_id): @@ -833,7 +889,13 @@ class DisplayController: # Use PluginExecutor if available for safe execution if hasattr(self.plugin_manager, 'plugin_executor'): - success = self.plugin_manager.plugin_executor.execute_update(plugin_instance, plugin_id) + # The remaining budget is the timeout, so the pass cannot + # run past its deadline. Bounding the loop alone did not do + # it: the last plugin to start could still block for the + # executor's full 30s, which turned a 20s budget into a 31.8s + # pass on the rig. + success = self.plugin_manager.plugin_executor.execute_update( + plugin_instance, plugin_id, timeout=update_timeout) if success and hasattr(self.plugin_manager, 'plugin_last_update'): self.plugin_manager.plugin_last_update[plugin_id] = time.time() else: @@ -852,6 +914,12 @@ class DisplayController: if hasattr(self.plugin_manager, 'health_tracker') and self.plugin_manager.health_tracker: self.plugin_manager.health_tracker.record_failure(plugin_id, exc) + if deferred: + logger.info( + "Initial update budget spent; %d plugin(s) left to the update " + "tick so the display can start: %s", + len(deferred), ", ".join(deferred)) + def _tick_plugin_updates_for_vegas(self) -> None: """Run scheduled plugin updates and tell Vegas mode which plugins actually got fresh data, so it can hot-swap them into the scroll @@ -1638,6 +1706,12 @@ class DisplayController: logger.warning("Error checking live priority for %s: %s", mode_name, e) return live + def _vegas_keeps_live_in_ticker(self) -> bool: + """Whether live content should stay in the ticker instead of preempting it.""" + coordinator = getattr(self, 'vegas_coordinator', None) + config = getattr(coordinator, 'vegas_config', None) + return bool(getattr(config, 'live_in_ticker', False)) + def _check_live_priority(self, advance=False): """Return the live-priority mode to display, or None if nothing is live. @@ -1687,11 +1761,12 @@ class DisplayController: # rebuilding available_modes happens here on the render thread so # it can't race with rendering. Deferred while on-demand is active # (the flag stays set) so we don't fight its temporary-enable. + # The lock-free read is a fast path only; it can be a false + # negative (the watcher setting the flag just after it is read + # is seen next iteration), never a false positive that loses a + # request. if self._pending_plugin_reconcile and not self.on_demand_active: - # Only clear the flag on success -- a retryable failure - # (e.g. discovery) leaves it set so the request isn't lost. - if self._reconcile_enabled_plugins(): - self._pending_plugin_reconcile = False + self._service_pending_reconcile() if not self.available_modes: # Nothing to render yet. Re-check _pending_plugin_reconcile @@ -1851,14 +1926,24 @@ class DisplayController: # Check for live priority content and switch to it immediately. # advance=True so multiple simultaneously-live games take turns # (round-robin) instead of pinning to the first plugin. - if not self.on_demand_active and not wifi_status_data: + # Skipped when the ticker is keeping live content: switching + # the rotation underneath Vegas would move current_mode_index + # and stash a resume point for a takeover that never happens. + if (not self.on_demand_active and not wifi_status_data + and not (self._is_vegas_mode_active() + and self._vegas_keeps_live_in_ticker())): live_priority_mode = self._check_live_priority(advance=True) self._apply_live_priority(live_priority_mode) # Vegas scroll mode - continuous ticker across all plugins # Priority: on-demand > wifi-status > live-priority > vegas > normal rotation if self._is_vegas_mode_active() and not wifi_status_data: - live_mode = self._check_live_priority() + # Live content normally preempts the ticker entirely. With + # vegas_scroll.live_in_ticker the marquee keeps running and + # the live plugin takes extra turns inside it instead -- + # see StreamManager._apply_priority_weights. + live_mode = (None if self._vegas_keeps_live_in_ticker() + else self._check_live_priority()) if not live_mode: try: # Run Vegas mode iteration @@ -2741,7 +2826,8 @@ class DisplayController: logger.debug("Using manifest display_modes for %s: %s", plugin_id, display_modes) if not (isinstance(display_modes, list) and display_modes): display_modes = [plugin_id] - self.plugin_display_modes[plugin_id] = list(display_modes) + with self._plugin_modes_lock: + self.plugin_display_modes[plugin_id] = list(display_modes) # Subscribe to config changes for per-plugin hot-reload. Bind plugin_id # and instance as defaults so each plugin's callback targets its own @@ -2775,7 +2861,8 @@ class DisplayController: def _unregister_plugin(self, plugin_id: str) -> None: """Remove a plugin's modes, config subscription and instance, then unload it. Used by live disable hot-reload.""" - modes = self.plugin_display_modes.pop(plugin_id, []) + with self._plugin_modes_lock: + modes = self.plugin_display_modes.pop(plugin_id, []) for mode in modes: if mode in self.available_modes: self.available_modes.remove(mode) @@ -2820,6 +2907,67 @@ class DisplayController: } return enabled_map(old_config) != enabled_map(new_config) + def _service_pending_reconcile(self) -> None: + """Consume a pending reconcile request and run it. + + The request is consumed BEFORE reconciling, not cleared after. Clearing + after would drop any config change that lands while reconcile is + running: reconcile has already read its config by then, so the clear + erases a request it never served and the newest config never + reconciles -- the same "your save did nothing" failure this whole path + exists to prevent. Consuming first means such a request stays set and + is picked up on the next pass. + + A retryable failure (e.g. discovery) re-arms the flag. + """ + with self._reconcile_flag_lock: + pending = self._pending_plugin_reconcile + self._pending_plugin_reconcile = False + if pending and not self._reconcile_enabled_plugins(): + with self._reconcile_flag_lock: + self._pending_plugin_reconcile = True + + def _enabled_plugin_not_running(self, new_config: Dict[str, Any]) -> bool: + """True when a discovered plugin is enabled in config but not running. + + ``_enabled_set_changed`` compares only top-level ``enabled`` flags, which + misses the case that strands a plugin: one whose ``validate_config()`` + returned False is absent from the running set, and the edit that fixes it + (enabling a league, filling in an API key) lives *nested* inside that + plugin's own section. No top-level flag changes, so no reconcile is + queued, and the save that should have fixed it appears to do nothing -- + only toggling some unrelated plugin recovers it. hockey-scoreboard sat + enabled-but-absent on a live rig for four days this way. + + Deliberately narrow: it fires only for ids the plugin manager has + actually discovered, so non-plugin sections that carry their own + ``enabled`` flag (``schedule``, ``display``, ...) don't queue a reconcile + on every save. In the steady state -- everything enabled is loaded -- + this is False and costs nothing. That matters because reconcile runs + ``discover_plugins()`` on the render thread, where a needless + filesystem scan per config save would show up as a frame hitch. + + Runs on the config-watcher thread, so both mappings it reads are + snapshotted under the lock that guards their writes. + """ + if self.plugin_manager is None: + return False + # Two snapshots, each taken under its own lock and never nested, so a + # half-written mapping is never observed and this can't deadlock + # against discovery (which holds the discovery lock while rebuilding). + try: + known = self.plugin_manager.discovered_plugin_ids() + except AttributeError: + # Older manager without the accessor: fall back to a plain read. + known = set(getattr(self.plugin_manager, 'plugin_manifests', ()) or ()) + with self._plugin_modes_lock: + running = set(self.plugin_display_modes) + for key, value in new_config.items(): + if (key in known and isinstance(value, dict) + and value.get('enabled', False) and key not in running): + return True + return False + def _reconcile_enabled_plugins(self) -> bool: """Load/unload plugins so the running set matches the enabled set in config. Runs on the main display thread (never the config-watcher diff --git a/src/display_manager.py b/src/display_manager.py index d96fef35..9cc7f622 100644 --- a/src/display_manager.py +++ b/src/display_manager.py @@ -25,6 +25,7 @@ the same object. import json import os +import socket import tempfile if os.getenv("EMULATOR", "false") == "true": from RGBMatrixEmulator import RGBMatrix, RGBMatrixOptions @@ -258,6 +259,26 @@ class DisplayManager: # Initialize managers # Calendar manager is now initialized by DisplayController + # Orientation setting -> rpi-rgb-led-matrix "Rotate:" pixel-mapper suffix. + # "normal" needs no suffix since 0 degrees is the identity transform. + _ORIENTATION_ROTATE_DEGREES = {'normal': None, '90': 90, '180': 180, '270': 270} + + def _build_pixel_mapper_config(self, hardware_config: dict) -> str: + """Compose the raw pixel_mapper_config string with the orientation setting. + + `pixel_mapper_config` stays available as a free-form advanced field (e.g. + for "U-mapper" chain layouts); `orientation` is the user-facing dropdown + for physical mounting (e.g. panels mounted upside down) and is appended as + a "Rotate:" mapper rather than overwriting any existing config. + """ + base_mapper = (hardware_config.get('pixel_mapper_config') or '').strip() + orientation = hardware_config.get('orientation', 'normal') + degrees = self._ORIENTATION_ROTATE_DEGREES.get(orientation) + if degrees is None: + return base_mapper + rotate_mapper = f'Rotate:{degrees}' + return f'{base_mapper};{rotate_mapper}' if base_mapper else rotate_mapper + def _setup_matrix(self): """Initialize the RGB matrix with configuration settings.""" _init_error_str = None @@ -283,7 +304,7 @@ class DisplayManager: options.pwm_bits = hardware_config.get('pwm_bits', 10) options.pwm_lsb_nanoseconds = hardware_config.get('pwm_lsb_nanoseconds', 150) options.led_rgb_sequence = hardware_config.get('led_rgb_sequence', 'RGB') - options.pixel_mapper_config = hardware_config.get('pixel_mapper_config', '') + options.pixel_mapper_config = self._build_pixel_mapper_config(hardware_config) options.row_address_type = hardware_config.get('row_address_type', 0) options.multiplexing = hardware_config.get('multiplexing', 0) options.panel_type = hardware_config.get('panel_type', '') @@ -497,6 +518,91 @@ class DisplayManager: logger.warning(f"[BRIGHTNESS] Matrix does not support brightness property: {e}", exc_info=True) return -1 + @staticmethod + def _local_ip() -> Optional[str]: + """This device's address on the network it routes through, or None. + + Deliberately not `hostname -I` or a systemctl probe for AP mode, which + is how the web launcher does it: both spawn processes with multi-second + timeouts, and this runs on the startup path the rest of this change + exists to shorten. Connecting a UDP socket sends no packets -- it only + asks the kernel which source address it would use -- so it costs + microseconds and works with the network down, as long as a route + exists. + """ + sock = None + try: + sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) + sock.settimeout(0.2) + sock.connect(("8.8.8.8", 80)) # nosec B104 - no traffic; selects a route + ip = sock.getsockname()[0] + return ip if ip and not ip.startswith("127.") else None + except OSError: + return None + finally: + if sock is not None: + try: + sock.close() + except OSError: + pass + + def _fitting_font(self, lines, width): + """The largest font from the usual ladder that fits every line.""" + candidates = [self.font, + ("assets/fonts/4x6-font.ttf", 6)] + for candidate in candidates: + try: + font = candidate + if isinstance(candidate, tuple): + font = ImageFont.truetype(candidate[0], candidate[1]) + if all(self.draw.textlength(t, font=font) <= width for t in lines): + return font + except (OSError, ValueError, AttributeError): + continue + return self.font + + def _draw_startup_banner(self, lines, width: int, height: int) -> None: + """Centre `lines` over whatever the test pattern already drew. + + This screen stays on the panel for the whole initial plugin update, and + on a headless Pi it is the only place the device's address appears + without going looking for it -- so it has to be readable off a wall, + not merely present. + + The font is chosen to fit rather than fixed at 8px: "Initializing" is + 96px in PressStart2P, which ran off the side of a 64px panel even + before an address was added. And the pattern is punched out behind the + text, because the diagonal runs through the middle of the panel, which + is exactly where this sits. + + The text stays blue. It is not decoration: the pattern draws one pure + channel per element -- red border, green diagonal, blue text -- so that + a glance at the panel says whether led_rgb_sequence is right. Swap the + wiring to BGR and the border comes up blue and this text red. Drawing + it white would light all three channels and destroy the only blue + reference on the screen, which is why it is worth a comment rather + than a quiet preference. + """ + if not lines: + return + font = self._fitting_font(lines, width - 2) + line_height = self.draw.textbbox((0, 0), "Ag", font=font)[3] + 1 + block_height = line_height * len(lines) + block_top = max(1, (height - block_height) // 2) + block_width = max(self.draw.textlength(t, font=font) for t in lines) + block_left = max(0, (width - block_width) // 2) + + self.draw.rectangle( + [block_left - 2, block_top - 1, + block_left + block_width + 1, block_top + block_height], + fill=(0, 0, 0)) + + for row, line in enumerate(lines): + line_width = self.draw.textlength(line, font=font) + self.draw.text( + (max(0, (width - line_width) // 2), block_top + row * line_height), + line, font=font, fill=(0, 0, 255)) + def _draw_test_pattern(self): """Draw a test pattern to verify the display is working.""" try: @@ -516,8 +622,11 @@ class DisplayManager: # Draw a diagonal line self.draw.line([0, 0, self.matrix.width-1, self.matrix.height-1], fill=(0, 255, 0)) - # Draw some text - changed from "TEST" to "Initializing" with smaller font - self.draw.text((10, 10), "Initializing", font=self.font, fill=(0, 0, 255)) + lines = ["Initializing"] + ip = self._local_ip() + if ip: + lines.append(ip) + self._draw_startup_banner(lines, self.matrix.width, self.matrix.height) # Update the display once after everything is drawn self.update_display() diff --git a/src/dynamic_team_resolver.py b/src/dynamic_team_resolver.py index ab969e17..7cbe09cd 100644 --- a/src/dynamic_team_resolver.py +++ b/src/dynamic_team_resolver.py @@ -167,10 +167,14 @@ class DynamicTeamResolver: # Sort by ranking (1, 2, 3, etc.) sorted_rankings = dict(sorted(rankings.items(), key=lambda x: x[1])) - - # Cache the results - self._rankings_cache = sorted_rankings - self._cache_timestamp = current_time + + # Cache the results ON THE CLASS. Assigning through self + # would create instance attributes that shadow the shared + # class-level cache, making it per-instance — and every + # scoreboard constructs its own resolver, so the cache + # would never actually be shared. + DynamicTeamResolver._rankings_cache = sorted_rankings + DynamicTeamResolver._cache_timestamp = current_time self.logger.info(f"Fetched rankings for {len(sorted_rankings)} teams") return sorted_rankings @@ -216,9 +220,11 @@ class DynamicTeamResolver: return any(pattern in team_name.upper() for pattern in dynamic_patterns) def clear_cache(self): - """Clear the rankings cache to force fresh data on next request.""" - self._rankings_cache = {} - self._cache_timestamp = 0 + """Clear the SHARED rankings cache to force fresh data on next + request. Writes through the class — assigning via self would only + shadow the shared cache for this instance.""" + DynamicTeamResolver._rankings_cache = {} + DynamicTeamResolver._cache_timestamp = 0 self.logger.info("Cleared dynamic team rankings cache") diff --git a/src/logging_config.py b/src/logging_config.py index 48eba67c..843257eb 100644 --- a/src/logging_config.py +++ b/src/logging_config.py @@ -5,6 +5,7 @@ Provides consistent logging configuration across the LEDMatrix application. Supports structured logging with context information and appropriate log levels. """ +import copy import logging import sys import os @@ -65,24 +66,29 @@ class ContextualFormatter(logging.Formatter): self.include_context = include_context def format(self, record: logging.LogRecord) -> str: - """Format log record with context.""" - # Add context to message if present + """Format log record with context. + + Works on a shallow copy of the record: a record is formatted once + PER HANDLER, so mutating record.msg in place (the old behavior) + prepended the context prefix again for every additional handler. + """ if self.include_context: context_parts = [] - + if hasattr(record, 'plugin_id'): context_parts.append(f"[Plugin: {record.plugin_id}]") - + if hasattr(record, 'operation_id'): context_parts.append(f"[Op: {record.operation_id}]") - + if hasattr(record, 'context') and isinstance(record.context, dict): for key, value in record.context.items(): context_parts.append(f"[{key}: {value}]") - + if context_parts: + record = copy.copy(record) record.msg = ' '.join(context_parts) + ' ' + str(record.msg) - + return super().format(record) @@ -124,7 +130,12 @@ def setup_logging( # Console handler (always add) console_handler = logging.StreamHandler(sys.stdout) console_handler.setLevel(level) - console_handler.setFormatter(formatter) + # Under systemd, tag each line so the journal records the real severity + # rather than filing everything as informational. The file handler below + # keeps the plain formatter: the prefix is meaningful to journald and noise + # anywhere else. + console_handler.setFormatter( + JournalPriorityFormatter(formatter) if _under_systemd() else formatter) root_logger.addHandler(console_handler) # File handler (if specified) @@ -139,6 +150,84 @@ def setup_logging( sys.stderr.write(f"Warning: Could not set up file logging to {log_file}: {e}\n") +#: syslog priorities, which is what systemd parses from a "" prefix on +#: stdout. Mapped from Python's levels. +_SYSLOG_PRIORITY = { + logging.CRITICAL: 2, # LOG_CRIT + logging.ERROR: 3, # LOG_ERR + logging.WARNING: 4, # LOG_WARNING + logging.INFO: 6, # LOG_INFO + logging.DEBUG: 7, # LOG_DEBUG +} + + +class JournalPriorityFormatter(logging.Formatter): + """Wraps a formatter, prefixing each line with its syslog priority. + + Under systemd everything this process writes to stdout lands in the journal + as PRIORITY=6, whatever the Python level was. Measured on a live rig: 55 + ERROR lines and 13 WARNING lines in a day, every one of them recorded as + informational, so `journalctl -p err -u ledmatrix` returned nothing at all + while errors were being logged. Anyone triaging has to grep the message + text instead, which is both slower and wrong -- a search for "oom" matches + the radar logging "zoom=9". + + systemd reads a leading "" on each line and uses it as the priority + (sd-daemon(3)), so this needs no extra dependency. Multi-line records get + the prefix on every line, since the journal splits them and an unprefixed + continuation would fall back to the default. + """ + + def __init__(self, inner: logging.Formatter): + super().__init__() + self._inner = inner + + @property + def inner(self) -> logging.Formatter: + """The formatter doing the actual work. + + Whether journald tagging is applied depends on JOURNAL_STREAM, so it is + on under systemd and off in a terminal -- and anything asserting which + formatter setup_logging() selected would otherwise get a different + answer in CI than on a developer's machine. Exposing the inner one lets + those checks stay about format_type, which is what they mean. + """ + return self._inner + + def format(self, record: logging.LogRecord) -> str: + text = self._inner.format(record) + prefix = f"<{_SYSLOG_PRIORITY.get(record.levelno, 6)}>" + return "\n".join(prefix + line for line in text.split("\n")) + + +def _under_systemd() -> bool: + """True when stdout really is the journal. + + systemd sets JOURNAL_STREAM to "dev:ino" for services whose output it + captures. Presence alone is not enough to act on: the variable is + inherited by child processes and survives redirection, so a subprocess + whose stdout is a pipe or a file still sees it and would emit the "" + priority prefixes as literal noise into that output. systemd's own + guidance is to fstat the descriptor and compare st_dev/st_ino, which is + what distinguishes "the journal is somewhere in my ancestry" from "my + stdout is the journal". + """ + declared = os.environ.get("JOURNAL_STREAM") + if not declared: + return False + try: + dev_text, ino_text = declared.split(":", 1) + declared_ids = (int(dev_text), int(ino_text)) + except (ValueError, AttributeError): + return False + try: + stat_result = os.fstat(sys.stdout.fileno()) + except (OSError, ValueError, AttributeError): + # No usable stdout: captured by pytest, detached, or already closed. + return False + return (stat_result.st_dev, stat_result.st_ino) == declared_ids + + class PluginLoggerAdapter(logging.LoggerAdapter): """LoggerAdapter that stamps every record with its plugin_id. @@ -224,8 +313,11 @@ def log_warning(logger: logging.Logger, message: str, **kwargs) -> None: def log_error(logger: logging.Logger, message: str, **kwargs) -> None: - """Log error message with context.""" - log_with_context(logger, logging.ERROR, message, **kwargs, exc_info=True) + """Log error message with context. Defaults exc_info=True; a caller + passing exc_info explicitly wins (the old hardcoded keyword raised + TypeError on that duplicate).""" + kwargs.setdefault('exc_info', True) + log_with_context(logger, logging.ERROR, message, **kwargs) def log_debug(logger: logging.Logger, message: str, **kwargs) -> None: diff --git a/src/logo_downloader.py b/src/logo_downloader.py index e4dad335..b799b7c1 100644 --- a/src/logo_downloader.py +++ b/src/logo_downloader.py @@ -118,7 +118,14 @@ class LogoDownloader: @staticmethod def normalize_abbreviation(abbreviation: str) -> str: - """Normalize team abbreviation for consistent filename usage.""" + """Normalize team abbreviation for consistent filename usage. + + Public API: sports scoreboard plugins call this directly. + NOTE: LogoHelper.normalize_abbreviation (src/common/logo_helper.py) + is a deliberately different variant (strips spaces, fewer character + replacements) — keep both behaviors stable; logo filenames on + existing installs depend on them. + """ # Handle special characters that can cause filesystem issues normalized = abbreviation.upper() diff --git a/src/plugin_system/base_plugin.py b/src/plugin_system/base_plugin.py index 8ec4414d..dbe14a49 100644 --- a/src/plugin_system/base_plugin.py +++ b/src/plugin_system/base_plugin.py @@ -364,8 +364,10 @@ class BasePlugin(ABC): # Handle None case if duration is None: pass # Fall through to config - # Try to convert to float if it's a number or numeric string - elif isinstance(duration, (int, float)): + # Try to convert to float if it's a number or numeric string. + # bool is excluded: it's an int subclass, and True would + # otherwise read as a 1-second duration. + elif isinstance(duration, (int, float)) and not isinstance(duration, bool): if duration > 0: return float(duration) else: @@ -403,8 +405,9 @@ class BasePlugin(ABC): # Fall back to config config_duration = self.config.get("display_duration", 15.0) try: - # Ensure config value is also a valid float - if isinstance(config_duration, (int, float)): + # Ensure config value is also a valid float (bool excluded — an + # int subclass that would otherwise read True as 1 second) + if isinstance(config_duration, (int, float)) and not isinstance(config_duration, bool): if config_duration > 0: return float(config_duration) else: @@ -552,6 +555,48 @@ class BasePlugin(ABC): """ return False + def get_vegas_priority_weight(self) -> Optional[int]: + """How many slots per Vegas cycle this plugin should get, or None. + + The Vegas ticker is otherwise a strict round robin: every plugin + appears exactly once per cycle. With a dozen plugins enabled that puts + minutes between a live score and its next appearance. A weight of N + gives the plugin N slots per cycle, spread evenly through it rather + than clumped together. + + Return ``None`` (the default) to let the core decide. It gives a + plugin ``vegas_scroll.live_weight`` when ``has_live_priority()`` and + ``has_live_content()`` are both true, and 1 otherwise -- so live sports + already get extra turns without implementing this at all. + + Implement it only when the plugin knows something the core cannot. The + motivating case is favorite teams: the core can see *that* a game is + live but not *whose*, so a scoreboard that wants its favorite's game + shown more often than other live games has to say so:: + + def get_vegas_priority_weight(self): + if not (self.has_live_priority() and self.has_live_content()): + return None # let the core decide + cfg = self.global_config.get('display', {}).get('vegas_scroll', {}) + if self._favorite_is_live(): + return cfg.get('favorite_live_weight', 5) + return cfg.get('live_weight', 3) + + The weight is per *plugin*, not per game. A scoreboard showing four + live games still occupies one slot at a time and rotates its own games + within that slot; this controls how often the plugin itself comes + round. + + Raising is safe: the core logs it and falls back to its own + live-content check, so a broken weight calculation costs the plugin + the favorite distinction but not the live boost. + + Returns: + Slots per cycle (clamped to 1..10 by the caller), or None to + defer to the core's own live-content weighting. + """ + return None + def get_live_modes(self) -> List[str]: """ Get list of display modes that should be used during live priority takeover. @@ -794,10 +839,12 @@ class BasePlugin(ABC): self.logger.error("'enabled' must be a boolean") return False - # Check display_duration if present + # Check display_duration if present. bool is excluded explicitly: + # it's an int subclass, and get_display_duration rejects it too. if "display_duration" in self.config: duration = self.config["display_duration"] - if not isinstance(duration, (int, float)) or duration <= 0: + if (not isinstance(duration, (int, float)) + or isinstance(duration, bool) or duration <= 0): self.logger.error("'display_duration' must be a positive number") return False diff --git a/src/plugin_system/compatibility.py b/src/plugin_system/compatibility.py index 784a6b6a..9b676cd2 100644 --- a/src/plugin_system/compatibility.py +++ b/src/plugin_system/compatibility.py @@ -180,6 +180,45 @@ def declared_min_version(manifest: Dict[str, Any]) -> Optional[str]: return None +def is_update_available(installed_version: str, latest_version: str) -> bool: + """Return True when the registry's ``latest_version`` is strictly newer + than the installed version. + + THE shared comparator for "should this plugin be updated?" — used by both + the web UI's update badge (`api_v3._is_plugin_update_available`) and the + store's `update_plugin` reinstall decision, so the two can never disagree. + + Uses PEP 440-aware comparison (``packaging``), which also normalizes + equivalent spellings: ``v1.2.0`` == ``1.2.0`` and ``1.2`` == ``1.2.0``, so + cosmetic differences never trigger a reinstall — and a locally modified + plugin whose version is *ahead* of the registry is never "updated" + (downgraded). If either version string can't be parsed the mismatch is + surfaced (True) so the user can reconcile, rather than silently hiding a + potential update. + """ + if not installed_version or not latest_version: + return False + if not isinstance(installed_version, str) or not isinstance(latest_version, str): + # A malformed manifest/registry can carry a number (1.2) or worse; + # packaging would raise TypeError. Surface the mismatch instead. + return True + if installed_version == latest_version: + return False + try: + from packaging.version import parse as _parse_version, InvalidVersion + except ImportError: + # packaging is a core dependency, but if it's somehow unavailable we + # can't compare semantically — surface the mismatch we already know + # exists (the two strings differ). + return True + try: + return _parse_version(latest_version) > _parse_version(installed_version) + except InvalidVersion: + # Unparseable version string: we can't tell direction, so surface the + # mismatch rather than silently hiding a potential update. + return True + + def check(manifest: Dict[str, Any], core_version: str) -> Tuple[bool, Optional[str]]: """Return ``(compatible, reason)``. diff --git a/src/plugin_system/plugin_health.py b/src/plugin_system/plugin_health.py index ecbec076..3908d9ca 100644 --- a/src/plugin_system/plugin_health.py +++ b/src/plugin_system/plugin_health.py @@ -7,7 +7,7 @@ and circuit breaker state. Provides automatic recovery mechanisms. import time import logging -from typing import Dict, Optional, Any +from typing import Dict, Optional, Any, Tuple from enum import Enum @@ -64,10 +64,48 @@ class PluginHealthTracker: cache_key, max_age=None, memory_ttl=0 if force_reload else None ) - if cached: - return cached - - # Default state + if isinstance(cached, dict) and cached: + # Complete it rather than trusting it: a persisted record can be + # missing fields the callers index directly (a partial write, a + # restored backup, an older schema), and returning it verbatim makes + # record_success / record_failure raise KeyError, which takes the + # display down in a restart loop that survives reboots because the + # bad entry is on disk. + state, repaired = self._repair_health_state(cached) + if repaired: + self.logger.warning( + f"Repaired health state for {plugin_id}: " + f"{sorted(repaired)} missing or invalid, using defaults for those." + ) + return state + + # Not a dict at all: written by something other than + # _save_health_state (a key collision, a corrupted entry). Nothing to + # salvage. + if cached is not None and not isinstance(cached, dict): + self.logger.warning( + f"Discarding malformed health state for {plugin_id}: expected " + f"dict, got {type(cached).__name__}. Falling back to defaults." + ) + + return self._default_health_state() + + def _save_health_state(self, plugin_id: str, state: Dict[str, Any]) -> None: + """Save health state to cache.""" + cache_key = self._get_health_key(plugin_id) + self.cache_manager.set(cache_key, state) # Persist indefinitely + self._health_state[plugin_id] = state + + # The fields callers index directly (state['circuit_state'] and friends). + # A cached dict missing any of them raises KeyError deep in record_success / + # record_failure, so the value is completed before it is handed out. + _COUNTER_FIELDS = ('consecutive_failures', 'total_failures', 'total_successes') + _TIMESTAMP_FIELDS = ('last_success_time', 'last_failure_time', + 'circuit_opened_time', 'half_open_start_time') + + @staticmethod + def _default_health_state() -> Dict[str, Any]: + """A fresh state with every field the callers expect.""" return { 'consecutive_failures': 0, 'total_failures': 0, @@ -77,15 +115,56 @@ class PluginHealthTracker: 'circuit_state': CircuitState.CLOSED.value, 'circuit_opened_time': None, 'half_open_start_time': None, - 'last_error': None + 'last_error': None, } - - def _save_health_state(self, plugin_id: str, state: Dict[str, Any]) -> None: - """Save health state to cache.""" - cache_key = self._get_health_key(plugin_id) - self.cache_manager.set(cache_key, state) # Persist indefinitely - self._health_state[plugin_id] = state - + + @classmethod + def _repair_health_state(cls, cached: Dict[str, Any]) -> Tuple[Dict[str, Any], list]: + """Return `cached` completed against the defaults, plus what was repaired. + + Per-field rather than all-or-nothing: a record that has real failure + counts but is missing `last_error` should keep the counts, not be reset + to healthy. Only values that are absent or the wrong type fall back to + the default, so a partial or older-schema record survives with whatever + it does carry, while every field the callers index is guaranteed present + and of a usable type. + """ + state = cls._default_health_state() + repaired = [] + for field, default in state.items(): + if field not in cached: + repaired.append(field) + continue + value = cached[field] + if field in cls._COUNTER_FIELDS: + ok = isinstance(value, int) and not isinstance(value, bool) and value >= 0 + elif field in cls._TIMESTAMP_FIELDS: + # bool is a subclass of int, so True would pass as a timestamp + # and then compare as 1.0 -- expiring a cooldown the instant it + # opens, or (False) making the elapsed check never fire. + ok = value is None or ( + isinstance(value, (int, float)) and not isinstance(value, bool) + ) + elif field == 'circuit_state': + # Membership first requires the value to be hashable: a list or + # dict here would raise TypeError out of the repair itself, + # which is the crash this whole path exists to prevent. + ok = isinstance(value, str) and value in { + member.value for member in CircuitState + } + else: # last_error + ok = value is None or isinstance(value, str) + if ok: + state[field] = value + else: + repaired.append(field) + # Anything the schema has since grown (degraded, degraded_reason) is + # read with .get() by its callers, so carry it through untouched. + for field, value in cached.items(): + if field not in state: + state[field] = value + return state, repaired + def get_health_state(self, plugin_id: str, force_reload: bool = False) -> Dict[str, Any]: """Get current health state for a plugin. @@ -99,11 +178,21 @@ class PluginHealthTracker: ) return self._health_state[plugin_id] + # Fields the circuit breaker is rebuilt from after a restart. Everything + # else in a health record is reporting, read only for display. + _DURABLE_FIELDS = ('consecutive_failures', 'circuit_state', + 'circuit_opened_time', 'half_open_start_time') + + def _durable(self, state: Dict[str, Any]) -> tuple: + """The part of a health record whose loss would change behaviour.""" + return tuple(state.get(field) for field in self._DURABLE_FIELDS) + def record_success(self, plugin_id: str) -> None: """Record a successful plugin execution.""" state = self.get_health_state(plugin_id) current_time = time.time() - + durable_before = self._durable(state) + # Reset consecutive failures state['consecutive_failures'] = 0 state['total_successes'] = state.get('total_successes', 0) + 1 @@ -119,9 +208,20 @@ class PluginHealthTracker: # Shouldn't happen, but handle it state['circuit_state'] = CircuitState.CLOSED.value state['circuit_opened_time'] = None - - self._save_health_state(plugin_id, state) - + + # A healthy plugin reports success every cycle, and in that steady state + # the only fields changed above are a counter and a timestamp that + # nothing reads back after a restart. Persisting them anyway rewrites a + # small file per plugin per cycle: on a rig running 24 plugins, a + # five-minute sample measured 22 rewrites, about 4.4 a minute or 6,300 a + # day. Those land on an SD card, where the cost is an erase-block cycle + # rather than the 400 bytes involved, and where wear is what eventually + # kills the card. + # In-memory state is still updated every time, so the health API and web + # UI show exactly what they did before; only the write is skipped. + if self._durable(state) != durable_before: + self._save_health_state(plugin_id, state) + def record_failure(self, plugin_id: str, error: Optional[Exception] = None) -> None: """Record a failed plugin execution.""" state = self.get_health_state(plugin_id) diff --git a/src/plugin_system/plugin_loader.py b/src/plugin_system/plugin_loader.py index bd748ef7..d8ae0e3d 100644 --- a/src/plugin_system/plugin_loader.py +++ b/src/plugin_system/plugin_loader.py @@ -14,7 +14,7 @@ import sys import subprocess import threading from pathlib import Path -from typing import Dict, Any, Optional, Tuple, Type +from typing import Dict, Any, List, Optional, Tuple, Type import logging from packaging.requirements import InvalidRequirement, Requirement @@ -45,6 +45,76 @@ def requirements_has_real_deps(requirements_file: str) -> bool: return False +def _extra_dependencies(dist_name: str, extras) -> Optional[List[Requirement]]: + """Dependencies a distribution declares *only* behind the given extras. + + Returns None when the installed metadata cannot be read or parsed, so the + caller can fall back to running pip rather than assuming anything. + """ + try: + meta = importlib.metadata.metadata(dist_name) + except importlib.metadata.PackageNotFoundError: + return None + + gated: List[Requirement] = [] + for raw in meta.get_all('Requires-Dist') or []: + try: + dep = Requirement(raw) + except InvalidRequirement: + return None + if dep.marker is None: + continue + # Keep only what the distribution gates behind an extra we asked for: + # satisfied when `extra` is that name, but not when no extra is + # requested. A marker that holds either way (python_version, sys_platform) + # belongs to the base install and is already covered by the version check. + if dep.marker.evaluate({'extra': ''}): + continue + if any(dep.marker.evaluate({'extra': extra}) for extra in extras): + gated.append(dep) + return gated + + +def _extras_are_satisfied(req: Requirement, _visited: Optional[set] = None) -> bool: + """Check the dependencies pulled in by req's extras are installed. + + Follows extras through nested extras. A gated dependency can itself request + one (`requests[socks]`), and checking only that `requests` is installed at + an acceptable version says nothing about whether the socks extra's own + dependency is there -- so the caller would skip pip and the plugin would + fail at import instead. Plain dependencies are still checked one level + deep, which is all that is needed to tell "the extra was installed" from + "the extra was never installed". + + `_visited` carries the (distribution, extras) pairs already seen, so a + dependency cycle between extras terminates instead of recursing forever. + Anything unreadable returns False, so the caller still falls through to pip. + """ + if _visited is None: + _visited = set() + marker = (req.name.lower(), frozenset(e.lower() for e in req.extras)) + if marker in _visited: + # Already accounted for higher up the chain; treating a cycle as + # satisfied here is safe because the outer frame still has to pass. + return True + _visited.add(marker) + + gated = _extra_dependencies(req.name, req.extras) + if gated is None: + return False + + for dep in gated: + try: + dep_version = importlib.metadata.version(dep.name) + except importlib.metadata.PackageNotFoundError: + return False + if dep.specifier and not dep.specifier.contains(dep_version, prereleases=True): + return False + if dep.extras and not _extras_are_satisfied(dep, _visited): + return False + return True + + def requirements_are_satisfied(requirements_file: str) -> bool: """ Check whether every real requirement line in requirements.txt is already @@ -76,9 +146,6 @@ def requirements_are_satisfied(requirements_file: str) -> bool: except InvalidRequirement: return False - if req.extras: - return False # verifying extras' sub-dependencies isn't worth it here - if req.marker is not None and not req.marker.evaluate(): continue # not applicable on this platform/interpreter @@ -90,6 +157,9 @@ def requirements_are_satisfied(requirements_file: str) -> bool: if req.specifier and not req.specifier.contains(installed_version, prereleases=True): return False + if req.extras and not _extras_are_satisfied(req): + return False + return True diff --git a/src/plugin_system/plugin_manager.py b/src/plugin_system/plugin_manager.py index 50d6419e..0dc7a428 100644 --- a/src/plugin_system/plugin_manager.py +++ b/src/plugin_system/plugin_manager.py @@ -71,11 +71,15 @@ class PluginManager: self.plugin_loader = PluginLoader(logger=self.logger) self.plugin_executor = PluginExecutor(default_timeout=30.0, logger=self.logger) self.state_manager = PluginStateManager(logger=self.logger) - self.schema_manager = SchemaManager(plugins_dir=self.plugins_dir, logger=self.logger) + self.schema_manager = SchemaManager(plugins_dir=self.plugins_dir, logger=self.logger, + config_manager=self.config_manager) # Lock protecting plugin_manifests and plugin_directories from # concurrent mutation (background reconciliation) and reads (requests). self._discovery_lock = threading.RLock() + #: Directories already reported as unloadable, so the warning is + #: emitted once rather than on every discovery scan. + self._skip_reported: set = set() # Lock protecting plugin_last_update from concurrent mutation/iteration. # It's written from run_scheduled_updates()/update_all_plugins() (main @@ -113,9 +117,26 @@ class PluginManager: self._update_queue: "queue.Queue[Optional[Tuple[str, float]]]" = queue.Queue() self._pending_updates: set = set() self._pending_lock = threading.Lock() + # Serializes the "is this plugin eligible?" -> "claim it (RUNNING)" + # transition. Two schedulers run concurrently in practice — the render + # loop's _tick_plugin_updates() and Vegas mode's vegas-plugin-tick + # daemon thread, which is never joined — so without this both can + # observe ENABLED and both call update() on the same plugin. Held only + # across the check and the state transition, never across update() + # itself: that would serialize slow plugins behind each other and + # reintroduce the stall the async worker exists to avoid. + self._reservation_lock = threading.Lock() self._plugin_locks: Dict[str, threading.Lock] = {} self._plugin_locks_guard = threading.Lock() self._update_worker: Optional[threading.Thread] = None + # Plugin ids whose update() has finished since the last time anyone + # asked. Updates are dispatched to a worker thread, so a caller that + # wants to know "whose data just changed" cannot learn it by diffing + # plugin_last_update around run_scheduled_updates() -- that call only + # enqueues, and the timestamp is stamped later, on the worker. See + # run_scheduled_updates_with_changes(). + self._completed_updates: set = set() + self._completed_updates_lock = threading.Lock() self._synchronous_updates = False if self.config_manager is not None: try: @@ -178,18 +199,59 @@ class PluginManager: continue manifest_path = item / "manifest.json" - if manifest_path.exists(): - try: - with open(manifest_path, 'r', encoding='utf-8') as f: - manifest = json.load(f) - plugin_id = manifest.get('id') - if plugin_id: - plugin_ids.append(plugin_id) - new_manifests[plugin_id] = manifest - new_directories[plugin_id] = item - except (json.JSONDecodeError, PermissionError, OSError) as e: - self.logger.warning("Error reading manifest from %s: %s", manifest_path, e, exc_info=True) - continue + if not manifest_path.exists(): + # Once per directory per process. Discovery runs on every + # web UI page load and every config reconcile, so warning + # unconditionally would put a line in the journal each + # time someone opened a page -- the same log-volume + # problem this is meant to help diagnose. + # A directory here that carries no manifest is not a + # plugin. Said once, because the alternative is a plugin + # that is enabled in config, enabled in plugin state, + # present on disk, and simply absent from the running + # process with nothing anywhere to say why. Working that + # out afterwards means reading cache-file mtimes. + if item.name not in self._skip_reported: + self._skip_reported.add(item.name) + self.logger.warning( + "Skipping %s: no manifest.json, so it cannot be " + "loaded as a plugin", item.name) + continue + try: + with open(manifest_path, 'r', encoding='utf-8') as f: + manifest = json.load(f) + except (json.JSONDecodeError, PermissionError, OSError) as e: + self.logger.warning("Error reading manifest from %s: %s", manifest_path, e, exc_info=True) + continue + + # json.load accepts any JSON value, so a manifest holding + # null, [] or "text" parses and then raises AttributeError on + # .get(). Nothing here catches that -- the outer handler takes + # OSError/PermissionError only -- so a single malformed + # manifest aborted the whole scan and every other plugin on + # disk, however healthy, silently failed to register. + if not isinstance(manifest, dict): + if item.name not in self._skip_reported: + self._skip_reported.add(item.name) + self.logger.warning( + "Skipping %s: its manifest.json is %s, not a JSON " + "object", item.name, type(manifest).__name__) + continue + + plugin_id = manifest.get('id') + if not plugin_id: + # Parsed but unusable. This was the quietest path of all: + # the manifest is read successfully and then dropped. + if item.name not in self._skip_reported: + self._skip_reported.add(item.name) + self.logger.warning( + "Skipping %s: its manifest.json has no \"id\", so " + "there is nothing to register it under", item.name) + continue + + plugin_ids.append(plugin_id) + new_manifests[plugin_id] = manifest + new_directories[plugin_id] = item except (OSError, PermissionError) as e: self.logger.error("Error scanning directory %s: %s", directory, e, exc_info=True) @@ -569,6 +631,17 @@ class PluginManager: return self.load_plugin(plugin_id) + def discovered_plugin_ids(self) -> set: + """Snapshot of the discovered plugin ids, taken under the discovery lock. + + Callers on other threads (the config watcher) must not iterate + ``plugin_manifests`` directly: discovery rebuilds it entry by entry, so + an unsynchronised reader can see a half-populated mapping or raise + "dictionary changed size during iteration". + """ + with self._discovery_lock: + return set(self.plugin_manifests) + def get_plugin(self, plugin_id: str) -> Optional[Any]: """ Get a loaded plugin instance by ID. @@ -808,25 +881,69 @@ class PluginManager: if self.health_tracker and self.health_tracker.should_skip_plugin(plugin_id): continue - # Check if plugin can execute - if not self.state_manager.can_execute(plugin_id): - continue - interval = self._get_plugin_update_interval(plugin_id, plugin_instance) if interval is None: continue - with self._plugin_last_update_lock: - last_update = self.plugin_last_update.get(plugin_id, 0.0) + # Eligibility check, due check and the RUNNING transition happen + # together, so a concurrent scheduler cannot claim the same plugin. + if not self._reserve_for_update(plugin_id, current_time, interval): + continue - if last_update == 0.0 or (current_time - last_update) >= interval: - if self._synchronous_updates: - # Kill-switch path: the original inline execution - # (blocks the caller until update() completes/times out) - self.state_manager.set_state(plugin_id, PluginState.RUNNING) - self._execute_update_now(plugin_id, plugin_instance, current_time) - else: - self._enqueue_update(plugin_id, current_time) + if self._synchronous_updates: + # Kill-switch path: the original inline execution + # (blocks the caller until update() completes/times out) + self._execute_update_now(plugin_id, plugin_instance, current_time) + else: + self._enqueue_update(plugin_id, current_time) + + def _reserve_for_update( + self, + plugin_id: str, + current_time: Optional[float] = None, + interval: Optional[float] = None, + ) -> bool: + """Atomically claim a plugin for update, returning True if we won it. + + can_execute() and the RUNNING transition have to happen under one lock. + As two separate calls, two scheduler threads can both see ENABLED and + both go on to run the same plugin's update() concurrently — unsafe for + any plugin that isn't reentrant (shared mutable state, a non-thread-safe + HTTP session or cache). + + The due-time check is inside the lock too. Leaving it outside would let + a second thread that had already decided "due" claim the plugin the + instant the first finished, running update() twice in one interval. + + Args: + plugin_id: Plugin to claim. + current_time: Now, for the due check. Omit to skip that check. + interval: Seconds between updates. Omit to skip the due check. + + Returns: + True if this caller reserved the plugin and must dispatch it, + False if it is ineligible, not yet due, or already claimed. + """ + with self._reservation_lock: + if not self.state_manager.can_execute(plugin_id): + return False + + if current_time is not None and interval is not None: + with self._plugin_last_update_lock: + last_update = self.plugin_last_update.get(plugin_id, 0.0) + if last_update != 0.0 and (current_time - last_update) < interval: + return False + + self.state_manager.set_state(plugin_id, PluginState.RUNNING) + return True + + def _release_reservation(self, plugin_id: str) -> None: + """Hand a claimed plugin back when it never got dispatched. + + Without this a plugin reserved but not queued would sit in RUNNING + forever, and can_execute() would refuse it on every later tick. + """ + self.state_manager.set_state(plugin_id, PluginState.ENABLED) def get_plugin_lock(self, plugin_id: str) -> threading.Lock: """Per-plugin lock keeping update() and display() mutually exclusive. @@ -843,16 +960,40 @@ class PluginManager: return lock def _enqueue_update(self, plugin_id: str, scheduled_time: float) -> None: - """Queue a due update for the background worker (dedup while pending).""" + """Queue an already-reserved update for the background worker. + + The caller has reserved the plugin (RUNNING), which is what blocks + re-entry and shows the truthful state in the web UI while the item + waits its turn. The pending set stays as a second line of defence; if + it ever fires the reservation has to be handed back, or the plugin + would sit in RUNNING with nothing queued to release it. + """ with self._pending_lock: if plugin_id in self._pending_updates: + self.logger.warning( + "Plugin %s reserved for update but already queued; " + "releasing the reservation", plugin_id) + self._release_reservation(plugin_id) return self._pending_updates.add(plugin_id) - # RUNNING is set at enqueue time so can_execute() blocks re-entry and - # the web UI shows the truthful state while the item waits its turn. - self.state_manager.set_state(plugin_id, PluginState.RUNNING) - self._ensure_update_worker() - self._update_queue.put((plugin_id, scheduled_time)) + try: + self._ensure_update_worker() + self._update_queue.put((plugin_id, scheduled_time)) + except Exception as exc: # pylint: disable=broad-except + # Thread.start() raises RuntimeError when the OS refuses a new + # thread — a real condition on a Pi under memory pressure. Nothing + # is queued to release the plugin at that point, so the claim has to + # be undone here, or it sits in RUNNING with nothing to clear it and + # can_execute() refuses it for the rest of the process. Swallowed + # rather than raised so the remaining plugins in this tick still get + # their turn. + self.logger.error( + "Could not queue update for plugin %s (%s: %s); releasing the " + "reservation so the next tick can retry", + plugin_id, type(exc).__name__, exc, exc_info=True) + with self._pending_lock: + self._pending_updates.discard(plugin_id) + self._release_reservation(plugin_id) def _ensure_update_worker(self) -> None: if self._update_worker is not None and self._update_worker.is_alive(): @@ -937,9 +1078,18 @@ class PluginManager: return finished['done'] = True try: + # Drop the queue reservation *before* the state goes back to + # ENABLED. The other order leaves a window where a scheduler + # sees ENABLED, reserves the plugin, then finds it still in + # _pending_updates -- the enqueue is dropped and the plugin + # would sit in RUNNING with nothing left to release it. + if lock is not None: + with self._pending_lock: + self._pending_updates.discard(plugin_id) if success: with self._plugin_last_update_lock: self.plugin_last_update[plugin_id] = scheduled_time + self._note_update_completed(plugin_id) self.state_manager.record_update(plugin_id) self.state_manager.set_state(plugin_id, PluginState.ENABLED) if self.health_tracker: @@ -949,8 +1099,6 @@ class PluginManager: finally: if lock is not None: lock.release() - with self._pending_lock: - self._pending_updates.discard(plugin_id) if lock is None: # Synchronous / no-lock path: unchanged behavior. @@ -1006,28 +1154,41 @@ class PluginManager: def run_scheduled_updates_with_changes(self, current_time: Optional[float] = None) -> List[str]: """ - Like run_scheduled_updates(), but also returns the plugin_ids whose - plugin_last_update timestamp actually advanced during this call. + Like run_scheduled_updates(), but also reports which plugins have + fresh data -- the ids whose update() has finished since the last + call, not necessarily the ones enqueued by this one. - The before/after snapshots and the update pass itself are each - individually lock-protected against concurrent plugin_last_update - mutation (Vegas mode calls this from its own background - update-tick thread, racing the main render loop's plugin updates), - so callers get an atomic "who got fresh data" answer without - reaching into plugin_last_update themselves. The lock is not held - across the update pass so slow/blocking plugin update() calls don't - serialize against other plugin_last_update readers. + That distinction is the whole point. This used to snapshot + plugin_last_update, call run_scheduled_updates(), and diff. But + run_scheduled_updates() only *enqueues*: the work runs on the + update worker and the timestamp is stamped there, after this method + has already returned. The two snapshots were therefore always + identical and the result was always empty, so Vegas never learned + that any plugin's data had changed and kept scrolling whatever a + segment was first built from -- last night's live game still drawn + as live the next morning. The only path that ever worked was the + synchronous kill-switch, where update() runs inline. + + Reporting completions instead of enqueues costs a poll's worth of + latency (the Vegas tick runs every ~4s) and is correct regardless of + which side of the queue the work lands on. """ - with self._plugin_last_update_lock: - old_times = dict(self.plugin_last_update) - self.run_scheduled_updates(current_time) + return self.drain_completed_updates() - with self._plugin_last_update_lock: - return [ - plugin_id for plugin_id, new_time in self.plugin_last_update.items() - if new_time > old_times.get(plugin_id, 0.0) - ] + def _note_update_completed(self, plugin_id: str) -> None: + """Record that a plugin's update() finished, for the next poll.""" + with self._completed_updates_lock: + self._completed_updates.add(plugin_id) + + def drain_completed_updates(self) -> List[str]: + """Return and clear the plugin ids whose update() has since finished.""" + with self._completed_updates_lock: + if not self._completed_updates: + return [] + done = sorted(self._completed_updates) + self._completed_updates.clear() + return done def update_all_plugins(self) -> None: """ @@ -1041,18 +1202,18 @@ class PluginManager: if not hasattr(plugin_instance, "update"): continue - # Check if plugin can execute - if not self.state_manager.can_execute(plugin_id): + # Eligibility check and the RUNNING transition together, so a + # concurrent scheduler cannot claim the same plugin (see + # _reserve_for_update). + if not self._reserve_for_update(plugin_id): continue - - # Update state to RUNNING - self.state_manager.set_state(plugin_id, PluginState.RUNNING) - + try: success = self.plugin_executor.execute_update(plugin_instance, plugin_id) if success: with self._plugin_last_update_lock: self.plugin_last_update[plugin_id] = time.time() + self._note_update_completed(plugin_id) self.state_manager.record_update(plugin_id) self.state_manager.set_state(plugin_id, PluginState.ENABLED) else: diff --git a/src/plugin_system/plugin_state.py b/src/plugin_system/plugin_state.py index 269bb423..f9b9d7d0 100644 --- a/src/plugin_system/plugin_state.py +++ b/src/plugin_system/plugin_state.py @@ -6,14 +6,40 @@ with state transitions and queries. """ import threading +import time +from collections import deque from enum import Enum -from typing import Optional, Dict, Any +from typing import Optional, Dict, Any, Deque, List, Tuple from datetime import datetime import logging from src.logging_config import get_logger +# The history is diagnostic only -- nothing reads the entries themselves, just +# their count -- but it is appended to on the hot scheduling path: every update +# cycle records RUNNING on reserve and ENABLED on finish. Unbounded, that is +# 2,880 entries per plugin per day at the default 60s interval, which on a 1 GB +# Pi exhausts memory in weeks. +# +# Two limits, because a single entry count answers the wrong question. What a +# reader wants is "the last couple of hours", and how many transitions that is +# depends entirely on the plugin's update interval -- which on a real board +# spans 2s to 3600s. A flat 200 entries is 4.2 days for the slowest plugin and +# 3.3 minutes for the fastest, so the plugin churning hardest, the one worth +# looking at, keeps the least history. +# +# So: trim by AGE first, which makes the retained window comparable across +# plugins whatever their cadence... +STATE_HISTORY_MAX_AGE_SECONDS = 2 * 60 * 60 + +# ...and cap by COUNT second, purely as a memory ceiling for the fast pollers +# whose age window would otherwise run to thousands of entries. At ~230 bytes +# an entry this is ~0.5 MB per plugin worst case, and only plugins updating +# faster than roughly every 4s can reach it. +MAX_STATE_HISTORY_PER_PLUGIN = 2000 + + class PluginState(Enum): """Plugin state enumeration.""" UNLOADED = "unloaded" # Plugin not loaded @@ -37,11 +63,43 @@ class PluginStateManager: self.logger = logger or get_logger(__name__) self._lock = threading.RLock() self._states: Dict[str, PluginState] = {} - self._state_history: Dict[str, list] = {} + # (monotonic timestamp, transition). The clock is monotonic so a DST + # shift or an NTP step cannot make entries look old and flush the + # history; the human-readable timestamp lives inside the transition. + self._state_history: Dict[str, Deque[Tuple[float, Dict[str, Any]]]] = {} + # Lifetime transition totals, kept separately so the count reported by + # get_state_info() stays truthful once the history above starts rolling. + self._state_transition_counts: Dict[str, int] = {} self._error_info: Dict[str, Dict[str, Any]] = {} self._last_update: Dict[str, datetime] = {} self._last_display: Dict[str, datetime] = {} + def _record_transition( + self, + plugin_id: str, + transition: Dict[str, Any] + ) -> None: + """Append a transition to the plugin's bounded history. + + Callers must already hold ``_lock``. The deque discards its oldest + entry once it is full, so the history cannot grow without bound; the + lifetime total is tracked separately for get_state_info(). + """ + history = self._state_history.get(plugin_id) + if history is None: + history = deque(maxlen=MAX_STATE_HISTORY_PER_PLUGIN) + self._state_history[plugin_id] = history + now = time.monotonic() + history.append((now, transition)) + # Age out first; the deque's maxlen is the backstop for plugins that + # produce more than the ceiling within the window. + cutoff = now - STATE_HISTORY_MAX_AGE_SECONDS + while history and history[0][0] < cutoff: + history.popleft() + self._state_transition_counts[plugin_id] = ( + self._state_transition_counts.get(plugin_id, 0) + 1 + ) + def set_state( self, plugin_id: str, @@ -60,16 +118,13 @@ class PluginStateManager: old_state = self._states.get(plugin_id, PluginState.UNLOADED) self._states[plugin_id] = state - if plugin_id not in self._state_history: - self._state_history[plugin_id] = [] - transition = { 'timestamp': datetime.now(), 'from': old_state.value, 'to': state.value, 'error': str(error) if error else None } - self._state_history[plugin_id].append(transition) + self._record_transition(plugin_id, transition) # Store error info if transitioning to ERROR state if state == PluginState.ERROR and error: @@ -126,17 +181,29 @@ class PluginStateManager: state = self.get_state(plugin_id) return state == PluginState.ENABLED - def get_state_history(self, plugin_id: str) -> list: + def get_state_history(self, plugin_id: str) -> List[Dict[str, Any]]: """ Get state transition history for a plugin. - + + Retention is by age first -- transitions older than + STATE_HISTORY_MAX_AGE_SECONDS are dropped -- and by count second, at + MAX_STATE_HISTORY_PER_PLUGIN, which only binds for plugins updating + fast enough to exceed it inside that window. + Args: plugin_id: Plugin identifier - + Returns: - List of state transitions + List of recent state transitions, oldest first. Both the list and + the transition dicts are copies, so callers cannot mutate the + manager's own history. The values inside a transition are all + immutable, so a shallow copy per entry is enough. """ - return self._state_history.get(plugin_id, []) + with self._lock: + return [ + dict(transition) + for _stamp, transition in self._state_history.get(plugin_id, ()) + ] def set_error_info(self, plugin_id: str, error_info: Dict[str, Any]) -> None: """ @@ -179,9 +246,7 @@ class PluginStateManager: old_state = self._states.get(plugin_id, PluginState.UNLOADED) self._states[plugin_id] = state - if plugin_id not in self._state_history: - self._state_history[plugin_id] = [] - self._state_history[plugin_id].append({ + self._record_transition(plugin_id, { 'timestamp': datetime.now(), 'from': old_state.value, 'to': state.value, @@ -241,26 +306,40 @@ class PluginStateManager: Returns: Dictionary with state information """ - state = self.get_state(plugin_id) - info = { - 'state': state.value, - 'is_loaded': self.is_loaded(plugin_id), - 'is_enabled': self.is_enabled(plugin_id), - 'is_running': self.is_running(plugin_id), - 'is_error': self.is_error(plugin_id), - 'can_execute': self.can_execute(plugin_id), - 'last_update': self.get_last_update(plugin_id), - 'last_display': self.get_last_display(plugin_id), - 'error_info': self.get_error_info(plugin_id), - 'state_history_count': len(self.get_state_history(plugin_id)) - } + # One snapshot, one critical section. Each field was read under its own + # lock, so an unload running concurrently could be observed half-done: + # 'state' read before clear_state() removed it and + # 'state_history_count' read after, giving a caller a plugin that is + # ENABLED with zero transitions. _lock is an RLock, so the helpers + # below can still take it. + with self._lock: + state = self.get_state(plugin_id) + info = { + 'state': state.value, + 'is_loaded': self.is_loaded(plugin_id), + 'is_enabled': self.is_enabled(plugin_id), + 'is_running': self.is_running(plugin_id), + 'is_error': self.is_error(plugin_id), + 'can_execute': self.can_execute(plugin_id), + 'last_update': self.get_last_update(plugin_id), + 'last_display': self.get_last_display(plugin_id), + 'error_info': self.get_error_info(plugin_id), + 'state_history_count': self._state_transition_counts.get(plugin_id, 0) + } return info def clear_state(self, plugin_id: str) -> None: - """Clear all state information for a plugin.""" - self._states.pop(plugin_id, None) - self._state_history.pop(plugin_id, None) - self._error_info.pop(plugin_id, None) - self._last_update.pop(plugin_id, None) - self._last_display.pop(plugin_id, None) + """Clear all state information for a plugin. + + Held under ``_lock`` so the five dicts are dropped as one unit: every + other mutator takes the lock, and without it a concurrent set_state() + could interleave and leave a plugin with history but no state. + """ + with self._lock: + self._states.pop(plugin_id, None) + self._state_history.pop(plugin_id, None) + self._state_transition_counts.pop(plugin_id, None) + self._error_info.pop(plugin_id, None) + self._last_update.pop(plugin_id, None) + self._last_display.pop(plugin_id, None) diff --git a/src/plugin_system/resource_monitor.py b/src/plugin_system/resource_monitor.py index 42ded4ec..1b63d74e 100644 --- a/src/plugin_system/resource_monitor.py +++ b/src/plugin_system/resource_monitor.py @@ -9,7 +9,7 @@ import time import logging import threading from typing import Dict, Optional, Any, Callable -from dataclasses import dataclass, field +from dataclasses import dataclass, field, fields try: import psutil @@ -49,6 +49,20 @@ class ResourceMetrics: self.total_execution_time = self.total_execution_time / self.call_count +#: How often a plugin's metrics are written to the cache, in seconds. +#: +#: Persisting on every call meant a small file rewritten roughly nine times a +#: minute per plugin. On a rig with fourteen active plugins that was ~126 +#: writes a minute for metrics alone, and since each ~350-byte file costs a +#: 4KB block plus an ext4 journal entry, it dominated the device's write +#: volume -- on an SD card, which wears out. +#: +#: The in-memory copy stays authoritative and exact; only the cross-process +#: snapshot the web UI reads is delayed, and telemetry up to half a minute old +#: is still a fair description of a long-running plugin. +_METRICS_PERSIST_INTERVAL = 30.0 + + class PluginResourceMonitor: """ Monitors resource usage for plugins. @@ -75,6 +89,10 @@ class PluginResourceMonitor: # Resource metrics per plugin self._metrics: Dict[str, ResourceMetrics] = {} self._limits: Dict[str, ResourceLimits] = {} + # When each plugin's metrics last reached the cache. Metrics change on + # every call, so they cannot be de-duplicated the way health state can; + # they are rate-limited instead. See _METRICS_PERSIST_INTERVAL. + self._metrics_persisted_at: Dict[str, float] = {} # Thread-local storage for execution tracking self._local = threading.local() @@ -102,6 +120,66 @@ class PluginResourceMonitor: "psutil not available - resource monitoring will be limited to execution time only" ) + def _metrics_from_cache(self, plugin_id: str, cached: Any) -> "ResourceMetrics": + """Build metrics from a cached record, ignoring anything unrecognised. + + ResourceMetrics(**cached) raises TypeError on a single unexpected key, + and that exception escapes into plugin_manager, which reports it as + "plugin operation failed". Every plugin fails, and the plugin + system never finishes initialising. + + Seen on a live rig: every plugin failing with + + ResourceMetrics.__init__() got an unexpected keyword argument + 'consecutive_failures' + + which is a plugin_health field, not a metrics one. How a health-shaped + record came to sit under a plugin_metrics key on that machine is not + established -- a restored backup that mixed two machines' caches is the + likeliest explanation -- but the loader should not be brittle enough for + it to matter. plugin_health already repairs its records field by field + rather than trusting whatever is on disk; this does the same. + + Unknown keys are dropped and named once, so a genuine schema change is + visible in the log instead of silently discarded. + """ + if not isinstance(cached, dict): + self.logger.warning( + "Ignoring cached metrics for %s: expected a mapping, got %s", + plugin_id, type(cached).__name__) + return ResourceMetrics() + + known = {f.name for f in fields(ResourceMetrics)} + unknown = sorted(set(cached) - known) + if unknown: + self.logger.warning( + "Dropping unrecognised field(s) from cached metrics for %s: %s", + plugin_id, ", ".join(unknown)) + # A dataclass does not enforce its annotations, so + # ResourceMetrics(call_count="not a number") builds happily and only + # blows up later, deep inside monitor_call ("can only concatenate str + # (not \"int\") to str"). Coerce here, where there is still a cache + # key to name in the warning. + declared = {f.name: f.type for f in fields(ResourceMetrics)} + usable = {} + for key, value in cached.items(): + if key not in known: + continue + try: + usable[key] = int(value) if declared[key] in ('int', int) else float(value) + except (TypeError, ValueError): + self.logger.warning( + "Cached metrics for %s have a bad %s (%r); starting fresh", + plugin_id, key, value) + return ResourceMetrics() + try: + return ResourceMetrics(**usable) + except (TypeError, ValueError) as e: + self.logger.warning( + "Cached metrics for %s unusable (%s); starting fresh", + plugin_id, e) + return ResourceMetrics() + def _get_metrics_key(self, plugin_id: str) -> str: """Get cache key for plugin metrics.""" return f"plugin_metrics:{plugin_id}" @@ -126,7 +204,7 @@ class PluginResourceMonitor: cache_key, max_age=None, memory_ttl=0 if force_reload else None ) if cached: - metrics = ResourceMetrics(**cached) + metrics = self._metrics_from_cache(plugin_id, cached) else: metrics = ResourceMetrics() self._metrics[plugin_id] = metrics @@ -232,18 +310,8 @@ class PluginResourceMonitor: # CPU is harder to measure per-call, so we track it separately metrics.cpu_percent = self._get_process_cpu_percent() - # Persist metrics - cache_key = self._get_metrics_key(plugin_id) - self.cache_manager.set(cache_key, { - 'memory_mb': metrics.memory_mb, - 'cpu_percent': metrics.cpu_percent, - 'execution_time': metrics.execution_time, - 'call_count': metrics.call_count, - 'total_execution_time': metrics.total_execution_time, - 'max_execution_time': metrics.max_execution_time, - 'min_execution_time': metrics.min_execution_time if metrics.min_execution_time != float('inf') else 0.0, - 'last_update_time': metrics.last_update_time - }) + # Persist metrics, at most once per interval per plugin. + self._persist_metrics(plugin_id, metrics) # Check limits if limits: @@ -363,6 +431,44 @@ class PluginResourceMonitor: summaries[plugin_id] = self.get_metrics_summary(plugin_id) return summaries + def _persist_metrics(self, plugin_id: str, metrics: ResourceMetrics, + force: bool = False) -> None: + """Write a plugin's metrics to the cache, at most once per interval. + + Caller must hold ``self._lock``. + """ + # Monotonic, not wall clock: these devices have no RTC, so the clock + # jumps by however far off boot-time was the moment NTP first syncs. + # A forward jump would allow an early write, a backward one would + # stall the snapshot well past the interval. + # + # The sentinel for "never written" is None, not 0.0. monotonic() is + # time since boot on Linux, and systemd starts this service *at* boot, + # so `now - 0.0 < 30` was true for the first half-minute of every + # single run -- the throttle swallowed the very first snapshot, which + # is the one that matters most after a restart. + now = time.monotonic() + last_written = self._metrics_persisted_at.get(plugin_id) + if (not force and last_written is not None + and now - last_written < _METRICS_PERSIST_INTERVAL): + return + cache_key = self._get_metrics_key(plugin_id) + self.cache_manager.set(cache_key, { + 'memory_mb': metrics.memory_mb, + 'cpu_percent': metrics.cpu_percent, + 'execution_time': metrics.execution_time, + 'call_count': metrics.call_count, + 'total_execution_time': metrics.total_execution_time, + 'max_execution_time': metrics.max_execution_time, + 'min_execution_time': (metrics.min_execution_time + if metrics.min_execution_time != float('inf') + else 0.0), + 'last_update_time': metrics.last_update_time, + }) + # Only after the write lands. Marking it first would mean a failed + # set() bought the next interval's silence without leaving a snapshot. + self._metrics_persisted_at[plugin_id] = now + def reset_metrics(self, plugin_id: str) -> None: """Reset metrics for a plugin.""" with self._lock: @@ -370,4 +476,7 @@ class PluginResourceMonitor: self._metrics[plugin_id] = ResourceMetrics() cache_key = self._get_metrics_key(plugin_id) self.cache_manager.delete(cache_key) + # Let the next call persist immediately rather than leaving the + # deleted key absent for the rest of the interval. + self._metrics_persisted_at.pop(plugin_id, None) diff --git a/src/plugin_system/saved_repositories.py b/src/plugin_system/saved_repositories.py index bd0e1872..c8da3c5b 100644 --- a/src/plugin_system/saved_repositories.py +++ b/src/plugin_system/saved_repositories.py @@ -6,6 +6,7 @@ Manages saved GitHub repository URLs for easy plugin discovery and installation. import json import logging +import os from pathlib import Path from typing import List, Dict, Optional @@ -43,20 +44,45 @@ class SavedRepositoriesManager: return [] def _save_repositories(self) -> bool: - """Save repositories to file.""" + """Save repositories to file atomically. + + Writes to a temp file in the same directory and os.replace()s it + over the target, so a failed write can never truncate or + half-overwrite an existing saved_repositories.json. + """ + tmp_path = self.config_path.with_suffix(self.config_path.suffix + '.tmp') try: # Ensure directory exists self.config_path.parent.mkdir(parents=True, exist_ok=True) - - with open(self.config_path, 'w') as f: + + with open(tmp_path, 'w') as f: json.dump(self.repositories, f, indent=2) - + f.flush() + os.fsync(f.fileno()) + os.replace(tmp_path, self.config_path) + self.logger.info(f"Saved {len(self.repositories)} repositories to {self.config_path}") return True except Exception as e: self.logger.error(f"Error saving repositories: {e}") + try: + tmp_path.unlink(missing_ok=True) + except OSError: + pass return False + @staticmethod + def _clean_url(repo_url: str) -> str: + """Normalize a repo URL: strip whitespace, trailing slashes, and a + trailing ``.git`` suffix ONLY. (The old ``.replace('.git', '')`` + was an unanchored substring replace that mangled URLs merely + containing ``.git``, e.g. ``https://github.com/user/my.github.io``.) + """ + repo_url = repo_url.strip().rstrip('/') + if repo_url.endswith('.git'): + repo_url = repo_url[:-4] + return repo_url + def get_all(self) -> List[Dict[str, str]]: """Get all saved repositories.""" return self.repositories.copy() @@ -72,15 +98,14 @@ class SavedRepositoriesManager: Returns: True if added successfully """ - # Clean URL - repo_url = repo_url.strip().rstrip('/').replace('.git', '') - + repo_url = self._clean_url(repo_url) + # Check if already exists for repo in self.repositories: if repo.get('url') == repo_url: self.logger.warning(f"Repository already exists: {repo_url}") return False - + # Extract name from URL if not provided if not name: parts = repo_url.split('/') @@ -88,15 +113,20 @@ class SavedRepositoriesManager: name = parts[-1] else: name = repo_url - + # Add repository self.repositories.append({ 'url': repo_url, 'name': name, 'type': 'registry' if 'plugins.json' in repo_url or 'ledmatrix-plugins' in repo_url.lower() else 'single' }) - - return self._save_repositories() + + if not self._save_repositories(): + # Keep memory consistent with disk: a failed save must not leave + # a phantom entry that only this process can see. + self.repositories.pop() + return False + return True def remove(self, repo_url: str) -> bool: """ @@ -108,21 +138,25 @@ class SavedRepositoriesManager: Returns: True if removed successfully """ - # Clean URL - repo_url = repo_url.strip().rstrip('/').replace('.git', '') - - original_count = len(self.repositories) - self.repositories = [r for r in self.repositories if r.get('url') != repo_url] - - if len(self.repositories) < original_count: - return self._save_repositories() + repo_url = self._clean_url(repo_url) + + previous = self.repositories + remaining = [r for r in previous if r.get('url') != repo_url] + + if len(remaining) < len(previous): + self.repositories = remaining + if not self._save_repositories(): + # Failed save: restore so memory matches disk. + self.repositories = previous + return False + return True else: self.logger.warning(f"Repository not found: {repo_url}") return False def has(self, repo_url: str) -> bool: """Check if a repository is already saved.""" - repo_url = repo_url.strip().rstrip('/').replace('.git', '') + repo_url = self._clean_url(repo_url) return any(r.get('url') == repo_url for r in self.repositories) def get_registry_repositories(self) -> List[Dict[str, str]]: diff --git a/src/plugin_system/schema_manager.py b/src/plugin_system/schema_manager.py index 5b894100..3db4a5a2 100644 --- a/src/plugin_system/schema_manager.py +++ b/src/plugin_system/schema_manager.py @@ -26,7 +26,25 @@ class SchemaManager: - Cache invalidation on plugin changes """ - def __init__(self, plugins_dir: Optional[Path] = None, project_root: Optional[Path] = None, logger: Optional[logging.Logger] = None): + # Plugin config keys that mean "where this device is". A plugin declaring + # any of these in its schema gets the device-wide ``location`` block from + # config.json as the *default* for that field, instead of whatever city the + # plugin author happened to ship. A value the user set on the plugin itself + # always wins -- this only ever replaces the schema default, so an explicit + # per-plugin location is still honoured. + # + # Only these fully-namespaced keys are substituted. A bare ``state`` or + # ``city`` key is deliberately left alone: plugins use those for unrelated + # things (ledmatrix-elections' ``state`` is a two-letter code, not a place + # name), and silently rewriting them would break those plugins. + DEVICE_LOCATION_KEYS: Dict[str, str] = { + 'location_city': 'city', + 'location_state': 'state', + 'location_country': 'country', + } + + def __init__(self, plugins_dir: Optional[Path] = None, project_root: Optional[Path] = None, + logger: Optional[logging.Logger] = None, config_manager: Optional[Any] = None): """ Initialize the Schema Manager. @@ -34,10 +52,14 @@ class SchemaManager: plugins_dir: Base plugins directory path project_root: Project root directory path logger: Optional logger instance + config_manager: Optional config manager, used to resolve the + device-wide ``location`` that seeds plugin location defaults. + Omitting it simply leaves schema defaults untouched. """ self.logger = logger or logging.getLogger(__name__) self.plugins_dir = plugins_dir self.project_root = project_root or Path.cwd() + self.config_manager = config_manager # Schema cache: plugin_id -> schema dict self._schema_cache: Dict[str, Dict[str, Any]] = {} @@ -212,10 +234,70 @@ class SchemaManager: return defaults + def get_device_location(self) -> Optional[Dict[str, Any]]: + """ + Return the device-wide ``location`` block from config.json, or None. + + This is the City/State/Country the user sets once under General + settings. Returns None when there is no config manager wired, the + config can't be read, or no location has been configured. + """ + if self.config_manager is None: + return None + try: + config = self.config_manager.load_config() + except Exception as e: + # A config that can't be read must never stop defaults being + # generated -- the plugin's own schema defaults still apply. + self.logger.debug(f"Could not read device location from config: {e}") + return None + if not isinstance(config, dict): + return None + location = config.get('location') + return location if isinstance(location, dict) else None + + def apply_device_location(self, defaults: Dict[str, Any]) -> Dict[str, Any]: + """ + Replace location-shaped schema defaults with the device's own location. + + Without this, a plugin that ships ``"location_city": "Dallas"`` as its + schema default silently reports Dallas weather (and centres its radar + there) for every user who never opened that plugin's config form -- + even though they set their real city under General settings. The + substituted value is still only a *default*: ``merge_with_defaults`` + lets any per-plugin value the user saved win over it. + + Mutates and returns ``defaults`` for convenience. + """ + if not defaults: + return defaults + if not any(key in defaults for key in self.DEVICE_LOCATION_KEYS): + return defaults + + location = self.get_device_location() + if not location: + return defaults + + for key, field in self.DEVICE_LOCATION_KEYS.items(): + if key not in defaults: + continue + value = location.get(field) + # Only a non-empty string is a real answer; a blank or missing + # field means "not configured", which leaves the schema default. + if isinstance(value, str) and value.strip(): + defaults[key] = value.strip() + + return defaults + def generate_default_config(self, plugin_id: str, use_cache: bool = True) -> Dict[str, Any]: """ Generate default configuration for a plugin from its schema. + Location fields (see ``DEVICE_LOCATION_KEYS``) default to the device's + configured location rather than the plugin author's. That substitution + is applied on the way out rather than being cached, so changing the + device location takes effect without invalidating the defaults cache. + Args: plugin_id: Plugin identifier use_cache: If True, return cached defaults if available @@ -225,7 +307,7 @@ class SchemaManager: """ # Check cache first if use_cache and plugin_id in self._defaults_cache: - return self._defaults_cache[plugin_id].copy() + return self.apply_device_location(self._defaults_cache[plugin_id].copy()) schema = self.load_schema(plugin_id, use_cache=use_cache) if not schema: @@ -249,10 +331,11 @@ class SchemaManager: if 'live_priority' not in defaults: defaults['live_priority'] = schema.get('properties', {}).get('live_priority', {}).get('default', False) - # Cache the defaults + # Cache the defaults *before* the device location is layered on, so a + # later change to the device location is picked up by the next call. self._defaults_cache[plugin_id] = defaults.copy() - return defaults + return self.apply_device_location(defaults) def validate_config_against_schema(self, config: Dict[str, Any], schema: Dict[str, Any], plugin_id: Optional[str] = None) -> Tuple[bool, List[str]]: diff --git a/src/plugin_system/store_manager.py b/src/plugin_system/store_manager.py index e13b3e5b..623270e3 100644 --- a/src/plugin_system/store_manager.py +++ b/src/plugin_system/store_manager.py @@ -1143,7 +1143,7 @@ class PluginStoreManager: """ registry = self.fetch_registry() plugins = registry.get('plugins', []) or [] - plugin_info = next((p for p in plugins if p['id'] == plugin_id), None) + plugin_info = self._match_registry_entry(plugins, plugin_id) if not plugin_info: return None @@ -1183,6 +1183,37 @@ class PluginStoreManager: return plugin_info + @staticmethod + def _match_registry_entry(plugins: List[Dict], plugin_id: str) -> Optional[Dict]: + """Find a registry entry by its id, or by the directory it installs to. + + Four shipped plugins have a registry ``id`` that differs from the ``id`` + in their own manifest: ``weather`` installs to ``plugins/ledmatrix-weather``, + and likewise stocks, music and leaderboard. Installation already prefers + the manifest id for the directory name, so on disk, in ``config.json`` + and in a backup manifest those plugins are called ``ledmatrix-weather``. + + Only the registry calls them ``weather``, and nothing resolved that in + reverse: restoring a backup asked the store for ``ledmatrix-weather`` + and got "Plugin not found in registry", silently dropping four enabled + plugins from a restored device. + + Matching ``plugin_path`` fixes it without renaming any published id, + which would orphan ``plugin_state.json`` entries keyed on the old ones. + Exact id always wins, so an entry whose *path* happens to collide with + another entry's id cannot shadow it. + """ + if not plugin_id: + return None + exact = next((p for p in plugins if p.get('id') == plugin_id), None) + if exact is not None: + return exact + for entry in plugins: + path = (entry.get('plugin_path') or '').rstrip('/') + if path and path.rsplit('/', 1)[-1] == plugin_id: + return entry + return None + def get_registry_info(self, plugin_id: str) -> Optional[Dict]: """ Get plugin information from the registry cache only (no GitHub API calls). @@ -1198,7 +1229,7 @@ class PluginStoreManager: """ registry = self.fetch_registry() plugins = registry.get('plugins', []) or [] - return next((p for p in plugins if p.get('id') == plugin_id), None) + return self._match_registry_entry(plugins, plugin_id) def install_plugin(self, plugin_id: str, branch: Optional[str] = None) -> bool: """Install a plugin, keeping any existing install until the new one is @@ -2552,6 +2583,85 @@ class PluginStoreManager: self.logger.error(f"Error uninstalling plugin {plugin_id}: {e}") return False + def _gate_pulled_commit(self, plugin_id: str, plugin_path: Path, + previous_sha: Optional[str]) -> bool: + """Apply the compatibility gate to a commit that arrived via git pull. + + Every other route into an installed plugin goes through + ``install_plugin``, which gates in ``_install_plugin_impl``. This one + did not: a ``git pull`` could deliver a manifest flooring above this + core and nothing would notice until the plugin failed to load, which + surfaces as one line in the journal and a scoreboard that silently + stopped appearing. + + Checked after the pull rather than before it, for the same reason + ``_install_plugin_impl`` checks after the download: the registry + carries no compatibility field, so the incoming floor is only knowable + once the new commit is on disk. + + Undone with ``git reset --hard`` rather than by removing the directory. + This is a live checkout, the previous commit is still in the object + store, and the reset leaves the user on the exact version they were + already running -- the same promise ``_reinstall_with_rollback`` makes, + reached by the means this path actually has. It is also the gentler + option: no window in which the plugin directory does not exist, and no + ``.standalone-backup-`` debris if the process dies mid-way. + + A manifest that cannot be read is not evidence of incompatibility, so + it allows. ``compatibility.check`` refuses only on evidence for the + same reason: a wrong refusal breaks a working install, while a wrong + allowance degrades to exactly the behaviour this path had before the + gate existed. + """ + manifest_path = plugin_path / "manifest.json" + try: + with open(manifest_path, 'r', encoding='utf-8') as mf: + manifest = json.load(mf) + except (OSError, ValueError) as e: + self.logger.warning( + "Could not read %s after updating %s (%s); allowing the " + "update, as an unreadable manifest declares no floor", + manifest_path, plugin_id, e) + return True + + from src import __version__ as core_version + from src.plugin_system import compatibility + + compatible, reason = compatibility.check(manifest, core_version) + if compatible: + return True + + self.logger.error("Refusing the update to %s: %s", plugin_id, reason) + + if not previous_sha: + self.logger.error( + "Cannot roll %s back: the commit it was on before the pull is " + "unknown. It is now on a version this core cannot run — " + "reinstall it from the plugin store.", plugin_id) + return False + + # Safe by construction: update_plugin returns before pulling unless the + # tree was clean or successfully stashed, so there are no uncommitted + # tracked edits for --hard to discard. The stash is not popped on the + # success path either, so the reset leaves the working tree exactly + # where a successful pull would have. Say "commit", not "changes". + reset = subprocess.run( + ['git', '-C', str(plugin_path), 'reset', '--hard', previous_sha], + capture_output=True, text=True, timeout=60, check=False) + if reset.returncode != 0: + self.logger.error( + "CRITICAL: could not roll %s back to commit %s: %s. It is left " + "on a version this core cannot run; " + "`git -C %s reset --hard %s` restores it.", + plugin_id, previous_sha[:7], + (reset.stderr or reset.stdout or '').strip(), + plugin_path, previous_sha) + else: + self.logger.info( + "Rolled %s back to commit %s; it stays on the version it was " + "already running.", plugin_id, previous_sha[:7]) + return False + def _reinstall_with_rollback(self, plugin_id: str, plugin_path: Path) -> bool: """Replace an installed plugin with a fresh install, atomically. @@ -2829,6 +2939,8 @@ class PluginStoreManager: status_result = type('obj', (object,), {'stdout': '', 'stderr': 'Status check timed out'})() stash_info = "" + # Whether the pull can be undone without destroying work. + tree_is_recoverable = not has_changes if has_changes: self.logger.info(f"Stashing local changes in {plugin_id} before update") try: @@ -2842,12 +2954,37 @@ class PluginStoreManager: ) if stash_result.returncode == 0: stash_info = " (local changes were stashed)" + tree_is_recoverable = True self.logger.info(f"Stashed local changes (including untracked files) for {plugin_id}") else: self.logger.warning(f"Failed to stash local changes for {plugin_id}: {stash_result.stderr}") except subprocess.TimeoutExpired: self.logger.warning(f"Stash operation timed out for {plugin_id}, proceeding with pull") + # Do not pull what cannot be un-pulled. + # + # The compatibility gate below can refuse the commit this + # pull brings down, and its only way back is `git reset + # --hard`, which discards uncommitted tracked edits. Those + # edits are exactly what the stash above exists to protect, + # so a stash that failed or timed out leaves the rollback + # unable to run without destroying them. + # + # A pull does not necessarily refuse on a dirty tree -- git + # merges happily as long as the incoming commit touches + # different files -- so without this the update would + # succeed, the gate would refuse, and the reset would take + # the user's work with it. Refusing here costs an update in + # a case that already went wrong; the alternative costs + # data. + if not tree_is_recoverable: + self.logger.error( + "Refusing to update %s: it has local changes that could " + "not be stashed, and an incompatible update could then " + "only be rolled back by discarding them. Commit or stash " + "them by hand, then update.", plugin_id) + return False + # Pull from the determined remote branch self.logger.info(f"Pulling from origin/{remote_pull_branch} for {plugin_id}...") pull_result = subprocess.run( @@ -2870,6 +3007,14 @@ class PluginStoreManager: elif updated_sha: self.logger.info(f"Plugin {plugin_id} updated to commit {updated_sha[:7]}{stash_info}") + # The install gate, at the only point on this path where + # it can be answered. Every other route in goes through + # install_plugin, which gates in _install_plugin_impl; this + # one did not, so a pull could deliver a manifest flooring + # above this core and nothing would notice. + if not self._gate_pulled_commit(plugin_id, plugin_path, local_sha): + return False + self._install_dependencies(plugin_path) return True @@ -2969,7 +3114,10 @@ class PluginStoreManager: remote_branch = plugin_info_remote.get('branch') or plugin_info_remote.get('default_branch') # Compare local manifest version against registry latest_version - # to avoid unnecessary reinstalls for monorepo plugins + # to avoid unnecessary reinstalls for monorepo plugins. Uses the + # same semantic comparator as the web UI's update badge, so + # equivalent spellings ("v1.2.0" vs "1.2.0") never trigger a + # reinstall and a locally-ahead version is never downgraded. try: local_manifest_path = plugin_path / "manifest.json" if local_manifest_path.exists(): @@ -2977,8 +3125,16 @@ class PluginStoreManager: local_manifest = json.load(f) local_version = local_manifest.get('version', '') remote_version = plugin_info_remote.get('latest_version', '') - if local_version and remote_version and local_version == remote_version: - self.logger.info(f"Plugin {plugin_id} already at latest version {local_version}") + from src.plugin_system.compatibility import is_update_available + # No truthiness gate: the shared comparator already treats + # a missing version on either side as "no update", and the + # store must agree with the UI badge in that case too. A + # missing manifest (not just a missing version field) + # still falls through to the reinstall recovery path. + if not is_update_available(local_version, remote_version): + self.logger.info( + f"Plugin {plugin_id} already at latest version " + f"(installed {local_version}, registry {remote_version})") return True except Exception as e: self.logger.debug(f"Could not compare versions for {plugin_id}: {e}") diff --git a/src/plugin_system/testing/harness.py b/src/plugin_system/testing/harness.py index ee4d6613..51c688a2 100644 --- a/src/plugin_system/testing/harness.py +++ b/src/plugin_system/testing/harness.py @@ -73,6 +73,11 @@ class RenderResult: golden_ok: Optional[bool] = None golden_diff_pixels: int = 0 golden_max_delta: int = 0 + # what display() handed back; the controller skips a mode only on False + display_returned: Any = None + # empty-frame check: rendered nothing while not reporting "no content" + empty_claimed: Optional[bool] = None # True when that happened + empty_ok: Optional[bool] = None # False only in strict mode # fill / scale-up check (populated only for sizes >= 2x the design size) fill_checked: bool = False fill_ok: Optional[bool] = None # False only in strict mode @@ -92,6 +97,8 @@ class RenderResult: return False if self.fill_ok is False: return False + if self.empty_ok is False: + return False return True @@ -132,21 +139,25 @@ def _instantiate(plugin_id: str, manifest: Dict[str, Any], plugin_dir: Path, return plugin_instance -def _render_mode(plugin_instance: Any, mode: str) -> None: +def _render_mode(plugin_instance: Any, mode: str) -> Any: """Render a specific screen. Prefer an explicit display_mode kwarg; otherwise drive the plugin's internal mode state machine (first display() call renders - modes[current_mode_index] when current_display_mode is None).""" + modes[current_mode_index] when current_display_mode is None). + + Returns whatever display() returned. The display controller skips a mode + whose display() returns False, so that value decides whether an empty mode + is rotated past or sat on -- which makes it worth reporting rather than + discarding.""" sig = inspect.signature(plugin_instance.display) if "display_mode" in sig.parameters: - plugin_instance.display(force_clear=True, display_mode=mode) - return + return plugin_instance.display(force_clear=True, display_mode=mode) modes = getattr(plugin_instance, "modes", None) if modes and mode in modes: plugin_instance.current_mode_index = list(modes).index(mode) if hasattr(plugin_instance, "current_display_mode"): plugin_instance.current_display_mode = None - plugin_instance.display(force_clear=False) + return plugin_instance.display(force_clear=False) def _freeze(freeze_time: Optional[str]): @@ -234,7 +245,7 @@ def _render_size(plugin_id, manifest, plugin_dir, config, mock_data, logger.warning("update() raised a non-connectivity error for %s [%s]: %s", plugin_id, mode, e) if result.error is None: - _render_mode(inst, mode) + result.display_returned = _render_mode(inst, mode) result.image = dm.get_image() result.overflow = dm.check_overflow() except Exception as e: # noqa: BLE001 — a display crash is a real failure @@ -341,6 +352,44 @@ def fill_metrics(image: Image.Image) -> Tuple[float, float, float]: return (extent_x, extent_y, ink) +def check_empty_claimed(results: List[RenderResult], + strict: bool = False) -> List[RenderResult]: + """Flag a mode that rendered nothing without reporting "no content". + + The display controller skips a mode whose ``display()`` returns False, and + treats anything else -- including None -- as "content was shown". A mode + that draws nothing and does not return False therefore holds whatever is on + the panel for its whole display duration. Since a mode switch clears first, + that is a blank screen. Two sports plugins shipped exactly this: their + ``display()`` returned None on every path, so an out-of-season league sat + blank for its full duration rather than being rotated past. + + Warn-only by default, because a blank frame is not automatically wrong: a + scroll mode whose first frame is its blank scroll-in buffer renders empty + and is behaving correctly. ``strict=True`` sets ``empty_claimed`` such that + ``RenderResult.ok`` fails -- opt in per plugin via harness.json + ``{"empty_check": "strict"}`` once its modes are known to draw on the + fixture data. + + Note this can only catch what the fixtures actually render. A plugin whose + harness fixture seeds content never exercises its empty path here; the + source-level gate in the plugins repo covers that case. + """ + for r in results: + if r.image is None or r.error is not None: + continue + # An explicit False is the plugin correctly saying "nothing to show". + if r.display_returned is False: + continue + if r.image.convert("L").point( + lambda p: 255 if p > _LIT_THRESHOLD else 0).getbbox() is not None: + continue + r.empty_claimed = True + if strict: + r.empty_ok = False + return results + + def check_scale_up(results: List[RenderResult], design_size: Tuple[int, int] = (128, 32), min_extent: float = _MIN_FILL_EXTENT, diff --git a/src/plugin_system/testing/plugin_test_base.py b/src/plugin_system/testing/plugin_test_base.py index f873ea9e..806f3d3f 100644 --- a/src/plugin_system/testing/plugin_test_base.py +++ b/src/plugin_system/testing/plugin_test_base.py @@ -2,6 +2,11 @@ Base test class for LEDMatrix plugins. Provides common fixtures and helper methods for plugin testing. + +Note: this is the plugin-author-facing base class shipped with the +core (importable as src.plugin_system.testing.plugin_test_base). The +repo's own plugin tests use a separate, richer harness in +test/plugins/test_plugin_base.py — the two are intentionally distinct. """ import unittest diff --git a/src/plugin_system/testing/visual_display_manager.py b/src/plugin_system/testing/visual_display_manager.py index e3dc2ec6..5211a226 100644 --- a/src/plugin_system/testing/visual_display_manager.py +++ b/src/plugin_system/testing/visual_display_manager.py @@ -10,6 +10,17 @@ without requiring hardware or the RGBMatrixEmulator. Used for: Unlike MockDisplayManager (which logs calls but doesn't render) or MagicMock (which tracks nothing visual), this class creates a real PIL Image canvas and draws text using the actual project fonts. + +MAINTENANCE WARNING: this class is a deliberate fork of +src/display_manager.py so it can run without hardware. It mirrors +these DisplayManager methods by name and behavior: _load_fonts, +_draw_bdf_text, get_font_height, get_text_width, draw_text, +draw_text_with_icons, draw_weather_icon (and the _draw_sun/_draw_cloud/ +_draw_rain/_draw_snow/_draw_storm family), format_date_with_ordinal, +capture_mode, set_scrolling_state, is_currently_scrolling, +process_deferred_updates, update_display, render_size. A behavior +change to any of those in DisplayManager must be mirrored here, or +plugin visual tests will pass against stale behavior. """ import math diff --git a/src/startup_validator.py b/src/startup_validator.py index 86c703f6..79275dc2 100644 --- a/src/startup_validator.py +++ b/src/startup_validator.py @@ -15,16 +15,23 @@ from src.logging_config import get_logger class StartupValidator: """Validates system state on startup.""" - def __init__(self, config_manager: Any, plugin_manager: Optional[Any] = None) -> None: + def __init__(self, config_manager: Any, plugin_manager: Optional[Any] = None, + cache_manager: Optional[Any] = None) -> None: """ Initialize the startup validator. - + Args: config_manager: ConfigManager instance plugin_manager: Optional PluginManager instance + cache_manager: The CacheManager the application will actually use. + Pass it. Without one this validator builds its own just to read + a directory path, which reports on a cache the app does not + use and leaves behind a cleanup thread that nothing stops -- + validation runs twice per startup, so that was two of them. """ self.config_manager = config_manager self.plugin_manager = plugin_manager + self.cache_manager = cache_manager self.logger = get_logger(__name__) self.errors: List[str] = [] self.warnings: List[str] = [] @@ -37,7 +44,12 @@ class StartupValidator: Tuple of (is_valid, errors, warnings) """ self.logger.info("Starting startup validation...") - + + # Fresh lists each run — without this, calling validate_all() twice + # duplicated every message. + self.errors = [] + self.warnings = [] + # Validate configuration self._validate_config() @@ -50,6 +62,9 @@ class StartupValidator: # Validate plugins if plugin manager is available if self.plugin_manager: self._validate_plugins() + + # Warn when the running systemd unit no longer matches the repo's + self._validate_systemd_units() is_valid = len(self.errors) == 0 @@ -62,6 +77,80 @@ class StartupValidator: return (is_valid, self.errors.copy(), self.warnings.copy()) + #: Units this project installs, and where each is installed to. + _UNITS = ( + ("systemd/ledmatrix.service", "/etc/systemd/system/ledmatrix.service"), + ("systemd/ledmatrix-web.service", "/etc/systemd/system/ledmatrix-web.service"), + ) + + def _validate_systemd_units(self) -> None: + """Warn when an installed unit has drifted from the repo's template. + + Nothing re-applies these after the first install. `git pull` -- which is + what the web UI's update button runs -- brings a new template into the + checkout, but nothing copies it to /etc/systemd/system and nothing runs + `systemctl daemon-reload`, so the unit that actually runs is whatever + first_time_install.sh wrote on day one. + + That makes every hardening added to a unit inert on existing installs. + Measured on one rig: the installed unit was thirteen days older than the + repo's and differed in content, so a MemoryMax the repo had specified + was not being enforced at all -- `systemctl show` reported + MemoryMax=infinity. + + A warning rather than an error, and certainly not a silent rewrite: + editing files under /etc and restarting services is the installer's job, + not something a display process should do to a machine while it boots. + The remedy is to re-run scripts/install/install_service.sh. + """ + try: + project_root = Path(__file__).resolve().parent.parent + for template_rel, installed_path in self._UNITS: + template = project_root / template_rel + installed = Path(installed_path) + if not template.is_file() or not installed.is_file(): + continue + + # The template carries placeholders the installer substitutes, + # so compare the substituted form rather than the raw file. + expected = template.read_text(encoding="utf-8") + expected = expected.replace("__PROJECT_ROOT_DIR__", str(project_root)) + expected = expected.replace("__USER__", "root") + + try: + actual = installed.read_text(encoding="utf-8") + except PermissionError: + continue + + if self._unit_body(expected) != self._unit_body(actual): + self.warnings.append( + f"{installed.name} differs from {template_rel}; the " + "installed unit is not refreshed by an update, so " + "settings added to the template are not in effect. " + "Re-run scripts/install/install_service.sh to apply them." + ) + except OSError as e: + self.logger.debug("Could not compare systemd units: %s", e) + + @staticmethod + def _unit_body(text: str) -> str: + """A unit's meaningful lines, in order: no comments, no blanks. + + Order is preserved deliberately. This used to sort, which made the + comparison insensitive to two changes that matter in a systemd unit: + repeated directives such as ExecStartPre= and ExecStartPost= run in + the order they appear, and a directive that moves between [Unit], + [Service] and [Install] means something different -- or nothing -- + where it lands. A drift check that normalises those away reports no + drift for a unit that has genuinely changed. + """ + lines = [] + for line in text.splitlines(): + line = line.strip() + if line and not line.startswith("#"): + lines.append(line) + return "\n".join(lines) + def _validate_config(self) -> None: """Validate configuration files.""" try: @@ -86,9 +175,21 @@ class StartupValidator: def _validate_cache_directory(self) -> None: """Validate cache directory permissions.""" try: - from src.cache_manager import CacheManager - cache_manager = CacheManager() - cache_dir = cache_manager.get_cache_dir() + cache_manager = self.cache_manager + if cache_manager is None: + # No caller supplied one (older embedders, direct use in a + # script). Build one, but do not leave its cleanup thread + # running behind us -- this instance is discarded on the next + # line but the thread is a closure over it, so it would never + # be collected. + from src.cache_manager import CacheManager + cache_manager = CacheManager() + try: + cache_dir = cache_manager.get_cache_dir() + finally: + cache_manager.stop_cleanup_thread() + else: + cache_dir = cache_manager.get_cache_dir() if not cache_dir: self.warnings.append("Cache directory not available - caching will be disabled") diff --git a/src/vegas_mode/config.py b/src/vegas_mode/config.py index 14786481..f1039f5b 100644 --- a/src/vegas_mode/config.py +++ b/src/vegas_mode/config.py @@ -104,15 +104,53 @@ class VegasModeConfig: overflow_mode: str = "rotate" # Cap on one plugin's share of a cycle, as a multiple of display width. - # A single ticker returning 7,000px would otherwise hold the panel for over - # two minutes. Overflow is deferred to later cycles rather than discarded. - # 0 disables the cap. - max_plugin_width_ratio: float = 3.0 + # 0 (the default) disables the cap, so every plugin contributes all of its + # content and is always entered at its beginning. + # + # Capping was the default until it proved to cost more than it bought. + # Measured over a 17-plugin fleet on a 512px panel, only four plugins were + # ever wide enough to hit a 3.0 cap; for those four it produced two visible + # faults. Content resumed mid-item on each appearance (a news ticker entered + # at column 6027 of its own strip), and the final window of a rotation was + # whatever happened to be left — 348px of a 1840px stocks ticker, seven + # seconds of panel time. Both read as the display being broken rather than + # as deferral working. + # + # A wide plugin does hold the panel for a long time uncapped: set the cap + # per plugin with vegas_max_width_screens where that matters, rather than + # globally where it mostly hurts plugins that were never the problem. + max_plugin_width_ratio: float = 0.0 # Plugin management plugin_order: List[str] = field(default_factory=list) excluded_plugins: Set[str] = field(default_factory=set) + # --- Live content in the ticker ------------------------------------- + # + # By default a live game preempts Vegas entirely: the display controller + # refuses to run the ticker while any plugin reports live priority, and you + # get the full-screen scoreboard instead. Set live_in_ticker to keep the + # marquee running and let live content take extra turns within it. + # + # The rotation is otherwise a strict round robin -- every plugin appears + # exactly once per cycle -- so with a dozen plugins enabled a live score + # comes round once a lap and can be minutes old on screen. Weighting lets a + # plugin claim several slots per cycle instead. + # + # Weights are per plugin, not per game: a scoreboard showing four live + # games still occupies one slot at a time, and rotates its own games within + # that slot using its own favorite_live_boost. + live_in_ticker: bool = False + + # Slots per cycle for a plugin reporting live content. 1 disables the boost + # and restores the plain round robin. + live_weight: int = 3 + + # Slots per cycle for a plugin whose live content involves a favorite team. + # Only plugins implementing get_vegas_priority_weight() can claim this -- + # the core cannot tell whose game is on, so the plugin reports it. + favorite_live_weight: int = 5 + # Performance settings target_fps: int = 125 # Target frame rate buffer_ahead: int = 2 # Number of plugins to buffer ahead @@ -159,10 +197,16 @@ class VegasModeConfig: lead_in_width=int(vegas_config.get('lead_in_width', 0)), plugins_per_cycle=int(vegas_config.get('plugins_per_cycle', 6)), max_plugin_width_ratio=float( - vegas_config.get('max_plugin_width_ratio', 3.0)), + vegas_config.get('max_plugin_width_ratio', 0.0)), overflow_mode=str(vegas_config.get('overflow_mode', 'rotate')), plugin_order=list(vegas_config.get('plugin_order', [])), excluded_plugins=set(vegas_config.get('excluded_plugins', [])), + live_in_ticker=bool(vegas_config.get('live_in_ticker', False)), + # Clamped: a weight below 1 would drop the plugin from the rotation + # entirely, and a very large one starves everything else. + live_weight=max(1, min(10, int(vegas_config.get('live_weight', 3)))), + favorite_live_weight=max( + 1, min(10, int(vegas_config.get('favorite_live_weight', 5)))), target_fps=int(vegas_config.get('target_fps', 125)), buffer_ahead=int(vegas_config.get('buffer_ahead', 2)), frame_based_scrolling=vegas_config.get('frame_based_scrolling', True), @@ -192,6 +236,9 @@ class VegasModeConfig: 'lead_in_width': self.lead_in_width, 'plugins_per_cycle': self.plugins_per_cycle, 'max_plugin_width_ratio': self.max_plugin_width_ratio, + 'live_in_ticker': self.live_in_ticker, + 'live_weight': self.live_weight, + 'favorite_live_weight': self.favorite_live_weight, 'overflow_mode': self.overflow_mode, 'plugin_order': self.plugin_order, 'excluded_plugins': list(self.excluded_plugins), @@ -359,6 +406,15 @@ class VegasModeConfig: if 'enabled' in vegas_config: self.enabled = vegas_config['enabled'] + if 'live_in_ticker' in vegas_config: + self.live_in_ticker = bool(vegas_config['live_in_ticker']) + # Clamped exactly as from_config does: a weight below 1 would drop the + # plugin from the rotation, and a huge one starves everything else. + if 'live_weight' in vegas_config: + self.live_weight = max(1, min(10, int(vegas_config['live_weight']))) + if 'favorite_live_weight' in vegas_config: + self.favorite_live_weight = max( + 1, min(10, int(vegas_config['favorite_live_weight']))) if 'scroll_speed' in vegas_config: self.scroll_speed = float(vegas_config['scroll_speed']) if 'separator_width' in vegas_config: diff --git a/src/vegas_mode/coordinator.py b/src/vegas_mode/coordinator.py index e75e8c51..b9af5ea6 100644 --- a/src/vegas_mode/coordinator.py +++ b/src/vegas_mode/coordinator.py @@ -12,6 +12,7 @@ Supports three display modes per plugin: """ import logging +import math import time import threading from typing import Optional, Dict, Any, List, Callable, TYPE_CHECKING @@ -30,6 +31,33 @@ if TYPE_CHECKING: logger = logging.getLogger(__name__) +#: Degradation threshold, as a fraction of target_fps. A marquee jitters a +#: little all the time, so "anything under target" would report constantly and +#: mean nothing; 90% of target is the point where a shortfall is real. At a +#: 60fps target that is 54fps -- 55fps is a normal wobble and stays at DEBUG, +#: which is deliberate, not an off-by-one. +_FPS_HEALTHY_FRACTION = 0.9 + +#: A healthy marquee still reports this often, so silence means stopped +#: rather than fine. +_FPS_HEARTBEAT_INTERVAL = 300.0 + + +def _percentile(ordered: List[float], fraction: float) -> float: + """Nearest-rank percentile of an already-sorted list. + + Index ceil(n * fraction) - 1, so 100 samples at 0.99 give the 99th-ranked + value. The obvious int(n * fraction) is off by one and, at exactly 100 + samples, lands on the maximum -- which is the number already reported + alongside this one as the worst frame, so the two columns would agree + precisely when the sample was smallest. + """ + if not ordered: + return 0.0 + index = math.ceil(len(ordered) * fraction) - 1 + return ordered[min(len(ordered) - 1, max(0, index))] + + class VegasModeCoordinator: """ Orchestrates Vegas scroll mode operation. @@ -80,6 +108,11 @@ class VegasModeCoordinator: self._is_active = False self._is_paused = False self._should_stop = False + # Frame-rate health, tracked across run_iteration() calls so the + # heartbeat is one-per-interval rather than one-per-cycle, and so a + # recovery spanning two cycles is still reported. Reset on start(). + self._fps_last_health_log = 0.0 + self._fps_was_degraded = False self._state_lock = threading.Lock() # Live priority tracking @@ -232,6 +265,11 @@ class VegasModeCoordinator: self._is_active = True self._should_stop = False self._start_time = time.time() + # A fresh run starts with a clean health slate: no stale + # "was degraded" from the previous run, and a heartbeat that is + # due immediately so the first sample confirms the marquee is up. + self._fps_last_health_log = 0.0 + self._fps_was_degraded = False # Line up the next group immediately, so the first extension is already # warm rather than stalling the scroll to fetch it. @@ -379,14 +417,36 @@ class VegasModeCoordinator: duration = self.render_pipeline.get_dynamic_duration() start_time = time.time() frame_count = 0 - fps_log_interval = 5.0 # Log FPS every 5 seconds - last_fps_log_time = start_time + fps_log_interval = 5.0 # Sample FPS every 5 seconds + # Health state lives on the coordinator, not here: run_iteration() is + # called once per cycle, so locals reset every few seconds. That made + # `last_fps_health_log = 0.0` fire the "heartbeat" on the first sample + # of every iteration rather than once per interval, and a recovery + # that crossed an iteration boundary was never reported at all -- + # was_degraded had already gone back to False. + # Monotonic, and deliberately not start_time: start_time is wall + # clock and is used below to report the iteration's duration. Mixing + # the two here would make every delta hugely negative and silence the + # frame-rate reporting altogether. + last_fps_log_time = time.monotonic() fps_frame_count = 0 + # A mean hides stutter completely. At 120fps a five-second window is + # ~600 frames, so a 200ms freeze -- plainly visible on a marquee -- + # moves the average from 120.0 to 115.4 and reads as healthy. What a + # viewer actually notices is the worst frame, so track that too. + frame_worst = 0.0 + frame_times: List[float] = [] logger.info("Starting Vegas iteration for %.1fs", duration) while True: - frame_started = time.time() + # Monotonic, like the FPS window below. These devices have no RTC, + # so the wall clock jumps by however wrong boot time was the moment + # NTP first syncs. A backward jump makes frame_elapsed negative, + # and `frame_interval - frame_elapsed` then sleeps for longer than + # the whole budget -- the render loop stalls for the size of the + # correction. A forward jump inflates p99 and worst-frame instead. + frame_started = time.monotonic() # Check for STATIC mode plugin that should pause scroll static_plugin = self._check_static_plugin_trigger() @@ -414,23 +474,58 @@ class VegasModeCoordinator: # quarter of the budget spent not rendering. Subtracting the work # already done keeps the pacing target while reclaiming that time, # and yields the GIL either way so other threads still run. - frame_elapsed = time.time() - frame_started + frame_elapsed = time.monotonic() - frame_started time.sleep(max(0.0, frame_interval - frame_elapsed)) + # Measured before the sleep: time spent working, not pacing. + if frame_elapsed > frame_worst: + frame_worst = frame_elapsed + frame_times.append(frame_elapsed) + # Increment frame count and check for interrupt periodically frame_count += 1 fps_frame_count += 1 - # Periodic FPS logging - current_time = time.time() + # Periodic FPS logging. Reported at INFO only when the frame rate + # is actually worth an operator's attention -- a shortfall against + # target, or the recovery from one -- with a slow heartbeat so a + # healthy marquee still shows a pulse. + # + # Measured over two hours on a running rig: 1410 samples, 98.5% + # of them within 10% of target. The 1.5% that were not included a + # reading of 8.6fps against a target of 60 -- a real stall, and + # completely invisible inside 1389 lines reading "59.6". + # Monotonic: every use of this value in the block below is a + # duration, and these devices have no RTC, so the wall clock jumps + # by however wrong boot time was the moment NTP first syncs. That + # would not only mis-fire the heartbeat, it would corrupt the + # frame rate itself, since fps is frames divided by this delta. + current_time = time.monotonic() if current_time - last_fps_log_time >= fps_log_interval: fps = fps_frame_count / (current_time - last_fps_log_time) - logger.info( - "Vegas FPS: %.1f (target: %d, frames: %d)", - fps, self.vegas_config.target_fps, fps_frame_count - ) + p99 = _percentile(sorted(frame_times), 0.99) + target = self.vegas_config.target_fps + degraded = target > 0 and fps < target * _FPS_HEALTHY_FRACTION + due = (current_time - self._fps_last_health_log + >= _FPS_HEARTBEAT_INTERVAL) + if degraded or self._fps_was_degraded or due: + logger.info( + "Vegas FPS: %.1f (target: %d, frames: %d) p99 %.1fms worst %.1fms", + fps, target, fps_frame_count, + p99 * 1000.0, frame_worst * 1000.0 + ) + self._fps_last_health_log = current_time + else: + logger.debug( + "Vegas FPS: %.1f (target: %d, frames: %d) p99 %.1fms worst %.1fms", + fps, target, fps_frame_count, + p99 * 1000.0, frame_worst * 1000.0 + ) + self._fps_was_degraded = degraded last_fps_log_time = current_time fps_frame_count = 0 + frame_worst = 0.0 + frame_times.clear() if (self._interrupt_check and frame_count % self._interrupt_check_interval == 0): @@ -497,6 +592,12 @@ class VegasModeCoordinator: if not self._live_priority_check: return False + if self.vegas_config.live_in_ticker: + # The ticker keeps live content rather than yielding to it; the + # extra turns are arranged in the rotation itself, so there is + # nothing to pause for. + return False + try: live_mode = self._live_priority_check() if live_mode: diff --git a/src/vegas_mode/plugin_adapter.py b/src/vegas_mode/plugin_adapter.py index 0da51ce2..01ef8cca 100644 --- a/src/vegas_mode/plugin_adapter.py +++ b/src/vegas_mode/plugin_adapter.py @@ -68,7 +68,22 @@ class PluginAdapter: # always the same opening items. self._item_offsets: dict = {} - logger.info( + # What the matching entry in _item_offsets is an offset *into*, as + # (kind, size). An offset only means anything against the content it + # was derived from, and there are three incompatible kinds: + # + # ('rows', n) index into a list of n images + # ('cuts', n) index into the n item boundaries of one image + # ('cols', w) pixel column in a w-wide image with no item boundaries + # + # Without this the offsets were reused across kinds — a plugin that + # returned one wide image on one fetch and several rows on the next had + # a pixel column of 1400 read back as a row index — and across content + # changes, where a column recorded against a 9,793px news strip pointed + # into unrelated headlines once the strip refreshed to 9,505px. + self._offset_shapes: dict = {} + + logger.debug( "PluginAdapter initialized: display=%dx%d", self.display_width, self.display_height ) @@ -94,7 +109,7 @@ class PluginAdapter: Returns: List of PIL Images representing plugin content, or None if no content """ - logger.info( + logger.debug( "[%s] Getting content (class=%s)", plugin_id, plugin.__class__.__name__ ) @@ -103,7 +118,7 @@ class PluginAdapter: cached = self._get_cached(plugin_id) if cached is not None: total_width = sum(img.width for img in cached) - logger.info( + logger.debug( "[%s] Using cached content: %d images, %dpx total", plugin_id, len(cached), total_width ) @@ -111,46 +126,46 @@ class PluginAdapter: # Try native Vegas content method first has_native = hasattr(plugin, 'get_vegas_content') - logger.info("[%s] Has get_vegas_content: %s", plugin_id, has_native) + logger.debug("[%s] Has get_vegas_content: %s", plugin_id, has_native) if has_native: content = self._get_native_content(plugin, plugin_id, offscreen_only) if content: total_width = sum(img.width for img in content) - logger.info( + logger.debug( "[%s] Native content SUCCESS: %d images, %dpx total", plugin_id, len(content), total_width ) return self._finalize(content, plugin_id, 'native', plugin) - logger.info("[%s] Native content returned None", plugin_id) + logger.debug("[%s] Native content returned None", plugin_id) # Try to get scroll_helper's cached image (for scrolling plugins like stocks/odds) has_scroll_helper = hasattr(plugin, 'scroll_helper') - logger.info("[%s] Has scroll_helper: %s", plugin_id, has_scroll_helper) + logger.debug("[%s] Has scroll_helper: %s", plugin_id, has_scroll_helper) content = self._get_scroll_helper_content(plugin, plugin_id, offscreen_only) if content: total_width = sum(img.width for img in content) - logger.info( + logger.debug( "[%s] ScrollHelper content SUCCESS: %d images, %dpx total", plugin_id, len(content), total_width ) return self._finalize(content, plugin_id, 'scroll_helper', plugin) if has_scroll_helper: - logger.info("[%s] ScrollHelper content returned None", plugin_id) + logger.debug("[%s] ScrollHelper content returned None", plugin_id) if offscreen_only: # Display capture needs the shared canvas; leave it to the caller. - logger.info( + logger.debug( "[%s] Needs display capture, deferring to the render thread", plugin_id ) return None # Fall back to display capture - logger.info("[%s] Trying fallback display capture...", plugin_id) + logger.debug("[%s] Trying fallback display capture...", plugin_id) content = self._capture_display_content(plugin, plugin_id) if content: total_width = sum(img.width for img in content) - logger.info( + logger.debug( "[%s] Fallback capture SUCCESS: %d images, %dpx total", plugin_id, len(content), total_width ) @@ -211,7 +226,7 @@ class PluginAdapter: kept.append(result.image) if not kept: - logger.info( + logger.debug( "[%s] All %d image(s) from %s were blank — contributing nothing", plugin_id, len(images), source ) @@ -220,14 +235,14 @@ class PluginAdapter: trimmed_width = sum(img.width for img in kept) if trimmed_width < self.config.min_plugin_width: - logger.info( + logger.debug( "[%s] Trimmed content %dpx is below min_plugin_width %dpx — skipping", plugin_id, trimmed_width, self.config.min_plugin_width ) return None if trimmed_width != original_width or dropped_blank: - logger.info( + logger.debug( "[%s] Trimmed %s content: %dpx -> %dpx (%.0f%% reclaimed), " "%d image(s) kept, %d blank dropped", plugin_id, source, original_width, trimmed_width, @@ -398,6 +413,88 @@ class PluginAdapter: return 0 return int(self.display_width * ratio) + def _resume_offset(self, plugin_id: str, shape: Tuple[str, int]) -> int: + """ + The plugin's stored rotation offset, if it still applies. + + An offset is only meaningful against content shaped the way it was + when the offset was recorded. When the shape has changed — a different + number of rows, a re-rendered strip with different item boundaries — + the stored value points somewhere arbitrary, so rotation restarts. + + Args: + plugin_id: Plugin identifier + shape: (kind, size) describing what an offset would index into now + + Returns: + The stored offset, or 0 when it no longer applies + """ + if self._offset_shapes.get(plugin_id) != shape: + if plugin_id in self._item_offsets: + logger.debug( + "[%s] Content is %s now, was %s — restarting the rotation " + "rather than resuming at a position that no longer means " + "anything", plugin_id, shape, + self._offset_shapes.get(plugin_id)) + self._item_offsets.pop(plugin_id, None) + self._offset_shapes[plugin_id] = shape + return 0 + return self._item_offsets.get(plugin_id, 0) + + def _record_offset( + self, plugin_id: str, offset: int, shape: Tuple[str, int] + ) -> None: + """Store where the next window should resume, with what it indexes.""" + if offset: + self._item_offsets[plugin_id] = offset + self._offset_shapes[plugin_id] = shape + else: + # A wrapped-to-zero rotation is the same as no state at all, and + # keeping the key would report a window as active when the next + # pass starts from the top anyway. + self._item_offsets.pop(plugin_id, None) + self._offset_shapes.pop(plugin_id, None) + + def _clear_offset(self, plugin_id: str) -> None: + """Forget any rotation state for a plugin.""" + self._item_offsets.pop(plugin_id, None) + self._offset_shapes.pop(plugin_id, None) + + def _merge_trailing_runt(self, end: int, width: int, budget: int) -> int: + """ + Extend a window to the end of the content when what would be left over + is too small to be worth its own pass. + + Windows were placed by walking forward from the last one, which makes + the final window whatever happens to remain. Measured on a live panel + that produced a 1,840px stocks ticker splitting 1,492 + 348 — the + second pass showing seven seconds of content before cutting, which + reads as the display failing rather than as a rotation. + + Absorbing the remainder overruns the budget by less than one window + floor, which is a better trade than a fragment: the budget is a guard + against one plugin holding the panel for minutes, not a hard limit. + + Args: + end: Column the window would otherwise end at + width: Full content width + budget: Width budget being applied + + Returns: + ``end``, or ``width`` when the remainder is below the floor + """ + remainder = width - end + # Measured against the budget rather than the panel: snapping to item + # boundaries means an ordinary window already lands short of the budget + # (a 512px budget over 182px-pitch items yields 348px windows), so an + # absolute floor would merge windows that were never fragments. Half a + # budget separates "a short last pass" from "a sliver", and caps the + # overrun this can cause at 1.5 budgets. + floor = budget // 2 + if 0 < remainder < floor: + return width + return end + def _apply_width_budget( self, images: List[Image.Image], plugin_id: str, plugin: Optional['BasePlugin'] = None @@ -435,45 +532,62 @@ class PluginAdapter: if not budget or total <= budget: # Fits, so reset rotation — the whole segment is being shown. - self._item_offsets.pop(plugin_id, None) + self._clear_offset(plugin_id) return images if len(images) == 1: return [self._crop_to_budget(images[0], budget, plugin_id, mode)] + shape = ('rows', len(images)) if mode == 'truncate': # Ordered content: always show from the top. Deliberately does not # advance the offset, so the same opening items appear every time # rather than the viewer being shown the middle of a ranked list. start = 0 else: - start = self._item_offsets.get(plugin_id, 0) % len(images) + start = self._resume_offset(plugin_id, shape) % len(images) selected: List[Image.Image] = [] used = 0 consumed = 0 # Walk forward from the rotation offset, taking whole items only, so a # cut never lands in the middle of one. + # + # A window may overrun the budget while it is still shorter than the + # runt floor, for the same reason _merge_trailing_runt exists on the + # single-image path: a pass far shorter than its neighbours reads as + # the display failing rather than as a rotation. Rows of 450, 450 and + # 100 against a 512px budget used to give the 100 a pass of its own -- + # two seconds against nine. Wrapping does not prevent that, because it + # only helps when the row wrapped to actually fits. + floor = budget // 2 for step in range(len(images)): img = images[(start + step) % len(images)] cost = img.width if selected: cost += self._row_gap(selected[-1], img) if selected and used + cost > budget: - break + # Keep the overrun bounded at the same 1.5 budgets the + # single-image path allows. A next row too wide to absorb + # leaves a short window standing -- better than a window of + # 1.9 budgets, and the same trade the always-take-the-first + # rule below already makes. + if used >= floor or used + cost > budget + floor: + break selected.append(img) used += cost consumed += 1 if mode == 'truncate': - logger.info( + logger.debug( "[%s] Width budget %dpx: showing the first %d of %d row(s) " "(%dpx incl. gaps); the rest are not shown (overflow=truncate)", plugin_id, budget, len(selected), len(images), used ) else: - self._item_offsets[plugin_id] = (start + consumed) % len(images) - logger.info( + self._record_offset( + plugin_id, (start + consumed) % len(images), shape) + logger.debug( "[%s] Width budget %dpx: showing %d of %d row(s) (%dpx incl. gaps) " "from offset %d; remainder deferred to a later cycle", plugin_id, budget, len(selected), len(images), used, start @@ -490,16 +604,13 @@ class PluginAdapter: The cut is snapped to the nearest blank column so it does not slice through a glyph or logo and leave half a character at the panel edge. - """ - if mode == 'truncate': - # Always the start of the strip, so a ranked table is never entered - # from the middle. - offset = 0 - else: - offset = self._item_offsets.get(plugin_id, 0) - if offset >= img.width: - offset = 0 + Rotation is tracked as an index into the strip's item boundaries rather + than as a pixel column, because a ticker re-renders between fetches. A + column recorded against one render points at unrelated content in the + next as soon as anything ahead of it changes width — a digit in a + price, a shorter headline. The Nth boundary stays the Nth boundary. + """ # Cut only where the plugin left a real gap between items. Snapping to # any blank column used to pick the single-column gaps between # characters, splitting a word and orphaning its tail into the next @@ -514,10 +625,18 @@ class PluginAdapter: # budget exactly. The gap rule exists to protect discrete items # (words, ticker entries); it would be wrong to let a solid image # escape the cap in its name. - end = min(offset + budget, img.width) + # + # With no items to index, the offset here has to stay a column, so + # it is only reusable while the image keeps its width. + shape = ('cols', img.width) + offset = 0 if mode == 'truncate' else self._resume_offset( + plugin_id, shape) + end = self._merge_trailing_runt( + min(offset + budget, img.width), img.width, budget) if mode != 'truncate': - self._item_offsets[plugin_id] = 0 if end >= img.width else end - logger.info( + self._record_offset( + plugin_id, 0 if end >= img.width else end, shape) + logger.debug( "[%s] Width budget %dpx: cropped continuous %dpx image to " "[%d:%d] (no item gaps of %dpx+ to align to)%s", plugin_id, budget, img.width, offset, end, min_run, @@ -528,8 +647,15 @@ class PluginAdapter: # Cut mid-gap so the content either side keeps some breathing room. cuts = sorted({0, img.width} | {(a + b) // 2 for a, b in gaps}) - start = max((c for c in cuts if c <= offset), default=0) - later = [c for c in cuts if c > start] + shape = ('cuts', len(cuts)) + index = 0 if mode == 'truncate' else self._resume_offset( + plugin_id, shape) + # Clamped rather than wrapped: a stale index past the end means the + # strip shrank, and restarting reads better than landing near the end. + start_index = index if 0 <= index < len(cuts) - 1 else 0 + start = cuts[start_index] + + later = cuts[start_index + 1:] if not later: end = img.width else: @@ -537,15 +663,22 @@ class PluginAdapter: # No boundary inside the budget: take the next one and overrun, # because the alternative is cutting through an item. end = max(within) if within else min(later) + end = self._merge_trailing_runt(end, img.width, budget) + # Every candidate for `end` came from `cuts` (which includes img.width), + # so this always resolves; the fallback is defensive only. + end_index = cuts.index(end) if end in cuts else len(cuts) - 1 if mode != 'truncate': - # Next cycle resumes where this one stopped; wrap when the strip ends. - self._item_offsets[plugin_id] = 0 if end >= img.width else end + # Next cycle resumes at the boundary this one stopped on; wrap when + # the strip ends. + self._record_offset( + plugin_id, 0 if end >= img.width else end_index, shape) - logger.info( + logger.debug( "[%s] Width budget %dpx: cropped single %dpx image to [%d:%d] " - "(%dpx) at item boundaries, %s", + "(%dpx) at item boundaries %d-%d of %d, %s", plugin_id, budget, img.width, start, end, end - start, + start_index, end_index, len(cuts) - 1, "showing the start only (overflow=truncate)" if mode == 'truncate' else "window advances next cycle" ) @@ -565,7 +698,7 @@ class PluginAdapter: List of images or None """ try: - logger.info("[%s] Native: calling get_vegas_content()", plugin_id) + logger.debug("[%s] Native: calling get_vegas_content()", plugin_id) # Tell the plugin how much width the ticker wants it to use, and # narrow the canvas for the duration of the call. A plugin that @@ -574,7 +707,7 @@ class PluginAdapter: # be explicit can read get_vegas_render_width(). render_width = self.resolve_render_width(plugin, plugin_id) if render_width != self.display_width: - logger.info( + logger.debug( "[%s] Native: requesting %dpx instead of %dpx", plugin_id, render_width, self.display_width ) @@ -602,19 +735,19 @@ class PluginAdapter: plugin._vegas_render_width = None if result is None: - logger.info("[%s] Native: get_vegas_content() returned None", plugin_id) + logger.debug("[%s] Native: get_vegas_content() returned None", plugin_id) return None # Normalize to list if isinstance(result, Image.Image): images = [result] - logger.info( + logger.debug( "[%s] Native: got single Image %dx%d", plugin_id, result.width, result.height ) elif isinstance(result, (list, tuple)): images = list(result) - logger.info( + logger.debug( "[%s] Native: got %d items in list/tuple", plugin_id, len(images) ) @@ -635,14 +768,14 @@ class PluginAdapter: ) continue - logger.info( + logger.debug( "[%s] Native: item[%d] is %dx%d, mode=%s", plugin_id, i, img.width, img.height, img.mode ) # Ensure correct height if img.height != self.display_height: - logger.info( + logger.debug( "[%s] Native: resizing item[%d]: %dx%d -> %dx%d", plugin_id, i, img.width, img.height, img.width, self.display_height @@ -660,13 +793,13 @@ class PluginAdapter: if valid_images: total_width = sum(img.width for img in valid_images) - logger.info( + logger.debug( "[%s] Native: SUCCESS - %d images, %dpx total width", plugin_id, len(valid_images), total_width ) return valid_images - logger.info("[%s] Native: no valid images after validation", plugin_id) + logger.debug("[%s] Native: no valid images after validation", plugin_id) return None except (AttributeError, TypeError, ValueError, OSError) as e: @@ -700,20 +833,20 @@ class PluginAdapter: logger.debug("[%s] No scroll_helper attribute", plugin_id) return None - logger.info( + logger.debug( "[%s] Found scroll_helper: %s", plugin_id, type(scroll_helper).__name__ ) cached_image = getattr(scroll_helper, 'cached_image', None) if cached_image is None: - logger.info( + logger.debug( "[%s] scroll_helper.cached_image is None, triggering content generation", plugin_id ) if offscreen_only: # Generating it calls display(), which needs the canvas. - logger.info( + logger.debug( "[%s] scroll_helper cache empty; deferring generation " "to the render thread", plugin_id ) @@ -726,13 +859,13 @@ class PluginAdapter: return None if not isinstance(cached_image, Image.Image): - logger.info( + logger.debug( "[%s] scroll_helper.cached_image is not an Image: %s", plugin_id, type(cached_image).__name__ ) return None - logger.info( + logger.debug( "[%s] scroll_helper.cached_image found: %dx%d, mode=%s", plugin_id, cached_image.width, cached_image.height, cached_image.mode ) @@ -755,7 +888,7 @@ class PluginAdapter: # Ensure correct height if img.height != self.display_height: - logger.info( + logger.debug( "[%s] Resizing scroll_helper content: %dx%d -> %dx%d", plugin_id, img.width, img.height, img.width, self.display_height @@ -769,7 +902,7 @@ class PluginAdapter: if img.mode != 'RGB': img = img.convert('RGB') - logger.info( + logger.debug( "[%s] ScrollHelper content ready: %dx%d", plugin_id, img.width, img.height ) @@ -869,7 +1002,7 @@ class PluginAdapter: with self._capture(): # Method 1: Try _create_scrolling_display (stocks pattern) if hasattr(plugin, '_create_scrolling_display'): - logger.info( + logger.debug( "[%s] Triggering via _create_scrolling_display()", plugin_id ) @@ -877,7 +1010,7 @@ class PluginAdapter: plugin._create_scrolling_display() cached_image = getattr(scroll_helper, 'cached_image', None) if cached_image is not None and isinstance(cached_image, Image.Image): - logger.info( + logger.debug( "[%s] _create_scrolling_display() SUCCESS: %dx%d", plugin_id, cached_image.width, cached_image.height ) @@ -889,7 +1022,7 @@ class PluginAdapter: # Method 2: Try display(force_clear=True) which typically builds scroll content if hasattr(plugin, 'display'): - logger.info( + logger.debug( "[%s] Triggering via display(force_clear=True)", plugin_id ) @@ -898,12 +1031,12 @@ class PluginAdapter: plugin.display(force_clear=True) cached_image = getattr(scroll_helper, 'cached_image', None) if cached_image is not None and isinstance(cached_image, Image.Image): - logger.info( + logger.debug( "[%s] display(force_clear=True) SUCCESS: %dx%d", plugin_id, cached_image.width, cached_image.height ) return cached_image - logger.info( + logger.debug( "[%s] display(force_clear=True) did not populate cached_image", plugin_id ) @@ -912,7 +1045,7 @@ class PluginAdapter: "[%s] display(force_clear=True) failed", plugin_id ) - logger.info( + logger.debug( "[%s] Could not trigger scroll content generation", plugin_id ) @@ -944,15 +1077,15 @@ class PluginAdapter: try: # Save current display state original_image = self.display_manager.image.copy() - logger.info("[%s] Fallback: saved original display state", plugin_id) + logger.debug("[%s] Fallback: saved original display state", plugin_id) # Ensure plugin has fresh data before capturing has_update_data = hasattr(plugin, 'update_data') - logger.info("[%s] Fallback: has update_data=%s", plugin_id, has_update_data) + logger.debug("[%s] Fallback: has update_data=%s", plugin_id, has_update_data) if has_update_data: try: plugin.update_data() - logger.info("[%s] Fallback: update_data() called", plugin_id) + logger.debug("[%s] Fallback: update_data() called", plugin_id) except (AttributeError, RuntimeError, OSError): logger.exception("[%s] Fallback: update_data() failed", plugin_id) @@ -964,41 +1097,41 @@ class PluginAdapter: # arrangement rather than one that has to be cropped afterwards. render_width = self.resolve_render_width(plugin, plugin_id) if render_width != self.display_width: - logger.info( + logger.debug( "[%s] Fallback: rendering at %dpx instead of %dpx", plugin_id, render_width, self.display_width ) with self._capture(), self._render_at(render_width): self.display_manager.clear() - logger.info("[%s] Fallback: display cleared, calling display()", plugin_id) + logger.debug("[%s] Fallback: display cleared, calling display()", plugin_id) # First try without force_clear (some plugins behave better this way) try: plugin.display() - logger.info("[%s] Fallback: display() called successfully", plugin_id) + logger.debug("[%s] Fallback: display() called successfully", plugin_id) except TypeError: # Plugin may require force_clear argument - logger.info("[%s] Fallback: display() failed, trying with force_clear=True", plugin_id) + logger.debug("[%s] Fallback: display() failed, trying with force_clear=True", plugin_id) plugin.display(force_clear=True) # Capture the result captured = self.display_manager.image.copy() - logger.info( + logger.debug( "[%s] Fallback: captured frame %dx%d, mode=%s", plugin_id, captured.width, captured.height, captured.mode ) # Check if captured image has content (not all black) is_blank, bright_ratio = self._is_blank_image(captured, return_ratio=True) - logger.info( + logger.debug( "[%s] Fallback: brightness check - %.3f%% bright pixels (threshold=0.5%%)", plugin_id, bright_ratio * 100 ) if is_blank: - logger.info( + logger.debug( "[%s] Fallback: first capture blank, retrying with force_clear", plugin_id ) @@ -1009,7 +1142,7 @@ class PluginAdapter: captured = self.display_manager.image.copy() is_blank, bright_ratio = self._is_blank_image(captured, return_ratio=True) - logger.info( + logger.debug( "[%s] Fallback: retry brightness - %.3f%% bright pixels", plugin_id, bright_ratio * 100 ) @@ -1026,7 +1159,7 @@ class PluginAdapter: if captured.mode != 'RGB': captured = captured.convert('RGB') - logger.info( + logger.debug( "[%s] Fallback: SUCCESS - captured %dx%d", plugin_id, captured.width, captured.height ) diff --git a/src/vegas_mode/stream_manager.py b/src/vegas_mode/stream_manager.py index a97420c9..59bd1134 100644 --- a/src/vegas_mode/stream_manager.py +++ b/src/vegas_mode/stream_manager.py @@ -406,6 +406,8 @@ class StreamManager: ) logger.info("Ordered plugins: %s", ordered_plugins) + ordered_plugins = self._apply_priority_weights(ordered_plugins) + # Atomically update shared state under lock to avoid races with prefetchers with self._buffer_lock: self._ordered_plugins = ordered_plugins @@ -417,6 +419,143 @@ class StreamManager: logger.info("=" * 60) + def _plugin_weight(self, plugin_id: str) -> int: + """Slots per cycle for one plugin. + + A plugin may answer for itself via get_vegas_priority_weight() -- the + only way favorite-team awareness can reach here, since the core can see + that a game is live but not whose. When it declines (returns None, the + default), live content earns ``live_weight`` and everything else 1. + """ + plugin = None + try: + plugin = self.plugin_manager.plugins.get(plugin_id) + except (AttributeError, TypeError): + return 1 + if plugin is None: + return 1 + + try: + if hasattr(plugin, 'get_vegas_priority_weight'): + declared = plugin.get_vegas_priority_weight() + if declared is not None: + return max(1, min(10, int(declared))) + except Exception: + # Deliberately falls through to the core's own live check rather + # than demoting to 1. The plugin's weight calculation is broken, + # but has_live_priority() and has_live_content() are separate + # methods guarded separately below -- a plugin that genuinely has + # a live game should still get live_weight for it. + logger.exception("[%s] get_vegas_priority_weight() failed", plugin_id) + + try: + if (hasattr(plugin, 'has_live_priority') + and hasattr(plugin, 'has_live_content') + and plugin.has_live_priority() + and plugin.has_live_content()): + return self.config.live_weight + except Exception: + logger.exception("[%s] live-content check failed", plugin_id) + return 1 + + def _apply_priority_weights(self, ordered: List[str]) -> List[str]: + """Expand the rotation so weighted plugins take several turns per cycle. + + Smooth Weighted Round-Robin, the same scheduler the sports plugins use + to rotate their own games: a plugin of weight N appears N times per + cycle, and the repeats are spaced through the cycle rather than + clumped, so a live score is never three-in-a-row followed by a long + silence. + + Returns the input unchanged when nothing is weighted, which is both the + common case and the pre-existing behaviour. + """ + if not ordered or not self.config.live_in_ticker: + return ordered + + weights = {pid: self._plugin_weight(pid) for pid in ordered} + total = sum(weights.values()) + if total <= len(ordered): + return ordered # nothing boosted; plain round robin + + current = {pid: 0 for pid in ordered} + schedule: List[str] = [] + for _ in range(total): + for pid in ordered: + current[pid] += weights[pid] + picked = max(current, key=lambda p: current[p]) + current[picked] -= total + schedule.append(picked) + + schedule = self._unclump_seam(schedule) + + boosted = {p: w for p, w in weights.items() if w > 1} + logger.info( + "Vegas rotation weighted: %d slots for %d plugins (boosted: %s)", + len(schedule), len(ordered), boosted) + return schedule + + @staticmethod + def _unclump_seam(schedule: List[str]) -> List[str]: + """Stop the heaviest plugin sitting on both ends of the cycle. + + Smooth Weighted Round-Robin spaces repeats well *within* a pass, but + it schedules the heaviest item first and often last too. The strip + loops, so those two are neighbours: the one place the marquee shows + the same plugin twice running is the seam between cycles. + + Rotating the list cannot fix this. Rotation preserves the cyclic order + exactly, so it only moves where the seam is drawn, not the adjacency + itself. The trailing entry has to be swapped with one from the middle + whose neighbours differ from it, which breaks the pair without + creating another. + + Left alone when no such position exists -- a rotation short enough or + lopsided enough to have none is one where the plugin is unavoidably + adjacent to itself anyway. + """ + if len(schedule) < 3 or schedule[0] != schedule[-1]: + return schedule + + repeated = schedule[-1] + size = len(schedule) + + def cyclic_doubles(seq) -> int: + return sum(1 for i in range(size) if seq[i] == seq[(i + 1) % size]) + + def clearance(seq, value) -> int: + """Smallest cyclic gap between appearances of `value`.""" + at = [i for i, v in enumerate(seq) if v == value] + if len(at) < 2: + return size + return min(min((b - a) % size, (a - b) % size) + for i, a in enumerate(at) for b in at[i + 1:]) + + # Try each swap and judge the result, rather than reasoning about which + # neighbours the two moved elements will end up with. That reasoning is + # where the first version went wrong: it guarded the slot `repeated` + # moves into but not the one the displaced element lands in, so + # ['a','b','c','d','x','y','x','a'] came back ending ['x','x'] -- the + # seam duplicate traded for a fresh one. + best = None + best_clearance = -1 + for j in range(1, size - 1): + candidate = list(schedule) + candidate[j], candidate[-1] = candidate[-1], candidate[j] + if cyclic_doubles(candidate): + continue + # Among the repairs that work, prefer the one that leaves the + # boosted plugin most evenly spread; taking the first that merely + # fits moved a repeat from a gap of 7 into a gap of 2. + spread = clearance(candidate, repeated) + if spread > best_clearance: + best, best_clearance = candidate, spread + + # None exists when the value is unavoidably adjacent to itself -- a + # plugin holding most of the slots has to be. Schedule it as it is + # rather than refuse. + return best if best is not None else schedule + def _prefetch_content(self, count: int = 1) -> None: """ Prefetch content for upcoming plugins. diff --git a/src/web_interface/api_helpers.py b/src/web_interface/api_helpers.py index 7ba6567a..3cff5293 100644 --- a/src/web_interface/api_helpers.py +++ b/src/web_interface/api_helpers.py @@ -29,18 +29,16 @@ def success_response( Flask jsonify response """ response_data = create_success_response(data, message, metadata) - - # Add request metadata if available - if metadata is None: - metadata = {} - - # Add timing if request start time is available + + # Timing is merged into whatever the caller passed, without inventing a + # metadata block for responses that have neither. + enriched = dict(metadata) if metadata is not None else {} if hasattr(request, 'start_time'): - metadata['response_time_ms'] = int((time.time() - request.start_time) * 1000) - - if metadata: - response_data['metadata'] = metadata - + enriched['response_time_ms'] = int((time.time() - request.start_time) * 1000) + + if metadata is not None or enriched: + response_data['metadata'] = enriched + return jsonify(response_data) diff --git a/src/web_interface/error_handler.py b/src/web_interface/error_handler.py index c15d373c..509c5679 100644 --- a/src/web_interface/error_handler.py +++ b/src/web_interface/error_handler.py @@ -1,11 +1,11 @@ """ Centralized error handling for web interface. -Provides decorators and helpers for consistent error handling across API endpoints. +Provides helpers for consistent error responses across API endpoints. """ -import functools -from typing import Callable, Any, Optional +import re +from typing import Any, Optional from flask import jsonify from src.web_interface.errors import ( @@ -17,68 +17,95 @@ from src.logging_config import get_logger logger = get_logger(__name__) -def handle_errors( - default_error_code: Optional[ErrorCode] = None, - default_category: Optional[ErrorCategory] = None, - log_error: bool = True -): +# Credentials that turn up inside exception text. A requests error quotes the +# URL it failed on, and plugins that authenticate by query string put their key +# there, so echoing an exception verbatim can hand out an API key. Redact the +# value, keep the parameter name -- knowing *which* credential was involved is +# part of the diagnosis. +_REDACT_CREDENTIAL = re.compile( + r'((?:api[_-]?key|access[_-]?token|auth|apikey|key|passwd|password|pwd|' + r'secret|sig|signature|token)["\']?\s*[=:]\s*["\']?)([^\s&"\'<>,}]+)', + re.IGNORECASE, +) + +# `Authorization: `. The scheme name is kept because it +# says which kind of credential failed; the credential goes. Any scheme +# matches, not a fixed list: ApiKey, Negotiate, NTLM, AWS4-HMAC-SHA256 and +# whatever a plugin's API invents next are all credentials, and a list would +# silently leak the ones nobody thought of. Not covered by the generic pattern +# above, whose value part stops at whitespace and so would keep the credential +# once a space follows the scheme. +_REDACT_AUTH_HEADER = re.compile( + r'((?:proxy-)?authorization["\']?\s*[=:]\s*["\']?\s*' + r'(?:[A-Za-z][\w.+-]*[ \t]+)?)' # optional scheme name, kept + r'([^\s,"\'<>}]+)', # the credential, redacted + re.IGNORECASE, +) + +# Credentials embedded in a URL: https://user:password@host. requests quotes +# the full URL in its exceptions, so this is a realistic leak. The username is +# kept -- it identifies which account failed without being the secret. +_REDACT_URL_USERINFO = re.compile(r'([a-z][a-z0-9+.-]*://[^/\s:@]+:)([^/\s@]+)(@)', + re.IGNORECASE) + +# Long enough for an errno string with a path, short enough not to dump a +# parser's worth of context into a JSON field. +_MAX_DETAIL_LENGTH = 400 + + +def describe_exception(exc: BaseException, + max_length: int = _MAX_DETAIL_LENGTH) -> str: """ - Decorator to handle errors in API endpoints. - - Catches exceptions and converts them to structured error responses. - + One-line, safe-to-return description of an exception. + + The generic "an error occurred; see logs for details" tells a user nothing + and, when the failure is bad enough, the logs are unreachable too: a device + whose storage was failing returned that message from every endpoint + *including* the log viewer, because journalctl could not be executed. The + underlying `[Errno 5] Input/output error` named the fault immediately. + + Returns "TypeName: message", credentials redacted and length capped. The + type alone is worth carrying -- a bare PermissionError says more than any + generic sentence. + Args: - default_error_code: Default error code if exception doesn't match known types - default_category: Default error category - log_error: Whether to log the error + exc: The exception to describe + max_length: Truncate beyond this many characters + + Returns: + A single-line description, never empty """ - def decorator(func: Callable) -> Callable: - @functools.wraps(func) - def wrapper(*args, **kwargs): - try: - return func(*args, **kwargs) - except WebInterfaceError as e: - # Already a structured error - if log_error: - logger.error( - f"Error in {func.__name__}: {e.message}", - extra={ - 'error_code': e.error_code.value, - 'category': e.category.value, - 'context': e.context - } - ) - return jsonify(e.to_dict()), 500 - - except Exception as e: - # Convert to structured error - web_error = WebInterfaceError.from_exception( - e, - error_code=default_error_code, - context={ - 'function': func.__name__, - 'endpoint': getattr(func, '__name__', 'unknown') - } - ) - - if default_category: - web_error.category = default_category - - if log_error: - logger.error( - f"Unhandled error in {func.__name__}: {e}", - exc_info=True, - extra={ - 'error_code': web_error.error_code.value, - 'category': web_error.category.value, - 'context': web_error.context - } - ) - - return jsonify(web_error.to_dict()), 500 - - return wrapper - return decorator + message = str(exc).strip() + text = f"{type(exc).__name__}: {message}" if message else type(exc).__name__ + return redact_text(text, max_length) + + +def redact_text(text: str, max_length: int = _MAX_DETAIL_LENGTH) -> str: + """Make arbitrary text safe to hand back over HTTP. + + Split out of describe_exception because exceptions are not the only thing + worth returning: a subprocess's stderr, or a message a helper script + printed, is just as useful to a user and just as capable of carrying a + token or a password in it. + + Args: + text: The text to redact + max_length: Truncate beyond this many characters + + Returns: + A single line, credentials replaced, length capped. + """ + text = text or '' + # Order matters: the URL and header forms are more specific than the + # generic key=value pattern, which would otherwise chew the scheme. + text = _REDACT_URL_USERINFO.sub(r'\1\3', text) + text = _REDACT_AUTH_HEADER.sub(r'\1', text) + text = _REDACT_CREDENTIAL.sub(r'\1', text) + # Collapse newlines/tabs so the detail stays one line in a JSON field. + text = ' '.join(text.split()) + if len(text) > max_length: + text = text[:max_length - 1].rstrip() + '…' + return text def create_error_response( @@ -134,14 +161,17 @@ def create_success_response( "status": "success" } + # All three use `is not None` rather than truthiness: "" and {} are + # values a caller chose to send, and dropping them silently would make + # the response shape depend on the data. if data is not None: response["data"] = data - - if message: + + if message is not None: response["message"] = message - - if metadata: + + if metadata is not None: response["metadata"] = metadata - + return response diff --git a/src/web_interface/errors.py b/src/web_interface/errors.py index 11397892..bb7c6e07 100644 --- a/src/web_interface/errors.py +++ b/src/web_interface/errors.py @@ -89,7 +89,11 @@ class WebInterfaceError: self.category = category or self._infer_category(error_code) self.details = details self.context = context or {} - self.suggested_fixes = suggested_fixes or self._get_default_suggestions(error_code) + # `is None`, not truthiness: an explicit [] means "this caller has + # no suggestions to offer", which the default list would override. + self.suggested_fixes = ( + suggested_fixes if suggested_fixes is not None + else self._get_default_suggestions(error_code)) self.original_error = original_error def _infer_category(self, error_code: ErrorCode) -> ErrorCategory: diff --git a/src/web_interface/logging_config.py b/src/web_interface/logging_config.py deleted file mode 100644 index 010130d1..00000000 --- a/src/web_interface/logging_config.py +++ /dev/null @@ -1,160 +0,0 @@ -""" -Structured logging configuration for web interface. - -Provides JSON-formatted structured logging for better debugging and monitoring. -""" - -import json -import logging -import sys -from datetime import datetime -from typing import Dict, Any, Optional - - -class StructuredFormatter(logging.Formatter): - """ - JSON formatter for structured logging. - - Formats log records as JSON for easy parsing and analysis. - """ - - def format(self, record: logging.LogRecord) -> str: - """Format log record as JSON.""" - log_data = { - 'timestamp': datetime.utcnow().isoformat(), - 'level': record.levelname, - 'logger': record.name, - 'message': record.getMessage(), - 'module': record.module, - 'function': record.funcName, - 'line': record.lineno - } - - # Add exception info if present - if record.exc_info: - log_data['exception'] = self.formatException(record.exc_info) - - # Add extra fields from record - if hasattr(record, 'extra'): - log_data.update(record.extra) - - # Add context from record - if hasattr(record, 'context'): - log_data['context'] = record.context - - return json.dumps(log_data) - - def formatException(self, exc_info) -> Dict[str, Any]: - """Format exception as structured data.""" - import traceback - return { - 'type': exc_info[0].__name__ if exc_info[0] else None, - 'message': str(exc_info[1]) if exc_info[1] else None, - 'traceback': traceback.format_exception(*exc_info) - } - - -def setup_structured_logging( - level: int = logging.INFO, - use_json: bool = False, - output_stream = sys.stdout -) -> None: - """ - Set up structured logging for web interface. - - Args: - level: Logging level - use_json: Whether to use JSON formatting - output_stream: Output stream for logs - """ - root_logger = logging.getLogger() - root_logger.setLevel(level) - - # Remove existing handlers - for handler in root_logger.handlers[:]: - root_logger.removeHandler(handler) - - # Create handler - handler = logging.StreamHandler(output_stream) - handler.setLevel(level) - - # Set formatter - if use_json: - formatter = StructuredFormatter() - else: - formatter = logging.Formatter( - '%(asctime)s - %(name)s - %(levelname)s - %(message)s' - ) - - handler.setFormatter(formatter) - root_logger.addHandler(handler) - - -def log_plugin_operation( - logger: logging.Logger, - operation: str, - plugin_id: str, - status: str, - context: Optional[Dict[str, Any]] = None -) -> None: - """ - Log a plugin operation with structured data. - - Args: - logger: Logger instance - operation: Operation name (install, update, uninstall, etc.) - plugin_id: Plugin identifier - status: Operation status (success, failed, etc.) - context: Optional additional context - """ - extra = { - 'operation': operation, - 'plugin_id': plugin_id, - 'status': status - } - - if context: - extra['context'] = context - - logger.info( - f"Plugin operation: {operation} for {plugin_id} - {status}", - extra=extra - ) - - -def log_config_change( - logger: logging.Logger, - config_key: str, - action: str, - before: Optional[Dict[str, Any]] = None, - after: Optional[Dict[str, Any]] = None, - context: Optional[Dict[str, Any]] = None -) -> None: - """ - Log a configuration change with before/after values. - - Args: - logger: Logger instance - config_key: Configuration key that changed - action: Action performed (save, update, delete, etc.) - before: Configuration before change - after: Configuration after change - context: Optional additional context - """ - extra = { - 'config_key': config_key, - 'action': action - } - - if before: - extra['before'] = before - if after: - extra['after'] = after - if context: - extra['context'] = context - - logger.info( - f"Config change: {action} on {config_key}", - extra=extra - ) - diff --git a/src/web_interface/secret_helpers.py b/src/web_interface/secret_helpers.py index 310d9848..9aa2eccc 100644 --- a/src/web_interface/secret_helpers.py +++ b/src/web_interface/secret_helpers.py @@ -5,7 +5,7 @@ Provides functions for identifying, masking, separating, and filtering secret fields in plugin configurations based on JSON Schema x-secret markers. """ -from typing import Any, Dict, Set, Tuple +from typing import Any, Dict, Optional, Set, Tuple def find_secret_fields(properties: Dict[str, Any], prefix: str = '') -> Set[str]: @@ -143,6 +143,12 @@ def mask_secret_fields(config: Dict[str, Any], schema_properties: Dict[str, Any] return result +#: What a masked secret looks like on the wire. Named because the write path +#: has to recognise it coming back: a client that renders the mask and posts +#: it unchanged must not store the mask as if it were the secret. +SECRET_MASK = '\u2022' * 8 + + def mask_all_secret_values(config: Dict[str, Any]) -> Dict[str, Any]: """Blanket-mask every non-empty value in a secrets config dict. @@ -156,15 +162,25 @@ def mask_all_secret_values(config: Dict[str, Any]) -> Dict[str, Any]: Returns: A copy with all real values replaced by ``'••••••••'``. """ - masked: Dict[str, Any] = {} - for k, v in config.items(): - if isinstance(v, dict): - masked[k] = mask_all_secret_values(v) - elif v not in (None, '') and not (isinstance(v, str) and v.startswith('YOUR_')): - masked[k] = '••••••••' - else: - masked[k] = v - return masked + return {k: _mask_value(v) for k, v in config.items()} + + +def _mask_value(value: Any) -> Any: + """Mask one value, recursing through dicts and lists. + + A list used to be masked as though it were a scalar, so + ``accounts: [{"name": "a", "token": "..."}]`` came back as a single + ``'••••••••'``. Nothing leaked, but the caller could no longer see how + many entries there were or any of their non-secret fields, and the raw + editor was shown a string where the file holds an array. + """ + if isinstance(value, dict): + return {k: _mask_value(v) for k, v in value.items()} + if isinstance(value, list): + return [_mask_value(item) for item in value] + if value in (None, '') or (isinstance(value, str) and value.startswith('YOUR_')): + return value + return SECRET_MASK def remove_empty_secrets(secrets: Dict[str, Any]) -> Dict[str, Any]: @@ -186,6 +202,133 @@ def remove_empty_secrets(secrets: Dict[str, Any]) -> Dict[str, Any]: nested = remove_empty_secrets(v) if nested: result[k] = nested + elif isinstance(v, list): + # Lists used to fall through to the scalar branch below and be + # kept verbatim, blanks and all. Because lists merge by + # *replacement*, saving any unrelated setting then wrote + # [{"token": ""}, ...] straight over the stored list and + # destroyed every credential in it. + pruned = _prune_secret_list(v) + if pruned is not None: + result[k] = pruned elif v is not None and not (isinstance(v, str) and v.strip() == ''): result[k] = v return result + + +def _prune_secret_list(items: list) -> Optional[list]: + """Strip blanks from inside a list of secrets, preserving every index. + + The rest of the system treats a secrets list as *parallel* to the regular + one -- ``sec[i]`` holds the secret fields of item ``i``, and ``{}`` means + "item i has none" (see ConfigManager._strip_secrets_recursive). So an + emptied dict item stays ``{}``: putting ``None`` there makes that list stop + looking parallel, and the stripper then drops the whole key from the main + config, taking the non-secret fields with it. + + A blank *scalar* becomes ``None``, meaning "no update at this index" -- + :func:`merge_secrets` substitutes whatever is stored there. Returns + ``None`` when nothing in the list carries a real value, so the caller drops + the key and leaves the stored list untouched. + """ + pruned: list = [] + has_real_value = False + for item in items: + if isinstance(item, dict): + kept = remove_empty_secrets(item) + pruned.append(kept) + has_real_value = has_real_value or bool(kept) + elif isinstance(item, list): + sub = _prune_secret_list(item) + pruned.append(sub if sub is not None else []) + has_real_value = has_real_value or sub is not None + elif item is not None and not (isinstance(item, str) and item.strip() == ''): + pruned.append(item) + has_real_value = True + else: + pruned.append(None) + return pruned if has_real_value else None + + +def merge_secrets(stored: Any, incoming: Any) -> Any: + """Merge submitted secrets over stored ones, element-wise inside lists. + + ``deep_merge`` replaces a list wholesale. For secrets that is destructive: + an incoming list that carries a real value for one entry and ``None`` for + the rest would drop the stored credentials of every other entry. Here a + list merges by index, and ``None`` means "keep what is stored". + + Entries are matched by *position*, which is what the config form gives us + -- there is no schema-declared identity to key on, and it is the same + contract ConfigManager._strip_secrets_recursive already relies on. The + incoming list's length wins, so deleting an item deletes its secrets; + an item the client left blank keeps whatever is stored at that index. + """ + if isinstance(stored, dict) and isinstance(incoming, dict): + merged = dict(stored) + for key, value in incoming.items(): + merged[key] = (merge_secrets(stored[key], value) + if key in stored else value) + return merged + if isinstance(stored, list) and isinstance(incoming, list): + # The incoming list sets the length -- the regular config's list is + # authoritative about how many items exist, and this one runs parallel + # to it. Removing an entry must therefore remove its secrets too. + merged_list = [] + for index, item in enumerate(incoming): + stored_item = stored[index] if index < len(stored) else None + merged_list.append(stored_item if item is None + else merge_secrets(stored_item, item)) + return merged_list + if incoming is None: + return stored + return incoming + + +def strip_masked_values(secrets: Dict[str, Any]) -> Dict[str, Any]: + """Remove values a client echoed back rather than changed. + + The counterpart to :func:`mask_all_secret_values`. A client that GETs the + masked secrets, edits one field and POSTs the whole object back is sending + ``SECRET_MASK`` for every field it did not touch. Storing those would + replace each untouched credential with eight bullet characters. + + Drops the mask and, like :func:`remove_empty_secrets`, blank values -- so + the caller can merge the result onto what is already stored and have + "unchanged" mean unchanged. Empty nested dicts are pruned. + """ + result: Dict[str, Any] = {} + for k, v in secrets.items(): + if isinstance(v, dict): + nested = strip_masked_values(v) + if nested: + result[k] = nested + elif isinstance(v, list): + # A list is merged by replacement, not element by element -- there + # is no identity to match entries on -- so a list that still holds + # a mask cannot be merged safely: keeping it would store bullets, + # and keeping the submitted entries alone would drop whichever the + # client did not send back. Dropping the key leaves the stored + # list untouched, which is what an untouched list should do. + # + # The consequence, deliberately: editing one secret inside a list + # through this endpoint requires sending real values for all of + # them. Sending some masks leaves the whole list as it was. + if not _contains_mask(v): + result[k] = v + elif v is None: + continue + elif isinstance(v, str) and (v.strip() == '' or v == SECRET_MASK): + continue + else: + result[k] = v + return result + + +def _contains_mask(value: Any) -> bool: + """True when a mask sentinel survives anywhere inside ``value``.""" + if isinstance(value, dict): + return any(_contains_mask(v) for v in value.values()) + if isinstance(value, list): + return any(_contains_mask(item) for item in value) + return value == SECRET_MASK diff --git a/src/web_interface/validators.py b/src/web_interface/validators.py index e383ae99..fa1772ef 100644 --- a/src/web_interface/validators.py +++ b/src/web_interface/validators.py @@ -43,10 +43,15 @@ def validate_image_url(url: str) -> Tuple[bool, Optional[str]]: if any(handler in url_lower for handler in ['onerror=', 'onload=', 'onclick=']): return False, "Event handlers not allowed in URLs" + # Reject directory traversal anywhere, not only in relative paths: + # http://host/../secret is as much a traversal attempt as /../secret. + if '..' in url: + return False, "Invalid path: directory traversal not allowed" + # Allow relative paths starting with / if url.startswith('/'): - # Validate it's a safe relative path (no directory traversal) - if '..' in url or url.startswith('//'): + # // would be a protocol-relative URL, not a local path + if url.startswith('//'): return False, "Invalid relative path" return True, None @@ -104,10 +109,11 @@ def validate_file_upload(filename: str, max_size_mb: int = 10, if '..' in filename or '/' in filename or '\\' in filename: return False, "Filename contains invalid characters" - # Check extension if specified + # Check extension if specified. Both sides are lowercased: the caller's + # list is as likely to hold '.TTF' as the filename is. if allowed_extensions: file_ext = Path(filename).suffix.lower() - if file_ext not in allowed_extensions: + if file_ext not in [ext.lower() for ext in allowed_extensions]: return False, f"File extension must be one of: {', '.join(allowed_extensions)}" return True, None @@ -147,7 +153,8 @@ def validate_numeric_range(value: float, min_val: Optional[float] = None, Returns: Tuple of (is_valid, error_message) """ - if not isinstance(value, (int, float)): + # bool is an int subclass, so True would otherwise validate as 1. + if not isinstance(value, (int, float)) or isinstance(value, bool): return False, "Value must be a number" if min_val is not None and value < min_val: @@ -183,11 +190,19 @@ def validate_string_length(text: str, min_length: Optional[int] = None, def sanitize_plugin_config(config: dict) -> dict: """ - Sanitize plugin configuration input to prevent injection. - + Restrict a plugin config to safe key names and value types. + + Drops keys that are not plain identifiers and values that are not + JSON-ish scalars, lists, or dicts, recursing into the latter two. + + String values are returned **unescaped**: output escaping is the + template layer's job, and escaping here would store the escaped form + in config.json. Do not read this function as XSS protection for + rendered output. + Args: config: Configuration dictionary - + Returns: Sanitized configuration dictionary """ diff --git a/start_display.sh b/start_display.sh old mode 100644 new mode 100755 diff --git a/stop_display.sh b/stop_display.sh old mode 100644 new mode 100755 diff --git a/systemd/ledmatrix-wifi-monitor.service b/systemd/ledmatrix-wifi-monitor.service index 2f08fde0..46b69bfe 100644 --- a/systemd/ledmatrix-wifi-monitor.service +++ b/systemd/ledmatrix-wifi-monitor.service @@ -10,8 +10,8 @@ WorkingDirectory=__PROJECT_ROOT_DIR__ ExecStart=/usr/bin/python3 __PROJECT_ROOT_DIR__/scripts/utils/wifi_monitor_daemon.py --interval 30 Restart=on-failure RestartSec=10 -StandardOutput=syslog -StandardError=syslog +StandardOutput=journal +StandardError=journal SyslogIdentifier=ledmatrix-wifi-monitor [Install] diff --git a/systemd/ledmatrix.service b/systemd/ledmatrix.service index f3bc9e28..ab5ddf4d 100644 --- a/systemd/ledmatrix.service +++ b/systemd/ledmatrix.service @@ -8,9 +8,38 @@ Type=simple User=root WorkingDirectory=__PROJECT_ROOT_DIR__ Environment=PYTHONDONTWRITEBYTECODE=1 +# glibc gives each allocating thread its own malloc arena, up to 8 x CPU count, +# and an arena that has grown is never handed back to the OS. This process runs +# 9 threads on a 3-core Pi, so the ceiling is 24 arenas -- and a rig measured at +# 1030 MB resident held 23 large anonymous mappings on 64 MB-aligned addresses, +# 920 MB of them, while the live data it was actually holding (widest scroll +# strip seen: 35,746 x 64) accounts for roughly 15 MB. That gap is arena bloat, +# not leaked objects: RSS was flat across repeated sampling, not climbing. +# +# Capping the arenas trades a little allocator concurrency for a large amount of +# resident memory on a device that has neither to spare. 2 is the usual value; +# raise it if frame times regress. +Environment=MALLOC_ARENA_MAX=2 ExecStart=/usr/bin/python3 __PROJECT_ROOT_DIR__/run.py -Restart=on-failure +# Restart=always, not on-failure: run.py exiting 0 (a clean shutdown path taken +# for a reason that no longer applies, e.g. a config reload) would otherwise leave +# the service stopped and the panel dark indefinitely, with systemd considering +# that a successful outcome and never bringing it back. +Restart=always RestartSec=10 +# Memory ceiling as a share of physical RAM, so one unit file suits a 512 MB +# Pi Zero 2 W and an 8 GB Pi 5 alike. This is a backstop, not a tuning knob: it +# turns "the board runs out of memory, stops being able to fork, and takes sshd +# and the panel down together until someone pulls the plug" into "this one +# service restarts". +# +# NOTE: Raspberry Pi firmware boots the kernel with cgroup_disable=memory, and +# systemd accepts this setting and then silently ignores it. Verify with: +# grep memory /sys/fs/cgroup/cgroup.controllers +# If that prints nothing, add "cgroup_enable=memory cgroup_memory=1" to +# /boot/firmware/cmdline.txt (all on line 1) and reboot. first_time_install.sh +# does this for you. +MemoryMax=85% StandardOutput=journal StandardError=journal SyslogIdentifier=ledmatrix diff --git a/test/_api_v3_test_helpers.py b/test/_api_v3_test_helpers.py new file mode 100644 index 00000000..9bcb7984 --- /dev/null +++ b/test/_api_v3_test_helpers.py @@ -0,0 +1,75 @@ +""" +Shared scaffolding for api_v3 blueprint tests. + +Not a test module (the leading underscore keeps pytest from collecting +it). It is the pytest-fixture equivalent of ``_make_client()`` in +test_uninstall_and_reconcile_endpoint.py, which is unittest-style and +requires ``self.addCleanup``. + +The api_v3 blueprint keeps its managers as attributes on a module-level +singleton, not in Flask app state, so replacing them with mocks leaks +into every later test that imports api_v3 unless the originals are put +back. ``api_v3_client`` snapshots and restores them around each test. +""" + +from unittest.mock import MagicMock + +import pytest +from flask import Flask + + +# Every manager attribute the blueprint reads. Anything missing here keeps +# whatever a previously-run test left on the singleton. +API_V3_MANAGER_ATTRS = ( + 'config_manager', 'plugin_manager', 'plugin_store_manager', + 'plugin_state_manager', 'saved_repositories_manager', 'schema_manager', + 'operation_queue', 'operation_history', 'cache_manager', +) + +_SENTINEL = object() + + +def build_app(blueprint): + app = Flask(__name__) + app.config['TESTING'] = True + app.config['SECRET_KEY'] = 'test' + app.register_blueprint(blueprint, url_prefix='/api/v3') + return app + + +@pytest.fixture +def api_v3_module(): + """The api_v3 module with every manager replaced by a MagicMock. + + Restores the original attributes afterwards. Tests point individual + managers at real objects (a ConfigManager over tmp_path, say) or set + them to None to exercise the not-initialized branches. + """ + from web_interface.blueprints import api_v3 as module + + originals = { + name: getattr(module.api_v3, name, _SENTINEL) + for name in API_V3_MANAGER_ATTRS + } + for name in API_V3_MANAGER_ATTRS: + setattr(module.api_v3, name, MagicMock()) + # Default to the direct path; queue tests opt in explicitly. + module.api_v3.operation_queue = None + + yield module + + for name, original in originals.items(): + if original is _SENTINEL: + if hasattr(module.api_v3, name): + try: + delattr(module.api_v3, name) + except AttributeError: + pass + else: + setattr(module.api_v3, name, original) + + +@pytest.fixture +def api_v3_client(api_v3_module): + """Flask test client wired to the mocked blueprint.""" + return build_app(api_v3_module.api_v3).test_client() diff --git a/test/debug_nba_api.py b/test/debug_nba_api.py deleted file mode 100644 index 5840f5e1..00000000 --- a/test/debug_nba_api.py +++ /dev/null @@ -1,166 +0,0 @@ -#!/usr/bin/env python3 -""" -Diagnostic script to examine NBA API data structure and identify the missing 'id' field issue. -""" -import requests -import logging -from typing import Dict, Any - -# Set up logging -logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s') -logger = logging.getLogger(__name__) - -def fetch_nba_teams_data() -> Dict[str, Any]: - """Fetch NBA teams data from ESPN API.""" - teams_url = "https://site.api.espn.com/apis/site/v2/sports/basketball/nba/teams" - - try: - logger.info(f"Fetching NBA teams data from: {teams_url}") - response = requests.get(teams_url, timeout=30) - response.raise_for_status() - data = response.json() - - logger.info(f"Successfully fetched NBA teams data") - logger.info(f"Response structure keys: {list(data.keys())}") - - # Examine the structure - sports = data.get('sports', []) - if sports: - logger.info(f"Number of sports: {len(sports)}") - sport = sports[0] - logger.info(f"Sport keys: {list(sport.keys())}") - - leagues = sport.get('leagues', []) - if leagues: - league = leagues[0] - logger.info(f"League keys: {list(league.keys())}") - - teams = league.get('teams', []) - logger.info(f"Number of teams: {len(teams)}") - - if teams: - # Examine first team structure - first_team = teams[0] - logger.info(f"First team keys: {list(first_team.keys())}") - - team_data = first_team.get('team', {}) - logger.info(f"Team data keys: {list(team_data.keys())}") - - # Check for id field - team_id = team_data.get('id') - team_abbr = team_data.get('abbreviation') - team_name = team_data.get('name') - - logger.info(f"Sample team: ID={team_id}, ABBR={team_abbr}, NAME={team_name}") - - if team_id: - logger.info(f"Team ID field exists: {team_id}") - else: - logger.error("Team ID field is missing!") - - # Check a few more teams to confirm structure - for i in range(min(5, len(teams))): - team = teams[i].get('team', {}) - logger.info(f"Team {i+1}: ID={team.get('id')}, ABBR={team.get('abbreviation')}") - - return data - - except Exception as e: - logger.error(f"Error fetching NBA teams data: {e}") - return {} - -def fetch_nba_standings_data() -> Dict[str, Any]: - """Fetch NBA standings data from ESPN API.""" - standings_url = "https://site.api.espn.com/apis/v2/sports/basketball/nba/standings" - - try: - logger.info(f"Fetching NBA standings data from: {standings_url}") - response = requests.get(standings_url, timeout=30) - response.raise_for_status() - data = response.json() - - logger.info(f"Successfully fetched NBA standings data") - logger.info(f"Response structure keys: {list(data.keys())}") - - # Check if standings has entries (direct structure) - if 'standings' in data and 'entries' in data['standings']: - entries = data['standings']['entries'] - logger.info(f"Number of standings entries (direct): {len(entries)}") - - if entries: - # Examine first entry structure - first_entry = entries[0] - logger.info(f"First entry keys: {list(first_entry.keys())}") - - team_data = first_entry.get('team', {}) - logger.info(f"Team data keys: {list(team_data.keys())}") - - # Check for id field - team_id = team_data.get('id') - team_abbr = team_data.get('abbreviation') - team_name = team_data.get('displayName') - - logger.info(f"Sample standings team: ID={team_id}, ABBR={team_abbr}, NAME={team_name}") - - if team_id: - logger.info(f"Standings team ID field exists: {team_id}") - else: - logger.error("Standings team ID field is missing!") - - # Check children structure (divisions/conferences) - if 'children' in data: - children = data.get('children', []) - logger.info(f"Number of children (divisions/conferences): {len(children)}") - - for i, child in enumerate(children): - logger.info(f"Child {i+1} keys: {list(child.keys())}") - child_name = child.get('displayName', 'Unknown') - logger.info(f"Child {i+1} name: {child_name}") - - if 'standings' in child and 'entries' in child['standings']: - entries = child['standings']['entries'] - logger.info(f"Child {i+1} has {len(entries)} entries") - - if entries: - # Examine first entry in this child - first_entry = entries[0] - logger.info(f"Child {i+1} first entry keys: {list(first_entry.keys())}") - - team_data = first_entry.get('team', {}) - logger.info(f"Child {i+1} team data keys: {list(team_data.keys())}") - - # Check for id field - team_id = team_data.get('id') - team_abbr = team_data.get('abbreviation') - team_name = team_data.get('displayName') - - logger.info(f"Child {i+1} sample team: ID={team_id}, ABBR={team_abbr}, NAME={team_name}") - - if team_id: - logger.info(f"Child {i+1} team ID field exists: {team_id}") - else: - logger.error(f"Child {i+1} team ID field is missing!") - - return data - - except Exception as e: - logger.error(f"Error fetching NBA standings data: {e}") - return {} - -def main(): - """Main diagnostic function.""" - logger.info("Starting NBA API data structure diagnosis") - - # Fetch teams data - teams_data = fetch_nba_teams_data() - - # Fetch standings data - standings_data = fetch_nba_standings_data() - - # Summary - logger.info("Diagnosis complete") - logger.info("Check the logs above to see if team 'id' fields are present") - logger.info("The leaderboard manager needs team 'id' fields for logo fetching") - -if __name__ == "__main__": - main() diff --git a/test/fixtures/plugins/ci-fixture-plugin/config_schema.json b/test/fixtures/plugins/ci-fixture-plugin/config_schema.json new file mode 100644 index 00000000..9410d90a --- /dev/null +++ b/test/fixtures/plugins/ci-fixture-plugin/config_schema.json @@ -0,0 +1,31 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "title": "CI Fixture Plugin", + "type": "object", + "properties": { + "enabled": { + "type": "boolean", + "default": true + }, + "display_duration": { + "type": "number", + "default": 5 + }, + "border_color": { + "type": "array", + "items": {"type": "integer", "minimum": 0, "maximum": 255}, + "minItems": 3, + "maxItems": 3, + "default": [0, 255, 0], + "description": "RGB color of the border rectangle." + }, + "diagonal_color": { + "type": "array", + "items": {"type": "integer", "minimum": 0, "maximum": 255}, + "minItems": 3, + "maxItems": 3, + "default": [255, 0, 0], + "description": "RGB color of the diagonals." + } + } +} diff --git a/test/fixtures/plugins/ci-fixture-plugin/manager.py b/test/fixtures/plugins/ci-fixture-plugin/manager.py new file mode 100644 index 00000000..02805f60 --- /dev/null +++ b/test/fixtures/plugins/ci-fixture-plugin/manager.py @@ -0,0 +1,44 @@ +""" +CI fixture plugin. + +Exists so the plugin safety harness (test/plugins/test_plugin_matrix.py and +the plugin-safety CI job) always has at least one real plugin to load and +render — without it, an empty plugins/ directory turns the whole job into a +green no-op. The render is deliberately trivial and fully deterministic: +a border rectangle plus both diagonals, sized from the display manager's +declared dimensions. No fonts, no network, no time dependence, so golden +images are stable across platforms. +""" + +from PIL import ImageDraw + +from src.plugin_system.base_plugin import BasePlugin + + +class CIFixturePlugin(BasePlugin): + """Deterministic CI-only fixture plugin: renders a border + diagonals + pattern sized from the display's declared dimensions. Never shipped to + devices; exists solely so the plugin safety harness has a real plugin + to exercise in CI.""" + + def update(self) -> None: + """Nothing to fetch — the render is self-contained.""" + + def display(self, force_clear: bool = False) -> None: + self.display_manager.clear() + width = self.display_manager.matrix.width + height = self.display_manager.matrix.height + border = tuple(self.config.get("border_color", [0, 255, 0])) + diagonal = tuple(self.config.get("diagonal_color", [255, 0, 0])) + + image = self.display_manager.image + draw = ImageDraw.Draw(image) + # Blank only the declared panel area, then draw edge-to-edge content: + # the border proves the plugin reads dynamic dimensions (any overflow + # or underfill at any size is a harness bug or a dimensions bug), the + # diagonals make golden comparisons sensitive to size/offset drift. + draw.rectangle([0, 0, width - 1, height - 1], fill=(0, 0, 0)) + draw.rectangle([0, 0, width - 1, height - 1], outline=border) + draw.line([0, 0, width - 1, height - 1], fill=diagonal) + draw.line([0, height - 1, width - 1, 0], fill=diagonal) + self.display_manager.update_display() diff --git a/test/fixtures/plugins/ci-fixture-plugin/manifest.json b/test/fixtures/plugins/ci-fixture-plugin/manifest.json new file mode 100644 index 00000000..40d188cb --- /dev/null +++ b/test/fixtures/plugins/ci-fixture-plugin/manifest.json @@ -0,0 +1,13 @@ +{ + "id": "ci-fixture-plugin", + "name": "CI Fixture Plugin", + "version": "1.0.0", + "description": "Bundled test fixture so the plugin safety harness always has at least one real plugin to render in CI. Draws a deterministic border + diagonals pattern at any panel size. Not installable from the store and never shipped to devices.", + "author": "LEDMatrix", + "entry_point": "manager.py", + "class_name": "CIFixturePlugin", + "display_modes": ["ci-fixture"], + "update_interval": 3600, + "min_ledmatrix_version": "2.0.0", + "compatible_versions": [">=2.0.0"] +} diff --git a/test/fixtures/plugins/ci-fixture-plugin/requirements.txt b/test/fixtures/plugins/ci-fixture-plugin/requirements.txt new file mode 100644 index 00000000..7fefb0f5 --- /dev/null +++ b/test/fixtures/plugins/ci-fixture-plugin/requirements.txt @@ -0,0 +1,7 @@ +# No dependencies — the fixture must load in any environment. +# +# Pillow is deliberately NOT pinned here even though manager.py imports +# PIL: it is a core LEDMatrix dependency (see the repo-root +# requirements.txt), so it is always present wherever the harness runs, +# and the harness loads plugins with install_deps=False anyway. Pinning +# it here would only invite a needless pip install during test runs. diff --git a/test/fixtures/plugins/ci-fixture-plugin/test/golden/128x32/ci-fixture.png b/test/fixtures/plugins/ci-fixture-plugin/test/golden/128x32/ci-fixture.png new file mode 100644 index 00000000..32608a8c Binary files /dev/null and b/test/fixtures/plugins/ci-fixture-plugin/test/golden/128x32/ci-fixture.png differ diff --git a/test/fixtures/plugins/ci-fixture-plugin/test/golden/128x64/ci-fixture.png b/test/fixtures/plugins/ci-fixture-plugin/test/golden/128x64/ci-fixture.png new file mode 100644 index 00000000..de738059 Binary files /dev/null and b/test/fixtures/plugins/ci-fixture-plugin/test/golden/128x64/ci-fixture.png differ diff --git a/test/fixtures/plugins/ci-fixture-plugin/test/golden/128x96/ci-fixture.png b/test/fixtures/plugins/ci-fixture-plugin/test/golden/128x96/ci-fixture.png new file mode 100644 index 00000000..3107e6da Binary files /dev/null and b/test/fixtures/plugins/ci-fixture-plugin/test/golden/128x96/ci-fixture.png differ diff --git a/test/fixtures/plugins/ci-fixture-plugin/test/golden/256x128/ci-fixture.png b/test/fixtures/plugins/ci-fixture-plugin/test/golden/256x128/ci-fixture.png new file mode 100644 index 00000000..dd43ab76 Binary files /dev/null and b/test/fixtures/plugins/ci-fixture-plugin/test/golden/256x128/ci-fixture.png differ diff --git a/test/fixtures/plugins/ci-fixture-plugin/test/golden/256x32/ci-fixture.png b/test/fixtures/plugins/ci-fixture-plugin/test/golden/256x32/ci-fixture.png new file mode 100644 index 00000000..01638b37 Binary files /dev/null and b/test/fixtures/plugins/ci-fixture-plugin/test/golden/256x32/ci-fixture.png differ diff --git a/test/fixtures/plugins/ci-fixture-plugin/test/golden/64x32/ci-fixture.png b/test/fixtures/plugins/ci-fixture-plugin/test/golden/64x32/ci-fixture.png new file mode 100644 index 00000000..9312505b Binary files /dev/null and b/test/fixtures/plugins/ci-fixture-plugin/test/golden/64x32/ci-fixture.png differ diff --git a/test/fixtures/plugins/ci-fixture-plugin/test/golden/64x64/ci-fixture.png b/test/fixtures/plugins/ci-fixture-plugin/test/golden/64x64/ci-fixture.png new file mode 100644 index 00000000..7095ed8e Binary files /dev/null and b/test/fixtures/plugins/ci-fixture-plugin/test/golden/64x64/ci-fixture.png differ diff --git a/test/fixtures/plugins/ci-fixture-plugin/test/golden/96x48/ci-fixture.png b/test/fixtures/plugins/ci-fixture-plugin/test/golden/96x48/ci-fixture.png new file mode 100644 index 00000000..239561e4 Binary files /dev/null and b/test/fixtures/plugins/ci-fixture-plugin/test/golden/96x48/ci-fixture.png differ diff --git a/test/plugins/conftest.py b/test/plugins/conftest.py index 27bb6718..cff6dd42 100644 --- a/test/plugins/conftest.py +++ b/test/plugins/conftest.py @@ -23,9 +23,17 @@ os.environ['EMULATOR'] = 'true' def plugins_dir() -> Path: """Get the plugins directory path. - Checks plugins/ first, then falls back to plugin-repos/ - for monorepo development environments. + Honors LEDMATRIX_PLUGINS_DIR (first entry) when set — the same override + test_plugin_matrix.py uses, so CI can point every plugin suite at the + bundled fixture plugins. Otherwise checks plugins/ first, then falls + back to plugin-repos/ for monorepo development environments. """ + env = os.environ.get('LEDMATRIX_PLUGINS_DIR') + if env: + first = env.split(os.pathsep)[0] + if first: + return Path(first) + plugins_path = project_root / 'plugins' plugin_repos_path = project_root / 'plugin-repos' diff --git a/test/plugins/test_basketball_scoreboard.py b/test/plugins/test_basketball_scoreboard.py index 0fbd8b79..b93fb89f 100644 --- a/test/plugins/test_basketball_scoreboard.py +++ b/test/plugins/test_basketball_scoreboard.py @@ -1,5 +1,10 @@ """ Integration tests for basketball-scoreboard plugin. + +Requires the real plugin to be installed (plugins/ or plugin-repos/, +or the dir named by LEDMATRIX_PLUGINS_DIR) — on machines without it, +every test here skips by design. CI covers plugin safety with the +bundled fixture plugin via test_plugin_matrix.py instead. """ import pytest diff --git a/test/plugins/test_calendar.py b/test/plugins/test_calendar.py index 18528fdb..4d874940 100644 --- a/test/plugins/test_calendar.py +++ b/test/plugins/test_calendar.py @@ -1,5 +1,10 @@ """ Integration tests for calendar plugin. + +Requires the real plugin to be installed (plugins/ or plugin-repos/, +or the dir named by LEDMATRIX_PLUGINS_DIR) — on machines without it, +every test here skips by design. CI covers plugin safety with the +bundled fixture plugin via test_plugin_matrix.py instead. """ import pytest diff --git a/test/plugins/test_clock_simple.py b/test/plugins/test_clock_simple.py index 507feec9..9b25a59e 100644 --- a/test/plugins/test_clock_simple.py +++ b/test/plugins/test_clock_simple.py @@ -1,5 +1,10 @@ """ Integration tests for clock-simple plugin. + +Requires the real plugin to be installed (plugins/ or plugin-repos/, +or the dir named by LEDMATRIX_PLUGINS_DIR) — on machines without it, +every test here skips by design. CI covers plugin safety with the +bundled fixture plugin via test_plugin_matrix.py instead. """ import pytest diff --git a/test/plugins/test_odds_ticker.py b/test/plugins/test_odds_ticker.py index 90209548..231ed7de 100644 --- a/test/plugins/test_odds_ticker.py +++ b/test/plugins/test_odds_ticker.py @@ -1,5 +1,10 @@ """ Integration tests for odds-ticker plugin. + +Requires the real plugin to be installed (plugins/ or plugin-repos/, +or the dir named by LEDMATRIX_PLUGINS_DIR) — on machines without it, +every test here skips by design. CI covers plugin safety with the +bundled fixture plugin via test_plugin_matrix.py instead. """ import pytest diff --git a/test/plugins/test_soccer_scoreboard.py b/test/plugins/test_soccer_scoreboard.py index 36212bfd..d5526fb8 100644 --- a/test/plugins/test_soccer_scoreboard.py +++ b/test/plugins/test_soccer_scoreboard.py @@ -1,5 +1,10 @@ """ Integration tests for soccer-scoreboard plugin. + +Requires the real plugin to be installed (plugins/ or plugin-repos/, +or the dir named by LEDMATRIX_PLUGINS_DIR) — on machines without it, +every test here skips by design. CI covers plugin safety with the +bundled fixture plugin via test_plugin_matrix.py instead. """ import pytest diff --git a/test/plugins/test_text_display.py b/test/plugins/test_text_display.py index a43815ea..34adfa49 100644 --- a/test/plugins/test_text_display.py +++ b/test/plugins/test_text_display.py @@ -1,5 +1,10 @@ """ Integration tests for text-display plugin. + +Requires the real plugin to be installed (plugins/ or plugin-repos/, +or the dir named by LEDMATRIX_PLUGINS_DIR) — on machines without it, +every test here skips by design. CI covers plugin safety with the +bundled fixture plugin via test_plugin_matrix.py instead. """ import pytest diff --git a/test/test_api_helper.py b/test/test_api_helper.py new file mode 100644 index 00000000..b3c24e99 --- /dev/null +++ b/test/test_api_helper.py @@ -0,0 +1,275 @@ +""" +Tests for src/common/api_helper.py (APIHelper). + +Covers rate limiting, cached GETs, ESPN URL/cache-key construction, +session header defaults and per-call merging, the retry adapter, and the +fixed clear_cache() behavior (real CacheManager surface: clear_cache / +delete / list_cache_files, with safe no-ops elsewhere). + +No real network: helper.session.get/post are always replaced with mocks. +""" + +import types +from unittest.mock import MagicMock, Mock + +import pytest +import requests +from freezegun import freeze_time + +import src.common.api_helper as api_helper_module +from src.common.api_helper import APIHelper + + +def _make_response(payload): + response = MagicMock() + response.json.return_value = payload + response.raise_for_status.return_value = None + return response + + +@pytest.fixture +def cache(): + cache = MagicMock() + cache.get.return_value = None + return cache + + +@pytest.fixture +def helper(cache): + helper = APIHelper(cache_manager=cache) + # Default min interval is 1.0s and would really sleep between requests. + helper.set_rate_limit(0) + return helper + + +# --------------------------------------------------------------------------- +# Rate limiting +# --------------------------------------------------------------------------- + +class TestRateLimiting: + def test_sleeps_for_remaining_interval(self, helper, monkeypatch): + fake_time = MagicMock() + fake_time.time.side_effect = [102.0, 105.0] + monkeypatch.setattr(api_helper_module, 'time', fake_time) + + helper.set_rate_limit(5) + helper._last_request_time = 100.0 + helper._enforce_rate_limit() + + # 2s elapsed of a 5s interval -> sleep the remaining 3s. + fake_time.sleep.assert_called_once() + assert fake_time.sleep.call_args[0][0] == pytest.approx(3.0) + assert helper._last_request_time == 105.0 + + def test_no_sleep_when_interval_elapsed(self, helper, monkeypatch): + fake_time = MagicMock() + fake_time.time.side_effect = [200.0, 201.0] + monkeypatch.setattr(api_helper_module, 'time', fake_time) + + helper.set_rate_limit(5) + helper._last_request_time = 100.0 + helper._enforce_rate_limit() + + fake_time.sleep.assert_not_called() + assert helper._last_request_time == 201.0 + + +# --------------------------------------------------------------------------- +# get() +# --------------------------------------------------------------------------- + +class TestGet: + def test_cache_hit_skips_request_and_rate_limit(self, helper, cache): + cache.get.return_value = {'cached': True} + helper.session.get = Mock() + rate_spy = Mock() + helper._enforce_rate_limit = rate_spy + + result = helper.get('https://example.com/api', cache_key='k') + + assert result == {'cached': True} + helper.session.get.assert_not_called() + rate_spy.assert_not_called() + + def test_cache_miss_fetches_and_caches_without_ttl(self, helper, cache): + cache.get.return_value = None + helper.session.get = Mock(return_value=_make_response({'a': 1})) + + result = helper.get('https://example.com/api', cache_key='k', + cache_ttl=999) + + assert result == {'a': 1} + # Pin the ttl-dropped contract: CacheManager.set is called with + # (key, data) only — the cache_ttl argument is discarded. + cache.set.assert_called_once_with('k', {'a': 1}) + + def test_request_exception_returns_none_and_caches_nothing( + self, helper, cache): + helper.session.get = Mock( + side_effect=requests.exceptions.RequestException('boom')) + + result = helper.get('https://example.com/api', cache_key='k') + + assert result is None + cache.set.assert_not_called() + + def test_timeout_zero_falls_back_to_default(self, helper): + # Quirk pin: `timeout or self.default_timeout` treats an explicit + # timeout=0 as falsy, so the default (30) is used instead. + helper.session.get = Mock(return_value=_make_response({})) + + helper.get('https://example.com/api', timeout=0) + + assert helper.session.get.call_args.kwargs['timeout'] == 30 + + def test_per_call_headers_merge_over_session_headers(self, helper): + helper.session.get = Mock(return_value=_make_response({})) + + helper.get('https://example.com/api', headers={'X-Custom': 'yes'}) + + sent = helper.session.get.call_args.kwargs['headers'] + # Merged, not replaced: session defaults survive alongside the + # per-call header. + assert sent['X-Custom'] == 'yes' + assert sent['User-Agent'] == ( + 'LEDMatrix/1.0 (+https://github.com/ChuckBuilds/LEDMatrix)') + assert sent['Accept'] == 'application/json' + # The session's own headers are not polluted by the per-call ones. + assert 'X-Custom' not in helper.session.headers + + +# --------------------------------------------------------------------------- +# ESPN helpers +# --------------------------------------------------------------------------- + +class TestEspnHelpers: + @freeze_time('2026-08-07') + def test_fetch_espn_scoreboard_url_params_and_cache_key(self, helper): + helper.get = Mock(return_value={'ok': 1}) + + result = helper.fetch_espn_scoreboard('football', 'nfl') + + assert result == {'ok': 1} + helper.get.assert_called_once_with( + 'https://site.api.espn.com/apis/site/v2/sports/football/nfl/scoreboard', + params={'dates': '20260807', 'limit': 1000}, + cache_key='espn_football_nfl_20260807', + cache_ttl=300, + ) + + def test_fetch_espn_scoreboard_explicit_date(self, helper): + helper.get = Mock(return_value=None) + + helper.fetch_espn_scoreboard('basketball', 'nba', date='20250115') + + kwargs = helper.get.call_args.kwargs + assert kwargs['params'] == {'dates': '20250115', 'limit': 1000} + assert kwargs['cache_key'] == 'espn_basketball_nba_20250115' + + def test_fetch_espn_standings_url_and_cache_key(self, helper): + helper.get = Mock(return_value={'ok': 1}) + + helper.fetch_espn_standings('football', 'nfl') + + helper.get.assert_called_once_with( + 'https://site.api.espn.com/apis/site/v2/sports/football/nfl/standings', + cache_key='espn_standings_football_nfl', + cache_ttl=3600, + ) + + def test_fetch_espn_rankings_url_and_cache_key(self, helper): + helper.get = Mock(return_value={'ok': 1}) + + helper.fetch_espn_rankings('football', 'college-football') + + helper.get.assert_called_once_with( + 'https://site.api.espn.com/apis/site/v2/sports/football/college-football/rankings', + cache_key='espn_rankings_football_college-football', + cache_ttl=3600, + ) + + +# --------------------------------------------------------------------------- +# Session setup +# --------------------------------------------------------------------------- + +class TestSessionSetup: + def test_user_agent_exact(self, helper): + # Regression guard: ESPN began 403ing other user agents; this exact + # string must be sent on every request. + assert helper.session.headers['User-Agent'] == ( + 'LEDMatrix/1.0 (+https://github.com/ChuckBuilds/LEDMatrix)') + + def test_retry_adapter_configuration(self): + helper = APIHelper(cache_manager=None, max_retries=7) + + retries = helper.session.get_adapter('https://x').max_retries + assert retries.total == 7 + assert {429, 500, 502, 503, 504} <= set(retries.status_forcelist) + + +# --------------------------------------------------------------------------- +# clear_cache (fixed behavior: real CacheManager surface) +# --------------------------------------------------------------------------- + +class TestClearCache: + def test_no_pattern_uses_clear_cache_method(self): + manager = types.SimpleNamespace(clear_cache=Mock()) + helper = APIHelper(cache_manager=manager) + helper.set_rate_limit(0) + + helper.clear_cache() + + manager.clear_cache.assert_called_once_with() + + def test_no_pattern_falls_back_to_clear(self): + manager = types.SimpleNamespace(clear=Mock()) + helper = APIHelper(cache_manager=manager) + helper.set_rate_limit(0) + + helper.clear_cache() + + manager.clear.assert_called_once_with() + + def test_no_pattern_manager_without_any_clear_is_noop(self): + helper = APIHelper(cache_manager=object()) + helper.set_rate_limit(0) + + helper.clear_cache() # must not raise + + def test_pattern_deletes_only_matching_keys(self): + manager = types.SimpleNamespace( + list_cache_files=Mock(return_value=[ + {'key': 'espn_nfl_x'}, + {'key': 'other'}, + ]), + delete=Mock(), + ) + helper = APIHelper(cache_manager=manager) + helper.set_rate_limit(0) + + helper.clear_cache(pattern='espn') + + manager.delete.assert_called_once_with('espn_nfl_x') + + def test_pattern_manager_without_list_cache_files_is_noop(self): + helper = APIHelper(cache_manager=object()) + helper.set_rate_limit(0) + + helper.clear_cache(pattern='espn') # must not raise + + +# --------------------------------------------------------------------------- +# No cache manager +# --------------------------------------------------------------------------- + +class TestNoCacheManager: + def test_all_cache_operations_safe_without_manager(self): + helper = APIHelper(cache_manager=None) + helper.set_rate_limit(0) + + assert helper.get_cache('k') is None + assert helper._get_from_cache('k') is None + assert helper.set_cache('k', {'a': 1}) is None + assert helper.clear_cache() is None + assert helper.clear_cache(pattern='espn') is None diff --git a/test/test_api_v3_calendar_credentials.py b/test/test_api_v3_calendar_credentials.py new file mode 100644 index 00000000..48274e01 --- /dev/null +++ b/test/test_api_v3_calendar_credentials.py @@ -0,0 +1,226 @@ +""" +Endpoint tests for POST /plugins/calendar/upload-credentials. + +The endpoint takes an uploaded Google OAuth credentials file, writes it +into the calendar plugin's directory as credentials.json at mode 0600, and +copies any previous file aside first. It had no tests. + +Regression coverage for two fixed bugs: +- The OAuth-shape check sat inside `except Exception: pass`, so a valid + JSON document that is not an object — a bare `42`, a list, a string — + raised TypeError on the membership test, was swallowed, and got saved + as credentials.json anyway. +- Each overwrite created a timestamped backup and nothing ever removed + them, so every re-upload left another complete copy of the user's OAuth + client credentials in the plugin directory, indefinitely. +""" + +import io +import json +import os +import stat +import sys +import time +from pathlib import Path +from types import SimpleNamespace + +import pytest + +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from test._api_v3_test_helpers import api_v3_client, api_v3_module # noqa: F401,E402 + +URL = "/api/v3/plugins/calendar/upload-credentials" + +VALID_CREDENTIALS = { + "installed": { + "client_id": "abc.apps.googleusercontent.com", + "client_secret": "shh", + "redirect_uris": ["http://localhost"], + } +} + + +@pytest.fixture +def plugin_dir(tmp_path, api_v3_module): + directory = tmp_path / "plugins" / "calendar" + directory.mkdir(parents=True) + api_v3_module.api_v3.plugin_manager.get_plugin_directory.return_value = str(directory) + return directory + + +def upload(client, content, filename="credentials.json"): + # bytes are sent verbatim (to exercise malformed input); anything else + # is serialized, so None becomes the JSON literal null rather than an + # empty body. + payload = content if isinstance(content, bytes) else json.dumps(content).encode() + return client.post( + URL, + data={"file": (io.BytesIO(payload), filename)}, + content_type="multipart/form-data", + ) + + +def backups(plugin_dir): + return sorted(plugin_dir.glob("credentials.json.backup.*")) + + +class TestRequestValidation: + def test_no_file_part_is_a_400(self, api_v3_client, plugin_dir): + response = api_v3_client.post(URL, data={}, content_type="multipart/form-data") + assert response.status_code == 400 + assert "No file provided" in response.get_json()["message"] + + def test_empty_filename_is_a_400(self, api_v3_client, plugin_dir): + response = upload(api_v3_client, VALID_CREDENTIALS, filename="") + assert response.status_code == 400 + + @pytest.mark.parametrize("filename", ["creds.txt", "creds.pem", "creds"]) + def test_non_json_extension_is_a_400(self, api_v3_client, plugin_dir, filename): + response = upload(api_v3_client, VALID_CREDENTIALS, filename=filename) + assert response.status_code == 400 + assert "JSON file" in response.get_json()["message"] + + def test_uppercase_json_extension_accepted(self, api_v3_client, plugin_dir): + assert upload(api_v3_client, VALID_CREDENTIALS, + filename="CREDENTIALS.JSON").status_code == 200 + + def test_oversized_file_is_a_400(self, api_v3_client, plugin_dir): + response = upload(api_v3_client, b"x" * (1024 * 1024 + 1)) + assert response.status_code == 400 + assert "1MB" in response.get_json()["message"] + assert not (plugin_dir / "credentials.json").exists() + + def test_invalid_json_is_a_400(self, api_v3_client, plugin_dir): + response = upload(api_v3_client, b"{not json") + assert response.status_code == 400 + assert "not valid JSON" in response.get_json()["message"] + assert not (plugin_dir / "credentials.json").exists() + + def test_missing_plugin_directory_is_a_404(self, api_v3_client, api_v3_module, tmp_path): + api_v3_module.api_v3.plugin_manager.get_plugin_directory.return_value = str( + tmp_path / "not-installed") + assert upload(api_v3_client, VALID_CREDENTIALS).status_code == 404 + + +class TestOAuthShapeValidation: + def test_installed_key_accepted(self, api_v3_client, plugin_dir): + assert upload(api_v3_client, VALID_CREDENTIALS).status_code == 200 + + def test_web_key_accepted(self, api_v3_client, plugin_dir): + assert upload(api_v3_client, {"web": {"client_id": "x"}}).status_code == 200 + + def test_object_without_oauth_keys_is_a_400(self, api_v3_client, plugin_dir): + response = upload(api_v3_client, {"something": "else"}) + assert response.status_code == 400 + assert "valid Google OAuth" in response.get_json()["message"] + assert not (plugin_dir / "credentials.json").exists() + + @pytest.mark.parametrize("content", [42, "a string", [1, 2, 3], True, None]) + def test_valid_json_that_is_not_an_object_is_rejected( + self, api_v3_client, plugin_dir, content): + # Regression: `'installed' not in 42` raises TypeError, which the + # bare `except Exception: pass` swallowed — the file was then saved + # as credentials.json despite being unusable as credentials. + response = upload(api_v3_client, content) + assert response.status_code == 400 + assert "valid Google OAuth" in response.get_json()["message"] + assert not (plugin_dir / "credentials.json").exists() + + +class TestSaving: + def test_file_written_with_contents_intact(self, api_v3_client, plugin_dir): + response = upload(api_v3_client, VALID_CREDENTIALS) + assert response.status_code == 200 + saved = json.loads((plugin_dir / "credentials.json").read_text()) + assert saved == VALID_CREDENTIALS + + def test_response_reports_the_path(self, api_v3_client, plugin_dir): + body = upload(api_v3_client, VALID_CREDENTIALS).get_json() + assert body["path"].endswith("credentials.json") + + def test_permissions_are_owner_only(self, api_v3_client, plugin_dir): + upload(api_v3_client, VALID_CREDENTIALS) + mode = stat.S_IMODE((plugin_dir / "credentials.json").stat().st_mode) + assert mode == 0o600 + + def test_first_upload_creates_no_backup(self, api_v3_client, plugin_dir): + upload(api_v3_client, VALID_CREDENTIALS) + assert backups(plugin_dir) == [] + + def test_overwrite_backs_up_the_previous_file(self, api_v3_client, plugin_dir): + (plugin_dir / "credentials.json").write_text(json.dumps({"installed": {"old": 1}})) + upload(api_v3_client, VALID_CREDENTIALS) + assert len(backups(plugin_dir)) == 1 + assert json.loads(backups(plugin_dir)[0].read_text()) == {"installed": {"old": 1}} + assert json.loads((plugin_dir / "credentials.json").read_text()) == VALID_CREDENTIALS + + +class TestBackupPruning: + def _seed(self, plugin_dir, count): + """Create `count` backups with distinct, increasing mtimes.""" + now = int(time.time()) + for i in range(count): + path = plugin_dir / f"credentials.json.backup.{now - (count - i) * 10}" + path.write_text(json.dumps({"installed": {"gen": i}})) + os.utime(path, (now - (count - i) * 10, now - (count - i) * 10)) + + def test_old_backups_are_pruned(self, api_v3_client, plugin_dir): + # Regression: nothing ever removed these, so a plugin directory + # accumulated one full copy of the user's OAuth credentials per + # re-upload, forever. + (plugin_dir / "credentials.json").write_text(json.dumps({"installed": {"cur": 1}})) + self._seed(plugin_dir, 7) + assert len(backups(plugin_dir)) == 7 + + upload(api_v3_client, VALID_CREDENTIALS) + assert len(backups(plugin_dir)) == 5 + + def test_the_newest_backups_are_the_ones_kept(self, api_v3_client, plugin_dir): + (plugin_dir / "credentials.json").write_text(json.dumps({"installed": {"cur": 1}})) + self._seed(plugin_dir, 7) + + upload(api_v3_client, VALID_CREDENTIALS) + remaining = backups(plugin_dir) + # The just-created backup (of "cur") plus the four newest seeds. + contents = [json.loads(p.read_text()) for p in remaining] + assert {"installed": {"cur": 1}} in contents + assert {"installed": {"gen": 0}} not in contents # oldest seed gone + + def test_under_the_limit_nothing_is_removed(self, api_v3_client, plugin_dir): + (plugin_dir / "credentials.json").write_text(json.dumps({"installed": {"cur": 1}})) + self._seed(plugin_dir, 2) + upload(api_v3_client, VALID_CREDENTIALS) + assert len(backups(plugin_dir)) == 3 # 2 seeded + 1 new + + def test_repeated_uploads_stay_bounded( + self, api_v3_client, plugin_dir, api_v3_module, monkeypatch): + # The backup filename carries int(time.time()), so uploads inside + # the same second all write the same name and overwrite each other. + # Advance a fake clock a second per round — otherwise this never + # reaches six backups and the bound holds for the wrong reason. + clock = {"now": int(time.time())} + monkeypatch.setattr( + api_v3_module, "time", SimpleNamespace(time=lambda: clock["now"])) + for i in range(10): + clock["now"] += 1 + upload(api_v3_client, {"installed": {"round": i}}) + os.utime(plugin_dir / "credentials.json", + (clock["now"], clock["now"])) + remaining = backups(plugin_dir) + assert len(remaining) == 5 + # And they are the five most recent rounds, not an arbitrary five. + kept = sorted(int(p.name.rsplit(".", 1)[1]) for p in remaining) + assert kept == [clock["now"] - 4 + i for i in range(5)] + + def test_unremovable_backup_does_not_fail_the_upload( + self, api_v3_client, plugin_dir, monkeypatch): + (plugin_dir / "credentials.json").write_text(json.dumps({"installed": {"cur": 1}})) + self._seed(plugin_dir, 7) + + def refuse(self): + raise OSError("read-only filesystem") + monkeypatch.setattr(Path, "unlink", refuse) + + # Pruning is housekeeping; failing it must not lose the upload. + assert upload(api_v3_client, VALID_CREDENTIALS).status_code == 200 diff --git a/test/test_api_v3_music_auth_endpoints.py b/test/test_api_v3_music_auth_endpoints.py new file mode 100644 index 00000000..f7c709af --- /dev/null +++ b/test/test_api_v3_music_auth_endpoints.py @@ -0,0 +1,302 @@ +""" +Endpoint tests for /plugins/authenticate/spotify and .../ytm. + +The Spotify step-2 handler writes a Python wrapper script to a temp file +with the user's redirect URL embedded in it, then runs that file through +subprocess. That is the most dangerous shape in the blueprint and had no +tests: the URL is user input reaching generated source code. + +The two endpoints are NOT symmetrical, despite the matching names. Only +Spotify has a two-step flow, a wrapper script, and a redirect_url; YTM +just runs its script directly. + +Regression coverage for one fixed bug: the wrapper file was unlinked in +the success/failure branch and again in the TimeoutExpired handler, so +any other failure from subprocess.run — the interpreter missing, a fork +failure, an interrupted call — left a temp file containing the user's +redirect URL behind. +""" + +import ast +import json +import os +import subprocess +import sys +from pathlib import Path +from unittest.mock import MagicMock, patch + +import pytest + +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from test._api_v3_test_helpers import api_v3_client, api_v3_module # noqa: F401,E402 + + +@pytest.fixture +def plugin_dir(tmp_path, api_v3_module): + """A plugin directory containing both auth scripts.""" + directory = tmp_path / "plugins" / "ledmatrix-music" + directory.mkdir(parents=True) + (directory / "authenticate_spotify.py").write_text("print('spotify')\n") + (directory / "authenticate_ytm.py").write_text("print('ytm')\n") + api_v3_module.api_v3.plugin_manager.get_plugin_directory.return_value = str(directory) + return directory + + +def completed(returncode=0, stdout="ok", stderr=""): + return subprocess.CompletedProcess( + args=["python3"], returncode=returncode, stdout=stdout, stderr=stderr) + + +class TestSpotifyPreconditions: + URL = "/api/v3/plugins/authenticate/spotify" + + def test_missing_plugin_directory_is_404(self, api_v3_client, api_v3_module, tmp_path): + api_v3_module.api_v3.plugin_manager.get_plugin_directory.return_value = str( + tmp_path / "not-installed") + response = api_v3_client.post(self.URL, json={}) + assert response.status_code == 404 + assert response.get_json()["message"] == "Plugin not found" + + def test_none_plugin_directory_is_404(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_manager.get_plugin_directory.return_value = None + assert api_v3_client.post(self.URL, json={}).status_code == 404 + + def test_missing_auth_script_is_404(self, api_v3_client, plugin_dir): + (plugin_dir / "authenticate_spotify.py").unlink() + response = api_v3_client.post(self.URL, json={}) + assert response.status_code == 404 + assert "script not found" in response.get_json()["message"] + + +class TestSpotifyStepTwo: + """redirect_url present — the wrapper-script path.""" + + URL = "/api/v3/plugins/authenticate/spotify" + + def test_success(self, api_v3_client, plugin_dir): + with patch.object(subprocess, "run", return_value=completed(0, "done")): + response = api_v3_client.post(self.URL, json={"redirect_url": "http://cb/?code=x"}) + assert response.status_code == 200 + body = response.get_json() + assert body["status"] == "success" + assert body["output"] == "done" + + def test_script_failure_is_a_400_with_combined_output(self, api_v3_client, plugin_dir): + with patch.object(subprocess, "run", return_value=completed(1, "out", "err")): + response = api_v3_client.post(self.URL, json={"redirect_url": "http://cb/?code=x"}) + assert response.status_code == 400 + assert response.get_json()["output"] == "outerr" + + def test_timeout_is_a_408(self, api_v3_client, plugin_dir): + with patch.object(subprocess, "run", + side_effect=subprocess.TimeoutExpired("python3", 120)): + response = api_v3_client.post(self.URL, json={"redirect_url": "http://cb/?code=x"}) + assert response.status_code == 408 + assert "timed out" in response.get_json()["message"] + + def test_runs_a_list_argv_never_a_shell(self, api_v3_client, plugin_dir): + with patch.object(subprocess, "run", return_value=completed()) as run: + api_v3_client.post(self.URL, json={"redirect_url": "http://cb/?code=x"}) + args, kwargs = run.call_args + assert isinstance(args[0], list) + assert args[0][0] == "python3" + assert kwargs.get("shell") in (None, False) + + def test_timeout_is_bounded(self, api_v3_client, plugin_dir): + with patch.object(subprocess, "run", return_value=completed()) as run: + api_v3_client.post(self.URL, json={"redirect_url": "http://cb/?code=x"}) + assert run.call_args.kwargs["timeout"] == 120 + + +class TestSpotifyWrapperCleanup: + URL = "/api/v3/plugins/authenticate/spotify" + + def _wrapper_paths_after(self, api_v3_client, run_mock): + """Run the endpoint and return the wrapper path subprocess saw.""" + seen = {} + + def capture(args, **kwargs): + seen["path"] = args[1] + return run_mock(args, **kwargs) + + with patch.object(subprocess, "run", side_effect=capture): + api_v3_client.post(self.URL, json={"redirect_url": "http://cb/?code=x"}) + return seen["path"] + + def test_removed_after_success(self, api_v3_client, plugin_dir): + path = self._wrapper_paths_after(api_v3_client, lambda *a, **kw: completed()) + assert not os.path.exists(path) + + def test_removed_after_script_failure(self, api_v3_client, plugin_dir): + path = self._wrapper_paths_after( + api_v3_client, lambda *a, **kw: completed(1, "out", "err")) + assert not os.path.exists(path) + + def test_removed_after_timeout(self, api_v3_client, plugin_dir): + def raise_timeout(*a, **kw): + raise subprocess.TimeoutExpired("python3", 120) + path = self._wrapper_paths_after(api_v3_client, raise_timeout) + assert not os.path.exists(path) + + def test_removed_when_subprocess_cannot_start(self, api_v3_client, plugin_dir): + # Regression: cleanup lived in the success/failure branch and in the + # TimeoutExpired handler only. An OSError from subprocess.run itself + # — no interpreter, fork failure — skipped both and left the wrapper, + # which contains the user's redirect URL, on disk. + def raise_oserror(*a, **kw): + raise OSError("[Errno 12] Cannot allocate memory") + path = self._wrapper_paths_after(api_v3_client, raise_oserror) + assert not os.path.exists(path) + + +class TestSpotifyRedirectUrlIsNotInjectable: + """The wrapper embeds redirect_url into generated Python source.""" + + URL = "/api/v3/plugins/authenticate/spotify" + + ADVERSARIAL = [ + '''http://cb/?code=x"''', + """http://cb/?code=x'""", + 'http://cb/?code=x\\', + 'http://cb/?code=x\nimport os; os.system("id")', + 'http://cb/?code=x"""\nimport os\n"""', + "http://cb/?code=x'''", + 'http://cb/?code=x\\"\\n', + '"; import os; os.system("id"); "', + ] + + def _wrapper_source(self, api_v3_client, redirect_url): + captured = {} + + def capture(args, **kwargs): + captured["source"] = Path(args[1]).read_text() + return completed() + + with patch.object(subprocess, "run", side_effect=capture): + api_v3_client.post(self.URL, json={"redirect_url": redirect_url}) + return captured["source"] + + @pytest.mark.parametrize("redirect_url", ADVERSARIAL) + def test_wrapper_is_still_valid_python(self, api_v3_client, plugin_dir, redirect_url): + # If escaping failed, the generated file would not parse at all. + source = self._wrapper_source(api_v3_client, redirect_url) + ast.parse(source) + + @pytest.mark.parametrize("redirect_url", ADVERSARIAL) + def test_url_survives_as_one_string_literal( + self, api_v3_client, plugin_dir, redirect_url): + # Stronger than "it parses": the URL must still be a single string + # assigned to redirect_url, not code that escaped into statements. + source = self._wrapper_source(api_v3_client, redirect_url) + tree = ast.parse(source) + assigned = [ + node.value.value for node in ast.walk(tree) + if isinstance(node, ast.Assign) + and isinstance(node.value, ast.Constant) + and any(getattr(t, "id", None) == "redirect_url" for t in node.targets) + ] + assert assigned == [redirect_url.strip()] + + def test_injected_call_does_not_become_a_statement(self, api_v3_client, plugin_dir): + source = self._wrapper_source( + api_v3_client, 'http://cb/\nimport os; os.system("id")') + tree = ast.parse(source) + imported = { + alias.name for node in ast.walk(tree) + if isinstance(node, ast.Import) for alias in node.names + } + # The wrapper legitimately imports sys, subprocess and os; what it + # must not gain is a *call* smuggled in through the URL. + calls = [ + node for node in ast.walk(tree) + if isinstance(node, ast.Call) + and isinstance(node.func, ast.Attribute) + and node.func.attr == "system" + ] + assert calls == [] + + +class TestSpotifyStepOne: + """No redirect_url — the OAuth-URL path, which imports the script.""" + + URL = "/api/v3/plugins/authenticate/spotify" + + def test_script_without_credentials_helper_is_an_error( + self, api_v3_client, plugin_dir): + # The stub script defines neither get_auth_url nor + # load_spotify_credentials, so no URL can be produced. + response = api_v3_client.post(self.URL, json={}) + assert response.status_code in (400, 500) + assert response.get_json()["status"] == "error" + + def test_unusable_credentials_do_not_leak_into_the_response( + self, api_v3_client, plugin_dir): + (plugin_dir / "authenticate_spotify.py").write_text( + "def load_spotify_credentials():\n" + " return ('id-abc', 'super-secret-value', None)\n" + ) + response = api_v3_client.post(self.URL, json={}) + assert "super-secret-value" not in response.get_data(as_text=True) + + def test_script_raising_on_import_is_handled(self, api_v3_client, plugin_dir): + (plugin_dir / "authenticate_spotify.py").write_text("raise RuntimeError('boom')\n") + response = api_v3_client.post(self.URL, json={}) + assert response.status_code == 500 + assert response.get_json()["status"] == "error" + + def test_bodyless_post_reaches_step_one(self, api_v3_client, plugin_dir): + # Covered by the silent=True fix: previously a 500 from body parsing. + response = api_v3_client.post(self.URL) + assert response.status_code in (400, 500) + assert response.get_json()["status"] == "error" + + def test_whitespace_redirect_url_is_treated_as_absent( + self, api_v3_client, plugin_dir): + with patch.object(subprocess, "run", return_value=completed()) as run: + api_v3_client.post(self.URL, json={"redirect_url": " "}) + # Step 2 never runs, so no wrapper is executed. + run.assert_not_called() + + +class TestYouTubeMusic: + """No wrapper script and no redirect_url — deliberately not symmetric.""" + + URL = "/api/v3/plugins/authenticate/ytm" + + def test_missing_plugin_directory_is_404(self, api_v3_client, api_v3_module, tmp_path): + api_v3_module.api_v3.plugin_manager.get_plugin_directory.return_value = str( + tmp_path / "not-installed") + assert api_v3_client.post(self.URL).status_code == 404 + + def test_missing_script_is_404(self, api_v3_client, plugin_dir): + (plugin_dir / "authenticate_ytm.py").unlink() + response = api_v3_client.post(self.URL) + assert response.status_code == 404 + assert "script not found" in response.get_json()["message"] + + def test_success(self, api_v3_client, plugin_dir): + with patch.object(subprocess, "run", return_value=completed(0, "authorized")): + response = api_v3_client.post(self.URL) + assert response.status_code == 200 + assert response.get_json()["output"] == "authorized" + + def test_failure_is_a_400_with_combined_output(self, api_v3_client, plugin_dir): + with patch.object(subprocess, "run", return_value=completed(1, "out", "err")): + response = api_v3_client.post(self.URL) + assert response.status_code == 400 + assert response.get_json()["output"] == "outerr" + + def test_timeout_is_a_408(self, api_v3_client, plugin_dir): + with patch.object(subprocess, "run", + side_effect=subprocess.TimeoutExpired("python3", 60)): + assert api_v3_client.post(self.URL).status_code == 408 + + def test_runs_the_script_directly_without_a_shell(self, api_v3_client, plugin_dir): + with patch.object(subprocess, "run", return_value=completed()) as run: + api_v3_client.post(self.URL) + args, kwargs = run.call_args + assert args[0][0] == "python3" + assert args[0][1].endswith("authenticate_ytm.py") + assert kwargs.get("shell") in (None, False) + assert kwargs["timeout"] == 60 diff --git a/test/test_api_v3_non_finite_numbers.py b/test/test_api_v3_non_finite_numbers.py new file mode 100644 index 00000000..4e6c217b --- /dev/null +++ b/test/test_api_v3_non_finite_numbers.py @@ -0,0 +1,85 @@ +"""Non-finite JSON numbers must be rejected, not raise. + +json.loads accepts Infinity/-Infinity/NaN by default (they are not valid JSON, +but Python's parser emits them) and Flask's get_json passes them straight +through. int(float('inf')) raises OverflowError, which is neither ValueError +nor TypeError -- so validation blocks that carefully caught those let it +through and Flask turned it into a 500. + +The damage was not the status code. /config/dim-schedule answered with +CONFIG_SAVE_FAILED and suggested "Check file permissions on config directory" +and "Check available disk space" for what was actually an invalid number. + +NaN already returned 400 (int(nan) raises ValueError), which is why this only +showed up for the infinities. +""" +import sys +from pathlib import Path + +import pytest + +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from test._api_v3_test_helpers import api_v3_client, api_v3_module # noqa: F401,E402 + + +#: (route, field) that returned 500 before OverflowError was caught. Both +#: infinity signs are exercised: int() raises OverflowError for either, but +#: only one of them was in the original report, and a guard that special-cased +#: the sign would pass a one-sided test. +NON_FINITE_ROUTES = [ + ('/api/v3/config/dim-schedule', 'dim_brightness'), + ('/api/v3/errors/clear', 'max_age_hours'), + ('/api/v3/config/main', 'multiplexing'), + ('/api/v3/config/main', 'row_address_type'), +] +NON_FINITE_CASES = [ + (route, '{"%s": %s}' % (field, literal)) + for route, field in NON_FINITE_ROUTES + for literal in ('Infinity', '-Infinity') +] + + +@pytest.mark.parametrize("route,body", NON_FINITE_CASES) +def test_infinity_is_a_client_error_not_a_server_error(api_v3_client, route, body): + """Exactly 400, not merely "some 4xx". + + Accepting any 4xx would let a 404 pass, so renaming one of these routes + would leave the test green while testing nothing -- the failure mode this + whole file exists to catch. + """ + response = api_v3_client.post(route, data=body, content_type='application/json') + assert response.status_code == 400, ( + f"{route} with {body} answered {response.status_code}; expected 400" + ) + + +@pytest.mark.parametrize("route,body", [ + ('/api/v3/config/dim-schedule', '{"dim_brightness": NaN}'), + ('/api/v3/errors/clear', '{"max_age_hours": NaN}'), +]) +def test_nan_is_also_a_client_error(api_v3_client, route, body): + """int(nan) raises ValueError so this path already worked -- pinned so a + refactor that narrows the except tuple cannot quietly break it.""" + response = api_v3_client.post(route, data=body, content_type='application/json') + assert response.status_code == 400 + + +def test_a_valid_number_is_accepted(api_v3_client, api_v3_module, monkeypatch): + """Prove the widened except did not start swallowing ordinary input. + + Asserting "not a 400" would not show that: the mocked save path fails for + any input, so the assertion would hold even if validation had rejected the + value. Give load_config a real dict and stub the atomic save, and the + endpoint reaches its success response -- which only happens if 30 passed + validation. + """ + api_v3_module.api_v3.config_manager.load_config.return_value = {} + monkeypatch.setattr(api_v3_module, '_save_config_atomic', + lambda *a, **k: (True, '')) + response = api_v3_client.post( + '/api/v3/config/dim-schedule', + data='{"dim_brightness": 30}', + content_type='application/json', + ) + assert response.status_code == 200, response.get_data(as_text=True)[:200] diff --git a/test/test_api_v3_optional_body.py b/test/test_api_v3_optional_body.py new file mode 100644 index 00000000..4b7eb133 --- /dev/null +++ b/test/test_api_v3_optional_body.py @@ -0,0 +1,136 @@ +""" +Regression tests: POST endpoints whose body is optional must accept a +request that has no body at all. + +Six handlers in api_v3 read their body as ``request.get_json() or {}``. +The ``or {}`` states the intent plainly — every field is optional, so a +bodyless POST should fall back to defaults. But ``get_json()`` without +``silent=True`` raises ``UnsupportedMediaType`` when the request carries +no JSON Content-Type, and it raises *before* ``or {}`` is evaluated. Each +handler's catch-all then turned that into a 500. + +So the natural way to call these endpoints — a POST with no body, which +is what curl, a fetch() without options, and most HTTP clients send by +default — failed on every one of them. The shipped UI always sends a JSON +object, which is why this went unnoticed. + +This file covers the endpoints whose bodyless behaviour is not already +tested in their own suite. +""" + +import re +import sys +from pathlib import Path +from unittest.mock import MagicMock + +import pytest + +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from test._api_v3_test_helpers import api_v3_client, api_v3_module # noqa: F401,E402 + + +class TestOnDemandStart: + URL = "/api/v3/display/on-demand/start" + + def test_bodyless_post_is_not_a_server_error(self, api_v3_client, api_v3_module): + response = api_v3_client.post(self.URL) + # The endpoint may still reject the request on its own terms (no + # plugin_id, nothing to display); what it must not do is fail with + # a 500 raised out of body parsing. + assert response.status_code != 500 + + def test_json_body_still_works(self, api_v3_client, api_v3_module): + assert api_v3_client.post(self.URL, json={}).status_code != 500 + + +class TestResetPluginConfig: + URL = "/api/v3/plugins/config/reset" + + def test_bodyless_post_is_not_a_server_error(self, api_v3_client, api_v3_module): + assert api_v3_client.post(self.URL).status_code != 500 + + def test_json_body_still_works(self, api_v3_client, api_v3_module): + assert api_v3_client.post(self.URL, json={}).status_code != 500 + + +class TestDeleteOfTheDayJson: + URL = "/api/v3/plugins/of-the-day/json/delete" + + def test_bodyless_post_is_not_a_server_error(self, api_v3_client, api_v3_module): + assert api_v3_client.post(self.URL).status_code != 500 + + def test_json_body_still_works(self, api_v3_client, api_v3_module): + assert api_v3_client.post(self.URL, json={}).status_code != 500 + + +class TestPluginLimits: + URL = "/api/v3/plugins/clock/limits" + + def test_bodyless_post_is_not_a_server_error(self, api_v3_client, api_v3_module): + assert api_v3_client.post(self.URL).status_code != 500 + + +class TestMissingBodyGivesTheDeclaredError: + """Handlers that answer "No data provided" must actually be able to. + + A second group of handlers reads `data = request.get_json()` and then + guards with `if not data: return 400`. That guard is unreachable for a + request with no JSON body, because get_json() raises first — so the + caller got a 500 "an error occurred; see logs for details" instead of + the 400 the handler plainly intends to send. + """ + + @pytest.mark.parametrize("url", [ + "/api/v3/plugins/install", + "/api/v3/plugins/install-from-url", + "/api/v3/plugins/registry-from-url", + "/api/v3/config/raw/main", + "/api/v3/config/raw/secrets", + "/api/v3/cache/delete", + ]) + def test_bodyless_post_gets_a_400_not_a_500(self, api_v3_client, api_v3_module, url): + response = api_v3_client.post(url) + assert response.status_code == 400, ( + f"{url} answered {response.status_code}: " + f"{response.get_data(as_text=True)[:200]}") + + @pytest.mark.parametrize("url", [ + "/api/v3/plugins/install", + "/api/v3/config/raw/main", + ]) + def test_malformed_json_gets_a_400_not_a_500(self, api_v3_client, api_v3_module, url): + response = api_v3_client.post( + url, data="{not json", content_type="application/json") + assert response.status_code == 400 + + +class TestNoBodyReadContradictsItsOwnGuard: + SOURCE = Path(__file__).parent.parent / "web_interface/blueprints/api_v3.py" + + def test_no_or_default_read_is_unguarded(self): + """`get_json() or ` is a contradiction without silent=True. + + Writing `or {}` declares the body optional; omitting silent=True + means the call raises before the default can apply. + """ + offenders = [ + line.strip() for line in self.SOURCE.read_text().splitlines() + if "request.get_json()" in line and " or " in line + ] + assert offenders == [], ( + "these reads declare a default but raise before reaching it; " + f"use get_json(silent=True): {offenders}") + + def test_no_not_data_guard_is_unreachable(self): + """A `if not data:` guard needs a read that can actually return None.""" + lines = self.SOURCE.read_text().splitlines() + offenders = [] + for i, line in enumerate(lines): + if re.search(r"=\s*request\.get_json\(\)\s*$", line): + window = "\n".join(lines[i + 1:i + 3]) + if re.search(r"if\s+(not\s+data\b|data\s+is\s+None)", window): + offenders.append(f"line {i + 1}: {line.strip()}") + assert offenders == [], ( + "these handlers guard on a missing body but raise before the " + f"guard runs; use get_json(silent=True): {offenders}") diff --git a/test/test_api_v3_plugin_install_endpoints.py b/test/test_api_v3_plugin_install_endpoints.py new file mode 100644 index 00000000..29f7f7dc --- /dev/null +++ b/test/test_api_v3_plugin_install_endpoints.py @@ -0,0 +1,302 @@ +""" +Endpoint tests for POST /plugins/install and POST /plugins/install-from-url. + +Both were only ever tested at the PluginStoreManager layer, so the route +logic — the queue-vs-direct branch, schema invalidation, plugin discovery, +state and history recording — was unexercised. + +/plugins/install carries the same install logic twice: once inside the +operation-queue callback and once in the direct fallback. The paired +tests below assert both branches produce the same side effects, so the +duplication cannot quietly drift. +""" + +import sys +from pathlib import Path +from unittest.mock import MagicMock + +import pytest + +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from test._api_v3_test_helpers import api_v3_client, api_v3_module # noqa: F401,E402 + +INSTALL = "/api/v3/plugins/install" +FROM_URL = "/api/v3/plugins/install-from-url" + + +@pytest.fixture +def queued(api_v3_module): + """Enable the operation queue and run its callback synchronously.""" + queue = MagicMock() + + def enqueue(operation_type, plugin_id, operation_callback=None): + queue.callback_result = operation_callback(MagicMock()) + return "op-123" + + queue.enqueue_operation.side_effect = enqueue + api_v3_module.api_v3.operation_queue = queue + return queue + + +def side_effects(module): + """The manager calls a successful install is expected to make.""" + api = module.api_v3 + return { + "schema_invalidated": api.schema_manager.invalidate_cache.call_args_list, + "discovered": api.plugin_manager.discover_plugins.call_count, + "loaded": api.plugin_manager.load_plugin.call_args_list, + "state_set": api.plugin_state_manager.set_plugin_installed.call_args_list, + "history": api.operation_history.record_operation.call_args_list, + } + + +class TestInstallValidation: + def test_uninitialized_store_manager_is_a_500(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager = None + response = api_v3_client.post(INSTALL, json={"plugin_id": "clock"}) + assert response.status_code == 500 + assert "not initialized" in response.get_json()["message"] + + def test_missing_plugin_id_is_a_400(self, api_v3_client, api_v3_module): + response = api_v3_client.post(INSTALL, json={}) + assert response.status_code == 400 + assert "plugin_id required" in response.get_json()["message"] + api_v3_module.api_v3.plugin_store_manager.install_plugin.assert_not_called() + + def test_empty_body_is_a_400(self, api_v3_client, api_v3_module): + assert api_v3_client.post(INSTALL, json=None).status_code == 400 + + +class TestInstallDirectPath: + """operation_queue is None — the fallback branch.""" + + def test_success(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.install_plugin.return_value = True + response = api_v3_client.post(INSTALL, json={"plugin_id": "clock"}) + assert response.status_code == 200 + assert response.get_json()["status"] == "success" + + def test_success_side_effects(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.install_plugin.return_value = True + api_v3_client.post(INSTALL, json={"plugin_id": "clock"}) + effects = side_effects(api_v3_module) + assert effects["schema_invalidated"] == [(("clock",), {})] + assert effects["discovered"] == 1 + assert effects["loaded"] == [(("clock",), {})] + assert effects["state_set"] == [(("clock",), {})] + assert effects["history"][0].kwargs["status"] == "success" + + def test_branch_forwarded_to_the_manager(self, api_v3_client, api_v3_module): + manager = api_v3_module.api_v3.plugin_store_manager + manager.install_plugin.return_value = True + api_v3_client.post(INSTALL, json={"plugin_id": "clock", "branch": "dev"}) + manager.install_plugin.assert_called_once_with("clock", branch="dev") + + def test_branch_named_in_the_message(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.install_plugin.return_value = True + response = api_v3_client.post(INSTALL, json={"plugin_id": "clock", "branch": "dev"}) + assert "(branch: dev)" in response.get_json()["message"] + + def test_failure_is_a_500(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.install_plugin.return_value = False + response = api_v3_client.post(INSTALL, json={"plugin_id": "clock"}) + assert response.status_code == 500 + assert "Failed to install" in response.get_json()["message"] + + def test_failure_mentions_missing_registry_entry(self, api_v3_client, api_v3_module): + manager = api_v3_module.api_v3.plugin_store_manager + manager.install_plugin.return_value = False + manager.get_plugin_info.return_value = None + response = api_v3_client.post(INSTALL, json={"plugin_id": "ghost"}) + assert "not found in registry" in response.get_json()["message"] + + def test_failure_omits_registry_note_when_plugin_is_known( + self, api_v3_client, api_v3_module): + manager = api_v3_module.api_v3.plugin_store_manager + manager.install_plugin.return_value = False + manager.get_plugin_info.return_value = {"id": "clock"} + response = api_v3_client.post(INSTALL, json={"plugin_id": "clock"}) + assert "not found in registry" not in response.get_json()["message"] + + def test_failure_recorded_in_history(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.install_plugin.return_value = False + api_v3_client.post(INSTALL, json={"plugin_id": "clock"}) + record = api_v3_module.api_v3.operation_history.record_operation.call_args + assert record.kwargs["status"] == "failed" + + def test_no_side_effects_on_failure(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.install_plugin.return_value = False + api_v3_client.post(INSTALL, json={"plugin_id": "clock"}) + effects = side_effects(api_v3_module) + assert effects["schema_invalidated"] == [] + assert effects["loaded"] == [] + assert effects["state_set"] == [] + + +class TestInstallQueuedPath: + """operation_queue present — the callback branch.""" + + def test_returns_an_operation_id(self, api_v3_client, api_v3_module, queued): + api_v3_module.api_v3.plugin_store_manager.install_plugin.return_value = True + response = api_v3_client.post(INSTALL, json={"plugin_id": "clock"}) + assert response.status_code == 200 + assert response.get_json()["data"]["operation_id"] == "op-123" + + def test_message_says_queued(self, api_v3_client, api_v3_module, queued): + api_v3_module.api_v3.plugin_store_manager.install_plugin.return_value = True + response = api_v3_client.post(INSTALL, json={"plugin_id": "clock"}) + assert "queued" in response.get_json()["message"] + + def test_callback_success_side_effects(self, api_v3_client, api_v3_module, queued): + api_v3_module.api_v3.plugin_store_manager.install_plugin.return_value = True + api_v3_client.post(INSTALL, json={"plugin_id": "clock"}) + effects = side_effects(api_v3_module) + assert effects["schema_invalidated"] == [(("clock",), {})] + assert effects["discovered"] == 1 + assert effects["loaded"] == [(("clock",), {})] + assert effects["state_set"] == [(("clock",), {})] + assert effects["history"][0].kwargs["status"] == "success" + + def test_callback_reports_success(self, api_v3_client, api_v3_module, queued): + api_v3_module.api_v3.plugin_store_manager.install_plugin.return_value = True + api_v3_client.post(INSTALL, json={"plugin_id": "clock"}) + assert queued.callback_result["success"] is True + + def test_callback_failure_raises_for_the_queue(self, api_v3_client, api_v3_module, queued): + api_v3_module.api_v3.plugin_store_manager.install_plugin.return_value = False + # The callback signals failure by raising, so the queue can mark the + # operation failed; the route's catch-all turns it into a 500. + response = api_v3_client.post(INSTALL, json={"plugin_id": "clock"}) + assert response.status_code == 500 + + def test_callback_failure_recorded_in_history(self, api_v3_client, api_v3_module, queued): + api_v3_module.api_v3.plugin_store_manager.install_plugin.return_value = False + api_v3_client.post(INSTALL, json={"plugin_id": "clock"}) + record = api_v3_module.api_v3.operation_history.record_operation.call_args + assert record.kwargs["status"] == "failed" + + def test_branch_forwarded_from_the_callback(self, api_v3_client, api_v3_module, queued): + manager = api_v3_module.api_v3.plugin_store_manager + manager.install_plugin.return_value = True + api_v3_client.post(INSTALL, json={"plugin_id": "clock", "branch": "dev"}) + manager.install_plugin.assert_called_once_with("clock", branch="dev") + + +class TestInstallPathsAgree: + """The queue callback and the direct fallback duplicate the same logic.""" + + def _run(self, client, module, install_ok, queue): + module.api_v3.plugin_store_manager.install_plugin.return_value = install_ok + client.post(INSTALL, json={"plugin_id": "clock", "branch": "dev"}) + return side_effects(module) + + def test_success_side_effects_match(self, api_v3_client, api_v3_module): + direct = self._run(api_v3_client, api_v3_module, True, None) + + # Reset and re-run through the queue. + for mock in (api_v3_module.api_v3.schema_manager, + api_v3_module.api_v3.plugin_manager, + api_v3_module.api_v3.plugin_state_manager, + api_v3_module.api_v3.operation_history): + mock.reset_mock() + queue = MagicMock() + queue.enqueue_operation.side_effect = ( + lambda t, p, operation_callback=None: operation_callback(MagicMock()) and "op") + api_v3_module.api_v3.operation_queue = queue + queued = self._run(api_v3_client, api_v3_module, True, queue) + + assert direct["schema_invalidated"] == queued["schema_invalidated"] + assert direct["discovered"] == queued["discovered"] + assert direct["loaded"] == queued["loaded"] + assert direct["state_set"] == queued["state_set"] + assert (direct["history"][0].kwargs["status"] + == queued["history"][0].kwargs["status"]) + assert (direct["history"][0].kwargs["details"] + == queued["history"][0].kwargs["details"]) + + def test_only_the_message_wording_differs(self, api_v3_client, api_v3_module): + # Characterized: the direct path says "Plugin installed + # successfully" while the queue callback says "Plugin clock + # installed successfully". Cosmetic, and the queue's text is + # internal to the operation record rather than the HTTP response. + api_v3_module.api_v3.plugin_store_manager.install_plugin.return_value = True + direct = api_v3_client.post(INSTALL, json={"plugin_id": "clock"}).get_json() + assert direct["message"] == "Plugin installed successfully" + + +class TestInstallFromUrl: + def test_uninitialized_store_manager_is_a_500(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager = None + assert api_v3_client.post(FROM_URL, json={"repo_url": "http://x"}).status_code == 500 + + def test_missing_repo_url_is_a_400(self, api_v3_client, api_v3_module): + response = api_v3_client.post(FROM_URL, json={}) + assert response.status_code == 400 + assert "repo_url required" in response.get_json()["message"] + + def test_success(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.install_from_url.return_value = { + "success": True, "plugin_id": "clock", "name": "Clock"} + response = api_v3_client.post(FROM_URL, json={"repo_url": "https://github.com/o/r"}) + assert response.status_code == 200 + body = response.get_json() + assert body["plugin_id"] == "clock" + assert body["name"] == "Clock" + + def test_all_optional_arguments_forwarded(self, api_v3_client, api_v3_module): + manager = api_v3_module.api_v3.plugin_store_manager + manager.install_from_url.return_value = {"success": True, "plugin_id": "clock"} + api_v3_client.post(FROM_URL, json={ + "repo_url": " https://github.com/o/r ", + "plugin_id": "clock", + "plugin_path": "plugins/clock", + "branch": "dev", + }) + manager.install_from_url.assert_called_once_with( + repo_url="https://github.com/o/r", + plugin_id="clock", + plugin_path="plugins/clock", + branch="dev", + ) + + def test_success_invalidates_schema_and_loads_plugin(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.install_from_url.return_value = { + "success": True, "plugin_id": "clock"} + api_v3_client.post(FROM_URL, json={"repo_url": "http://x"}) + api_v3_module.api_v3.schema_manager.invalidate_cache.assert_called_once_with("clock") + api_v3_module.api_v3.plugin_manager.load_plugin.assert_called_once_with("clock") + + def test_success_without_plugin_id_skips_discovery(self, api_v3_client, api_v3_module): + # install_from_url can succeed without naming the plugin; there is + # then nothing to invalidate or load. + api_v3_module.api_v3.plugin_store_manager.install_from_url.return_value = { + "success": True, "plugin_id": None} + api_v3_client.post(FROM_URL, json={"repo_url": "http://x"}) + api_v3_module.api_v3.schema_manager.invalidate_cache.assert_not_called() + api_v3_module.api_v3.plugin_manager.load_plugin.assert_not_called() + + def test_branch_from_result_included(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.install_from_url.return_value = { + "success": True, "plugin_id": "clock", "branch": "dev"} + body = api_v3_client.post(FROM_URL, json={"repo_url": "http://x"}).get_json() + assert body["branch"] == "dev" + assert "(branch: dev)" in body["message"] + + def test_failure_reports_the_managers_error(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.install_from_url.return_value = { + "success": False, "error": "repo not found"} + response = api_v3_client.post(FROM_URL, json={"repo_url": "http://x"}) + assert response.status_code == 500 + assert response.get_json()["message"] == "repo not found" + + def test_failure_without_error_uses_fallback_text(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.install_from_url.return_value = { + "success": False} + response = api_v3_client.post(FROM_URL, json={"repo_url": "http://x"}) + assert "Failed to install plugin from URL" in response.get_json()["message"] + + def test_manager_exception_is_a_500(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.install_from_url.side_effect = ( + RuntimeError("boom")) + assert api_v3_client.post(FROM_URL, json={"repo_url": "http://x"}).status_code == 500 diff --git a/test/test_api_v3_registry_endpoints.py b/test/test_api_v3_registry_endpoints.py new file mode 100644 index 00000000..85f4b377 --- /dev/null +++ b/test/test_api_v3_registry_endpoints.py @@ -0,0 +1,179 @@ +""" +Endpoint tests for the plugin-registry routes in api_v3: +POST /plugins/store/refresh and POST /plugins/registry-from-url. + +Both reach out to the network through PluginStoreManager (mocked here) and +had no endpoint-level coverage; registry-from-url in particular takes a +user-supplied URL and hands it straight to the manager. +""" + +import sys +from pathlib import Path + +import pytest + +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from test._api_v3_test_helpers import api_v3_client, api_v3_module # noqa: F401,E402 + + +class TestRefreshPluginStore: + URL = "/api/v3/plugins/store/refresh" + + def test_uninitialized_manager_is_a_500(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager = None + response = api_v3_client.post(self.URL, json={}) + assert response.status_code == 500 + assert "not initialized" in response.get_json()["message"] + + def test_success_reports_plugin_count(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.fetch_registry.return_value = { + "plugins": [{"id": "a"}, {"id": "b"}, {"id": "c"}]} + response = api_v3_client.post(self.URL, json={}) + assert response.status_code == 200 + assert response.get_json()["plugin_count"] == 3 + + def test_forces_a_refresh_rather_than_using_cache(self, api_v3_client, api_v3_module): + manager = api_v3_module.api_v3.plugin_store_manager + manager.fetch_registry.return_value = {"plugins": []} + api_v3_client.post(self.URL, json={}) + manager.fetch_registry.assert_called_once_with(force_refresh=True) + + def test_empty_registry_reports_zero(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.fetch_registry.return_value = {} + response = api_v3_client.post(self.URL, json={}) + assert response.get_json()["plugin_count"] == 0 + + def test_no_body_is_accepted(self, api_v3_client, api_v3_module): + # Regression: `request.get_json() or {}` says a missing body is + # fine, but get_json() raises UnsupportedMediaType before `or {}` + # is reached, so a bodyless POST — the natural way to call a + # refresh endpoint — came back 500. + api_v3_module.api_v3.plugin_store_manager.fetch_registry.return_value = {"plugins": []} + assert api_v3_client.post(self.URL).status_code == 200 + + def test_body_without_json_content_type_is_accepted( + self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.fetch_registry.return_value = {"plugins": []} + response = api_v3_client.post(self.URL, data="", content_type="text/plain") + assert response.status_code == 200 + + def test_malformed_json_body_falls_back_to_defaults( + self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.fetch_registry.return_value = {"plugins": []} + response = api_v3_client.post( + self.URL, data="{not json", content_type="application/json") + assert response.status_code == 200 + + @pytest.mark.parametrize("key", ["fetch_commit_info", "fetch_latest_versions"]) + def test_either_commit_info_key_extends_the_message( + self, api_v3_client, api_v3_module, key): + # fetch_latest_versions is the older spelling; both must work. + api_v3_module.api_v3.plugin_store_manager.fetch_registry.return_value = {"plugins": []} + response = api_v3_client.post(self.URL, json={key: True}) + assert "commit metadata" in response.get_json()["message"] + + def test_message_stays_plain_without_the_flag(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.fetch_registry.return_value = {"plugins": []} + response = api_v3_client.post(self.URL, json={}) + assert response.get_json()["message"] == "Plugin store refreshed" + + def test_network_failure_is_a_500(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.fetch_registry.side_effect = ( + ConnectionError("github unreachable")) + response = api_v3_client.post(self.URL, json={}) + assert response.status_code == 500 + assert response.get_json()["message"] == "An error occurred; see logs for details" + + def test_failure_body_carries_no_traceback_or_paths( + self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.fetch_registry.side_effect = ( + RuntimeError("failed at /home/user/LEDMatrix/src/secret.py line 42")) + body = api_v3_client.post(self.URL, json={}).get_json() + assert "Traceback" not in str(body) + # `details` is describe_exception output: one line, type-named, + # credential-redacted. It may quote the message, but never a stack. + assert body["details"].startswith("RuntimeError:") + assert "\n" not in body["details"] + + +class TestRegistryFromUrl: + URL = "/api/v3/plugins/registry-from-url" + + def test_uninitialized_manager_is_a_500(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager = None + response = api_v3_client.post(self.URL, json={"repo_url": "http://x"}) + assert response.status_code == 500 + + def test_missing_repo_url_is_a_400(self, api_v3_client, api_v3_module): + response = api_v3_client.post(self.URL, json={}) + assert response.status_code == 400 + assert "repo_url required" in response.get_json()["message"] + api_v3_module.api_v3.plugin_store_manager.fetch_registry_from_url.assert_not_called() + + def test_success_returns_the_plugin_list(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.fetch_registry_from_url.return_value = { + "plugins": [{"id": "clock"}]} + response = api_v3_client.post( + self.URL, json={"repo_url": "https://github.com/o/r"}) + assert response.status_code == 200 + body = response.get_json() + assert body["plugins"] == [{"id": "clock"}] + assert body["registry_url"] == "https://github.com/o/r" + + def test_url_is_trimmed_before_use(self, api_v3_client, api_v3_module): + manager = api_v3_module.api_v3.plugin_store_manager + manager.fetch_registry_from_url.return_value = {"plugins": []} + api_v3_client.post(self.URL, json={"repo_url": " https://github.com/o/r "}) + manager.fetch_registry_from_url.assert_called_once_with("https://github.com/o/r") + + def test_registry_without_plugins_key_returns_empty_list( + self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.fetch_registry_from_url.return_value = { + "other": 1} + response = api_v3_client.post(self.URL, json={"repo_url": "http://x"}) + assert response.get_json()["plugins"] == [] + + def test_no_registry_found_is_a_400(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.fetch_registry_from_url.return_value = None + response = api_v3_client.post(self.URL, json={"repo_url": "http://x/not-a-registry"}) + assert response.status_code == 400 + assert "Failed to fetch registry" in response.get_json()["message"] + + @pytest.mark.parametrize("url", [ + "not a url", + "javascript:alert(1)", + "file:///etc/passwd", + "http://localhost:8080/admin", + ]) + def test_unusable_urls_fail_cleanly(self, api_v3_client, api_v3_module, url): + # Characterization: the handler performs no URL validation of its + # own — whatever the manager makes of the URL decides the outcome. + # What is pinned here is that a rejected URL produces a clean 400 + # rather than a traceback or a 500. + api_v3_module.api_v3.plugin_store_manager.fetch_registry_from_url.return_value = None + response = api_v3_client.post(self.URL, json={"repo_url": url}) + assert response.status_code == 400 + assert "Traceback" not in str(response.get_json()) + + def test_fetch_exception_is_a_500_without_internals( + self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.fetch_registry_from_url.side_effect = ( + ValueError("parse failed in /srv/app/internal.py")) + response = api_v3_client.post(self.URL, json={"repo_url": "http://x"}) + assert response.status_code == 500 + body = response.get_json() + assert body["message"] == "An error occurred; see logs for details" + assert "Traceback" not in str(body) + + def test_non_string_repo_url_is_rejected(self, api_v3_client, api_v3_module): + # Regression: .strip() on a non-string raised, and the catch-all + # reported the caller's own mistake as a server fault. + response = api_v3_client.post(self.URL, json={"repo_url": 12345}) + assert response.status_code == 400 + api_v3_module.api_v3.plugin_store_manager.fetch_registry_from_url.assert_not_called() + + def test_blank_repo_url_is_rejected(self, api_v3_client, api_v3_module): + response = api_v3_client.post(self.URL, json={"repo_url": " "}) + assert response.status_code == 400 + api_v3_module.api_v3.plugin_store_manager.fetch_registry_from_url.assert_not_called() diff --git a/test/test_api_v3_wifi_endpoints.py b/test/test_api_v3_wifi_endpoints.py new file mode 100644 index 00000000..0c713ae2 --- /dev/null +++ b/test/test_api_v3_wifi_endpoints.py @@ -0,0 +1,240 @@ +""" +Endpoint tests for the /wifi/* routes in api_v3. + +These routes drive the host's actual networking — connecting, dropping a +connection, switching the radio off — and had no endpoint-level tests at +all. WiFiManager is mocked throughout; nothing here may touch real +networking. + +Each handler does `from src.wifi_manager import WiFiManager` inside the +function body, so the patch target is the class at its definition site. +""" + +import sys +from pathlib import Path +from unittest.mock import MagicMock, patch + +import pytest + +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from test._api_v3_test_helpers import api_v3_client, api_v3_module # noqa: F401,E402 + + +@pytest.fixture +def wifi_manager(): + """Patch WiFiManager where it is defined; yield the instance mock.""" + with patch("src.wifi_manager.WiFiManager") as cls: + instance = MagicMock() + cls.return_value = instance + yield instance + + +class TestConnect: + URL = "/api/v3/wifi/connect" + + def test_success(self, api_v3_client, wifi_manager): + wifi_manager.connect_to_network.return_value = (True, "Connected to HomeNet") + response = api_v3_client.post(self.URL, json={"ssid": "HomeNet", "password": "pw"}) + assert response.status_code == 200 + assert response.get_json()["message"] == "Connected to HomeNet" + wifi_manager.connect_to_network.assert_called_once_with("HomeNet", "pw") + + def test_missing_body_rejected(self, api_v3_client, wifi_manager): + response = api_v3_client.post(self.URL, json={}) + assert response.status_code == 400 + wifi_manager.connect_to_network.assert_not_called() + + def test_missing_ssid_rejected(self, api_v3_client, wifi_manager): + response = api_v3_client.post(self.URL, json={"password": "pw"}) + assert response.status_code == 400 + assert "SSID is required" in response.get_json()["message"] + wifi_manager.connect_to_network.assert_not_called() + + @pytest.mark.parametrize("ssid", ["", " ", "\t"]) + def test_blank_ssid_rejected(self, api_v3_client, wifi_manager, ssid): + response = api_v3_client.post(self.URL, json={"ssid": ssid}) + assert response.status_code == 400 + wifi_manager.connect_to_network.assert_not_called() + + def test_ssid_is_trimmed(self, api_v3_client, wifi_manager): + wifi_manager.connect_to_network.return_value = (True, "ok") + api_v3_client.post(self.URL, json={"ssid": " HomeNet "}) + wifi_manager.connect_to_network.assert_called_once_with("HomeNet", "") + + def test_missing_password_becomes_empty_string(self, api_v3_client, wifi_manager): + wifi_manager.connect_to_network.return_value = (True, "ok") + api_v3_client.post(self.URL, json={"ssid": "OpenNet"}) + wifi_manager.connect_to_network.assert_called_once_with("OpenNet", "") + + def test_null_password_becomes_empty_string(self, api_v3_client, wifi_manager): + wifi_manager.connect_to_network.return_value = (True, "ok") + api_v3_client.post(self.URL, json={"ssid": "OpenNet", "password": None}) + wifi_manager.connect_to_network.assert_called_once_with("OpenNet", "") + + def test_failure_reports_the_managers_reason(self, api_v3_client, wifi_manager): + wifi_manager.connect_to_network.return_value = (False, "Bad password") + response = api_v3_client.post(self.URL, json={"ssid": "HomeNet"}) + assert response.status_code == 400 + assert response.get_json()["message"] == "Bad password" + + def test_failure_without_reason_uses_fallback_text(self, api_v3_client, wifi_manager): + wifi_manager.connect_to_network.return_value = (False, None) + response = api_v3_client.post(self.URL, json={"ssid": "HomeNet"}) + assert response.status_code == 400 + assert response.get_json()["message"] == "Failed to connect to network" + + def test_manager_exception_is_a_500_without_leaking_internals( + self, api_v3_client, wifi_manager): + wifi_manager.connect_to_network.side_effect = RuntimeError( + "/usr/lib/secret/path blew up") + response = api_v3_client.post(self.URL, json={"ssid": "HomeNet"}) + assert response.status_code == 500 + body = response.get_json() + assert body["message"] == "An error occurred; see logs for details" + # `details` comes from describe_exception, which is deliberately + # safe to return (redacted, capped) — it names the type. + assert "RuntimeError" in body["details"] + + +class TestDisconnect: + URL = "/api/v3/wifi/disconnect" + + def test_success(self, api_v3_client, wifi_manager): + wifi_manager.disconnect_from_network.return_value = (True, "Disconnected") + response = api_v3_client.post(self.URL) + assert response.status_code == 200 + assert response.get_json()["message"] == "Disconnected" + + def test_failure(self, api_v3_client, wifi_manager): + wifi_manager.disconnect_from_network.return_value = (False, "Not connected") + response = api_v3_client.post(self.URL) + assert response.status_code == 400 + assert response.get_json()["message"] == "Not connected" + + def test_failure_without_reason_uses_fallback(self, api_v3_client, wifi_manager): + wifi_manager.disconnect_from_network.return_value = (False, "") + response = api_v3_client.post(self.URL) + assert response.get_json()["message"] == "Failed to disconnect from network" + + def test_exception_is_a_500(self, api_v3_client, wifi_manager): + wifi_manager.disconnect_from_network.side_effect = OSError("nmcli missing") + assert api_v3_client.post(self.URL).status_code == 500 + + +class TestApMode: + ENABLE = "/api/v3/wifi/ap/enable" + DISABLE = "/api/v3/wifi/ap/disable" + + def test_enable_success(self, api_v3_client, wifi_manager): + wifi_manager.enable_ap_mode.return_value = (True, "AP enabled") + response = api_v3_client.post(self.ENABLE, json={}) + assert response.status_code == 200 + wifi_manager.enable_ap_mode.assert_called_once_with(force=False) + + @pytest.mark.parametrize("raw,expected", [ + (True, True), (False, False), + ("true", True), ("TRUE", True), ("1", True), + ("false", False), ("no", False), ("yes", False), + (1, False), # only real True or the listed strings count + ]) + def test_force_coercion(self, api_v3_client, wifi_manager, raw, expected): + wifi_manager.enable_ap_mode.return_value = (True, "ok") + api_v3_client.post(self.ENABLE, json={"force": raw}) + wifi_manager.enable_ap_mode.assert_called_once_with(force=expected) + + def test_enable_without_body(self, api_v3_client, wifi_manager): + wifi_manager.enable_ap_mode.return_value = (True, "ok") + assert api_v3_client.post(self.ENABLE).status_code == 200 + + def test_enable_failure(self, api_v3_client, wifi_manager): + wifi_manager.enable_ap_mode.return_value = (False, "hostapd missing") + response = api_v3_client.post(self.ENABLE, json={}) + assert response.status_code == 400 + assert response.get_json()["message"] == "hostapd missing" + + def test_disable_success(self, api_v3_client, wifi_manager): + wifi_manager.disable_ap_mode.return_value = (True, "AP disabled") + assert api_v3_client.post(self.DISABLE).status_code == 200 + + def test_disable_failure(self, api_v3_client, wifi_manager): + wifi_manager.disable_ap_mode.return_value = (False, "not running") + assert api_v3_client.post(self.DISABLE).status_code == 400 + + def test_enable_exception_is_a_500(self, api_v3_client, wifi_manager): + wifi_manager.enable_ap_mode.side_effect = RuntimeError("boom") + assert api_v3_client.post(self.ENABLE, json={}).status_code == 500 + + +class TestRadio: + URL = "/api/v3/wifi/radio" + + def test_get_state(self, api_v3_client, wifi_manager): + wifi_manager.get_wifi_radio_state.return_value = { + "enabled": True, "ethernet_connected": False} + response = api_v3_client.get(self.URL) + assert response.status_code == 200 + assert response.get_json()["data"]["enabled"] is True + + def test_get_state_exception_is_a_500(self, api_v3_client, wifi_manager): + wifi_manager.get_wifi_radio_state.side_effect = OSError("rfkill missing") + assert api_v3_client.get(self.URL).status_code == 500 + + def test_enabled_is_required(self, api_v3_client, wifi_manager): + response = api_v3_client.post(self.URL, json={}) + assert response.status_code == 400 + assert "enabled is required" in response.get_json()["message"] + wifi_manager.set_wifi_radio.assert_not_called() + + def test_enable_success(self, api_v3_client, wifi_manager): + wifi_manager.set_wifi_radio.return_value = (True, "Radio on", None) + wifi_manager.get_wifi_radio_state.return_value = {"enabled": True} + response = api_v3_client.post(self.URL, json={"enabled": True}) + assert response.status_code == 200 + wifi_manager.set_wifi_radio.assert_called_once_with(True, force=False) + + @pytest.mark.parametrize("raw,expected", [ + (True, True), ("true", True), ("1", True), ("yes", True), + (False, False), ("false", False), ("off", False), (0, False), + ]) + def test_enabled_coercion_is_string_aware( + self, api_v3_client, wifi_manager, raw, expected): + # bool("false") is True, so the endpoint parses strings explicitly + # rather than trusting truthiness — it is a public contract, not + # only the shipped UI which always sends real JSON booleans. + wifi_manager.set_wifi_radio.return_value = (True, "ok", None) + wifi_manager.get_wifi_radio_state.return_value = {} + api_v3_client.post(self.URL, json={"enabled": raw}) + wifi_manager.set_wifi_radio.assert_called_once_with(expected, force=False) + + def test_force_passed_through(self, api_v3_client, wifi_manager): + wifi_manager.set_wifi_radio.return_value = (True, "ok", None) + wifi_manager.get_wifi_radio_state.return_value = {} + api_v3_client.post(self.URL, json={"enabled": False, "force": "true"}) + wifi_manager.set_wifi_radio.assert_called_once_with(False, force=True) + + def test_refusal_reports_reason(self, api_v3_client, wifi_manager): + # Disabling the radio without Ethernet would lock the user out of + # this very interface, so the manager can refuse with a reason. + wifi_manager.set_wifi_radio.return_value = ( + False, "Refusing: no wired fallback", "no_ethernet") + response = api_v3_client.post(self.URL, json={"enabled": False}) + assert response.status_code == 400 + body = response.get_json() + assert body["reason"] == "no_ethernet" + assert "Refusing" in body["message"] + + def test_exception_is_a_500(self, api_v3_client, wifi_manager): + wifi_manager.set_wifi_radio.side_effect = RuntimeError("boom") + assert api_v3_client.post(self.URL, json={"enabled": True}).status_code == 500 + + +class TestNoRealNetworking: + def test_wifi_manager_is_never_constructed_for_real(self, api_v3_client): + # Guard against a future refactor moving the import to module level, + # where the fixture's patch of the definition site would stop + # applying and the tests would start driving real networking. + with patch("src.wifi_manager.WiFiManager") as cls: + cls.return_value.disconnect_from_network.return_value = (True, "ok") + api_v3_client.post("/api/v3/wifi/disconnect") + assert cls.called diff --git a/test/test_background_fetch_dedupe.py b/test/test_background_fetch_dedupe.py new file mode 100644 index 00000000..4398a474 --- /dev/null +++ b/test/test_background_fetch_dedupe.py @@ -0,0 +1,450 @@ +"""A second request for a key already being fetched must join, not duplicate. + +request_id embeds a millisecond timestamp and active_requests is keyed by it, +so every submit looked new and nothing compared what was actually being +fetched. On a real board the season-schedule cache_key is requested by both +the Recent and the Upcoming manager: they miss the cache in the same +millisecond and each start a full download and parse of the same payload. +Measured on a running board, 138 background fetches in 24 hours arriving in +pairs at identical timestamps -- half of them redundant. + +The cost of a duplicate is a second download, a second JSON parse (the +expensive part on a Pi), and a second parsed copy resident at the same time. +Schedules on that board run from 256KB to 20MB. It also consumes a second of +the three executor slots with identical work, which is what makes two large +parses peak simultaneously. +""" + +import threading +import time +from unittest.mock import MagicMock, Mock, patch + +import pytest +import requests + +from src.background_data_service import BackgroundDataService, FetchStatus + + +PAYLOAD = {"events": [{"id": f"g{i}"} for i in range(20)]} + + +@pytest.fixture +def cache(): + m = MagicMock() + m.get.return_value = None # always a miss: force the fetch path + m.set.return_value = None + return m + + +@pytest.fixture +def service(cache): + svc = BackgroundDataService(cache, max_workers=3, request_timeout=5) + yield svc + svc.shutdown(wait=False) + + +def _resp(): + r = Mock() + r.json.return_value = PAYLOAD + r.raise_for_status.return_value = None + return r + + +def _wait(service, req_id, timeout=5): + deadline = time.time() + timeout + while not service.is_request_complete(req_id) and time.time() < deadline: + time.sleep(0.02) + + +class _BlockingSession: + """Holds the first fetch open so a second can be submitted mid-flight.""" + + def __init__(self): + self.calls = 0 + self.release = threading.Event() + self.started = threading.Event() + + def get(self, *a, **k): + self.calls += 1 + self.started.set() + self.release.wait(timeout=5) + return _resp() + + +def test_a_second_submit_for_the_same_key_does_not_fetch_twice(service): + session = _BlockingSession() + with patch.object(service, "session", session): + first = service.submit_fetch_request( + sport="nba", year=2026, url="https://x/s", cache_key="nba_2026", + callback=lambda r: None, max_retries=0) + assert session.started.wait(timeout=5) + + second = service.submit_fetch_request( + sport="nba", year=2026, url="https://x/s", cache_key="nba_2026", + callback=lambda r: None, max_retries=0) + + assert second == first, "the joiner should share the in-flight request id" + session.release.set() + _wait(service, first) + + assert session.calls == 1, f"the payload was fetched {session.calls} times" + + +def test_the_joiner_still_gets_its_callback(service): + session = _BlockingSession() + seen = [] + with patch.object(service, "session", session): + first = service.submit_fetch_request( + sport="nba", year=2026, url="https://x/s", cache_key="k", + callback=lambda r: seen.append("first"), max_retries=0) + assert session.started.wait(timeout=5) + joined = service.submit_fetch_request( + sport="nba", year=2026, url="https://x/s", cache_key="k", + callback=lambda r: seen.append("second"), max_retries=0) + # Assert the coalescing happened, otherwise this passes trivially: + # two independent requests would each fire their own callback and the + # test would say nothing about the joined path. + assert joined == first + session.release.set() + _wait(service, first) + + deadline = time.time() + 5 + while len(seen) < 2 and time.time() < deadline: + time.sleep(0.02) + assert sorted(seen) == ["first", "second"], ( + f"both submitters must be called back, got {seen}") + + +def test_one_callback_raising_does_not_silence_the_other(service): + session = _BlockingSession() + seen = [] + + def boom(result): + raise RuntimeError("consumer blew up") + + with patch.object(service, "session", session): + first = service.submit_fetch_request( + sport="nba", year=2026, url="https://x/s", cache_key="k", + callback=boom, max_retries=0) + assert session.started.wait(timeout=5) + joined = service.submit_fetch_request( + sport="nba", year=2026, url="https://x/s", cache_key="k", + callback=lambda r: seen.append("survivor"), max_retries=0) + # Same reason: without coalescing these are separate requests and + # neither callback can affect the other. + assert joined == first + session.release.set() + _wait(service, first) + + deadline = time.time() + 5 + while not seen and time.time() < deadline: + time.sleep(0.02) + assert seen == ["survivor"] + + +def test_different_keys_are_not_coalesced(service): + session = _BlockingSession() + with patch.object(service, "session", session): + a = service.submit_fetch_request( + sport="nba", year=2026, url="https://x/a", cache_key="key_a", + callback=lambda r: None, max_retries=0) + assert session.started.wait(timeout=5) + b = service.submit_fetch_request( + sport="nhl", year=2026, url="https://x/b", cache_key="key_b", + callback=lambda r: None, max_retries=0) + assert a != b, "different cache keys must not share a request" + session.release.set() + _wait(service, a) + _wait(service, b) + assert session.calls == 2 + + +def test_a_later_submit_after_completion_fetches_again(service): + """Dedupe is for concurrent requests only, not a second cache layer.""" + with patch.object(service.session, "get", side_effect=[_resp(), _resp()]) as get: + first = service.submit_fetch_request( + sport="nba", year=2026, url="https://x/s", cache_key="k", + callback=lambda r: None, max_retries=0) + _wait(service, first) + second = service.submit_fetch_request( + sport="nba", year=2026, url="https://x/s", cache_key="k", + callback=lambda r: None, max_retries=0) + _wait(service, second) + assert first != second + assert get.call_count == 2 + + +def test_cancelling_releases_the_key(service): + """A cancelled request must not wedge its key against future fetches.""" + session = _BlockingSession() + with patch.object(service, "session", session): + first = service.submit_fetch_request( + sport="nba", year=2026, url="https://x/s", cache_key="k", + callback=lambda r: None, max_retries=0) + assert session.started.wait(timeout=5) + service.cancel_request(first) + assert "k" not in service._inflight_by_cache_key + session.release.set() + + +def test_a_stranded_index_entry_cannot_wedge_a_key(service): + """Defensive: the request is looked up, not trusted from the id alone.""" + service._inflight_by_cache_key["ghost"] = "no_such_request" + with patch.object(service.session, "get", return_value=_resp()): + req = service.submit_fetch_request( + sport="nba", year=2026, url="https://x/s", cache_key="ghost", + callback=lambda r: None, max_retries=0) + _wait(service, req) + assert service.get_result(req).success is True + + +def test_the_deduplicated_count_is_reported(service): + session = _BlockingSession() + with patch.object(service, "session", session): + first = service.submit_fetch_request( + sport="nba", year=2026, url="https://x/s", cache_key="k", + callback=lambda r: None, max_retries=0) + assert session.started.wait(timeout=5) + service.submit_fetch_request( + sport="nba", year=2026, url="https://x/s", cache_key="k", + max_retries=0) + session.release.set() + _wait(service, first) + assert service.get_statistics().get("deduplicated_requests") == 1 + + +def test_a_cancelled_worker_cannot_overwrite_its_replacement(service, cache): + """Cancelling frees the key, so a replacement may already own it. + + The worker cannot abort an HTTP call in flight, so when the cancelled one + finally returns it must discard its response rather than write it. Without + that, the sequence is: cancel A, submit B for the same key, B fetches and + caches fresh data, A returns and overwrites it with the response nobody + wanted -- and calls A's callbacks too. + """ + slow = _BlockingSession() + stale = {"events": [{"id": "STALE"}]} + slow_resp = Mock() + slow_resp.json.return_value = stale + slow_resp.raise_for_status.return_value = None + + def blocked_get(*a, **k): + slow.calls += 1 + slow.started.set() + slow.release.wait(timeout=5) + return slow_resp + + called = [] + with patch.object(service.session, "get", side_effect=blocked_get): + first = service.submit_fetch_request( + sport="nba", year=2026, url="https://x/s", cache_key="k", + callback=lambda r: called.append("cancelled_one"), max_retries=0) + assert slow.started.wait(timeout=5) + + service.cancel_request(first) + assert "k" not in service._inflight_by_cache_key + + # The replacement writes the fresh value while the cancelled fetch is held. + fresh = {"events": [{"id": "FRESH"}]} + fresh_resp = Mock() + fresh_resp.json.return_value = fresh + fresh_resp.raise_for_status.return_value = None + with patch.object(service.session, "get", return_value=fresh_resp): + second = service.submit_fetch_request( + sport="nba", year=2026, url="https://x/s", cache_key="k", + callback=lambda r: called.append("replacement"), max_retries=0) + _wait(service, second) + + assert cache.set.call_args[0][1] == fresh, "replacement must own the cache" + + # Now let the cancelled fetch finish. It must write nothing and call nobody. + # Wait for the worker to actually finish rather than sleeping: a fixed + # sleep is a race under load, and a slow worker would make this pass for + # the wrong reason. A cancelled request is still filed in + # completed_requests, so that is the signal it has run to completion. + writes_before = cache.set.call_count + slow.release.set() + deadline = time.time() + 5 + while first not in service.completed_requests and time.time() < deadline: + time.sleep(0.02) + assert first in service.completed_requests, "cancelled worker never finished" + + assert cache.set.call_count == writes_before, ( + "the cancelled worker wrote to the cache after its replacement") + assert cache.set.call_args[0][1] == fresh, "stale data overwrote fresh" + assert "cancelled_one" not in called, ( + "a cancelled request must not deliver callbacks") + + +def test_request_ids_are_unique_within_a_millisecond(service): + """request_id was sport_year_milliseconds, which collides. + + Two submits inside the same millisecond produced the SAME id, so one + silently replaced the other in active_requests and completed_requests. + Dedupe hands this id back to every joiner as their handle for + get_result(), so uniqueness is now load-bearing rather than incidental. + """ + # Stub the executor rather than the session: this is about what submit + # hands back, and letting 50 workers loose would outlive the patch and + # make real network calls. + with patch.object(service.executor, "submit"): + ids = [ + service.submit_fetch_request( + sport="nba", year=2026, url="https://x/s", + cache_key=f"key_{i}", # distinct keys: no dedupe + callback=lambda r: None, max_retries=0) + for i in range(50) + ] + assert len(set(ids)) == len(ids), "request ids collided" + + +# --- cancellation must be terminal ------------------------------------------ +# +# Cancelling used to be advisory: three separate paths wrote request.status +# without checking whether the request had already been cancelled, so a cancel +# could be silently undone and the work it was meant to stop went ahead. + +URL = "http://example.invalid/scores" +KEY = "sched_nfl_2025" + + +class _CountingSession: + """Records whether an HTTP fetch was ever attempted.""" + + def __init__(self): + self.calls = 0 + + def get(self, *a, **k): + self.calls += 1 + return _resp() + + +class _BlockingFailingSession(_CountingSession): + """Holds the fetch open, then fails it. + + Cancelling while the worker is parked inside the HTTP call is the only way + to reach the exception handler as a cancelled request. Cancel it before + the call starts and the worker returns at the pre-start branch instead, + which would leave the except path untested. + """ + + def __init__(self): + super().__init__() + self.started = threading.Event() + self.release = threading.Event() + + def get(self, *a, **k): + self.calls += 1 + self.started.set() + self.release.wait(timeout=5) + raise requests.RequestException("connection reset") + + +def _fill_every_worker_slot(service, slots=3): + """Occupy the pool so the next submit is queued rather than started. + + This is what makes "cancel before the worker runs" deterministic instead + of a race the test would win only sometimes. Returns the gate that + releases the pool. + """ + gate = threading.Event() + for _ in range(slots): + service.executor.submit(gate.wait, 5) + return gate + + +def test_cancelling_before_the_worker_starts_stops_the_fetch(service, cache): + """The queued worker must honour a cancel, not overwrite it with IN_PROGRESS. + + Between submit and the worker picking the job up, the request sits in the + executor queue. Cancelling there is the cheapest possible cancel -- nothing + has been downloaded yet -- and it was the one that did not work. + """ + gate = _fill_every_worker_slot(service) + session = _CountingSession() + delivered = [] + + with patch.object(service, 'session', session): + rid = service.submit_fetch_request( + "nfl", 2025, URL, KEY, max_retries=0, callback=delivered.append + ) + assert service.cancel_request(rid) is True + gate.set() # let the queued worker run + _wait(service, rid) + + assert session.calls == 0, ( + "cancelled before it started, yet the worker still downloaded the payload" + ) + assert cache.set.call_count == 0, "a cancelled request wrote to the cache" + assert delivered == [], "a cancelled request invoked its callbacks" + + +def test_a_cancel_during_the_commit_is_refused(service, cache): + """Once the worker has claimed the commit, cancelling is too late. + + The claim and the cancelled-check happen in one critical section, so a + cancel arriving after it cannot retroactively abandon data already on its + way to the cache. Letting it through stranded every joiner: the payload + landed in the cache but the callbacks were suppressed, so a manager that + joined this fetch waited for a call that never came. + """ + gate = _fill_every_worker_slot(service) + late = {} + delivered = [] + + def cancel_mid_write(key, data, *a, **k): + late['returned'] = service.cancel_request(late['rid']) + + cache.set.side_effect = cancel_mid_write + + # Read the payload inside the callback. The service releases result.data + # once every callback has been delivered, so inspecting the FetchResult + # afterwards sees the released object, not what the caller was handed. + def record(result): + delivered.append((result.success, result.data)) + + with patch.object(service, 'session', _CountingSession()): + late['rid'] = service.submit_fetch_request( + "nfl", 2025, URL, KEY, max_retries=0, callback=record + ) + gate.set() # only now can the worker reach the commit + _wait(service, late['rid']) + + assert late.get('returned') is False, ( + "cancelled a request that had already committed" + ) + assert cache.set.call_count == 1, "the commit itself was lost" + assert delivered == [(True, PAYLOAD)], ( + "data reached the cache but the callbacks were suppressed -- " + "every joined submitter is left waiting forever" + ) + + +def test_a_failure_after_cancelling_stays_cancelled(service, cache): + """A cancelled request that then errors must not resurface as FAILED. + + The except path overwrote CANCELLED with FAILED, and the callback gate in + the finally block only suppresses callbacks for CANCELLED -- so cancelling + a request that was about to time out delivered a spurious error callback. + """ + session = _BlockingFailingSession() + delivered = [] + + with patch.object(service, 'session', session): + rid = service.submit_fetch_request( + "nfl", 2025, URL, KEY, max_retries=0, callback=delivered.append + ) + # Assert the worker is inside the HTTP call before cancelling, + # otherwise this silently degrades into the pre-start case and the + # exception handler is never exercised. + assert session.started.wait(timeout=5) + assert service.cancel_request(rid) is True + session.release.set() + _wait(service, rid) + + assert session.calls == 1, "the fetch never started, so nothing could fail" + + assert delivered == [], "a cancelled request delivered a failure callback" + assert service.get_request_status(rid) is FetchStatus.CANCELLED, ( + "a cancelled request that then errored was reported as FAILED" + ) diff --git a/test/test_background_payload_release.py b/test/test_background_payload_release.py new file mode 100644 index 00000000..2c90a8da --- /dev/null +++ b/test/test_background_payload_release.py @@ -0,0 +1,280 @@ +"""A delivered fetch payload must not stay resident on the stored result. + +BackgroundDataService kept the fetched body on the FetchResult it filed in +`completed_requests`, which is swept only hourly and capped at 500 entries by +count. For status records that is free; for a season schedule it is not. NCAA +football's 2026 schedule is 946 games, and on a 1GB Pi 3B+ the parsed payload +measured ~90MB -- a tenth of the board's memory, pinned for an hour after the +consumer had already been handed it. + +The cache-hit path was the worse of the two. It runs once per update interval +per sport, mints a fresh request_id each time, and hands back whatever the +cache returns -- so a memory-tier miss (the tier is capped at 150 entries) +re-parses the payload from disk into a genuinely new object. Those accumulate +as separate copies rather than shared references, which is the staircase seen +in the field: RSS stepping up ~90MB per sport as seasons loaded and never +coming back down. + +Releasing is safe because the payload is written to the cache under the +request's cache_key before the result is built, and that is where consumers +read it from -- the callback is handed the object directly and the plugins use +it only in passing before reading the cache back. + +Requests submitted *without* a callback keep their payload: polling +get_result() is then the only way to collect it, so releasing would break that +contract. + +And the release must happen after EVERY callback, not after each one. Callers +that joined an in-flight fetch share a single FetchResult, so releasing per +delivery strips the payload out from under everyone still queued -- see +TestJoinersAllGetTheData. +""" + +import threading +import time +import pytest +from unittest.mock import MagicMock, Mock, patch + +from src.background_data_service import BackgroundDataService + + +PAYLOAD = {"events": [{"id": f"g{i}"} for i in range(50)]} + + +@pytest.fixture +def cache(): + m = MagicMock() + m.get.return_value = None + m.set.return_value = None + return m + + +@pytest.fixture +def service(cache): + svc = BackgroundDataService(cache, max_workers=2, request_timeout=5) + yield svc + svc.shutdown(wait=False) + + +def _wait(service, req_id, timeout=5): + """Wait for the result to be FILED. + + Enough for anything that is true by the time the worker stores the result: + its success flag, its error, the cache write that happened during the + fetch. + """ + deadline = time.time() + timeout + while not service.is_request_complete(req_id) and time.time() < deadline: + time.sleep(0.02) + + +def _wait_for_release(service, req_id, timeout=5): + """Wait for the payload to be RELEASED, which is strictly later. + + The worker files the result, then runs the callback, then releases. So + is_request_complete() goes true while the callback still has not run -- + waiting on it alone leaves a window in which `seen` is empty and the + payload is still resident, and the assertions race the worker. It passes + in practice only because a one-line callback usually beats the 20ms poll. + + Release happens after the callback returns, so a released payload also + means the callback has finished: one wait covers both. + """ + deadline = time.time() + timeout + while time.time() < deadline: + result = service.get_result(req_id) + if result is not None and result.data is None: + return + time.sleep(0.02) + raise AssertionError( + f"payload for {req_id} was never released (callback may not have run)") + + +def _resp(): + r = Mock() + r.json.return_value = PAYLOAD + r.raise_for_status.return_value = None + return r + + +class TestFetchPath: + def test_callback_receives_the_payload_then_it_is_released(self, service, cache): + seen = {} + + def callback(result): + # The consumer's one look at the data happens here. + seen['events'] = len(result.data['events']) + + with patch.object(service.session, "get", return_value=_resp()): + req_id = service.submit_fetch_request( + sport="ncaa_fb", year=2026, url="https://example.com/s", + cache_key="ncaa_fb_2026", callback=callback, max_retries=0, + ) + _wait_for_release(service, req_id) + + assert seen['events'] == 50, "callback must still be handed the payload" + + stored = service.get_result(req_id) + assert stored is not None + assert stored.success is True + assert stored.data is None, "payload must not stay on the stored result" + + def test_nothing_is_lost_the_cache_holds_it(self, service, cache): + with patch.object(service.session, "get", return_value=_resp()): + req_id = service.submit_fetch_request( + sport="ncaa_fb", year=2026, url="https://example.com/s", + cache_key="ncaa_fb_2026", callback=lambda r: None, max_retries=0, + ) + _wait(service, req_id) + + cache.set.assert_called_once() + key, written = cache.set.call_args[0][:2] + assert key == "ncaa_fb_2026" + assert written == PAYLOAD, "the payload must be persisted before release" + + def test_without_a_callback_the_payload_is_kept(self, service, cache): + # Polling get_result() is then the only delivery mechanism. + with patch.object(service.session, "get", return_value=_resp()): + req_id = service.submit_fetch_request( + sport="nfl", year=2026, url="https://example.com/s", + cache_key="nfl_2026", max_retries=0, + ) + _wait(service, req_id) + + assert service.get_result(req_id).data == PAYLOAD + + def test_a_failed_fetch_still_records_its_error(self, service, cache): + with patch.object(service.session, "get", side_effect=Exception("boom")): + req_id = service.submit_fetch_request( + sport="nfl", year=2026, url="https://example.com/s", + cache_key="nfl_2026", callback=lambda r: None, max_retries=0, + ) + _wait(service, req_id) + + stored = service.get_result(req_id) + assert stored.success is False + assert stored.error is not None + + +class TestCacheHitPath: + def test_cache_hit_releases_after_the_callback(self, service, cache): + cache.get.return_value = PAYLOAD + seen = {} + + req_id = service.submit_fetch_request( + sport="ncaa_fb", year=2026, url="https://example.com/s", + cache_key="ncaa_fb_2026", + callback=lambda r: seen.update(events=len(r.data['events'])), + ) + + assert seen['events'] == 50 + assert service.get_result(req_id).data is None + + def test_repeated_cache_hits_do_not_accumulate_payloads(self, service, cache): + # The staircase: one entry per update interval per sport, each one + # potentially a freshly parsed copy after a memory-tier miss. + cache.get.return_value = PAYLOAD + + for _ in range(25): + service.submit_fetch_request( + sport="ncaa_fb", year=2026, url="https://example.com/s", + cache_key="ncaa_fb_2026", callback=lambda r: None, + ) + + retained = [r for r in service.completed_requests.values() if r.data is not None] + assert retained == [], f"{len(retained)} payloads still resident" + + def test_cache_hit_without_a_callback_is_unchanged(self, service, cache): + cache.get.return_value = PAYLOAD + req_id = service.submit_fetch_request( + sport="nfl", year=2026, url="https://example.com/s", + cache_key="nfl_2026", + ) + assert service.get_result(req_id).data == PAYLOAD + + +class TestJoinersAllGetTheData: + """Deduplicated callers share one FetchResult; releasing between them + empties it for the rest. + + Not a corner case. A sport's recent, upcoming and live managers all ask for + the same season schedule, so the second and third are joiners on almost + every cycle. Releasing inside the delivery loop handed the payload to + whichever ran first and gave the others `result.data is None`. + + The consequence was worse than a quiet degradation, because consumers do + `result.data.get('events')`: they raised AttributeError, the delivery loop + caught it, and the whole failure surfaced as a single + "Error in callback for request ..." line while that manager silently never + received its schedule. + """ + + class _BlockingSession: + """Holds the fetch open so a second submit lands while in flight.""" + + def __init__(self): + self.release = threading.Event() + self.started = threading.Event() + + def get(self, *a, **k): + self.started.set() + self.release.wait(timeout=5) + return _resp() + + def test_every_joiner_is_handed_the_payload(self, service): + session = self._BlockingSession() + seen = {} + + def record(name): + # Read it the way the sport managers do. `result.data['events']` + # would raise TypeError on None; `.get` raises AttributeError, + # which is the error actually seen in the field. + def cb(result): + seen[name] = result.data.get('events') if result.data else None + return cb + + with patch.object(service, "session", session): + first = service.submit_fetch_request( + sport="nhl", year=2026, url="https://x/s", cache_key="nhl_2026", + callback=record("first"), max_retries=0) + assert session.started.wait(timeout=5) + + joined = service.submit_fetch_request( + sport="nhl", year=2026, url="https://x/s", cache_key="nhl_2026", + callback=record("second"), max_retries=0) + # Without coalescing these are two independent fetches that each + # own their result, and the test proves nothing about sharing. + assert joined == first, "the joiner should share the in-flight id" + + session.release.set() + _wait(service, first) + + deadline = time.time() + 5 + while len(seen) < 2 and time.time() < deadline: + time.sleep(0.02) + + assert set(seen) == {"first", "second"}, f"both must be called, got {seen}" + for name, events in seen.items(): + assert events is not None, ( + f"{name!r} was handed a released payload: the result was " + f"emptied before every callback had been delivered") + assert len(events) == 50, f"{name!r} got {events!r}" + + def test_the_payload_is_still_released_once_they_have_all_had_it(self, service): + """The memory fix must survive the ordering fix.""" + session = self._BlockingSession() + + with patch.object(service, "session", session): + first = service.submit_fetch_request( + sport="nhl", year=2026, url="https://x/s", cache_key="nhl_2026", + callback=lambda r: None, max_retries=0) + assert session.started.wait(timeout=5) + service.submit_fetch_request( + sport="nhl", year=2026, url="https://x/s", cache_key="nhl_2026", + callback=lambda r: None, max_retries=0) + session.release.set() + _wait_for_release(service, first) + + stored = service.get_result(first) + assert stored is not None and stored.data is None, ( + "the payload must still be dropped once every callback has run") diff --git a/test/test_backup_manager.py b/test/test_backup_manager.py index fef10d82..e6533d69 100644 --- a/test/test_backup_manager.py +++ b/test/test_backup_manager.py @@ -3,6 +3,7 @@ from __future__ import annotations import json +import stat import zipfile from pathlib import Path @@ -41,6 +42,13 @@ def _make_project(root: Path) -> Path: json.dumps({"ap_mode": {"ssid": "LEDMatrix"}}), encoding="utf-8", ) + # Device-local auth that lives in config/ like the three above. It was + # omitted from backups, so a restore silently signed the user out of + # YouTube Music and they had to re-authenticate by hand. + (root / "config" / "ytm_auth.json").write_text( + json.dumps({"token": "YTM-TOKEN"}), + encoding="utf-8", + ) fonts = root / "assets" / "fonts" fonts.mkdir(parents=True) @@ -240,6 +248,10 @@ def test_restore_roundtrip(project: Path, empty_project: Path, tmp_path: Path) - restored_secrets = json.loads((empty_project / "config" / "config_secrets.json").read_text()) assert restored_secrets["ledmatrix-weather"]["api_key"] == "SECRET" + assert "ytm_auth" in result.restored + restored_ytm = json.loads((empty_project / "config" / "ytm_auth.json").read_text()) + assert restored_ytm["token"] == "YTM-TOKEN" + # User font restored, bundled font untouched. assert (empty_project / "assets" / "fonts" / "my-custom-font.ttf").read_bytes() == b"\x00\x01USER" assert (empty_project / "assets" / "fonts" / "5x7.bdf").read_text() == "BUNDLED" @@ -271,6 +283,10 @@ def test_restore_honors_options(project: Path, empty_project: Path, tmp_path: Pa assert result.plugins_to_install == [] assert "secrets" in result.skipped assert "wifi" in result.skipped + # ytm_auth rides on restore_wifi rather than its own flag -- disabling + # wifi restore must not leave a stale session token behind. + assert "ytm_auth" in result.skipped + assert not (empty_project / "config" / "ytm_auth.json").exists() def test_restore_rejects_malicious_zip(empty_project: Path, tmp_path: Path) -> None: @@ -282,3 +298,39 @@ def test_restore_rejects_malicious_zip(empty_project: Path, tmp_path: Path) -> N # validate_backup catches it before extraction. assert not result.success assert any("unsafe" in e.lower() for e in result.errors) + + +def test_restore_over_a_file_the_user_cannot_write( + project: Path, empty_project: Path, tmp_path: Path +) -> None: + """Restore must not need write permission on the destination *file*. + + Reproduces what a fresh install leaves behind: config files owned by root + and only group-readable, while the web interface that performs the restore + runs as a non-root user. shutil.copy2 opens the destination for writing and + failed with EACCES; writing alongside and renaming needs only directory + permission, which that account has. + + Simulated here by making the destination read-only — the owner cannot + open it for writing either, but can still replace it within its directory. + """ + zip_path = create_backup(project, output_dir=tmp_path / "exports") + + # Pre-existing, read-only destinations. + (empty_project / "config").mkdir(parents=True, exist_ok=True) + for name in ("config.json", "config_secrets.json", "wifi_config.json", "ytm_auth.json"): + target = empty_project / "config" / name + target.write_text("{}", encoding="utf-8") + target.chmod(0o444) + + result = restore_backup(zip_path, empty_project, RestoreOptions()) + + assert result.success, result.errors + for section in ("config", "secrets", "wifi", "ytm_auth"): + assert section in result.restored, f"{section} not restored: {result.errors}" + + restored = json.loads((empty_project / "config" / "config.json").read_text()) + assert restored["my-plugin"]["favorites"] == ["A", "B"] + + # The destination's mode is preserved rather than widened to the umask. + assert stat.S_IMODE((empty_project / "config" / "config_secrets.json").stat().st_mode) == 0o444 diff --git a/test/test_base_odds_manager.py b/test/test_base_odds_manager.py new file mode 100644 index 00000000..e53a5036 --- /dev/null +++ b/test/test_base_odds_manager.py @@ -0,0 +1,365 @@ +""" +Tests for src/base_odds_manager.py (BaseOddsManager). + +Covers get_odds validation/caching/URL construction, the null-safe +_extract_espn_data fix (ESPN sends explicit JSON nulls for absent sides), +the no_odds sentinel, stale-cache fallback on request failure, +is_odds_available's ML-blind truth table, the fixed format_odds_summary +gate (money-line-only odds now format), get_odds_for_games, and +configuration loading. + +No real network: requests.Session.get is always patched. The odds path sends +its requests through a session so it can identify itself to ESPN, so patching +the module-level requests.get would no longer intercept anything. +""" + +from unittest.mock import MagicMock, patch + +import pytest +import requests + +from src.base_odds_manager import BaseOddsManager + + +FULL_ITEM = { + 'details': 'DAL -3.5', + 'overUnder': 47.5, + 'spread': -3.5, + 'homeTeamOdds': {'moneyLine': -150, 'current': {'pointSpread': {'value': -3.5}}}, + 'awayTeamOdds': {'moneyLine': 130, 'current': {'pointSpread': {'value': 3.5}}}, +} + +FULL_EXTRACTED = { + 'details': 'DAL -3.5', + 'over_under': 47.5, + 'spread': -3.5, + 'home_team_odds': {'money_line': -150, 'spread_odds': -3.5}, + 'away_team_odds': {'money_line': 130, 'spread_odds': 3.5}, +} + + +def _make_response(payload): + response = MagicMock() + response.json.return_value = payload + response.raise_for_status.return_value = None + return response + + +@pytest.fixture +def cache_manager(): + cm = MagicMock() + # A bare MagicMock returns truthy Mocks from every call, so every + # get_odds() would look like a cache hit. Explicitly wire a miss. + cm.get_with_auto_strategy.return_value = None + return cm + + +@pytest.fixture +def manager(cache_manager): + return BaseOddsManager(cache_manager) + + +@pytest.fixture +def mock_get(): + with patch('src.base_odds_manager.requests.Session.get') as m: + m.return_value = _make_response({'items': [dict(FULL_ITEM)]}) + yield m + + +# --------------------------------------------------------------------------- +# get_odds +# --------------------------------------------------------------------------- + +class TestGetOdds: + def test_none_sport_raises(self, manager): + with pytest.raises(ValueError): + manager.get_odds(None, 'nfl', '1') + + def test_none_league_raises(self, manager): + with pytest.raises(ValueError): + manager.get_odds('football', None, '1') + + def test_cache_key_and_url(self, manager, cache_manager, mock_get): + manager.get_odds('football', 'nfl', '401') + + cache_manager.get_with_auto_strategy.assert_called_once_with( + 'odds_espn_football_nfl_401') + url = mock_get.call_args[0][0] + # Event id appears twice: /events//competitions//odds + assert '/events/401/competitions/401/odds' in url + assert url == ('https://sports.core.api.espn.com/v2/sports/football/' + 'leagues/nfl/events/401/competitions/401/odds') + # The number matters less than the property: a single stalled request + # must not be able to consume the plugin executor's 30s operation + # budget, since odds are fetched per live game inside update(). + assert mock_get.call_args.kwargs['timeout'] == 5 + assert mock_get.call_args.kwargs['timeout'] < 30 + + def test_ncaa_fb_maps_to_college_football(self, manager, mock_get): + manager.get_odds('football', 'ncaa_fb', '401') + + url = mock_get.call_args[0][0] + assert '/leagues/college-football/' in url + + def test_unknown_league_passes_through(self, manager, mock_get): + manager.get_odds('football', 'xfl', '401') + + assert '/leagues/xfl/' in mock_get.call_args[0][0] + + def test_cache_hit_skips_http(self, manager, cache_manager, mock_get): + cache_manager.get_with_auto_strategy.return_value = {'spread': -3.0} + + result = manager.get_odds('football', 'nfl', '401') + + assert result == {'spread': -3.0} + mock_get.assert_not_called() + + def test_cached_no_odds_sentinel_returned_verbatim( + self, manager, cache_manager, mock_get): + cache_manager.get_with_auto_strategy.return_value = {'no_odds': True} + + result = manager.get_odds('football', 'nfl', '401') + + assert result == {'no_odds': True} + mock_get.assert_not_called() + assert manager.is_odds_available(result) is False + + def test_success_caches_extracted_data_with_interval_ttl( + self, manager, cache_manager, mock_get): + result = manager.get_odds('football', 'nfl', '401', + update_interval_seconds=100) + + assert result == FULL_EXTRACTED + cache_manager.set.assert_called_once_with( + 'odds_espn_football_nfl_401', FULL_EXTRACTED, ttl=100) + + def test_no_odds_caches_sentinel(self, manager, cache_manager, mock_get): + mock_get.return_value = _make_response({'count': 0, 'items': []}) + + result = manager.get_odds('football', 'nfl', '401') + + assert result is None + cache_manager.set.assert_called_once_with( + 'odds_espn_football_nfl_401', {'no_odds': True}, ttl=3600) + + def test_zero_interval_falls_back_to_default( + self, manager, cache_manager, mock_get): + # Quirk pin: `update_interval_seconds or self.update_interval` + # treats an explicit 0 as falsy, so the 3600 default wins. + manager.get_odds('football', 'nfl', '401', update_interval_seconds=0) + + assert cache_manager.set.call_args.kwargs['ttl'] == 3600 + + def test_request_exception_falls_back_to_stale_cache( + self, manager, cache_manager, mock_get): + cache_manager.get_with_auto_strategy.side_effect = [ + None, {'stale': True}] + mock_get.side_effect = requests.exceptions.RequestException('boom') + + result = manager.get_odds('football', 'nfl', '401') + + assert result == {'stale': True} + assert cache_manager.get_with_auto_strategy.call_count == 2 + + +# --------------------------------------------------------------------------- +# _extract_espn_data +# --------------------------------------------------------------------------- + +class TestExtractEspnData: + def test_full_item_extracts_all_fields(self, manager): + result = manager._extract_espn_data({'items': [dict(FULL_ITEM)]}) + assert result == FULL_EXTRACTED + + def test_explicit_nulls_do_not_raise(self, manager): + # Post-fix: ESPN sends explicit JSON nulls for absent sides + # ("homeTeamOdds": null, "current": null); extraction must not + # raise and yields None fields. + payload = {'items': [{ + 'homeTeamOdds': None, + 'awayTeamOdds': {'moneyLine': 150, 'current': None}, + }]} + + result = manager._extract_espn_data(payload) + + assert result is not None + assert result['home_team_odds']['money_line'] is None + assert result['home_team_odds']['spread_odds'] is None + assert result['away_team_odds']['money_line'] == 150 + assert result['away_team_odds']['spread_odds'] is None + + def test_valid_empty_response_returns_none(self, manager): + assert manager._extract_espn_data({'count': 0, 'items': []}) is None + + def test_unexpected_structure_returns_none(self, manager): + assert manager._extract_espn_data({'unexpected': True}) is None + + def test_item_without_odds_fields_cached_as_data_not_sentinel( + self, manager, cache_manager, mock_get): + # Characterization pin: an item with no odds fields still extracts + # to a truthy dict of all-None values, so get_odds caches it as + # real data (NOT the no_odds sentinel) — but is_odds_available + # correctly reports False for it. + mock_get.return_value = _make_response({'items': [{}]}) + + result = manager.get_odds('football', 'nfl', '401') + + assert result == { + 'details': None, + 'over_under': None, + 'spread': None, + 'home_team_odds': {'money_line': None, 'spread_odds': None}, + 'away_team_odds': {'money_line': None, 'spread_odds': None}, + } + cache_manager.set.assert_called_once_with( + 'odds_espn_football_nfl_401', result, ttl=3600) + assert manager.is_odds_available(result) is False + + +# --------------------------------------------------------------------------- +# is_odds_available +# --------------------------------------------------------------------------- + +class TestIsOddsAvailable: + def test_none_is_false(self, manager): + assert manager.is_odds_available(None) is False + + def test_empty_dict_is_false(self, manager): + assert manager.is_odds_available({}) is False + + def test_no_odds_sentinel_is_false(self, manager): + assert manager.is_odds_available({'no_odds': True}) is False + + def test_spread_is_true(self, manager): + assert manager.is_odds_available({'spread': -3.5}) is True + + def test_over_under_is_true(self, manager): + assert manager.is_odds_available({'over_under': 47.5}) is True + + def test_nested_home_spread_odds_is_true(self, manager): + assert manager.is_odds_available( + {'home_team_odds': {'spread_odds': -3.5}}) is True + + def test_nested_away_spread_odds_is_true(self, manager): + assert manager.is_odds_available( + {'away_team_odds': {'spread_odds': 3.5}}) is True + + def test_moneyline_only_is_false(self, manager): + # Pinned ML-blind contract: is_odds_available ignores money lines + # (its callers decide whether to render an odds widget). Note that + # format_odds_summary deliberately uses a DIFFERENT gate — it will + # still format money-line-only odds (see TestFormatOddsSummary). + ml_only = { + 'home_team_odds': {'money_line': -120}, + 'away_team_odds': {'money_line': 100}, + } + assert manager.is_odds_available(ml_only) is False + + +# --------------------------------------------------------------------------- +# format_odds_summary (fixed gate: empty / no_odds only) +# --------------------------------------------------------------------------- + +class TestFormatOddsSummary: + def test_moneyline_only_formats(self, manager): + result = manager.format_odds_summary({ + 'home_team_odds': {'money_line': -120}, + 'away_team_odds': {'money_line': 100}, + }) + assert result == 'Home ML: -120 | Away ML: 100' + + def test_full_data_formats_all_parts(self, manager): + result = manager.format_odds_summary(FULL_EXTRACTED) + assert result == 'Spread: -3.5 | O/U: 47.5 | Home ML: -150 | Away ML: 130' + + def test_none_is_no_odds(self, manager): + assert manager.format_odds_summary(None) == 'No odds available' + + def test_empty_dict_is_no_odds(self, manager): + assert manager.format_odds_summary({}) == 'No odds available' + + def test_no_odds_sentinel_is_no_odds(self, manager): + assert manager.format_odds_summary( + {'no_odds': True}) == 'No odds available' + + +# --------------------------------------------------------------------------- +# get_odds_for_games +# --------------------------------------------------------------------------- + +class TestGetOddsForGames: + def test_missing_fields_get_none_odds_without_http(self, manager, mock_get): + games = [ + {'sport': 'football'}, + {'league': 'nfl'}, + {'id': '9'}, + {}, + ] + + result = manager.get_odds_for_games(games) + + assert all(g['odds'] is None for g in result) + mock_get.assert_not_called() + + def test_per_game_exception_continues_loop(self, manager, monkeypatch): + def fake_get_odds(sport, league, event_id, + update_interval_seconds=None): + if event_id == 'bad': + raise RuntimeError('boom') + return {'spread': -1.0} + + monkeypatch.setattr(manager, 'get_odds', fake_get_odds) + games = [ + {'sport': 'football', 'league': 'nfl', 'id': 'bad'}, + {'sport': 'football', 'league': 'nfl', 'id': 'ok'}, + ] + + result = manager.get_odds_for_games(games) + + assert len(result) == 2 + assert result[0]['odds'] is None + assert result[1]['odds'] == {'spread': -1.0} + + def test_input_dicts_mutated_in_place_and_returned(self, manager, mock_get): + # Pin: get_odds_for_games mutates the caller's game dicts in place + # and returns the same objects, not copies. + game = {'sport': 'football', 'league': 'nfl', 'id': '401'} + + result = manager.get_odds_for_games([game]) + + assert result[0] is game + assert game['odds'] == FULL_EXTRACTED + + +# --------------------------------------------------------------------------- +# _load_configuration +# --------------------------------------------------------------------------- + +class TestLoadConfiguration: + def test_loads_values_from_config(self, cache_manager): + config_manager = MagicMock() + config_manager.get_config.return_value = { + 'base_odds_manager': { + 'update_interval': 100, + 'timeout': 5, + 'cache_ttl': 42, + } + } + + manager = BaseOddsManager(cache_manager, config_manager=config_manager) + + assert manager.update_interval == 100 + # Key/attr mismatch pin: the config key is 'timeout' but the + # attribute is request_timeout. + assert manager.request_timeout == 5 + assert manager.cache_ttl == 42 + + def test_get_config_raising_keeps_defaults(self, cache_manager): + config_manager = MagicMock() + config_manager.get_config.side_effect = RuntimeError('boom') + + manager = BaseOddsManager(cache_manager, config_manager=config_manager) + + assert manager.update_interval == 3600 + assert manager.request_timeout == 5 + assert manager.cache_ttl == 1800 diff --git a/test/test_base_plugin_duration.py b/test/test_base_plugin_duration.py new file mode 100644 index 00000000..af7fe976 --- /dev/null +++ b/test/test_base_plugin_duration.py @@ -0,0 +1,146 @@ +""" +Tests for BasePlugin.get_display_duration — ~100 lines of type coercion that +every plugin's rotation slot depends on, previously untested. + +The contract: a positive number wins wherever it comes from; everything else +falls through instance attr → config → the 15.0 default, logging on the way. +""" + +from unittest.mock import MagicMock + +import pytest + +from src.plugin_system.base_plugin import BasePlugin + + +class _MinimalPlugin(BasePlugin): + def update(self): + pass + + def display(self, force_clear=False): + pass + + +def make_plugin(config=None, instance_duration="__unset__"): + plugin = _MinimalPlugin( + plugin_id="duration-test", + config=config or {}, + display_manager=MagicMock(), + cache_manager=MagicMock(), + plugin_manager=MagicMock(), + ) + if instance_duration != "__unset__": + plugin.display_duration = instance_duration + return plugin + + +class TestInstanceVariable: + def test_positive_int_wins(self): + assert make_plugin(instance_duration=30).get_display_duration() == 30.0 + + def test_positive_float_wins(self): + assert make_plugin(instance_duration=12.5).get_display_duration() == 12.5 + + def test_returns_float_type(self): + result = make_plugin(instance_duration=30).get_display_duration() + assert isinstance(result, float) + + def test_numeric_string_wins(self): + assert make_plugin(instance_duration="25").get_display_duration() == 25.0 + + def test_zero_falls_through_to_config(self): + plugin = make_plugin(config={"display_duration": 20}, + instance_duration=0) + assert plugin.get_display_duration() == 20.0 + + def test_negative_falls_through_to_config(self): + plugin = make_plugin(config={"display_duration": 20}, + instance_duration=-5) + assert plugin.get_display_duration() == 20.0 + + def test_none_falls_through_to_config(self): + plugin = make_plugin(config={"display_duration": 20}, + instance_duration=None) + assert plugin.get_display_duration() == 20.0 + + def test_garbage_string_falls_through(self): + plugin = make_plugin(config={"display_duration": 20}, + instance_duration="abc") + assert plugin.get_display_duration() == 20.0 + + def test_non_positive_string_falls_through(self): + plugin = make_plugin(config={"display_duration": 20}, + instance_duration="0") + assert plugin.get_display_duration() == 20.0 + + def test_unexpected_type_falls_through(self): + plugin = make_plugin(config={"display_duration": 20}, + instance_duration=[30]) + assert plugin.get_display_duration() == 20.0 + + def test_bool_true_falls_through_like_any_non_number(self): + # bool is an int subclass, but a boolean is not a duration: True + # must NOT read as 1 second — it falls through to config/default. + assert make_plugin(instance_duration=True).get_display_duration() == 15.0 + + def test_bool_true_falls_through_to_config(self): + plugin = make_plugin(config={"display_duration": 20}, + instance_duration=True) + assert plugin.get_display_duration() == 20.0 + + def test_bool_false_still_falls_through(self): + plugin = make_plugin(config={"display_duration": 20}, + instance_duration=False) + assert plugin.get_display_duration() == 20.0 + + +class TestConfigFallback: + def test_config_number(self): + assert make_plugin({"display_duration": 20}).get_display_duration() == 20.0 + + def test_config_numeric_string(self): + assert make_plugin({"display_duration": "12.5"}).get_display_duration() == 12.5 + + def test_missing_config_uses_default(self): + assert make_plugin({}).get_display_duration() == 15.0 + + def test_config_zero_uses_default(self): + assert make_plugin({"display_duration": 0}).get_display_duration() == 15.0 + + def test_config_negative_uses_default(self): + assert make_plugin({"display_duration": -10}).get_display_duration() == 15.0 + + def test_config_garbage_string_uses_default(self): + assert make_plugin({"display_duration": "soon"}).get_display_duration() == 15.0 + + def test_config_unexpected_type_uses_default(self): + assert make_plugin({"display_duration": {"s": 5}}).get_display_duration() == 15.0 + + def test_config_none_uses_default(self): + assert make_plugin({"display_duration": None}).get_display_duration() == 15.0 + + def test_config_bool_uses_default(self): + assert make_plugin({"display_duration": True}).get_display_duration() == 15.0 + assert make_plugin({"display_duration": False}).get_display_duration() == 15.0 + + +class TestValidateConfigDuration: + # validate_config must agree with get_display_duration about what a + # valid duration is — a config it accepts must not then be rejected + # (or silently defaulted) when the duration is actually read. + + def test_positive_number_valid(self): + assert make_plugin({"display_duration": 20}).validate_config() is True + + def test_zero_and_negative_invalid(self): + assert make_plugin({"display_duration": 0}).validate_config() is False + assert make_plugin({"display_duration": -5}).validate_config() is False + + def test_bool_invalid(self): + # bool is an int subclass; True would otherwise pass as "positive + # number" here while get_display_duration rejects it. + assert make_plugin({"display_duration": True}).validate_config() is False + assert make_plugin({"display_duration": False}).validate_config() is False + + def test_missing_duration_valid(self): + assert make_plugin({}).validate_config() is True diff --git a/test/test_cache_cleanup_thread_ownership.py b/test/test_cache_cleanup_thread_ownership.py new file mode 100644 index 00000000..2efa38c6 --- /dev/null +++ b/test/test_cache_cleanup_thread_ownership.py @@ -0,0 +1,146 @@ +"""Tests that one cache directory gets one cleanup thread per process. + +The sweep lists a directory and deletes from it, so a second thread over the +same directory only duplicates the scan. Nothing enforced that: every +CacheManager started its own, and since the loop closes over `self`, a +discarded manager could never be collected -- its thread stayed alive and +re-scanned the same directory every 24 hours for the life of the process. + +On the dev rig a display process carried three, for one cache directory: + + 14:22:59.954 display_controller (the real one) + 14:22:59.973 startup validation, run 1 (discarded) + 14:23:01.055 startup validation, run 2 (discarded) + +Startup validation runs twice and built a throwaway manager each time, purely +to read a directory path. +""" + +import threading + +import pytest + +from src.cache_manager import CacheManager + + +@pytest.fixture(autouse=True) +def _clean_registry(): + CacheManager._cleanup_owners.clear() + yield + for owner in list(CacheManager._cleanup_owners.values()): + owner.stop_cleanup_thread() + CacheManager._cleanup_owners.clear() + + +def _live_cleanup_threads(): + return [t for t in threading.enumerate() + if t.name == 'DiskCacheCleanup' and t.is_alive()] + + +@pytest.fixture +def manager(tmp_path, monkeypatch): + """A CacheManager pinned to a temp dir, so tests never touch the real one.""" + monkeypatch.setattr(CacheManager, '_get_writable_cache_dir', + lambda self: str(tmp_path)) + return CacheManager + + +class TestOneThreadPerDirectory: + def test_a_single_manager_starts_one(self, manager): + before = len(_live_cleanup_threads()) + m = manager() + try: + assert len(_live_cleanup_threads()) == before + 1 + finally: + m.stop_cleanup_thread() + + def test_three_managers_still_start_one(self, manager): + # Exactly the rig's shape: the real manager plus two throwaways. + before = len(_live_cleanup_threads()) + managers = [manager() for _ in range(3)] + try: + assert len(_live_cleanup_threads()) == before + 1 + finally: + for m in managers: + m.stop_cleanup_thread() + + def test_the_first_one_owns_it(self, manager): + first, second = manager(), manager() + try: + assert CacheManager._cleanup_owners[first.cache_dir] is first + assert second._cleanup_thread is None + finally: + first.stop_cleanup_thread() + second.stop_cleanup_thread() + + def test_the_survivor_can_take_over(self, manager): + first = manager() + first.stop_cleanup_thread() + assert not _live_cleanup_threads() + + second = manager() + try: + # Ownership was released, so the directory is swept again rather + # than being left permanently unclaimed by a dead owner. + assert len(_live_cleanup_threads()) == 1 + assert CacheManager._cleanup_owners[second.cache_dir] is second + finally: + second.stop_cleanup_thread() + + def test_stopping_a_non_owner_does_not_unclaim_the_directory(self, manager): + first, second = manager(), manager() + try: + second.stop_cleanup_thread() # never owned it + assert CacheManager._cleanup_owners[first.cache_dir] is first + assert len(_live_cleanup_threads()) == 1 + finally: + first.stop_cleanup_thread() + + def test_separate_directories_get_separate_threads(self, tmp_path, monkeypatch): + a, b = tmp_path / 'a', tmp_path / 'b' + a.mkdir() + b.mkdir() + dirs = iter([str(a), str(b)]) + monkeypatch.setattr(CacheManager, '_get_writable_cache_dir', + lambda self: next(dirs)) + first, second = CacheManager(), CacheManager() + try: + assert first.cache_dir != second.cache_dir + assert len(_live_cleanup_threads()) == 2 + finally: + first.stop_cleanup_thread() + second.stop_cleanup_thread() + + def test_no_thread_leaks_across_many_constructions(self, manager): + before = len(_live_cleanup_threads()) + made = [manager() for _ in range(12)] + try: + assert len(_live_cleanup_threads()) == before + 1 + finally: + for m in made: + m.stop_cleanup_thread() + assert len(_live_cleanup_threads()) == before + + +class TestValidatorDoesNotBuildItsOwn: + def test_it_uses_the_cache_manager_it_is_given(self, manager): + from src.startup_validator import StartupValidator + + shared = manager() + try: + before = len(_live_cleanup_threads()) + v = StartupValidator(config_manager=object(), cache_manager=shared) + v._validate_cache_directory() + assert len(_live_cleanup_threads()) == before, ( + "validation started another cleanup thread") + finally: + shared.stop_cleanup_thread() + + def test_without_one_it_cleans_up_after_itself(self, manager): + from src.startup_validator import StartupValidator + + before = len(_live_cleanup_threads()) + v = StartupValidator(config_manager=object()) + v._validate_cache_directory() + assert len(_live_cleanup_threads()) == before, ( + "the fallback manager left its cleanup thread running") diff --git a/test/test_cache_manager.py b/test/test_cache_manager.py index 8a93a39d..9d1195a8 100644 --- a/test/test_cache_manager.py +++ b/test/test_cache_manager.py @@ -458,3 +458,26 @@ class TestDiskCacheWriteEconomy: cache = DiskCache(cache_dir=str(tmp_path)) cache.set("k", {"when": datetime(2026, 7, 12, 10, 30)}) assert cache.get("k") == {"when": "2026-07-12T10:30:00"} + + +# --- the ceiling has to hold between cleanup sweeps --------------------------- + +def test_memory_cache_enforces_ceiling_on_every_write(): + """_cleanup_memory_cache only runs every cleanup_interval seconds (300 by + default). If set() accepted entries without bound in between, a burst could + take the cache far past max_size -- which is the unbounded growth the limit + exists to prevent, and on a 1GB board the difference between a bounded cache + and a Pi that cannot fork. + """ + from src.cache.memory_cache import MemoryCache + + cache = MemoryCache(max_size=150, cleanup_interval=300.0) + for i in range(1000): + cache.set(f"k{i}", {"v": i}) + + assert len(cache._cache) <= 150 + # The timestamp map has to be evicted alongside the values, or it becomes + # the leak instead. + assert len(cache._timestamps) <= 150 + assert cache.get("k999") is not None, "the newest write must survive" + assert cache.get("k0") is None, "the oldest must be the one evicted" diff --git a/test/test_cache_orphan_temp_sweep.py b/test/test_cache_orphan_temp_sweep.py new file mode 100644 index 00000000..f35579d6 --- /dev/null +++ b/test/test_cache_orphan_temp_sweep.py @@ -0,0 +1,188 @@ +"""Tests that abandoned cache temp files get collected. + +DiskCache.set() writes through tempfile.mkstemp and os.replace, removing its +own temp file in a finally. That covers a failed write, but not a process that +dies between the two -- a SIGKILL, a lost restart race, a power cut, all +ordinary on a Pi. Nothing collected what was left behind: the temp names are +"..json.", and the expiry sweep only listed names ending in +.json, so they accumulated for as long as the card had been in service. + +Measured on a live rig before this fix: 76 orphans totalling 1,050 MB -- 81% +of the entire cache directory -- the oldest six months old. + +The predicate that decides what to delete is tested harder than the sweep +itself, because a false positive here destroys real data. +""" + +import os +import time + +import pytest + +from src.cache.disk_cache import DiskCache, _ORPHAN_TEMP_MAX_AGE_SECONDS + + +class FakeStrategy: + @staticmethod + def get_data_type_from_key(key): + return 'default' + + +POLICIES = {'default': 30} + + +@pytest.fixture +def cache(tmp_path): + return DiskCache(str(tmp_path)) + + +def _age(path, seconds): + old = time.time() - seconds + os.utime(path, (old, old)) + + +def _write(tmp_path, name, body='{}'): + p = tmp_path / name + p.write_text(body, encoding='utf-8') + return p + + +class TestWhatCountsAsAnOrphan: + @pytest.mark.parametrize('name', [ + '.weather.json.a1b2c3d4', + '.odds_espn_football_nfl_401.json.xyz00000', + '.a.json.b', + ]) + def test_our_temp_files_are_orphans(self, name): + assert DiskCache._is_orphaned_temp(name) + + @pytest.mark.parametrize('name', [ + 'weather.json', # real data + '.weather.json', # a dotted key that completed + '.gitignore', # not ours + '.hidden', # not ours + 'weather.json.bak', # no leading dot: someone else's + '.json.abc', # no key between the dot and .json. + '.weather.json.', # no random component + 'notes.txt', + ]) + def test_everything_else_is_left_alone(self, name): + assert not DiskCache._is_orphaned_temp(name) + + def test_the_names_set_actually_creates_are_matched(self, cache, tmp_path): + """Guard against the predicate and the writer drifting apart.""" + created = [] + real = os.replace + + def capture(src, dst): + created.append(os.path.basename(src)) + return real(src, dst) + + import src.cache.disk_cache as mod + mod.os.replace = capture + try: + cache.set('weather', {'v': 1}) + finally: + mod.os.replace = real + + assert created, "set() did not go through the temp-file path" + assert all(DiskCache._is_orphaned_temp(n) for n in created), created + + +class TestTheSweep: + def test_an_old_orphan_is_removed(self, cache, tmp_path): + p = _write(tmp_path, '.weather.json.a1b2c3d4', 'x' * 5000) + _age(p, _ORPHAN_TEMP_MAX_AGE_SECONDS + 60) + + stats = cache.cleanup_expired_files(FakeStrategy(), POLICIES) + + assert not p.exists() + assert stats['orphan_temp_files_deleted'] == 1 + assert stats['space_freed_bytes'] >= 5000 + + def test_an_in_flight_write_is_not_snatched_away(self, cache, tmp_path): + # The whole risk of this sweep: deleting a temp file another thread is + # about to os.replace into place. + p = _write(tmp_path, '.weather.json.inflight') + + cache.cleanup_expired_files(FakeStrategy(), POLICIES) + + assert p.exists() + + def test_real_cache_files_survive(self, cache, tmp_path): + fresh = _write(tmp_path, 'weather.json') + dotted = _write(tmp_path, '.weather.json') + _age(dotted, _ORPHAN_TEMP_MAX_AGE_SECONDS + 60) + + cache.cleanup_expired_files(FakeStrategy(), POLICIES) + + assert fresh.exists() + assert dotted.exists(), "a completed .json was treated as a temp file" + + def test_unrelated_dotfiles_survive(self, cache, tmp_path): + keep = _write(tmp_path, '.gitignore') + _age(keep, 400 * 86400) + + cache.cleanup_expired_files(FakeStrategy(), POLICIES) + + assert keep.exists() + + def test_expiry_still_works_alongside_it(self, cache, tmp_path): + stale = _write(tmp_path, 'old.json') + _age(stale, 40 * 86400) # past the 30-day default + orphan = _write(tmp_path, '.old.json.zz999999') + _age(orphan, _ORPHAN_TEMP_MAX_AGE_SECONDS + 60) + + stats = cache.cleanup_expired_files(FakeStrategy(), POLICIES) + + assert not stale.exists() + assert not orphan.exists() + assert stats['files_deleted'] == 2 + assert stats['orphan_temp_files_deleted'] == 1 + + def test_the_rig_scenario(self, cache, tmp_path): + """76 orphans of assorted ages, none of them reachable before.""" + for i in range(76): + p = _write(tmp_path, '.sched_%d.json.r%06d' % (i, i), 'x' * 1000) + _age(p, (i + 2) * 86400) + keep = _write(tmp_path, 'sched.json') + + stats = cache.cleanup_expired_files(FakeStrategy(), POLICIES) + + assert stats['orphan_temp_files_deleted'] == 76 + assert keep.exists() + assert not list(tmp_path.glob('.sched_*')) + # The summary line is "/", so an orphan that is + # deleted but never counted as scanned renders as "76/1". + assert stats['files_scanned'] == 77 + assert stats['files_deleted'] <= stats['files_scanned'] + + def test_deleted_never_exceeds_scanned(self, cache, tmp_path): + p = _write(tmp_path, '.only.json.a1b2c3d4') + _age(p, _ORPHAN_TEMP_MAX_AGE_SECONDS + 60) + + stats = cache.cleanup_expired_files(FakeStrategy(), POLICIES) + + assert stats['files_deleted'] == 1 + assert stats['files_scanned'] == 1 + + def test_a_missing_file_mid_sweep_is_not_an_error(self, cache, tmp_path): + p = _write(tmp_path, '.weather.json.a1b2c3d4') + _age(p, _ORPHAN_TEMP_MAX_AGE_SECONDS + 60) + + import src.cache.disk_cache as mod + real = mod.os.path.getsize + + def vanish(path): + if path.endswith('.a1b2c3d4'): + os.remove(path) + raise FileNotFoundError(path) + return real(path) + + mod.os.path.getsize = vanish + try: + stats = cache.cleanup_expired_files(FakeStrategy(), POLICIES) + finally: + mod.os.path.getsize = real + + assert stats['errors'] == 0 diff --git a/test/test_cache_ttl_honoured.py b/test/test_cache_ttl_honoured.py new file mode 100644 index 00000000..080cbe2e --- /dev/null +++ b/test/test_cache_ttl_honoured.py @@ -0,0 +1,130 @@ +"""Tests that a per-entry ttl actually controls expiry. + +Regression under test: `CacheManager.set(key, data, ttl=...)` stored the value +and no read path ever consulted it. Expiry came from a `max_age` inferred from +substrings in the key ("live", "odds", "stock"), so every caller passing `ttl=` +-- 48 sites across the plugins and 4 in the core -- was writing a number that +did nothing. The old docstring admitted as much: "stored for compatibility but +expiration is still controlled via max_age when reading". + +Measured against a real device's cache (8,873 entries carrying a ttl), the +inferred value and the intended one disagreed almost everywhere: + + stocks max_age 600 vs ttl 1800 4903 entries + news max_age 3600 vs ttl 600 1770 entries + odds max_age 1800 vs ttl 3600 1301 entries + images max_age 300 vs ttl 2592000 20 entries + +No `sports_live` entry carries a ttl, so live scores keep their inferred +30-second freshness either way. +""" + +import time + +import pytest + +from src.cache.memory_cache import MemoryCache +from src.cache.disk_cache import DiskCache + + +@pytest.fixture +def disk(tmp_path): + return DiskCache(cache_dir=str(tmp_path)) + + +def _record(ttl=None, age=0.0): + rec = {"data": {"v": 1}, "timestamp": time.time() - age} + if ttl is not None: + rec["ttl"] = ttl + return rec + + +class TestDiskCacheHonoursTtl: + def test_ttl_longer_than_max_age_keeps_the_entry(self, disk): + # The odds case: written wanting an hour, expired at 30 minutes. + disk.set("odds_espn_football_nfl_401", _record(ttl=3600, age=1900)) + assert disk.get("odds_espn_football_nfl_401", max_age=1800) is not None + + def test_ttl_shorter_than_max_age_expires_the_entry(self, disk): + # The news case: written wanting 10 minutes, kept for an hour. + disk.set("news_NHL_1", _record(ttl=600, age=900)) + assert disk.get("news_NHL_1", max_age=3600) is None + + def test_without_a_ttl_max_age_still_applies(self, disk): + disk.set("plain_key", _record(age=400)) + assert disk.get("plain_key", max_age=300) is None + disk.set("plain_key2", _record(age=100)) + assert disk.get("plain_key2", max_age=300) is not None + + def test_a_fresh_entry_within_its_ttl_survives(self, disk): + disk.set("k", _record(ttl=600, age=10)) + assert disk.get("k", max_age=30) is not None + + def test_ttl_zero_expires_immediately(self, disk): + # 0 means zero seconds, not "forever" -- max_age=None is how a caller + # asks for no expiry. + disk.set("k", _record(ttl=0, age=1)) + assert disk.get("k", max_age=99999) is None + + @pytest.mark.parametrize("bad", ["600", None, True, False, -5, {"a": 1}]) + def test_a_nonsense_ttl_falls_back_to_max_age(self, disk, bad): + # Including bools: True is an int in Python and must not become a 1s ttl. + rec = _record(age=400) + rec["ttl"] = bad + disk.set("k_%s" % type(bad).__name__, rec) + assert disk.get("k_%s" % type(bad).__name__, max_age=300) is None + + +class TestMemoryCacheHonoursTtl: + def test_ttl_longer_than_max_age_keeps_the_entry(self): + m = MemoryCache() + m.set("k", _record(ttl=3600)) + m._timestamps["k"] = time.time() - 1900 + assert m.get("k", max_age=1800) is not None + + def test_ttl_shorter_than_max_age_expires_the_entry(self): + m = MemoryCache() + m.set("k", _record(ttl=600)) + m._timestamps["k"] = time.time() - 900 + assert m.get("k", max_age=3600) is None + + def test_without_a_ttl_max_age_still_applies(self): + m = MemoryCache() + m.set("k", _record()) + m._timestamps["k"] = time.time() - 400 + assert m.get("k", max_age=300) is None + + def test_both_layers_agree(self, tmp_path): + """A record must not be live in one layer and expired in the other.""" + rec = _record(ttl=3600, age=1900) + d = DiskCache(cache_dir=str(tmp_path)) + d.set("k", rec) + m = MemoryCache() + m.set("k", rec) + m._timestamps["k"] = rec["timestamp"] + assert (d.get("k", max_age=1800) is not None) == (m.get("k", max_age=1800) is not None) + + +class TestEndToEnd: + def test_set_then_get_respects_the_ttl(self, tmp_path, monkeypatch): + """The behaviour a caller of CacheManager.set(ttl=...) expects.""" + from src.cache_manager import CacheManager + + cm = CacheManager() + cm._disk_cache_component = DiskCache(cache_dir=str(tmp_path)) + cm._memory_cache_component = MemoryCache() + + cm.set("odds_espn_football_nfl_401", {"spread": 6.5}, ttl=3600) + + # Age the stored record past the inferred max_age for odds (1800s) but + # within the ttl the caller asked for. + path = cm._disk_cache_component.get_cache_path("odds_espn_football_nfl_401") + import json + rec = json.load(open(path)) + rec["timestamp"] = time.time() - 1900 + json.dump(rec, open(path, "w")) + cm._memory_cache_component.clear() if hasattr( + cm._memory_cache_component, "clear") else None + + got = cm.get_with_auto_strategy("odds_espn_football_nfl_401") + assert got is not None, "the ttl the caller asked for was ignored" diff --git a/test/test_checkbox_group_stale_values.py b/test/test_checkbox_group_stale_values.py new file mode 100644 index 00000000..c852372a --- /dev/null +++ b/test/test_checkbox_group_stale_values.py @@ -0,0 +1,113 @@ +"""A checkbox group must not post back options it cannot show. + +The enum that lets the widget draw checkboxes is also what validates the +saved value. When a league retires a team code -- OAK for the Athletics, ARI +for the Coyotes -- or a schema drops an option, a config that still holds the +old value has nothing to render for it. The value stayed in the hidden +``_data`` input regardless, because that input is seeded from the stored array +and only rebuilt by ``updateCheckboxGroupData()`` on change. Editing any other +field on that plugin therefore posted the stale value back, the schema +rejected it, and the save endpoint returned 400 +``CONFIG_VALIDATION_FAILED`` -- so the whole plugin became uneditable until +the user worked out which invisible entry was at fault. + +Runtime was never affected: plugin loading treats schema violations as +warn/degrade, and the stale code already matched no team. Only the web UI +blocked. + +These tests render the checkbox-group block lifted *out of the shipped +template*, following test_enum_option_labels.py, so they exercise the +production expression rather than a copy that could drift from it. +""" +import json +import re +from pathlib import Path + +from jinja2 import DictLoader, Environment + +PROJECT_ROOT = Path(__file__).resolve().parent.parent +CONFIG_FORM = (PROJECT_ROOT / 'web_interface' / 'templates' / 'v3' / 'partials' + / 'plugin_config.html') + +# The checkbox-group branch: from its `{% elif %}` guard through the sentinel +# hidden input that closes it. Anchored on the guard so the match cannot run on +# into a neighbouring widget branch. +BLOCK_RE = re.compile( + r"\{%\s*elif x_widget == 'checkbox-group'\s*%\}(.*?)" + r"", + re.S, +) + + +def _shipped_block() -> str: + """Return the live checkbox-group block lifted from plugin_config.html.""" + source = CONFIG_FORM.read_text(encoding='utf-8') + match = BLOCK_RE.search(source) + assert match, ( + 'could not find the checkbox-group block in plugin_config.html — the ' + 'template changed shape and this guard needs updating' + ) + block = match.group(1) + assert 'data-option-value' in block, 'extracted the wrong branch' + assert '{% elif' not in block, 'extraction ran past the checkbox-group branch' + return block + + +def _render(prop: dict, value=None) -> str: + env = Environment(loader=DictLoader({'f': _shipped_block()}), autoescape=True) + return env.get_template('f').render( + prop=prop, value=value, field_id='fid', full_key='k' + ) + + +def _submitted(html: str) -> list: + """The array the form will actually post: the hidden _data input.""" + match = re.search(r'id="fid_data"[^>]*\svalue=\'([^\']*)\'', html) + assert match, f'hidden _data input not found in:\n{html}' + return json.loads(match.group(1).replace(''', "'")) + + +def _checked(html: str) -> list: + return re.findall(r'data-option-value="([^"]+)"[^>]*checked', html) + + +MLB = {'type': 'array', 'items': {'type': 'string', 'enum': ['NYY', 'BOS', 'ATH']}, + 'x-widget': 'checkbox-group'} + + +def test_a_retired_code_is_not_posted_back() -> None: + """The regression: OAK became ATH, and OAK used to ride along on save.""" + html = _render(MLB, ['NYY', 'OAK']) + assert _submitted(html) == ['NYY'], 'stale value would still be submitted' + + +def test_the_dropped_value_is_named_rather_than_vanishing() -> None: + html = _render(MLB, ['NYY', 'OAK']) + assert 'OAK' in html + assert 'data-stale-options' in html + + +def test_valid_values_are_untouched_and_still_checked() -> None: + html = _render(MLB, ['NYY', 'ATH']) + assert _submitted(html) == ['NYY', 'ATH'] + assert sorted(_checked(html)) == ['ATH', 'NYY'] + assert 'data-stale-options' not in html + + +def test_an_all_stale_selection_clears_rather_than_blocking() -> None: + html = _render(MLB, ['OAK', 'SD']) + assert _submitted(html) == [] + + +def test_an_empty_enum_leaves_the_value_alone() -> None: + """No options means nothing to validate against — filtering would wipe it.""" + prop = {'type': 'array', 'items': {'type': 'string'}, 'x-widget': 'checkbox-group'} + html = _render(prop, ['ANYTHING', 'GOES']) + assert _submitted(html) == ['ANYTHING', 'GOES'] + + +def test_unset_value_falls_back_to_the_default() -> None: + prop = dict(MLB, default=['BOS']) + html = _render(prop, None) + assert _submitted(html) == ['BOS'] + assert _checked(html) == ['BOS'] diff --git a/test/test_compatibility.py b/test/test_compatibility.py new file mode 100644 index 00000000..34f88869 --- /dev/null +++ b/test/test_compatibility.py @@ -0,0 +1,271 @@ +""" +Tests for src/plugin_system/compatibility.py — the "can this plugin run on +this core?" gate used by both the plugin loader (advisory) and the store +manager (blocking at install/update time). + +This module had zero direct test coverage despite guarding every install. +These tests pin the documented contract: refuse only on evidence, resolve +every uncertain case (unparseable versions, missing fields, untrustworthy +core) to compatible. +""" + +import pytest + +from src.plugin_system.compatibility import ( + TRUSTWORTHY_FLOOR, + parse_semver, + _parse_strict, + _satisfies_range, + satisfies_compatible_versions, + declared_min_version, + check, +) + + +class TestParseSemver: + def test_plain_triplet(self): + assert parse_semver("1.2.3") == (1, 2, 3) + + def test_leading_v_tolerated(self): + assert parse_semver("v3.2.1") == (3, 2, 1) + + def test_prerelease_suffix_stripped(self): + # "3.2.0-rc1" must NOT parse as (3, 2, 1) — a release candidate must + # not rank above its own release. + assert parse_semver("3.2.0-rc1") == (3, 2, 0) + + def test_build_suffix_stripped(self): + # "3.2.0+build42" must NOT parse as (3, 2, 42). + assert parse_semver("3.2.0+build42") == (3, 2, 0) + + def test_two_part_version_pads_zero(self): + assert parse_semver("1.2") == (1, 2, 0) + + def test_one_part_version_pads_zeros(self): + assert parse_semver("2") == (2, 0, 0) + + def test_extra_parts_ignored(self): + assert parse_semver("1.2.3.4") == (1, 2, 3) + + def test_non_string_returns_none(self): + assert parse_semver(None) is None + assert parse_semver(123) is None + assert parse_semver((1, 2, 3)) is None + + def test_garbage_with_no_digits_is_lenient_zero(self): + # Documented leniency: digit-scraping yields (0, 0, 0) for pure + # garbage. Fine for a floor (0.0.0 never blocks), wrong for ranges — + # which is why ranges go through _parse_strict instead. + assert parse_semver("garbage") == (0, 0, 0) + + def test_whitespace_stripped(self): + assert parse_semver(" 1.2.3 ") == (1, 2, 3) + + +class TestParseStrict: + def test_accepts_real_versions(self): + assert _parse_strict("1.2.3") == (1, 2, 3) + assert _parse_strict("v1.2.3-rc1") == (1, 2, 3) + assert _parse_strict("2.0") == (2, 0, 0) + + def test_rejects_garbage(self): + assert _parse_strict("not-a-version") is None + assert _parse_strict("") is None + + def test_rejects_non_string(self): + assert _parse_strict(None) is None + + +class TestSatisfiesRange: + CORE = (3, 1, 0) + + @pytest.mark.parametrize("spec,expected", [ + (">=3.0.0", True), + (">=3.1.0", True), + (">=3.2.0", False), + ("<=3.1.0", True), + ("<=3.0.9", False), + (">3.0.9", True), + (">3.1.0", False), + ("<3.2.0", True), + ("<3.1.0", False), + ]) + def test_comparison_operators(self, spec, expected): + assert _satisfies_range(self.CORE, spec) is expected + + def test_tilde_allows_patch_only(self): + # ~3.1.0 means >=3.1.0, <3.2.0 + assert _satisfies_range((3, 1, 5), "~3.1.0") is True + assert _satisfies_range((3, 2, 0), "~3.1.0") is False + assert _satisfies_range((3, 0, 9), "~3.1.0") is False + + def test_caret_allows_minor_and_patch(self): + # ^3.1.0 means >=3.1.0, <4.0.0 + assert _satisfies_range((3, 9, 9), "^3.1.0") is True + assert _satisfies_range((4, 0, 0), "^3.1.0") is False + assert _satisfies_range((3, 0, 0), "^3.1.0") is False + + def test_bare_exact_version(self): + assert _satisfies_range((3, 1, 0), "3.1.0") is True + assert _satisfies_range((3, 1, 1), "3.1.0") is False + + def test_inclusive_dash_range(self): + assert _satisfies_range((2, 5, 0), "2.0.0 - 3.1.0") is True + assert _satisfies_range((2, 0, 0), "2.0.0 - 3.1.0") is True + assert _satisfies_range((3, 1, 0), "2.0.0 - 3.1.0") is True + assert _satisfies_range((3, 1, 1), "2.0.0 - 3.1.0") is False + + def test_unparseable_spec_returns_none_not_false(self): + # Garbage must read as "no evidence", never as a refusal — an + # unrecognised spelling must not cost a user a working install. + assert _satisfies_range(self.CORE, "banana") is None + assert _satisfies_range(self.CORE, ">=banana") is None + assert _satisfies_range(self.CORE, "") is None + assert _satisfies_range(self.CORE, "banana - 3.0.0") is None + + +class TestSatisfiesCompatibleVersions: + def test_any_entry_satisfying_wins(self): + manifest = {"compatible_versions": ["<1.0.0", ">=3.0.0"]} + assert satisfies_compatible_versions(manifest, (3, 1, 0)) is True + + def test_all_entries_failing_is_false(self): + manifest = {"compatible_versions": ["<1.0.0", "2.0.0 - 2.9.9"]} + assert satisfies_compatible_versions(manifest, (3, 1, 0)) is False + + def test_absent_field_returns_none(self): + assert satisfies_compatible_versions({}, (3, 1, 0)) is None + + def test_empty_list_returns_none(self): + assert satisfies_compatible_versions( + {"compatible_versions": []}, (3, 1, 0)) is None + + def test_non_list_returns_none(self): + assert satisfies_compatible_versions( + {"compatible_versions": ">=2.0.0"}, (3, 1, 0)) is None + + def test_all_unparseable_entries_returns_none(self): + manifest = {"compatible_versions": ["banana", 42, None]} + assert satisfies_compatible_versions(manifest, (3, 1, 0)) is None + + def test_mixed_parseable_and_garbage_uses_parseable(self): + manifest = {"compatible_versions": ["banana", ">=3.0.0"]} + assert satisfies_compatible_versions(manifest, (3, 1, 0)) is True + + +class TestDeclaredMinVersion: + def test_top_level_field(self): + assert declared_min_version({"min_ledmatrix_version": "2.1.0"}) == "2.1.0" + + def test_requires_dict_fallback(self): + manifest = {"requires": {"min_ledmatrix_version": "2.2.0"}} + assert declared_min_version(manifest) == "2.2.0" + + def test_versions_array_fallback(self): + manifest = {"versions": [{"ledmatrix_min_version": "2.3.0"}]} + assert declared_min_version(manifest) == "2.3.0" + + def test_versions_array_deprecated_spelling(self): + manifest = {"versions": [{"ledmatrix_min": "2.4.0"}]} + assert declared_min_version(manifest) == "2.4.0" + + def test_top_level_wins_over_versions_array(self): + manifest = { + "min_ledmatrix_version": "2.1.0", + "versions": [{"ledmatrix_min_version": "9.9.9"}], + } + assert declared_min_version(manifest) == "2.1.0" + + def test_requires_as_list_does_not_raise(self): + # A hand-edited manifest can carry `requires` as a list; this used to + # raise AttributeError and one malformed manifest would take down the + # whole install path. + assert declared_min_version({"requires": ["something"]}) is None + + def test_versions_as_dict_does_not_raise(self): + # Same for `versions` as a mapping (used to raise KeyError). + assert declared_min_version({"versions": {"0": {}}}) is None + + def test_nothing_declared_returns_none(self): + assert declared_min_version({}) is None + + +class TestCheck: + def test_compatible_when_nothing_declared(self): + assert check({}, "3.1.0") == (True, None) + + def test_min_version_blocks_older_core(self): + manifest = {"name": "Test Plugin", "min_ledmatrix_version": "3.2.0"} + ok, reason = check(manifest, "3.1.0") + assert ok is False + assert "3.2.0" in reason and "3.1.0" in reason + + def test_min_version_allows_equal_core(self): + manifest = {"min_ledmatrix_version": "3.1.0"} + assert check(manifest, "3.1.0") == (True, None) + + def test_compatible_versions_upper_bound_blocks(self): + # A range is the only field that can express "not compatible with + # newer cores" — it must win even when the floor passes. + manifest = { + "name": "Old Plugin", + "min_ledmatrix_version": "2.0.0", + "compatible_versions": ["2.0.0 - 2.9.9"], + } + ok, reason = check(manifest, "3.1.0") + assert ok is False + assert "2.0.0 - 2.9.9" in reason + + def test_unparseable_core_with_high_floor_is_blocked(self): + manifest = {"min_ledmatrix_version": "3.2.0", + "compatible_versions": [">=3.2.0"]} + # An unparseable core version is "unknown", not "old"... but note + # parse_semver("garbage") == (0,0,0) which is below TRUSTWORTHY_FLOOR, + # so this rides the untrustworthy-core branch: floor > 2.0.0 blocks. + ok, reason = check(manifest, "garbage") + assert ok is False + assert "too old to identify reliably" in reason + + def test_untrustworthy_core_allows_ecosystem_baseline_floor(self): + # A core reporting 1.0.0 may really be v3.1.0 (which shipped with a + # wrong __version__). Floors at or below TRUSTWORTHY_FLOOR must not + # block, or that population could install nothing. + manifest = {"min_ledmatrix_version": "2.0.0", + "compatible_versions": [">=2.0.0"]} + assert check(manifest, "1.0.0") == (True, None) + + def test_untrustworthy_core_blocks_floor_above_baseline(self): + # But a floor above 2.0.0 needs modules that no core reporting below + # the floor can have — the one refusal on that branch. + manifest = {"name": "New Plugin", "min_ledmatrix_version": "3.2.0"} + ok, reason = check(manifest, "1.0.0") + assert ok is False + assert "too old to identify reliably" in reason + + def test_untrustworthy_core_ignores_compatible_versions(self): + # On the untrustworthy branch only the declared floor is consulted; + # ranges cannot be evaluated against a version that isn't evidence. + manifest = {"compatible_versions": ["2.0.0 - 2.9.9"]} + assert check(manifest, "1.0.0") == (True, None) + + def test_floor_exactly_at_trustworthy_floor_is_allowed(self): + floor = ".".join(str(n) for n in TRUSTWORTHY_FLOOR) + manifest = {"min_ledmatrix_version": floor} + assert check(manifest, "1.0.0") == (True, None) + + def test_reason_uses_manifest_name(self): + manifest = {"name": "Fancy Clock", "min_ledmatrix_version": "9.0.0"} + ok, reason = check(manifest, "3.1.0") + assert ok is False + assert reason.startswith("Fancy Clock") + + def test_reason_falls_back_to_id(self): + manifest = {"id": "fancy-clock", "min_ledmatrix_version": "9.0.0"} + ok, reason = check(manifest, "3.1.0") + assert ok is False + assert reason.startswith("fancy-clock") + + def test_prerelease_core_compares_equal_to_release(self): + # Documented: prereleases compare equal to their release. + manifest = {"min_ledmatrix_version": "3.2.0"} + assert check(manifest, "3.2.0-rc1") == (True, None) diff --git a/test/test_config_helper.py b/test/test_config_helper.py new file mode 100644 index 00000000..4931fccc --- /dev/null +++ b/test/test_config_helper.py @@ -0,0 +1,253 @@ +""" +Tests for src/common/config_helper.py — pins the ConfigHelper contract. + +Covers: load/save round trips (missing/malformed files return {} rather +than raising, non-ASCII preserved via ensure_ascii=False, top-level JSON +lists returned as-is), dot-notation get/set including the silent-failure +contract when an intermediate key holds a non-dict, merge_configs deep +semantics with NO aliasing of the base config (the fixed bug — the old +shallow copy let mutations of the merged result leak into base's nested +dicts), simplified schema validation including the caught-TypeError path +when a schema 'type' is given as a string, plugin config key conventions +('{plugin_id}_config', enabled defaults True), and required-key checks +where a key present with value None counts as present. +""" + +import json + +import pytest + +from src.common.config_helper import ConfigHelper + + +@pytest.fixture +def helper(): + return ConfigHelper() + + +class TestLoadConfig: + def test_missing_file_returns_empty_dict(self, helper, tmp_path): + assert helper.load_config(tmp_path / "nope.json") == {} + + def test_malformed_json_returns_empty_dict(self, helper, tmp_path): + path = tmp_path / "bad.json" + path.write_text("{ this is not json", encoding="utf-8") + assert helper.load_config(path) == {} + + def test_top_level_list_returned_as_is(self, helper, tmp_path): + # load_config does not enforce a dict shape: a JSON list comes + # straight back. Pinned as a characterization of current behavior. + path = tmp_path / "list.json" + path.write_text("[1, 2, 3]", encoding="utf-8") + assert helper.load_config(path) == [1, 2, 3] + + +class TestSaveConfig: + def test_round_trip(self, helper, tmp_path): + path = tmp_path / "config.json" + config = {'display': {'hardware': {'rows': 32}}, 'timezone': 'UTC'} + assert helper.save_config(config, path) is True + assert helper.load_config(path) == config + + def test_creates_parent_directories(self, helper, tmp_path): + path = tmp_path / "deep" / "nested" / "config.json" + assert helper.save_config({'a': 1}, path) is True + assert path.exists() + assert helper.load_config(path) == {'a': 1} + + def test_non_ascii_survives_round_trip(self, helper, tmp_path): + path = tmp_path / "config.json" + config = {'city': 'Zürich', 'note': 'météo ☀'} + assert helper.save_config(config, path) is True + assert helper.load_config(path) == config + # ensure_ascii=False: characters are written raw, not \u-escaped + assert 'Zürich' in path.read_text(encoding='utf-8') + + def test_directory_path_returns_false_not_raise(self, helper, tmp_path): + assert helper.save_config({'a': 1}, tmp_path) is False + + +class TestGetConfigValue: + def test_dot_notation_hit(self, helper): + config = {'display': {'hardware': {'rows': 32}}} + assert helper.get_config_value(config, 'display.hardware.rows') == 32 + + def test_missing_returns_default(self, helper): + sentinel = object() + assert helper.get_config_value({}, 'display.rows', default=sentinel) is sentinel + + def test_intermediate_non_dict_returns_default(self, helper): + config = {'display': 'not-a-dict'} + assert helper.get_config_value(config, 'display.hardware.rows', default=64) == 64 + + def test_required_missing_raises_keyerror(self, helper): + with pytest.raises(KeyError): + helper.get_config_value({}, 'display.rows', required=True) + + +class TestSetConfigValue: + def test_sets_top_level(self, helper): + config = {} + helper.set_config_value(config, 'timezone', 'UTC') + assert config == {'timezone': 'UTC'} + + def test_auto_creates_intermediates(self, helper): + config = {} + helper.set_config_value(config, 'display.hardware.rows', 32) + assert config == {'display': {'hardware': {'rows': 32}}} + + def test_silent_failure_on_non_dict_intermediate(self, helper): + # 'a' exists but holds an int; the assignment attempt raises + # TypeError internally, which set_config_value swallows and logs. + # The config is left unchanged — pinned silent-failure contract. + config = {'a': 5} + helper.set_config_value(config, 'a.b', 1) + assert config == {'a': 5} + + +class TestMergeConfigs: + def test_nested_dicts_merge_recursively(self, helper): + base = {'display': {'rows': 32, 'cols': 64}, 'timezone': 'UTC'} + override = {'display': {'cols': 128, 'brightness': 90}} + merged = helper.merge_configs(base, override) + assert merged == { + 'display': {'rows': 32, 'cols': 128, 'brightness': 90}, + 'timezone': 'UTC', + } + + def test_scalar_override_wins_over_dict(self, helper): + merged = helper.merge_configs({'display': {'rows': 32}}, {'display': 7}) + assert merged['display'] == 7 + + def test_dict_override_wins_over_scalar(self, helper): + merged = helper.merge_configs({'display': 7}, {'display': {'rows': 32}}) + assert merged['display'] == {'rows': 32} + + def test_no_aliasing_of_base(self, helper): + # Post-fix: merge deep-copies base, so mutating the result never + # leaks back into the caller's base config. + base = {'display': {'x': 1}} + merged = helper.merge_configs(base, {}) + assert merged['display'] is not base['display'] + merged['display']['x'] = 99 + assert base['display']['x'] == 1 + + def test_inputs_unchanged(self, helper): + base = {'a': {'b': 1}} + override = {'a': {'c': 2}} + helper.merge_configs(base, override) + assert base == {'a': {'b': 1}} + assert override == {'a': {'c': 2}} + + def test_no_aliasing_of_override_values(self, helper): + # The non-recursive branch must deep-copy the override value too: + # mutating a merged-in list or dict must not reach back into + # override_config. + override = {'teams': ['A', 'B'], 'nested': {'x': [1]}} + merged = helper.merge_configs({}, override) + merged['teams'].append('C') + merged['nested']['x'].append(2) + assert override == {'teams': ['A', 'B'], 'nested': {'x': [1]}} + + +class TestValidateConfig: + def test_no_schema_dict_is_valid(self, helper): + assert helper.validate_config({'a': 1}) is True + + def test_no_schema_list_is_invalid(self, helper): + assert helper.validate_config([1, 2]) is False + + def test_required_key_missing_is_invalid(self, helper): + schema = {'rows': {'required': True, 'type': int}} + assert helper.validate_config({}, schema) is False + + def test_optional_key_missing_is_valid(self, helper): + schema = {'rows': {'required': False, 'type': int}} + assert helper.validate_config({}, schema) is True + + def test_wrong_type_is_invalid(self, helper): + schema = {'rows': {'type': int}} + assert helper.validate_config({'rows': 'thirty-two'}, schema) is False + assert helper.validate_config({'rows': 32}, schema) is True + + def test_allowed_values_violation_is_invalid(self, helper): + schema = {'mode': {'allowed_values': ['clock', 'weather']}} + assert helper.validate_config({'mode': 'stocks'}, schema) is False + assert helper.validate_config({'mode': 'clock'}, schema) is True + + def test_string_type_in_schema_is_invalid_via_typeerror(self, helper): + # 'type' given as the STRING "int" makes isinstance() raise + # TypeError; validate_config catches it and returns False rather + # than raising. Pinned characterization. + schema = {'rows': {'type': 'int'}} + assert helper.validate_config({'rows': 32}, schema) is False + + +class TestPluginConfigHelpers: + def test_get_plugin_config_uses_suffixed_key(self, helper): + plugin_cfg = {'enabled': True, 'display_duration': 30} + assert helper.get_plugin_config({'clock_config': plugin_cfg}, 'clock') == plugin_cfg + + def test_get_plugin_config_bare_id_key_not_found(self, helper): + # Only '{plugin_id}_config' is consulted — a bare 'clock' section + # is invisible to this helper. Pinned key contract. + assert helper.get_plugin_config({'clock': {'enabled': True}}, 'clock') == {} + + def test_create_default_config_wraps_in_suffixed_key(self, helper): + defaults = {'enabled': True} + assert helper.create_default_config('clock', defaults) == {'clock_config': defaults} + + def test_is_plugin_enabled_defaults_true_for_unknown(self, helper): + assert helper.is_plugin_enabled({}, 'clock') is True + + def test_is_plugin_enabled_false_when_disabled(self, helper): + config = {'clock_config': {'enabled': False}} + assert helper.is_plugin_enabled(config, 'clock') is False + + def test_is_plugin_enabled_ignores_bare_id_key(self, helper): + # Disabled under the wrong key -> still reported enabled (default). + config = {'clock': {'enabled': False}} + assert helper.is_plugin_enabled(config, 'clock') is True + + +class TestSportsAndDisplayHelpers: + def test_get_display_config(self, helper): + display = {'hardware': {'rows': 32}} + assert helper.get_display_config({'display': display}) == display + assert helper.get_display_config({}) == {} + + def test_get_sports_config_uses_scoreboard_suffix(self, helper): + sport_cfg = {'favorite_teams': ['TB']} + config = {'football_scoreboard': sport_cfg} + assert helper.get_sports_config(config, 'football') == sport_cfg + assert helper.get_sports_config(config, 'hockey') == {} + + def test_get_favorite_teams(self, helper): + config = {'football_scoreboard': {'favorite_teams': ['TB', 'DAL']}} + assert helper.get_favorite_teams(config, 'football') == ['TB', 'DAL'] + assert helper.get_favorite_teams({}, 'football') == [] + + def test_get_display_modes(self, helper): + modes = {'live': True, 'recent': False} + config = {'football_scoreboard': {'display_modes': modes}} + assert helper.get_display_modes(config, 'football') == modes + assert helper.get_display_modes({}, 'football') == {} + + +class TestValidateRequiredKeys: + def test_returns_missing_subset(self, helper): + config = {'a': 1, 'c': {'d': 2}} + missing = helper.validate_required_keys(config, ['a', 'b', 'c.d', 'c.e']) + assert missing == ['b', 'c.e'] + + def test_dot_notation_present(self, helper): + config = {'display': {'hardware': {'rows': 32}}} + assert helper.validate_required_keys(config, ['display.hardware.rows']) == [] + + def test_empty_requirements(self, helper): + assert helper.validate_required_keys({'a': 1}, []) == [] + + def test_present_with_none_counts_as_present(self, helper): + # _has_key checks key membership, not truthiness — a key set to + # None is NOT reported missing. Pinned semantics. + assert helper.validate_required_keys({'a': None}, ['a']) == [] diff --git a/test/test_config_main_redacts_secrets.py b/test/test_config_main_redacts_secrets.py new file mode 100644 index 00000000..15172df2 --- /dev/null +++ b/test/test_config_main_redacts_secrets.py @@ -0,0 +1,143 @@ +"""GET /config/main must not hand out credentials. + +The endpoint returned the raw config to anyone who could reach the port, and +this web interface has no authentication of any kind. Measured against a live +rig, an unauthenticated request returned: + + github.api_token 40 chars + incoming-packages.ha_token 183 chars + jellyfin-now-playing.api_key 32 chars + ledmatrix-weather.api_key 32 chars + on-air.mqtt_password 8 chars + youtube.api_key 20 chars + youtube-stats.api_key 39 chars + +A GitHub token and a Home Assistant long-lived token among them. + +The x-secret masking the plugin config endpoints use does not apply here: this +endpoint never consults a schema, and core keys such as github.api_token have +no schema to carry the marker. Several of those fields *are* tagged x-secret in +their plugin's schema and were still returned in full, which is what makes the +schema route the wrong one to rely on for this endpoint. + +Matching on field name is blunt. For a whole-config dump it is the right +default: anything named like a credential should not leave the process, and a +new plugin that adds a differently-shaped secret is covered without anyone +remembering to tag it. +""" +import pytest + +from web_interface.blueprints.api_v3 import ( + _looks_like_a_credential, + _redact_credentials, +) + + +@pytest.mark.parametrize("name", [ + "password", "mqtt_password", "opensky_password", "passwd", + "api_key", "apikey", "API_KEY", "flightaware_api_key", + "token", "ha_token", "api_token", "access_token", + "secret", "client_secret", "spotify_client_secret", + "access_key", "private_key", +]) +def test_credential_names_are_recognised(name): + assert _looks_like_a_credential(name) + + +@pytest.mark.parametrize("name", [ + "timezone", "city", "brightness", "enabled", "update_interval", + "favorite_teams", "display_duration", "keyword", +]) +def test_ordinary_names_are_left_alone(name): + assert not _looks_like_a_credential(name) + + +def test_the_measured_leak_is_closed(): + """The exact shape taken off the rig.""" + config = { + "github": {"api_token": "ghp_" + "x" * 36}, + "incoming-packages": {"ha_token": "y" * 183, "enabled": True}, + "jellyfin-now-playing": {"api_key": "z" * 32}, + "on-air": {"mqtt_password": "hunter22"}, + "youtube": {"api_key": "k" * 20}, + "timezone": "America/New_York", + } + out = _redact_credentials(config) + assert out["github"]["api_token"] == "" + assert out["incoming-packages"]["ha_token"] == "" + assert out["jellyfin-now-playing"]["api_key"] == "" + assert out["on-air"]["mqtt_password"] == "" + assert out["youtube"]["api_key"] == "" + # Everything else survives, or the config editor breaks. + assert out["timezone"] == "America/New_York" + assert out["incoming-packages"]["enabled"] is True + + +def test_nested_and_listed_credentials_are_reached(): + config = {"a": {"b": {"c": {"password": "p"}}}, + "feeds": [{"name": "x", "api_key": "k"}, {"name": "y"}]} + out = _redact_credentials(config) + assert out["a"]["b"]["c"]["password"] == "" + assert out["feeds"][0]["api_key"] == "" + assert out["feeds"][0]["name"] == "x" + + +def test_the_original_is_not_mutated(): + """The caller holds the live config; redaction must not edit it in place.""" + config = {"github": {"api_token": "keepme"}} + _redact_credentials(config) + assert config["github"]["api_token"] == "keepme" + + +def test_a_credential_shaped_container_is_still_walked(): + """`secrets: {...}` is a section name, not a value to blank.""" + config = {"secrets": {"api_key": "k", "note": "keep"}} + out = _redact_credentials(config) + assert out["secrets"]["api_key"] == "" + assert out["secrets"]["note"] == "keep" + + +def test_non_dict_input_passes_through(): + assert _redact_credentials("plain") == "plain" + assert _redact_credentials(7) == 7 + assert _redact_credentials(None) is None + + +def test_the_endpoint_itself_redacts(): + """Through the view function, not the helper. + + The helper tests above all passed with the route still returning + `config` -- reverting the one line that calls the redactor changed + nothing, because nothing exercised the route. A property asserted on a + helper is not a property asserted on the endpoint, and it is the endpoint + that is exposed to the network. + """ + import json as _json + from unittest.mock import MagicMock + + import flask + + from web_interface.blueprints import api_v3 as mod + + raw = {"github": {"api_token": "ghp_secret_value"}, + "timezone": "America/New_York"} + + manager = MagicMock() + manager.load_config.return_value = raw + previous = getattr(mod.api_v3, "config_manager", None) + mod.api_v3.config_manager = manager + + app = flask.Flask(__name__) + try: + with app.test_request_context("/config/main"): + response = mod.get_main_config() + payload = response.get_json() if hasattr(response, "get_json") else _json.loads(response[0].data) + finally: + mod.api_v3.config_manager = previous + + data = payload["data"] + assert data["github"]["api_token"] == "", ( + "the endpoint returned the token; the redactor is not wired in") + assert data["timezone"] == "America/New_York" + # And the config the manager handed over is untouched. + assert raw["github"]["api_token"] == "ghp_secret_value" diff --git a/test/test_config_manager.py b/test/test_config_manager.py index 29705f36..68c411e9 100644 --- a/test/test_config_manager.py +++ b/test/test_config_manager.py @@ -383,19 +383,6 @@ class TestConfigHelpers: display_config = manager.get_display_config() assert display_config["hardware"]["rows"] == 32 - def test_get_clock_config(self, tmp_path): - """Test getting clock config.""" - config_file = tmp_path / "config.json" - config_data = {"clock": {"format": "12h"}} - - with open(config_file, 'w') as f: - json.dump(config_data, f) - - manager = ConfigManager(config_path=str(config_file)) - manager.load_config() - - clock_config = manager.get_clock_config() - assert clock_config["format"] == "12h" class TestPluginConfigManagement: diff --git a/test/test_config_manager_secrets.py b/test/test_config_manager_secrets.py new file mode 100644 index 00000000..f7070c9c --- /dev/null +++ b/test/test_config_manager_secrets.py @@ -0,0 +1,335 @@ +""" +Tests for the ConfigManager secrets round-trip and the load_config fast path. + +The contract under test: config_secrets.json values are deep-merged INTO the +in-memory config at load time, and stripped back OUT before anything is +written to config.json — so secrets live in exactly one file on disk. This +suite pins that round-trip plus its sharp edges, including the guard that a +save REFUSES (ConfigError) when the secrets file exists but can't be loaded, +rather than leaking merged secrets into config.json in plaintext. + +Complements test_config_manager.py, which covers loading/migration/validation. +""" + +import json +import os + +import pytest + +from src.config_manager import ConfigManager +from src.exceptions import ConfigError + + +def make_manager(tmp_path, config=None, secrets=None): + """A ConfigManager over tmp_path files, template migration neutralized.""" + config_file = tmp_path / "config.json" + secrets_file = tmp_path / "config_secrets.json" + config_file.write_text(json.dumps(config if config is not None else {})) + if secrets is not None: + secrets_file.write_text(json.dumps(secrets)) + manager = ConfigManager(config_path=str(config_file), + secrets_path=str(secrets_file)) + # Point the (CWD-relative) template at nothing so migration never runs — + # these tests assert exact on-disk contents. + manager.template_path = str(tmp_path / "no-template.json") + return manager + + +class TestLoadMergesSecrets: + def test_secrets_deep_merged_into_config(self, tmp_path): + manager = make_manager( + tmp_path, + config={"weather": {"city": "Austin"}, "timezone": "UTC"}, + secrets={"weather": {"api_key": "s3cret"}}, + ) + loaded = manager.load_config() + assert loaded["weather"] == {"city": "Austin", "api_key": "s3cret"} + assert loaded["timezone"] == "UTC" + + def test_secret_scalar_overrides_config_value(self, tmp_path): + manager = make_manager( + tmp_path, + config={"weather": {"api_key": "YOUR_API_KEY"}}, + secrets={"weather": {"api_key": "real-key"}}, + ) + assert manager.load_config()["weather"]["api_key"] == "real-key" + + def test_missing_secrets_file_loads_config_fine(self, tmp_path): + manager = make_manager(tmp_path, config={"timezone": "UTC"}) + assert manager.load_config() == {"timezone": "UTC"} + + def test_corrupt_secrets_file_loads_config_without_secrets(self, tmp_path): + manager = make_manager(tmp_path, config={"timezone": "UTC"}) + (tmp_path / "config_secrets.json").write_text("{not json") + loaded = manager.load_config() + assert loaded["timezone"] == "UTC" + + +class TestSaveStripsSecrets: + def test_round_trip_keeps_secrets_out_of_config_json(self, tmp_path): + manager = make_manager( + tmp_path, + config={"weather": {"city": "Austin"}}, + secrets={"weather": {"api_key": "s3cret"}}, + ) + loaded = manager.load_config() + assert loaded["weather"]["api_key"] == "s3cret" # merged in memory + + manager.save_config(loaded) + + on_disk = json.loads((tmp_path / "config.json").read_text()) + assert "api_key" not in on_disk.get("weather", {}) + assert on_disk["weather"]["city"] == "Austin" + # In-memory config still carries the secret for runtime use. + assert manager.config["weather"]["api_key"] == "s3cret" + + def test_group_dropped_when_only_secrets_remain(self, tmp_path): + # _strip_secrets_recursive drops a group entirely when nothing + # non-secret is left in it. + manager = make_manager( + tmp_path, + config={}, + secrets={"weather": {"api_key": "s3cret"}}, + ) + manager.save_config({"weather": {"api_key": "s3cret"}, "timezone": "UTC"}) + on_disk = json.loads((tmp_path / "config.json").read_text()) + assert on_disk == {"timezone": "UTC"} + + def test_scalar_secret_key_stripped_at_top_level(self, tmp_path): + manager = make_manager(tmp_path, config={}, secrets={"token": "t"}) + manager.save_config({"token": "t", "timezone": "UTC"}) + on_disk = json.loads((tmp_path / "config.json").read_text()) + assert on_disk == {"timezone": "UTC"} + + def test_corrupt_secrets_file_refuses_save_no_plaintext_leak(self, tmp_path): + # Regression guard: when the secrets file exists but is corrupt at + # save time, stripping is impossible — the save must raise instead of + # writing the merged secrets into config.json in plaintext (the + # historical behavior). + manager = make_manager( + tmp_path, + config={"weather": {"city": "Austin"}}, + secrets={"weather": {"api_key": "s3cret"}}, + ) + loaded = manager.load_config() + (tmp_path / "config_secrets.json").write_text("{corrupt") + + with pytest.raises(ConfigError): + manager.save_config(loaded) + + # On-disk config untouched: no secret leaked. + on_disk = json.loads((tmp_path / "config.json").read_text()) + assert "api_key" not in on_disk.get("weather", {}) + + def test_corrupt_secrets_file_refuses_atomic_save_too(self, tmp_path): + # Same refusal on the atomic save path, which shared the leak. + manager = make_manager( + tmp_path, + config={"weather": {"city": "Austin"}}, + secrets={"weather": {"api_key": "s3cret"}}, + ) + loaded = manager.load_config() + (tmp_path / "config_secrets.json").write_text("{corrupt") + + with pytest.raises(ConfigError): + manager.save_config_atomic(loaded) + + on_disk = json.loads((tmp_path / "config.json").read_text()) + assert "api_key" not in on_disk.get("weather", {}) + + +class TestLoadFastPath: + def test_unchanged_files_return_cached_dict(self, tmp_path): + manager = make_manager(tmp_path, config={"timezone": "UTC"}) + first = manager.load_config() + second = manager.load_config() + assert second is first # same aliased dict, no re-read + + def test_touching_secrets_file_invalidates_cache(self, tmp_path): + manager = make_manager( + tmp_path, + config={"weather": {}}, + secrets={"weather": {"api_key": "old"}}, + ) + assert manager.load_config()["weather"]["api_key"] == "old" + + secrets_file = tmp_path / "config_secrets.json" + secrets_file.write_text(json.dumps({"weather": {"api_key": "new"}})) + # Force a different mtime_ns in case the write landed within the + # filesystem's timestamp granularity. + os.utime(secrets_file, ns=(1, 1)) + + assert manager.load_config()["weather"]["api_key"] == "new" + + def test_same_mtime_same_size_change_served_stale(self, tmp_path): + # Characterized fast-path blind spot: the signature is (mtime_ns, + # size) only, so a same-length content swap with a forged identical + # mtime is not detected. Real writes bump mtime_ns, so this is + # acceptable — but it is a contract worth pinning. + manager = make_manager(tmp_path, config={"timezone": "AAA"}) + config_file = tmp_path / "config.json" + os.utime(config_file, ns=(1_000_000_000, 1_000_000_000)) + manager._loaded_sig = None + first = manager.load_config() + assert first["timezone"] == "AAA" + + config_file.write_text(json.dumps({"timezone": "BBB"})) # same length + os.utime(config_file, ns=(1_000_000_000, 1_000_000_000)) + + assert manager.load_config()["timezone"] == "AAA" # stale, by design + + +class TestArraySecretStripAndMerge: + """Array-item secrets round-trip (parallel-placeholder lists). + + secret_helpers.separate_secrets emits array secrets as a list parallel + to the regular list, with {} for items that carry no secrets. Strip + must remove the secret fields from config.json while preserving item + indices; load must merge them back into the right items. The regular + list's length is authoritative in both directions. + """ + + def test_strip_removes_array_item_secrets_keeps_indices(self, tmp_path): + manager = make_manager(tmp_path) + data = {"plugin": {"accounts": [ + {"name": "a", "token": "ta"}, + {"name": "b"}, + ]}} + secrets = {"plugin": {"accounts": [{"token": "ta"}, {}]}} + stripped = manager._strip_secrets_recursive(data, secrets) + assert stripped == {"plugin": {"accounts": [{"name": "a"}, {"name": "b"}]}} + + def test_strip_keeps_all_placeholder_items(self, tmp_path): + # Even when every item strips to nothing extra, the list survives + # with its indices — required for merge-on-load alignment. + manager = make_manager(tmp_path) + data = {"accounts": [{"token": "t1"}, {"token": "t2"}]} + secrets = {"accounts": [{"token": "t1"}, {"token": "t2"}]} + stripped = manager._strip_secrets_recursive(data, secrets) + assert stripped == {"accounts": [{}, {}]} + + def test_strip_whole_scalar_array_secret_drops_key(self, tmp_path): + # A list of secret scalars is a whole-key secret, not the parallel + # shape — the key must vanish from config.json entirely. + manager = make_manager(tmp_path) + data = {"recovery_codes": ["a", "b"], "city": "Austin"} + secrets = {"recovery_codes": ["a", "b"]} + stripped = manager._strip_secrets_recursive(data, secrets) + assert stripped == {"city": "Austin"} + + def test_strip_shape_mismatch_drops_key(self, tmp_path): + # Conservative contract: if the shapes disagree, never leak. + manager = make_manager(tmp_path) + data = {"accounts": {"name": "not-a-list"}} + secrets = {"accounts": [{"token": "t"}]} + stripped = manager._strip_secrets_recursive(data, secrets) + assert stripped == {} + + def test_strip_ignores_extra_secrets_entries(self, tmp_path): + # Regular list length is authoritative: a user deleted an item. + manager = make_manager(tmp_path) + data = {"accounts": [{"name": "a", "token": "ta"}]} + secrets = {"accounts": [{"token": "ta"}, {"token": "tb"}]} + stripped = manager._strip_secrets_recursive(data, secrets) + assert stripped == {"accounts": [{"name": "a"}]} + + def test_merge_restores_array_item_secrets(self, tmp_path): + manager = make_manager(tmp_path) + target = {"accounts": [{"name": "a"}, {"name": "b"}]} + manager._deep_merge(target, {"accounts": [{"token": "ta"}, {}]}) + assert target == {"accounts": [ + {"name": "a", "token": "ta"}, + {"name": "b"}, + ]} + + def test_merge_ignores_extra_secrets_entries_with_warning(self, tmp_path, caplog): + manager = make_manager(tmp_path) + target = {"accounts": [{"name": "a"}]} + with caplog.at_level("WARNING"): + manager._deep_merge( + target, {"accounts": [{"token": "ta"}, {"token": "ghost"}]}) + assert target == {"accounts": [{"name": "a", "token": "ta"}]} + assert any("longer than the config list" in r.message for r in caplog.records) + + def test_merge_non_dict_item_replaced_by_secret(self, tmp_path): + # Shape drift inside the list: the secret wins for that index. + manager = make_manager(tmp_path) + target = {"accounts": ["oddball", {"name": "b"}]} + manager._deep_merge(target, {"accounts": [{"token": "ta"}, {}]}) + assert target == {"accounts": [{"token": "ta"}, {"name": "b"}]} + + def test_merge_whole_scalar_array_still_replaces(self, tmp_path): + # Legacy behavior preserved: a non-parallel list replaces wholesale. + manager = make_manager(tmp_path) + target = {"recovery_codes": ["old"]} + manager._deep_merge(target, {"recovery_codes": ["new1", "new2"]}) + assert target == {"recovery_codes": ["new1", "new2"]} + + def test_full_save_load_round_trip(self, tmp_path): + # End to end on real files: save strips array secrets out of + # config.json; load merges them back into the right items. + manager = make_manager( + tmp_path, + config={"plugin": {"accounts": [ + {"name": "a", "token": "s3cret-a"}, + {"name": "b", "token": "s3cret-b"}, + ]}}, + secrets={"plugin": {"accounts": [ + {"token": "s3cret-a"}, {"token": "s3cret-b"}, + ]}}, + ) + loaded = manager.load_config() + assert loaded["plugin"]["accounts"][0]["token"] == "s3cret-a" + + manager.save_config(loaded) + + raw = (tmp_path / "config.json").read_text() + assert "s3cret" not in raw + on_disk = json.loads(raw) + assert on_disk["plugin"]["accounts"] == [{"name": "a"}, {"name": "b"}] + + # A fresh manager (constructed directly — make_manager would + # overwrite the just-saved config.json) re-merges from the secrets + # file on load. + fresh = ConfigManager(config_path=str(tmp_path / "config.json"), + secrets_path=str(tmp_path / "config_secrets.json")) + fresh.template_path = str(tmp_path / "no-template.json") + reloaded = fresh.load_config() + assert reloaded["plugin"]["accounts"] == [ + {"name": "a", "token": "s3cret-a"}, + {"name": "b", "token": "s3cret-b"}, + ] + + def test_whole_item_secret_list_never_leaks_values(self, tmp_path): + # When the ENTIRE array item is secret (schema marks both key[] + # and key[].field), separate_secrets stores the full item dicts in + # the secrets file. That shape also matches the parallel-list + # discriminator — which is safe: strip drops every leaf key that + # appears in the secret item, so only empty {} skeletons (item + # count, no values) can reach config.json, and merge-on-load + # restores the full items from those skeletons. + from src.web_interface.secret_helpers import ( + find_secret_fields, separate_secrets) + schema_props = {"accounts": { + "type": "array", + "items": {"type": "object", "x-secret": True, "properties": { + "id": {"type": "string"}, + "token": {"type": "string", "x-secret": True}, + }}, + }} + paths = find_secret_fields(schema_props) + assert paths == {"accounts[]", "accounts[].token"} + full = {"accounts": [{"id": "i1", "token": "s3cret-a"}, + {"id": "i2", "token": "s3cret-b"}]} + _, secrets = separate_secrets(full, paths) + assert secrets == full # whole items are secret + + manager = make_manager(tmp_path) + stripped = manager._strip_secrets_recursive(full, secrets) + assert stripped == {"accounts": [{}, {}]} + + raw = json.dumps(stripped) + assert "s3cret" not in raw and "i1" not in raw + + manager._deep_merge(stripped, secrets) + assert stripped == full # round trip restores the items diff --git a/test/test_discovery_path_contract.py b/test/test_discovery_path_contract.py new file mode 100644 index 00000000..7c60b88c --- /dev/null +++ b/test/test_discovery_path_contract.py @@ -0,0 +1,167 @@ +""" +Drift guard: three components independently answer "where is plugin X?" and +their answers must stay coherent — plus the `.standalone-backup-` naming +contract that install/rollback shares with discovery. + +The three resolvers: +1. PluginManager._scan_directory_for_plugins — scans ONLY the configured dir. +2. PluginStoreManager._find_plugin_path — configured dir, then a sibling + `plugins/` fallback derived from the configured dir's parent. +3. SchemaManager.get_schema_path — configured dir, then project-root + `plugins/`, then `plugin-repos/`, then case-insensitive scans. + +The divergence is characterized (a plugin visible to the store/schema +fallbacks but invisible to discovery is a real support-issue shape) so any +change to the fallback chains is a deliberate one. + +The `.standalone-backup-` contract: store_manager renames a plugin dir aside +with that substring during install/rollback; discovery MUST skip such dirs +or a half-finished install would surface a ghost plugin. The substring is +duplicated as a literal in both files — this test breaks if either side +changes it unilaterally. +""" + +import json +import logging +import threading +from pathlib import Path + +import pytest + +from src.plugin_system.plugin_manager import PluginManager +from src.plugin_system.schema_manager import SchemaManager +from src.plugin_system.store_manager import PluginStoreManager + + +def _write_plugin(base: Path, plugin_id: str, dir_name: str = None): + plugin_dir = base / (dir_name or plugin_id) + plugin_dir.mkdir(parents=True) + (plugin_dir / "manifest.json").write_text(json.dumps({ + "id": plugin_id, "name": plugin_id, "version": "1.0.0", + })) + (plugin_dir / "config_schema.json").write_text(json.dumps({ + "type": "object", "properties": {"enabled": {"type": "boolean"}}, + })) + return plugin_dir + + +def _scanner(): + """A PluginManager stripped to just its discovery machinery — the full + constructor wires config/schema/health managers this test doesn't need.""" + pm = object.__new__(PluginManager) + pm.logger = logging.getLogger("test_discovery_path_contract") + pm._discovery_lock = threading.Lock() + pm.plugin_manifests = {} + pm.plugin_directories = {} + return pm + + +class TestResolversAgreeOnConfiguredDir: + def test_all_three_find_a_plugin_in_the_configured_dir(self, tmp_path): + plugins_dir = tmp_path / "plugin-repos" + plugin_dir = _write_plugin(plugins_dir, "demo-plugin") + + found = _scanner()._scan_directory_for_plugins(plugins_dir) + assert found == ["demo-plugin"] + + store = PluginStoreManager( + plugins_dir=str(plugins_dir), + uninstalled_registry_path=str(tmp_path / "uninstalled.json")) + assert store._find_plugin_path("demo-plugin") == plugin_dir + + schema = SchemaManager(plugins_dir=plugins_dir, project_root=tmp_path) + assert schema.get_schema_path("demo-plugin") == \ + plugin_dir / "config_schema.json" + + +class TestFallbackDivergence: + def test_plugin_only_in_plugins_dir_fallback(self, tmp_path): + """Characterized divergence: configured dir is plugin-repos/, but the + plugin sits in a sibling plugins/. The store and schema fallbacks + find it; discovery does NOT — so the plugin is installable/ + configurable but never loads. Pinned so a change to any fallback + chain shows up here.""" + configured = tmp_path / "plugin-repos" + configured.mkdir() + legacy_dir = _write_plugin(tmp_path / "plugins", "legacy-plugin") + + # Discovery: invisible. + assert _scanner()._scan_directory_for_plugins(configured) == [] + + # Store fallback: visible (parent-of-configured / 'plugins'). + store = PluginStoreManager( + plugins_dir=str(configured), + uninstalled_registry_path=str(tmp_path / "uninstalled.json")) + assert store._find_plugin_path("legacy-plugin") == legacy_dir + + # Schema fallback: visible (project_root / 'plugins'). + schema = SchemaManager(plugins_dir=configured, project_root=tmp_path) + assert schema.get_schema_path("legacy-plugin") == \ + legacy_dir / "config_schema.json" + + def test_schema_manager_probes_plugins_before_plugin_repos(self, tmp_path): + # Documented order (also in CLAUDE.md): plugins/ wins over + # plugin-repos/ when the same id exists in both. + in_plugins = _write_plugin(tmp_path / "plugins", "dupe") + _write_plugin(tmp_path / "plugin-repos", "dupe") + schema = SchemaManager(plugins_dir=None, project_root=tmp_path) + assert schema.get_schema_path("dupe") == \ + in_plugins / "config_schema.json" + + def test_schema_manager_case_insensitive_fallback(self, tmp_path): + plugin_dir = _write_plugin(tmp_path / "plugins", "MyPlugin", + dir_name="MyPlugin") + schema = SchemaManager(plugins_dir=None, project_root=tmp_path) + assert schema.get_schema_path("myplugin") == \ + plugin_dir / "config_schema.json" + + +class TestStandaloneBackupContract: + def test_discovery_skips_backup_dirs(self, tmp_path): + plugins_dir = tmp_path / "plugins" + _write_plugin(plugins_dir, "real-plugin") + # A rollback-in-progress dir with a valid manifest must NOT surface. + _write_plugin(plugins_dir, "real-plugin", + dir_name="real-plugin.standalone-backup-migrating") + + found = _scanner()._scan_directory_for_plugins(plugins_dir) + assert found == ["real-plugin"] + + def test_backup_substring_literal_matches_across_files(self): + """The substring is duplicated in plugin_manager (skip check) and + store_manager (rename-aside names). If either side changes it, the + other silently stops honoring the contract — this test is the + tripwire.""" + root = Path(__file__).resolve().parents[1] + pm_text = (root / "src/plugin_system/plugin_manager.py").read_text() + sm_text = (root / "src/plugin_system/store_manager.py").read_text() + assert "'.standalone-backup-'" in pm_text.replace('"', "'") + assert ".standalone-backup-" in sm_text + + +class TestSkinTargetResolution: + def _store(self, tmp_path): + return PluginStoreManager( + plugins_dir=str(tmp_path / "plugins"), + uninstalled_registry_path=str(tmp_path / "uninstalled.json")) + + def test_valid_skin_id_resolves_inside_skins_dir(self, tmp_path): + from src.skin_system import skin_runtime + store = self._store(tmp_path) + target = store._resolve_skin_target("my-skin") + assert target is not None + assert target.parent == skin_runtime.get_skins_directory().resolve() + + @pytest.mark.parametrize("bad_id", [ + "../evil", + "..", + "a/../../etc", + "/etc/passwd", + "skin/../../outside", + "", + None, + 123, + ]) + def test_traversal_and_malformed_ids_rejected(self, tmp_path, bad_id): + store = self._store(tmp_path) + assert store._resolve_skin_target(bad_id) is None diff --git a/test/test_display_controller.py b/test/test_display_controller.py index 782ce69e..da551b93 100644 --- a/test/test_display_controller.py +++ b/test/test_display_controller.py @@ -15,13 +15,6 @@ class TestDisplayControllerInitialization: assert test_display_controller.plugin_manager is not None assert test_display_controller.available_modes == [] - @pytest.mark.skip(reason="No assertions; init logic is covered by test_init_success and fixture setup") - def test_plugin_discovery_and_loading(self, test_display_controller): - """Test plugin discovery and loading during initialization.""" - pm = test_display_controller.plugin_manager - pm.discover_plugins.return_value = ["plugin1", "plugin2"] - pm.get_plugin.return_value = MagicMock() - class TestDisplayControllerModeRotation: """Test display mode rotation logic.""" @@ -345,31 +338,48 @@ class TestDisplayControllerSchedule: """Test schedule management.""" def test_schedule_disabled(self, test_display_controller): - """Test when schedule is disabled.""" + """schedule.enabled=False keeps the display active even outside the + configured window. (This test used to patch config_service, which + _check_schedule never reads — it asserted the init default.)""" controller = test_display_controller - schedule_config = {"schedule": {"enabled": False}} - with patch.object(controller.config_service, 'get_config', return_value=schedule_config): + controller.config['schedule'] = { + "enabled": False, + "start_time": "09:00", + "end_time": "17:00", + } + controller._schedule_checked_minute = None + controller._tz = None + controller.is_display_active = False # prove the method flips it back + + with patch('src.display_controller.datetime') as mock_datetime: + mock_datetime.now.return_value.strftime.return_value.lower.return_value = "monday" + mock_datetime.now.return_value.time.return_value = datetime.strptime("20:00", "%H:%M").time() + mock_datetime.strptime = datetime.strptime + controller._check_schedule() assert controller.is_display_active is True def test_active_hours(self, test_display_controller): - """Test active hours check.""" + """A time inside the window activates the display. (This test used + to patch config_service, which _check_schedule never reads — it + asserted the init default.)""" controller = test_display_controller + controller.config['schedule'] = { + "enabled": True, + "start_time": "09:00", + "end_time": "17:00", + } + controller._schedule_checked_minute = None + controller._tz = None + controller.is_display_active = False # prove the method flips it on + with patch('src.display_controller.datetime') as mock_datetime: mock_datetime.now.return_value.strftime.return_value.lower.return_value = "monday" mock_datetime.now.return_value.time.return_value = datetime.strptime("12:00", "%H:%M").time() mock_datetime.strptime = datetime.strptime - schedule_config = { - "schedule": { - "enabled": True, - "start_time": "09:00", - "end_time": "17:00" - } - } - with patch.object(controller.config_service, 'get_config', return_value=schedule_config): - controller._check_schedule() - assert controller.is_display_active is True + controller._check_schedule() + assert controller.is_display_active is True def test_inactive_hours(self, test_display_controller): """Test inactive hours check.""" diff --git a/test/test_display_controller_plugin_toggle.py b/test/test_display_controller_plugin_toggle.py index 0980e729..e09933d0 100644 --- a/test/test_display_controller_plugin_toggle.py +++ b/test/test_display_controller_plugin_toggle.py @@ -6,6 +6,7 @@ These tests cover the reconcile path that loads/unloads plugins and rebuilds the dispatch maps on the main thread when the enabled set changes. """ +import copy from unittest.mock import MagicMock @@ -253,3 +254,182 @@ class TestEnabledSetChanged: {"a": {"enabled": True, "duration": 30}}, {"a": {"enabled": True, "duration": 45}}, ) is False + + +class TestEnabledPluginNotRunning: + """A plugin that fails validate_config() is enabled but absent, and the + config edit that fixes it is nested inside the plugin's own section -- so + the top-level ``enabled`` comparison never sees it. These cover the second + gate that queues a reconcile in that case. + """ + + def test_nested_edit_is_invisible_to_the_enabled_set_check(self, test_display_controller): + """The original gate: proves why a second one is needed.""" + controller = test_display_controller + old = {"hockey-scoreboard": {"enabled": True, "nhl": {"enabled": False}}} + new = {"hockey-scoreboard": {"enabled": True, "nhl": {"enabled": True}}} + # Enabling a league changes no top-level flag. + assert controller._enabled_set_changed(old, new) is False + + def test_queues_reconcile_when_enabled_plugin_is_absent(self, test_display_controller): + controller = test_display_controller + controller.plugin_manager.plugin_manifests = {"hockey-scoreboard": {}} + controller.plugin_manager.discovered_plugin_ids.return_value = {"hockey-scoreboard"} + controller.plugin_display_modes = {} # failed to load + cfg = {"hockey-scoreboard": {"enabled": True, "nhl": {"enabled": True}}} + assert controller._enabled_plugin_not_running(cfg) is True + + def test_quiet_when_every_enabled_plugin_is_running(self, test_display_controller): + controller = test_display_controller + controller.plugin_manager.plugin_manifests = {"hockey-scoreboard": {}} + controller.plugin_manager.discovered_plugin_ids.return_value = {"hockey-scoreboard"} + controller.plugin_display_modes = {"hockey-scoreboard": ["nhl"]} + cfg = {"hockey-scoreboard": {"enabled": True}} + assert controller._enabled_plugin_not_running(cfg) is False + + def test_disabled_plugin_does_not_queue(self, test_display_controller): + controller = test_display_controller + controller.plugin_manager.plugin_manifests = {"hockey-scoreboard": {}} + controller.plugin_manager.discovered_plugin_ids.return_value = {"hockey-scoreboard"} + controller.plugin_display_modes = {} + cfg = {"hockey-scoreboard": {"enabled": False}} + assert controller._enabled_plugin_not_running(cfg) is False + + def test_non_plugin_sections_do_not_queue(self, test_display_controller): + """``schedule``/``display`` carry their own ``enabled`` and are never + in plugin_display_modes -- without the manifest check they would queue + a reconcile, and therefore a filesystem scan, on every config save.""" + controller = test_display_controller + controller.plugin_manager.plugin_manifests = {"hockey-scoreboard": {}} + controller.plugin_manager.discovered_plugin_ids.return_value = {"hockey-scoreboard"} + controller.plugin_display_modes = {"hockey-scoreboard": ["nhl"]} + cfg = { + "hockey-scoreboard": {"enabled": True}, + "schedule": {"enabled": True}, + "display": {"enabled": True}, + } + assert controller._enabled_plugin_not_running(cfg) is False + + def test_non_dict_section_is_ignored(self, test_display_controller): + controller = test_display_controller + controller.plugin_manager.plugin_manifests = {"hockey-scoreboard": {}} + controller.plugin_manager.discovered_plugin_ids.return_value = {"hockey-scoreboard"} + controller.plugin_display_modes = {} + assert controller._enabled_plugin_not_running({"hockey-scoreboard": "nonsense"}) is False + + def test_no_plugin_manager_is_quiet(self, test_display_controller): + controller = test_display_controller + controller.plugin_manager = None + assert controller._enabled_plugin_not_running({"x": {"enabled": True}}) is False + + +class TestReconcileQueuedThroughSubscriber: + """End-to-end through the real config-change subscriber, not the helper. + + Without the second gate this is the four-day-outage path: the plugin is + enabled, absent, and the save that enables its league sets no flag. + """ + + @staticmethod + def _subscriber(controller): + subs = controller.config_service._subscribers['*'] + for cb in subs: + if getattr(cb, '__name__', '') == '_controller_config_change': + return cb + raise AssertionError(f"controller subscriber not found among {subs}") + + @staticmethod + def _configs(controller, plugin_section_old, plugin_section_new): + """Build two full configs differing only inside the plugin section -- + the subscriber refreshes its cache from these, so they must be real.""" + base = copy.deepcopy(controller.config) + old = copy.deepcopy(base) + new = copy.deepcopy(base) + old["hockey-scoreboard"] = plugin_section_old + new["hockey-scoreboard"] = plugin_section_new + return old, new + + def test_nested_edit_queues_reconcile_for_absent_plugin(self, test_display_controller): + controller = test_display_controller + controller.plugin_manager.plugin_manifests = {"hockey-scoreboard": {}} + controller.plugin_manager.discovered_plugin_ids.return_value = {"hockey-scoreboard"} + controller.plugin_display_modes = {} # validate_config() said False + controller._pending_plugin_reconcile = False + + old, new = self._configs( + controller, + {"enabled": True, "nhl": {"enabled": False}}, + {"enabled": True, "nhl": {"enabled": True}}, + ) + # The original gate is blind to this edit ... + assert controller._enabled_set_changed(old, new) is False + self._subscriber(controller)(old, new) + # ... but the reconcile is queued anyway. + assert controller._pending_plugin_reconcile is True + + def test_steady_state_does_not_queue_reconcile(self, test_display_controller): + """Everything enabled is running: an unrelated edit must not queue a + reconcile, or every config save drags a filesystem scan onto the + render thread.""" + controller = test_display_controller + controller.plugin_manager.plugin_manifests = {"hockey-scoreboard": {}} + controller.plugin_manager.discovered_plugin_ids.return_value = {"hockey-scoreboard"} + controller.plugin_display_modes = {"hockey-scoreboard": ["nhl"]} + controller._pending_plugin_reconcile = False + + old, new = self._configs( + controller, + {"enabled": True, "scroll_speed": 1}, + {"enabled": True, "scroll_speed": 2}, + ) + self._subscriber(controller)(old, new) + + assert controller._pending_plugin_reconcile is False + + +class TestPendingReconcileNotLost: + """A config change arriving *during* reconcile must not be discarded. + + The flag used to be cleared after a successful reconcile. Reconcile has + already read its config by then, so that clear erased a request it never + served and the newest config never reconciled -- the same "my save did + nothing" symptom this path exists to prevent. + """ + + def test_request_arriving_during_reconcile_survives(self, test_display_controller): + controller = test_display_controller + controller._pending_plugin_reconcile = True + + def reconcile_and_race(): + # The watcher thread queues another change while we are mid-flight. + with controller._reconcile_flag_lock: + controller._pending_plugin_reconcile = True + return True + + controller._reconcile_enabled_plugins = reconcile_and_race + controller._service_pending_reconcile() + + assert controller._pending_plugin_reconcile is True, \ + "a config change landing during reconcile was discarded" + + def test_flag_cleared_on_a_quiet_success(self, test_display_controller): + controller = test_display_controller + controller._pending_plugin_reconcile = True + controller._reconcile_enabled_plugins = lambda: True + controller._service_pending_reconcile() + assert controller._pending_plugin_reconcile is False + + def test_retryable_failure_rearms(self, test_display_controller): + controller = test_display_controller + controller._pending_plugin_reconcile = True + controller._reconcile_enabled_plugins = lambda: False + controller._service_pending_reconcile() + assert controller._pending_plugin_reconcile is True + + def test_no_reconcile_when_nothing_pending(self, test_display_controller): + controller = test_display_controller + controller._pending_plugin_reconcile = False + calls = [] + controller._reconcile_enabled_plugins = lambda: calls.append(1) or True + controller._service_pending_reconcile() + assert calls == [] diff --git a/test/test_display_controller_schedule.py b/test/test_display_controller_schedule.py new file mode 100644 index 00000000..a7237d4b --- /dev/null +++ b/test/test_display_controller_schedule.py @@ -0,0 +1,277 @@ +""" +Behavioral tests for DisplayController._check_schedule and +_check_dim_schedule — the on/off window and night-dimming logic. + +test_display_controller_optimizations.py::TestScheduleMinuteGate already +covers the once-per-minute gating; this file covers what it doesn't: +midnight-crossing windows, mode selection (global / per-day / legacy +inference), per-day disabled days, invalid time strings, unknown +timezones, boundary equality, and the transition-tracking flags. + +Both methods read only self.config and a handful of instance attributes, +so a bare stub via object.__new__ (the test_display_controller_vegas_tick +pattern) is enough — no managers needed. +""" + +import os +from datetime import datetime + +from unittest.mock import patch + +import pytest + +os.environ.setdefault("EMULATOR", "true") + +from src.display_controller import DisplayController # noqa: E402 + + +def make_controller(config=None, *, normal_brightness=90): + dc = object.__new__(DisplayController) + dc.config = config or {} + dc._tz = None + dc._schedule_checked_minute = None + dc.is_display_active = True + dc._was_display_active = True + dc._normal_brightness = normal_brightness + dc._dim_checked_minute = None + dc._cached_target_brightness = None + dc.is_dimmed = False + dc._was_dimmed = False + return dc + + +def at(time_str, day="monday"): + """Context manager patching the controller module's clock.""" + patcher = patch("src.display_controller.datetime") + mock_dt = patcher.start() + mock_dt.strptime = datetime.strptime + mock_dt.now.return_value.time.return_value = ( + datetime.strptime(time_str, "%H:%M").time()) + mock_dt.now.return_value.strftime.return_value.lower.return_value = day + mock_dt.now.return_value.hour = int(time_str.split(":")[0]) + mock_dt.now.return_value.minute = int(time_str.split(":")[1]) + return patcher + + +@pytest.fixture +def clock(): + patchers = [] + + def _at(time_str, day="monday"): + patchers.append(p := at(time_str, day)) + return p + + yield _at + for p in patchers: + p.stop() + + +def check_at(dc, time_str, day="monday", clock=None): + """Run _check_schedule at a mocked wall time, resetting the minute gate.""" + dc._schedule_checked_minute = None + p = at(time_str, day) + try: + dc._check_schedule() + finally: + p.stop() + return dc.is_display_active + + +def dim_at(dc, time_str, day="monday"): + dc._dim_checked_minute = None + p = at(time_str, day) + try: + return dc._check_dim_schedule() + finally: + p.stop() + + +class TestScheduleWindows: + def _config(self, start, end, **extra): + return {"schedule": {"enabled": True, "start_time": start, + "end_time": end, **extra}, + "timezone": "UTC"} + + def test_same_day_window(self): + dc = make_controller(self._config("09:00", "17:00")) + assert check_at(dc, "12:00") is True + assert check_at(dc, "20:00") is False + assert check_at(dc, "08:59") is False + + def test_boundaries_are_inclusive(self): + dc = make_controller(self._config("09:00", "17:00")) + assert check_at(dc, "09:00") is True # now == start + assert check_at(dc, "17:00") is True # now == end + + def test_midnight_crossing_window(self): + # 21:00 -> 07:00: active late evening AND early morning, inactive + # mid-day. + dc = make_controller(self._config("21:00", "07:00")) + assert check_at(dc, "23:00") is True + assert check_at(dc, "03:00") is True + assert check_at(dc, "12:00") is False + assert check_at(dc, "21:00") is True # boundary + assert check_at(dc, "07:00") is True # boundary + + def test_no_schedule_config_is_always_active(self): + dc = make_controller({"timezone": "UTC"}) + dc.is_display_active = False + dc._check_schedule() + assert dc.is_display_active is True + + def test_invalid_time_string_falls_back_to_active(self): + dc = make_controller(self._config("9 o'clock", "17:00")) + dc.is_display_active = False + assert check_at(dc, "03:00") is True # ValueError -> stay on + + def test_unknown_timezone_falls_back_to_utc(self): + dc = make_controller({"schedule": {"enabled": True, + "start_time": "09:00", + "end_time": "17:00"}, + "timezone": "Mars/Olympus_Mons"}) + assert check_at(dc, "12:00") is True + import pytz + assert dc._tz is pytz.UTC + + +class TestScheduleModes: + DAYS = { + "monday": {"enabled": True, "start_time": "10:00", + "end_time": "18:00"}, + "tuesday": {"enabled": False}, + } + + def test_global_mode_ignores_days(self): + dc = make_controller({"schedule": { + "enabled": True, "mode": "global", + "start_time": "09:00", "end_time": "17:00", + "days": self.DAYS}, "timezone": "UTC"}) + # 09:30 is inside the global window but outside monday's per-day one. + assert check_at(dc, "09:30", day="monday") is True + + def test_per_day_mode_uses_day_window(self): + dc = make_controller({"schedule": { + "enabled": True, "mode": "per-day", + "start_time": "09:00", "end_time": "17:00", + "days": self.DAYS}, "timezone": "UTC"}) + assert check_at(dc, "09:30", day="monday") is False # before 10:00 + assert check_at(dc, "12:00", day="monday") is True + + def test_per_day_underscore_spelling_accepted(self): + dc = make_controller({"schedule": { + "enabled": True, "mode": "per_day", + "days": self.DAYS}, "timezone": "UTC"}) + assert check_at(dc, "12:00", day="monday") is True + + def test_legacy_no_mode_infers_per_day_from_days_config(self): + dc = make_controller({"schedule": { + "enabled": True, + "start_time": "09:00", "end_time": "17:00", + "days": self.DAYS}, "timezone": "UTC"}) + assert check_at(dc, "09:30", day="monday") is False # per-day won + + def test_per_day_disabled_day_turns_display_off(self): + dc = make_controller({"schedule": { + "enabled": True, "mode": "per-day", + "days": self.DAYS}, "timezone": "UTC"}) + assert check_at(dc, "12:00", day="tuesday") is False + + def test_per_day_missing_day_falls_back_to_global(self): + dc = make_controller({"schedule": { + "enabled": True, "mode": "per-day", + "start_time": "09:00", "end_time": "17:00", + "days": self.DAYS}, "timezone": "UTC"}) + # Wednesday has no per-day entry -> global window applies. + assert check_at(dc, "09:30", day="wednesday") is True + + def test_missing_enabled_key_means_enabled(self): + # Backward compat: schedules written before the enabled flag. + dc = make_controller({"schedule": { + "start_time": "09:00", "end_time": "17:00"}, "timezone": "UTC"}) + assert check_at(dc, "20:00") is False + + +class TestScheduleTransitions: + def test_was_display_active_tracks_state(self): + dc = make_controller({"schedule": {"enabled": True, + "start_time": "09:00", + "end_time": "17:00"}, + "timezone": "UTC"}) + check_at(dc, "12:00") + assert dc._was_display_active is True + check_at(dc, "20:00") + assert dc._was_display_active is False + check_at(dc, "12:05") + assert dc._was_display_active is True + + +class TestDimSchedule: + def _config(self, start="20:00", end="07:00", **extra): + return {"dim_schedule": {"enabled": True, "start_time": start, + "end_time": end, "dim_brightness": 25, + **extra}, + "timezone": "UTC"} + + def test_disabled_by_default(self): + dc = make_controller({"dim_schedule": {"start_time": "20:00", + "end_time": "07:00"}, + "timezone": "UTC"}) + # Unlike the on/off schedule, dimming defaults to DISABLED when the + # enabled key is missing. + assert dim_at(dc, "23:00") == 90 + assert dc.is_dimmed is False + + def test_overnight_dim_window(self): + dc = make_controller(self._config()) + assert dim_at(dc, "23:00") == 25 + assert dc.is_dimmed is True + assert dim_at(dc, "03:00") == 25 + assert dim_at(dc, "12:00") == 90 + assert dc.is_dimmed is False + + def test_dim_brightness_defaults_to_30(self): + dc = make_controller({"dim_schedule": {"enabled": True, + "start_time": "20:00", + "end_time": "07:00"}, + "timezone": "UTC"}) + assert dim_at(dc, "23:00") == 30 + + def test_inactive_display_short_circuits_undimmed(self): + dc = make_controller(self._config()) + dc.is_display_active = False + dc.is_dimmed = True + assert dim_at(dc, "23:00") == 90 + assert dc.is_dimmed is False + + def test_per_day_mode(self): + dc = make_controller(self._config(mode="per-day", days={ + "monday": {"enabled": True, "start_time": "22:00", + "end_time": "06:00"}, + "tuesday": {"enabled": False}, + })) + assert dim_at(dc, "23:00", day="monday") == 25 + assert dim_at(dc, "21:00", day="monday") == 90 # before per-day start + assert dim_at(dc, "23:00", day="tuesday") == 90 # day disabled + assert dc.is_dimmed is False + + def test_no_legacy_inference_for_dim(self): + # Unlike _check_schedule, dim mode defaults to GLOBAL even when a + # days config exists — no legacy inference. + dc = make_controller(self._config(days={ + "monday": {"enabled": True, "start_time": "22:00", + "end_time": "06:00"}, + })) + # 21:00 is inside the global 20:00-07:00 window but outside monday's + # per-day 22:00 start; global mode wins. + assert dim_at(dc, "21:00", day="monday") == 25 + + def test_invalid_time_string_returns_normal(self): + dc = make_controller(self._config(start="late")) + assert dim_at(dc, "23:00") == 90 + + def test_was_dimmed_tracks_transitions(self): + dc = make_controller(self._config()) + dim_at(dc, "23:00") + assert dc._was_dimmed is True + dim_at(dc, "12:00") + assert dc._was_dimmed is False diff --git a/test/test_display_controller_vegas_tick.py b/test/test_display_controller_vegas_tick.py index 643359f3..538e9272 100644 --- a/test/test_display_controller_vegas_tick.py +++ b/test/test_display_controller_vegas_tick.py @@ -11,9 +11,18 @@ orphaning VegasModeCoordinator.mark_plugin_updated() -- it has had zero callers since. """ +import os from typing import Dict, List, Optional from unittest.mock import MagicMock +# display_controller imports display_manager, which imports the hardware +# rgbmatrix module unless EMULATOR=true was set before import. Use the +# emulator (same convention as test_display_dirty_tracking.py and +# test/plugins/conftest.py) so this file collects on machines without the +# hardware library — and so display_manager gets the emulator binding no +# matter which test module imports it first. +os.environ.setdefault("EMULATOR", "true") + from src.display_controller import DisplayController diff --git a/test/test_display_helper.py b/test/test_display_helper.py new file mode 100644 index 00000000..b8c9be63 --- /dev/null +++ b/test/test_display_helper.py @@ -0,0 +1,307 @@ +"""Tests for src/common/display_helper.py (DisplayHelper). + +Pure-PIL tests, no hardware or mocks required. Pixel assertions rely on +getbbox()/getpixel() rather than exact text pixel counts, because the +default-font metrics vary across Pillow versions. + +These tests pin the FIXED behaviors on this branch: +- draw_error_message / draw_no_data_message return a rendered image + (they previously crashed with AttributeError), +- draw_scorebug_layout draws period/status/clock as one combined top + line (previously overprinted at the same y), +- draw_ticker_layout draws at x=0 (previously started at + x=display_width, i.e. entirely off-canvas -> blank frames). +""" + +from PIL import Image, ImageDraw, ImageFont + +from src.common.display_helper import DisplayHelper + + +def default_font(): + return ImageFont.load_default() + + +def make_helper(width=128, height=32): + return DisplayHelper(width, height) + + +class TestCreateBaseImage: + def test_default_is_black_rgb_display_sized(self): + helper = make_helper() + img = helper.create_base_image() + assert img.size == (128, 32) + assert img.mode == 'RGB' + assert img.getpixel((0, 0)) == (0, 0, 0) + assert img.getpixel((127, 31)) == (0, 0, 0) + # Entirely black -> no bounding box in luminance + assert img.convert('L').getbbox() is None + + def test_custom_background_color(self): + helper = make_helper() + img = helper.create_base_image(background_color=(10, 20, 30)) + assert img.getpixel((0, 0)) == (10, 20, 30) + assert img.getpixel((64, 16)) == (10, 20, 30) + + def test_mode_rgba_is_honored(self): + helper = make_helper() + img = helper.create_base_image(mode='RGBA') + assert img.mode == 'RGBA' + assert img.size == (128, 32) + + +class TestCreateOverlay: + def test_overlay_is_transparent_rgba(self): + helper = make_helper() + overlay = helper.create_overlay() + assert overlay.mode == 'RGBA' + assert overlay.size == (128, 32) + assert overlay.getpixel((0, 0)) == (0, 0, 0, 0) + assert overlay.getpixel((127, 31)) == (0, 0, 0, 0) + + +class TestCompositeImages: + def test_rgb_inputs_are_upconverted_and_result_is_rgba(self): + helper = make_helper() + base = Image.new('RGB', (128, 32), (0, 0, 0)) + overlay = Image.new('RGB', (128, 32), (255, 0, 0)) + result = helper.composite_images(base, overlay) + assert result.mode == 'RGBA' + assert result.size == base.size + # RGB->RGBA conversion yields a fully opaque overlay + assert result.getpixel((0, 0)) == (255, 0, 0, 255) + + def test_transparent_overlay_leaves_base_visible(self): + helper = make_helper() + base = Image.new('RGB', (128, 32), (5, 6, 7)) + overlay = helper.create_overlay() + result = helper.composite_images(base, overlay) + assert result.mode == 'RGBA' + assert result.getpixel((64, 16)) == (5, 6, 7, 255) + + +class TestScorebugLayout: + def test_full_game_data_renders(self): + helper = make_helper() + font = default_font() + fonts = {'time': font, 'status': font, 'score': font, 'team': font} + game_data = { + 'home_score': 3, 'away_score': 2, + 'home_abbr': 'NYY', 'away_abbr': 'BOS', + 'status_text': 'LIVE', 'period_text': 'T9', 'clock': '2:30', + } + img = helper.draw_scorebug_layout(game_data, fonts) + assert img.mode == 'RGB' + assert img.size == (128, 32) + assert img.convert('L').getbbox() is not None + + def test_empty_game_data_uses_defaults_without_raising(self): + helper = make_helper() + font = default_font() + fonts = {'time': font, 'status': font, 'score': font, 'team': font} + img = helper.draw_scorebug_layout({}, fonts) + assert img.mode == 'RGB' + assert img.size == (128, 32) + # Defaults '0'/'HOME'/'AWAY' actually render something + assert img.convert('L').getbbox() is not None + + def test_empty_fonts_dict_falls_back_to_default_font(self): + # Pin: fonts={} must not raise — PIL falls back to the default + # font when font=None is passed through. + helper = make_helper() + img = helper.draw_scorebug_layout( + {'status_text': 'FINAL', 'period_text': 'Q4', 'clock': '0:00'}, {}) + assert img.size == (128, 32) + assert img.convert('L').getbbox() is not None + + def test_top_line_is_one_combined_centered_draw(self): + # FIXED behavior: period/status/clock are joined into a single + # top line drawn once at y=1 instead of three overprinted draws. + helper = make_helper() + calls = [] + original = helper._draw_centered_text + + def spy(draw, text, font, y_position): + calls.append({'text': text, 'y_position': y_position}) + original(draw, text, font, y_position) + + helper._draw_centered_text = spy + font = default_font() + fonts = {'time': font, 'status': font, 'score': font, 'team': font} + helper.draw_scorebug_layout( + {'period_text': 'Q4', 'status_text': 'LIVE', 'clock': '2:30'}, + fonts) + + top_calls = [c for c in calls if c['y_position'] == 1] + assert len(top_calls) == 1 + text = top_calls[0]['text'] + assert 'Q4' in text + assert 'LIVE' in text + assert '2:30' in text + + def test_no_top_line_when_all_parts_empty(self): + helper = make_helper() + calls = [] + original = helper._draw_centered_text + + def spy(draw, text, font, y_position): + calls.append(y_position) + original(draw, text, font, y_position) + + helper._draw_centered_text = spy + font = default_font() + helper.draw_scorebug_layout({}, {'score': font, 'team': font}) + assert 1 not in calls # no combined top line drawn + + def test_logo_positions_bleed_off_edges(self): + # Home logo pastes at x = width - logo.width + 10 (right edge, + # bleeding off-screen right); away at x = -10 (bleeding left). + helper = make_helper() + home_logo = Image.new('RGBA', (20, 20), (0, 0, 255, 255)) # blue + away_logo = Image.new('RGBA', (20, 20), (255, 0, 0, 255)) # red + # Empty abbrs/status so text can't land on the probed pixels. + game_data = {'home_abbr': '', 'away_abbr': ''} + font = default_font() + img = helper.draw_scorebug_layout(game_data, {'score': font}, + home_logo=home_logo, + away_logo=away_logo) + # center_y = 16; logos span y 6..25 -> probe y=16 at both edges. + assert img.getpixel((0, 16)) == (255, 0, 0) # away (left edge) + assert img.getpixel((127, 16)) == (0, 0, 255) # home (right edge) + # And the off-screen parts are truly clipped: image is still 128 wide + assert img.size == (128, 32) + + +class TestTickerLayout: + def test_frame_is_not_blank(self): + # FIXED behavior: text now starts at x=0. Previously it was drawn + # at x=display_width, entirely off-canvas, so frames were blank. + helper = make_helper() + img = helper.draw_ticker_layout('HELLO WORLD', default_font()) + assert img.size == (128, 32) + assert img.mode == 'RGB' + assert img.convert('L').getbbox() is not None + + def test_text_starts_at_left_edge(self): + helper = make_helper() + img = helper.draw_ticker_layout('HELLO', default_font()) + bbox = img.convert('L').getbbox() + assert bbox is not None + # Text is positioned at x=0 (outline extends 1px left, clipped), + # so ink begins hugging the left edge. Allow a couple of pixels of + # slack for font-dependent left-side bearing. + assert bbox[0] <= 2 + + def test_scroll_speed_does_not_affect_frame(self): + # Pin: scroll_speed is accepted for API compatibility only. + helper = make_helper() + font = default_font() + img1 = helper.draw_ticker_layout('SCROLLING', font, scroll_speed=1) + img5 = helper.draw_ticker_layout('SCROLLING', font, scroll_speed=5) + assert img1.tobytes() == img5.tobytes() + + def test_custom_colors(self): + helper = make_helper() + img = helper.draw_ticker_layout('X', default_font(), + background_color=(0, 0, 40), + text_color=(0, 255, 0)) + assert img.getpixel((127, 0)) == (0, 0, 40) # background corner + colors = {img.getpixel((x, y)) + for x in range(img.width) for y in range(img.height)} + # Text color appears somewhere (anti-aliasing may blend it, so + # check for a green-dominant pixel rather than the exact color). + assert any(g > 150 and r < 100 for (r, g, b) in colors) + + +class TestCenteredText: + def test_renders_centered_text_on_background(self): + helper = make_helper() + img = helper.draw_centered_text('HI', default_font(), + background_color=(0, 0, 60), + text_color=(255, 255, 0)) + assert img.size == (128, 32) + assert img.convert('L').getbbox() is not None + # Corners stay pure background + assert img.getpixel((0, 0)) == (0, 0, 60) + assert img.getpixel((127, 0)) == (0, 0, 60) + assert img.getpixel((0, 31)) == (0, 0, 60) + assert img.getpixel((127, 31)) == (0, 0, 60) + + +class TestErrorAndNoDataMessages: + def test_draw_error_message_returns_rendered_image(self): + # FIXED behavior: used to crash with AttributeError; now returns + # a rendered image on a dark red background. + helper = make_helper() + img = helper.draw_error_message('Boom') + assert img.size == (128, 32) + assert img.mode == 'RGB' + assert img.convert('L').getbbox() is not None + assert img.getpixel((0, 0)) == (50, 0, 0) # dark red background + + def test_draw_error_message_default_text(self): + helper = make_helper() + img = helper.draw_error_message() + assert img.size == (128, 32) + assert img.getpixel((127, 31)) == (50, 0, 0) + + def test_draw_no_data_message_returns_rendered_image(self): + helper = make_helper() + img = helper.draw_no_data_message() + assert img.size == (128, 32) + assert img.mode == 'RGB' + assert img.convert('L').getbbox() is not None + assert img.getpixel((0, 0)) == (0, 0, 0) # black background + + +class TestDrawTextWithOutline: + def test_fill_color_appears_in_output(self): + helper = make_helper() + img = Image.new('RGB', (40, 20), (0, 0, 255)) + draw = ImageDraw.Draw(img) + helper._draw_text_with_outline(draw, 'X', (5, 2), default_font(), + fill=(255, 0, 0)) + pixels = {img.getpixel((x, y)) + for x in range(img.width) for y in range(img.height)} + # Anti-aliased fonts blend edge pixels, so look for red-dominant + # (fill) and near-black (outline) pixels rather than exact colors. + assert any(r > 150 and g < 50 for (r, g, b) in pixels) # fill + assert any(max(p) < 80 for p in pixels) # outline + + def test_default_fill_is_white(self): + helper = make_helper() + img = Image.new('RGB', (40, 20), (0, 0, 255)) + draw = ImageDraw.Draw(img) + helper._draw_text_with_outline(draw, 'X', (5, 2), default_font()) + pixels = {img.getpixel((x, y)) + for x in range(img.width) for y in range(img.height)} + # White-dominant pixel present (exact white may be anti-aliased) + assert any(r > 200 and g > 200 for (r, g, b) in pixels) + + +class TestOrientationAndDimensions: + def test_landscape_display(self): + helper = DisplayHelper(128, 32) + assert helper.is_landscape() is True + assert helper.is_portrait() is False + + def test_portrait_display(self): + helper = DisplayHelper(32, 128) + assert helper.is_portrait() is True + assert helper.is_landscape() is False + + def test_square_display_is_neither(self): + # Pin: a square display is neither portrait nor landscape. + helper = DisplayHelper(64, 64) + assert helper.is_portrait() is False + assert helper.is_landscape() is False + + def test_get_center_position(self): + assert DisplayHelper(128, 32).get_center_position() == (64, 16) + + def test_get_center_position_floors_odd_dimensions(self): + assert DisplayHelper(65, 33).get_center_position() == (32, 16) + + def test_get_display_dimensions(self): + assert DisplayHelper(128, 32).get_display_dimensions() == (128, 32) + assert DisplayHelper(64, 64).get_display_dimensions() == (64, 64) diff --git a/test/test_display_manager.py b/test/test_display_manager.py index 63c49114..9ccb7773 100644 --- a/test/test_display_manager.py +++ b/test/test_display_manager.py @@ -1,6 +1,15 @@ +import os import pytest from unittest.mock import MagicMock, patch from PIL import ImageDraw + +# display_manager imports the hardware rgbmatrix module at import time unless +# EMULATOR=true. Use the emulator (same convention as +# test_display_dirty_tracking.py) so this file collects standalone instead of +# relying on collection order — the tests below patch RGBMatrix/ +# RGBMatrixOptions explicitly, so the underlying binding doesn't matter here. +os.environ.setdefault("EMULATOR", "true") + from src.display_manager import DisplayManager @pytest.fixture @@ -77,18 +86,26 @@ class TestDisplayManagerDrawing: assert dm.matrix.Clear.called def test_draw_text(self, test_config, mock_rgb_matrix): - """Test text drawing.""" + """Text drawn through draw_text must actually light pixels.""" + from PIL import Image, ImageDraw, ImageFont + import src.display_manager as dm_mod with patch.dict('os.environ', {'EMULATOR': 'false'}): - dm = DisplayManager(test_config) - - # Mock font - font = MagicMock() - - dm.draw_text("Test", 0, 0, font) - - # Verify draw_text was called (DisplayManager uses freetype/PIL) - # The actual implementation uses freetype or PIL, not graphics module - assert True # draw_text should execute without error + DisplayManager._instance = None + dm = DisplayManager(test_config, suppress_test_pattern=True) + # The fixture replaces the module's freetype with a MagicMock, + # which breaks draw_text's isinstance(font, freetype.Face) check + # (and silently swallows the draw). Give the mock a real class so + # isinstance works and the PIL path is taken. + dm_mod.freetype.Face = type("_FakeFace", (), {}) + # Start from a known-black canvas so the assertion below can only + # pass if draw_text itself lit something. + dm.image = Image.new('RGB', (dm.width, dm.height)) + dm.draw = ImageDraw.Draw(dm.image) + + dm.draw_text("Test", 0, 0, font=ImageFont.load_default()) + + assert dm.image.convert("L").getbbox() is not None, \ + "draw_text lit no pixels" def test_draw_image(self, test_config, mock_rgb_matrix): """Test image drawing.""" @@ -220,3 +237,45 @@ class TestDisplayManagerDoubleSided: suppress_test_pattern=True) assert dm.set_brightness(70) is True assert mock_rgb_matrix['matrix_instance'].brightness == 70 + + +class TestDisplayManagerOrientation: + """The orientation setting composes onto pixel_mapper_config for panels + mounted upside down, without disturbing a custom pixel_mapper_config.""" + + def _config(self, **hardware_overrides): + config = { + 'display': { + 'hardware': { + 'rows': 32, 'cols': 64, 'chain_length': 2, 'parallel': 1, + 'hardware_mapping': 'adafruit-hat-pwm', 'brightness': 90, + }, + 'runtime': {'gpio_slowdown': 2}, + }, + 'timezone': 'UTC', + 'plugin_system': {'plugins_directory': 'plugins'}, + } + config['display']['hardware'].update(hardware_overrides) + return config + + def test_default_orientation_leaves_pixel_mapper_config_untouched(self, mock_rgb_matrix): + DisplayManager._instance = None + with patch.dict('os.environ', {'EMULATOR': 'false'}): + DisplayManager(self._config(), suppress_test_pattern=True) + options = mock_rgb_matrix['options_class'].return_value + assert options.pixel_mapper_config == '' + + def test_orientation_180_appends_rotate_mapper(self, mock_rgb_matrix): + DisplayManager._instance = None + with patch.dict('os.environ', {'EMULATOR': 'false'}): + DisplayManager(self._config(orientation='180'), suppress_test_pattern=True) + options = mock_rgb_matrix['options_class'].return_value + assert options.pixel_mapper_config == 'Rotate:180' + + def test_orientation_180_composes_with_existing_pixel_mapper_config(self, mock_rgb_matrix): + DisplayManager._instance = None + with patch.dict('os.environ', {'EMULATOR': 'false'}): + DisplayManager(self._config(orientation='180', pixel_mapper_config='U-mapper'), + suppress_test_pattern=True) + options = mock_rgb_matrix['options_class'].return_value + assert options.pixel_mapper_config == 'U-mapper;Rotate:180' diff --git a/test/test_doc_links.py b/test/test_doc_links.py new file mode 100644 index 00000000..08f904af --- /dev/null +++ b/test/test_doc_links.py @@ -0,0 +1,43 @@ +"""Guard: relative markdown links in active docs must resolve. + +Scans repo-root *.md and docs/ (excluding docs/archive/, which is allowed +to rot). External URLs, mailto links, and pure anchors are skipped, as are +links inside fenced code blocks. +""" +import re +from pathlib import Path +from typing import Iterator + +PROJECT_ROOT = Path(__file__).resolve().parent.parent +LINK_RE = re.compile(r'\[[^\]]*\]\(([^)\s]+)\)') +FENCE_RE = re.compile(r'^(```|~~~)') + + +def _md_files() -> Iterator[Path]: + """Yield active markdown files (repo root + docs/, excluding docs/archive/).""" + yield from PROJECT_ROOT.glob('*.md') + for path in PROJECT_ROOT.glob('docs/**/*.md'): + if 'archive' not in path.parts: + yield path + + +def test_relative_markdown_links_resolve() -> None: + """Every relative markdown link outside code fences must resolve on disk.""" + broken = [] + for md in _md_files(): + in_fence = False + for lineno, line in enumerate(md.read_text(encoding='utf-8').splitlines(), 1): + if FENCE_RE.match(line.strip()): + in_fence = not in_fence + continue + if in_fence: + continue + for target in LINK_RE.findall(line): + if target.startswith(('http://', 'https://', 'mailto:', '#')): + continue + resolved = (md.parent / target.split('#')[0]).resolve() + if not resolved.exists(): + broken.append( + f'{md.relative_to(PROJECT_ROOT)}:{lineno} -> {target}' + ) + assert not broken, 'Broken relative markdown links:\n' + '\n'.join(broken) diff --git a/test/test_dynamic_team_resolver.py b/test/test_dynamic_team_resolver.py new file mode 100644 index 00000000..c562d0d9 --- /dev/null +++ b/test/test_dynamic_team_resolver.py @@ -0,0 +1,259 @@ +""" +Tests for src/dynamic_team_resolver.py (DynamicTeamResolver). + +Covers dynamic team expansion (AP_TOP_5/10/25), order-preserving dedup, +unknown dynamic-name dropping, rankings parsing, the fixed genuinely +class-shared rankings cache (fetch and clear_cache write through +DynamicTeamResolver._rankings_cache / _cache_timestamp), TTL expiry, +network-failure resilience, and the resolve_dynamic_teams module function. + +No real network: src.dynamic_team_resolver.requests.get is always patched. +""" + +import types +from unittest.mock import MagicMock, patch + +import pytest +import requests + +import src.dynamic_team_resolver as dtr_module +from src.dynamic_team_resolver import DynamicTeamResolver, resolve_dynamic_teams + + +TOP_TEAMS = ['UGA', 'MICH', 'OSU', 'TEX', 'ALA', 'ORE', 'PSU', 'ND', 'FSU', 'OU'] + + +def _rankings_payload(teams=None): + teams = TOP_TEAMS if teams is None else teams + return { + 'rankings': [{ + 'name': 'AP Top 25', + 'ranks': [ + {'current': i + 1, 'team': {'abbreviation': abbr}} + for i, abbr in enumerate(teams) + ], + }] + } + + +def _make_response(payload): + response = MagicMock() + response.json.return_value = payload + response.raise_for_status.return_value = None + return response + + +@pytest.fixture(autouse=True) +def reset_class_cache(): + """Reset the CLASS-level shared cache between tests.""" + DynamicTeamResolver._rankings_cache = {} + DynamicTeamResolver._cache_timestamp = 0 + yield + DynamicTeamResolver._rankings_cache = {} + DynamicTeamResolver._cache_timestamp = 0 + + +@pytest.fixture +def mock_get(): + with patch('src.dynamic_team_resolver.requests.get') as m: + m.return_value = _make_response(_rankings_payload()) + yield m + + +@pytest.fixture +def resolver(): + return DynamicTeamResolver() + + +# --------------------------------------------------------------------------- +# resolve_teams basics +# --------------------------------------------------------------------------- + +class TestResolveTeamsBasics: + def test_empty_list_returns_empty_no_http(self, resolver, mock_get): + assert resolver.resolve_teams([]) == [] + mock_get.assert_not_called() + + def test_no_dynamic_names_passthrough_no_http(self, resolver, mock_get): + assert resolver.resolve_teams(['UGA', 'AUB', 'LSU']) == [ + 'UGA', 'AUB', 'LSU'] + mock_get.assert_not_called() + + def test_expansion_inserted_in_place_order_preserved( + self, resolver, mock_get): + result = resolver.resolve_teams(['UGA', 'AP_TOP_5', 'AUB']) + + # UGA is also ranked #1, so dedup keeps its first occurrence; the + # top-5 expansion lands where AP_TOP_5 appeared, AUB stays after. + assert result == ['UGA', 'MICH', 'OSU', 'TEX', 'ALA', 'AUB'] + + def test_order_preserving_dedup(self, resolver, mock_get): + result = resolver.resolve_teams(['UGA', 'AP_TOP_5', 'UGA']) + + assert result == ['UGA', 'MICH', 'OSU', 'TEX', 'ALA'] + assert result.count('UGA') == 1 + assert result[0] == 'UGA' + + +# --------------------------------------------------------------------------- +# AP_TOP_N slicing +# --------------------------------------------------------------------------- + +class TestSlicing: + def test_top_n_counts_and_order(self, resolver, mock_get): + teams_25 = [f'T{i:02d}' for i in range(1, 26)] + mock_get.return_value = _make_response(_rankings_payload(teams_25)) + + top5 = resolver.resolve_teams(['AP_TOP_5']) + top10 = resolver.resolve_teams(['AP_TOP_10']) + top25 = resolver.resolve_teams(['AP_TOP_25']) + + assert top5 == teams_25[:5] + assert top10 == teams_25[:10] + assert top25 == teams_25 + + +# --------------------------------------------------------------------------- +# Unknown dynamic-looking names +# --------------------------------------------------------------------------- + +class TestUnknownDynamicNames: + def test_unknown_dynamic_looking_names_dropped(self, resolver, mock_get): + result = resolver.resolve_teams( + ['AP_TOP_100', 'TOP_10', 'RANKED_ALL', 'PLAYOFF_TEAMS']) + + assert result == [] + mock_get.assert_not_called() + + def test_top_substring_hazard(self, resolver, mock_get): + # Hazard pin: _is_potential_dynamic_team matches the substring + # 'TOP_' anywhere in the (upper-cased) name, so a team literally + # named 'TOP_GUN' is dropped as an unknown dynamic team too. + assert resolver.resolve_teams(['TOP_GUN']) == [] + + +# --------------------------------------------------------------------------- +# Rankings parsing +# --------------------------------------------------------------------------- + +class TestRankingsParsing: + def test_drops_zero_rank_and_empty_abbreviation_sorts_ascending( + self, resolver, mock_get): + payload = { + 'rankings': [{ + 'name': 'AP Top 25', + 'ranks': [ + {'current': 3, 'team': {'abbreviation': 'C3'}}, + {'current': 1, 'team': {'abbreviation': 'A1'}}, + {'current': 0, 'team': {'abbreviation': 'ZERO'}}, + {'current': 4, 'team': {'abbreviation': ''}}, + {'current': 2, 'team': {'abbreviation': 'B2'}}, + ], + }] + } + mock_get.return_value = _make_response(payload) + + rankings = resolver._fetch_ncaa_fb_rankings() + + assert list(rankings.keys()) == ['A1', 'B2', 'C3'] + assert list(rankings.values()) == [1, 2, 3] + + def test_empty_rankings_returns_empty_and_caches_nothing( + self, resolver, mock_get): + mock_get.return_value = _make_response({'rankings': []}) + + assert resolver._fetch_ncaa_fb_rankings() == {} + # Nothing was cached, so the next call hits HTTP again. + assert resolver._fetch_ncaa_fb_rankings() == {} + assert mock_get.call_count == 2 + + +# --------------------------------------------------------------------------- +# Shared class cache (fixed behavior) +# --------------------------------------------------------------------------- + +class TestSharedCache: + def test_cache_shared_across_instances(self, mock_get): + resolver1 = DynamicTeamResolver() + resolver1.resolve_teams(['AP_TOP_5']) + assert mock_get.call_count == 1 + + resolver2 = DynamicTeamResolver() + result = resolver2.resolve_teams(['AP_TOP_5']) + + # Post-fix: the class-level cache serves the second instance with + # ZERO additional HTTP calls. + assert result == TOP_TEAMS[:5] + assert mock_get.call_count == 1 + + def test_ttl_expiry_refetches(self, resolver, mock_get, monkeypatch): + resolver.resolve_teams(['AP_TOP_5']) + assert mock_get.call_count == 1 + + stamp = DynamicTeamResolver._cache_timestamp + monkeypatch.setattr( + dtr_module, 'time', types.SimpleNamespace(time=lambda: stamp + 3601)) + + resolver.resolve_teams(['AP_TOP_5']) + assert mock_get.call_count == 2 + + def test_clear_cache_through_one_instance_affects_all(self, mock_get): + resolver1 = DynamicTeamResolver() + resolver1.resolve_teams(['AP_TOP_5']) + assert mock_get.call_count == 1 + + resolver2 = DynamicTeamResolver() + resolver2.clear_cache() + + # Post-fix: clear_cache writes through the class, so resolver1 + # must refetch even though resolver2 did the clearing. + resolver1.resolve_teams(['AP_TOP_5']) + assert mock_get.call_count == 2 + + def test_module_function_benefits_from_class_cache(self, mock_get): + # resolve_dynamic_teams constructs a fresh resolver per call, but + # the class-shared cache means only the first call hits HTTP. + first = resolve_dynamic_teams(['AP_TOP_5']) + second = resolve_dynamic_teams(['AP_TOP_5']) + + assert first == second == TOP_TEAMS[:5] + assert mock_get.call_count == 1 + + +# --------------------------------------------------------------------------- +# Failure handling +# --------------------------------------------------------------------------- + +class TestFailureHandling: + def test_network_failure_drops_dynamic_keeps_static_caches_nothing( + self, resolver, mock_get): + mock_get.side_effect = [ + requests.exceptions.RequestException('boom'), + _make_response(_rankings_payload()), + ] + + result = resolver.resolve_teams(['UGA', 'AP_TOP_5']) + + # Dynamic name silently dropped, static name kept, nothing raises. + assert result == ['UGA'] + + # Nothing was cached on failure: a subsequent call refetches and + # succeeds. + result = resolver.resolve_teams(['UGA', 'AP_TOP_5']) + assert result == ['UGA', 'MICH', 'OSU', 'TEX', 'ALA'] + assert mock_get.call_count == 2 + + +# --------------------------------------------------------------------------- +# sport argument +# --------------------------------------------------------------------------- + +class TestSportArgument: + def test_sport_arg_ignored_for_expansion(self, resolver, mock_get): + # Pin: the sport argument is effectively ignored — each pattern + # carries its own sport ('ncaa_fb'), so passing sport='nfl' still + # expands from the college-football rankings. + result = resolver.resolve_teams(['AP_TOP_5'], sport='nfl') + + assert result == TOP_TEAMS[:5] + assert mock_get.call_count == 1 diff --git a/test/test_enum_option_labels.py b/test/test_enum_option_labels.py new file mode 100644 index 00000000..1e8d248e --- /dev/null +++ b/test/test_enum_option_labels.py @@ -0,0 +1,134 @@ +"""Guard: enum dropdowns in the plugin config form honour x-options.labels. + +The form derives an option's visible text from its value — underscores +replaced, title case applied ("day_first" -> "Day First"). That cannot +express every label a schema needs: "vs" reads as "Vs", and "abbrev" says +nothing about the "Sep 19" it produces. Schemas can supply x-options.labels +instead, the same convention the checkbox-group widget already uses. + +These tests extract the enum `. +ENUM_BLOCK_RE = re.compile( + r"(\{%\s*set enum_labels\s*=.*?)", re.S +) + + +def _shipped_enum_block() -> str: + """Return the live enum block in plugin_config.html — the ' + 'template changed shape and this guard needs updating' + ) + return match.group(1) + + +def _render(prop: dict, value=None) -> str: + """Render the shipped enum block with a minimal fixture.""" + env = Environment(loader=DictLoader({'f': _shipped_enum_block()}), + autoescape=True) + return env.get_template('f').render( + prop=prop, value=value, field_id='fid', full_key='k' + ) + + +def _option_labels(html: str) -> dict: + """Map each rendered option's value to its visible text.""" + return { + value: text.strip() + for value, text in re.findall( + r'', html, re.S + ) + } + + +def test_labels_are_used_when_supplied() -> None: + html = _render({ + 'enum': ['vs', 'date_time'], + 'x-options': {'labels': {'vs': 'VS', 'date_time': 'Date and time'}}, + }) + assert _option_labels(html) == {'vs': 'VS', 'date_time': 'Date and time'} + + +def test_unlabelled_values_keep_the_humanised_fallback() -> None: + """Schemas without labels must render exactly as they did before.""" + html = _render({'enum': ['day_first', 'weekday']}) + assert _option_labels(html) == {'day_first': 'Day First', 'weekday': 'Weekday'} + + +def test_partial_labels_fall_back_per_value() -> None: + """A labels map covering some values leaves the rest humanised.""" + html = _render({'enum': ['vs', 'day_first'], + 'x-options': {'labels': {'vs': 'VS'}}}) + assert _option_labels(html) == {'vs': 'VS', 'day_first': 'Day First'} + + +def test_option_values_are_unchanged_by_labelling() -> None: + """Labels are display-only: the submitted value stays the enum value.""" + html = _render({'enum': ['abbrev'], + 'x-options': {'labels': {'abbrev': 'Sep 19'}}}) + assert _option_labels(html) == {'abbrev': 'Sep 19'} + + +def test_selected_option_still_tracks_the_current_value() -> None: + """Labelling must not disturb which option is marked selected.""" + html = _render({'enum': ['abbrev', 'numeric'], + 'x-options': {'labels': {'abbrev': 'Sep 19'}}}, + value='numeric') + selected = re.search(r'