diff --git a/.cursor/README.md b/.cursor/README.md deleted file mode 100644 index 0b6d621f..00000000 --- a/.cursor/README.md +++ /dev/null @@ -1,145 +0,0 @@ -# Cursor Helper Files for LEDMatrix Plugin Development - -This directory contains Cursor-specific helper files to assist with plugin development in the LEDMatrix project. - -## Files Overview - -### `.cursorrules` -Comprehensive rules file that Cursor uses to understand plugin development patterns, best practices, and workflows. This file is automatically loaded by Cursor and helps guide AI-assisted development. - -### `plugins_guide.md` -Detailed guide covering: -- Plugin system overview -- Creating new plugins -- Running plugins (emulator and hardware) -- Loading and configuring plugins -- Development workflow -- Testing strategies -- Troubleshooting - -### `plugin_templates/` -Template files for quick plugin creation: -- `manifest.json.template` - Plugin metadata template -- `manager.py.template` - Plugin class template -- `config_schema.json.template` - Configuration schema template -- `README.md.template` - Plugin documentation template -- `requirements.txt.template` - Dependencies template -- `QUICK_START.md` - Quick start guide for using templates - -## Quick Reference - -### Creating a New Plugin - -1. **Using templates** (recommended): -```bash -# See QUICK_START.md in plugin_templates/ -cd plugins -mkdir my-plugin -cd my-plugin -cp ../../.cursor/plugin_templates/*.template . -# Edit files, replacing PLUGIN_ID and other placeholders -``` - -2. **Using dev_plugin_setup.sh**: -```bash -# Link from GitHub -./scripts/dev/dev_plugin_setup.sh link-github my-plugin - -# Link local repo -./scripts/dev/dev_plugin_setup.sh link my-plugin /path/to/repo -``` - -### Running the Display - -```bash -# Emulator mode (development, no hardware required) -python3 run.py --emulator -# (equivalent: EMULATOR=true python3 run.py) - -# Hardware (production, requires the rpi-rgb-led-matrix submodule built) -python3 run.py - -# As a systemd service -sudo systemctl start ledmatrix - -# Dev preview server (renders plugins to a browser without running run.py) -python3 scripts/dev_server.py # then open http://localhost:5001 -``` - -The `-e`/`--emulator` CLI flag is defined in `run.py:19-20` and -sets `os.environ["EMULATOR"] = "true"` before any display imports, -which `src/display_manager.py:2` then reads to switch between the -hardware and emulator backends. - -### Managing Plugins - -```bash -# List plugins -./scripts/dev/dev_plugin_setup.sh list - -# Check status -./scripts/dev/dev_plugin_setup.sh status - -# Update plugin(s) -./scripts/dev/dev_plugin_setup.sh update [plugin-name] - -# Unlink plugin -./scripts/dev/dev_plugin_setup.sh unlink -``` - -## Using These Files with Cursor - -### `.cursorrules` -Cursor automatically reads this file to understand: -- Plugin structure and requirements -- Development workflows -- Best practices -- Common patterns -- API reference - -When asking Cursor to help with plugins, it will use this context to provide better assistance. - -### Plugin Templates -Use templates when creating new plugins: -1. Copy templates from `.cursor/plugin_templates/` -2. Replace placeholders (PLUGIN_ID, PluginClassName, etc.) -3. Customize for your plugin's needs -4. Follow the guide in `plugins_guide.md` - -### Documentation -Refer to `plugins_guide.md` for: -- Detailed explanations -- Troubleshooting steps -- Best practices -- Examples and patterns - -## Plugin Development Workflow - -1. **Plan**: Determine plugin functionality and requirements -2. **Create**: Use templates or dev_plugin_setup.sh to create plugin structure -3. **Develop**: Implement plugin logic following BasePlugin interface -4. **Test**: Test with emulator first, then on hardware -5. **Configure**: Add plugin config to config/config.json -6. **Iterate**: Refine based on testing and feedback - -## Resources - -- **Plugin System**: `src/plugin_system/` -- **Base Plugin**: `src/plugin_system/base_plugin.py` -- **Plugin Manager**: `src/plugin_system/plugin_manager.py` -- **Example Plugins**: see the - [`ledmatrix-plugins`](https://github.com/ChuckBuilds/ledmatrix-plugins) - repo for canonical sources (e.g. `plugins/hockey-scoreboard/`, - `plugins/football-scoreboard/`). Installed plugins land in - `plugin-repos/` (default) or `plugins/` (dev fallback). -- **Architecture Docs**: `docs/PLUGIN_ARCHITECTURE_SPEC.md` -- **Development Setup**: `scripts/dev/dev_plugin_setup.sh` - -## Getting Help - -1. Check `plugins_guide.md` for detailed documentation -2. Review `.cursorrules` for development patterns -3. Look at existing plugins for examples -4. Check logs for error messages -5. Review plugin system code in `src/plugin_system/` - diff --git a/.cursor/plugin_templates/QUICK_START.md b/.cursor/plugin_templates/QUICK_START.md deleted file mode 100644 index 9a9f9591..00000000 --- a/.cursor/plugin_templates/QUICK_START.md +++ /dev/null @@ -1,247 +0,0 @@ -# Quick Start: Creating a New Plugin - -This guide will help you create a new plugin using the templates in `.cursor/plugin_templates/`. - -## Step 1: Create Plugin Directory - -```bash -cd /path/to/LEDMatrix -mkdir -p plugins/my-plugin -cd plugins/my-plugin -``` - -## Step 2: Copy Templates - -```bash -# Copy all template files -cp ../../.cursor/plugin_templates/manifest.json.template ./manifest.json -cp ../../.cursor/plugin_templates/manager.py.template ./manager.py -cp ../../.cursor/plugin_templates/config_schema.json.template ./config_schema.json -cp ../../.cursor/plugin_templates/README.md.template ./README.md -cp ../../.cursor/plugin_templates/requirements.txt.template ./requirements.txt -``` - -## Step 3: Customize Files - -### manifest.json - -Replace placeholders: -- `PLUGIN_ID` → `my-plugin` (lowercase, use hyphens) -- `Plugin Name` → Your plugin's display name -- `PluginClassName` → `MyPlugin` (PascalCase) -- Update description, author, homepage, etc. - -### manager.py - -Replace placeholders: -- `PluginClassName` → `MyPlugin` (must match manifest) -- Implement `_fetch_data()` method -- Implement `_render_content()` method -- Add any custom validation in `validate_config()` - -### config_schema.json - -Customize: -- Update description -- Add/remove configuration properties -- Set default values -- Add validation rules - -### README.md - -Replace placeholders: -- `PLUGIN_ID` → `my-plugin` -- `Plugin Name` → Your plugin's name -- Fill in features, installation, configuration sections - -### requirements.txt - -Add your plugin's dependencies: -```txt -requests>=2.28.0 -pillow>=9.0.0 -``` - -## Step 4: Enable Plugin - -Edit `config/config.json`: - -```json -{ - "my-plugin": { - "enabled": true, - "display_duration": 15 - } -} -``` - -## Step 5: Test Plugin - -### Test with Emulator - -```bash -cd /path/to/LEDMatrix -python run.py --emulator -``` - -### Check Plugin Loading - -Look for logs like: -``` -[INFO] Discovered 1 plugin(s) -[INFO] Loaded plugin: my-plugin v1.0.0 -[INFO] Added plugin mode: my-plugin -``` - -### Test Plugin Display - -The plugin should appear in the display rotation. Check logs for any errors. - -## Step 6: Develop and Iterate - -1. Edit `manager.py` to implement your plugin logic -2. Test with emulator: `python run.py --emulator` -3. Check logs for errors -4. Iterate until working correctly - -## Step 7: Test on Hardware (Optional) - -When ready, test on Raspberry Pi: - -```bash -# Deploy to Pi -rsync -avz plugins/my-plugin/ pi@raspberrypi:/path/to/LEDMatrix/plugins/my-plugin/ - -# Or if using git -ssh pi@raspberrypi "cd /path/to/LEDMatrix/plugins/my-plugin && git pull" - -# Restart service -ssh pi@raspberrypi "sudo systemctl restart ledmatrix" -``` - -## Common Customizations - -### Adding API Integration - -1. Add API key to `config_schema.json`: -```json -{ - "api_key": { - "type": "string", - "description": "API key for service" - } -} -``` - -2. Implement API call in `_fetch_data()`: -```python -import requests - -def _fetch_data(self): - response = requests.get( - "https://api.example.com/data", - headers={"Authorization": f"Bearer {self.api_key}"} - ) - return response.json() -``` - -3. Store API key in `config/config_secrets.json`: -```json -{ - "my-plugin": { - "api_key": "your-secret-key" - } -} -``` - -### Adding Image Rendering - -There is no `draw_image()` helper on `DisplayManager`. To render an -image, paste it directly onto the underlying PIL `Image` -(`display_manager.image`) and then call `update_display()`: - -```python -def _render_content(self): - # Load and paste image onto the display canvas - image = Image.open("assets/logo.png").convert("RGB") - self.display_manager.image.paste(image, (0, 0)) - - # Draw text overlay - self.display_manager.draw_text( - "Text", - x=10, y=20, - color=(255, 255, 255) - ) - - self.display_manager.update_display() -``` - -For transparency, paste with a mask: - -```python -icon = Image.open("assets/icon.png").convert("RGBA") -self.display_manager.image.paste(icon, (5, 5), icon) -``` - - -### Adding Live Priority - -1. Enable in config: -```json -{ - "my-plugin": { - "live_priority": true - } -} -``` - -2. Implement `has_live_content()`: -```python -def has_live_content(self) -> bool: - return self.data and self.data.get("is_live", False) -``` - -3. Override `get_live_modes()` if needed: -```python -def get_live_modes(self) -> list: - return ["my_plugin_live_mode"] -``` - -## Troubleshooting - -### Plugin Not Loading - -- Check `manifest.json` syntax (must be valid JSON) -- Verify `entry_point` file exists -- Ensure `class_name` matches class name in manager.py -- Check for import errors in logs - -### Configuration Errors - -- Validate config against `config_schema.json` -- Check required fields are present -- Verify data types match schema - -### Display Issues - -- Check display dimensions: `display_manager.width`, `display_manager.height` -- Verify coordinates are within bounds -- Ensure `update_display()` is called -- Test with emulator first - -## Next Steps - -- Review existing plugins for patterns: - - `plugins/hockey-scoreboard/` - Sports scoreboard example - - `plugins/ledmatrix-music/` - Real-time data example - - `plugins/ledmatrix-stocks/` - Data display example - -- Read full documentation: - - `.cursor/plugins_guide.md` - Comprehensive guide - - `docs/PLUGIN_ARCHITECTURE_SPEC.md` - Architecture details - - `.cursorrules` - Development rules - -- Check plugin system code: - - `src/plugin_system/base_plugin.py` - Base class - - `src/plugin_system/plugin_manager.py` - Plugin manager - diff --git a/.cursor/plugin_templates/README.md.template b/.cursor/plugin_templates/README.md.template deleted file mode 100644 index ba26fd1b..00000000 --- a/.cursor/plugin_templates/README.md.template +++ /dev/null @@ -1,156 +0,0 @@ -# Plugin Name - -Brief description of what this plugin does. - -## Features - -- Feature 1 -- Feature 2 -- Feature 3 - -## Installation - -1. Link the plugin to your LEDMatrix installation: - -```bash -cd /path/to/LEDMatrix -./scripts/dev/dev_plugin_setup.sh link-github PLUGIN_ID -``` - -Or for local development: - -```bash -./scripts/dev/dev_plugin_setup.sh link PLUGIN_ID /path/to/plugin/repo -``` - -2. Install dependencies: - -```bash -pip install -r plugins/PLUGIN_ID/requirements.txt -``` - -3. Configure the plugin in `config/config.json`: - -```json -{ - "PLUGIN_ID": { - "enabled": true, - "display_duration": 15 - } -} -``` - -**Note:** API keys and other sensitive credentials must be stored in `config/config_secrets.json`, not in `config/config.json`. - -4. Store API keys in `config/config_secrets.json`: - -```json -{ - "PLUGIN_ID": { - "api_key": "your-secret-api-key" - } -} -``` - -## Configuration - -### Required Settings - -- `enabled` (boolean): Enable or disable the plugin -- `api_key` (string): API key for external service (if required) - -### Optional Settings - -- `display_duration` (number): How long to display this plugin (default: 15 seconds) -- `refresh_interval` (integer): How often to refresh data in seconds (default: 60) -- `live_priority` (boolean): Enable live priority takeover (default: false) - -## Display Modes - -This plugin provides the following display modes: - -- `PLUGIN_ID`: Main display mode - -## API Requirements - -This plugin requires: - -- **API Name**: Description of API requirements - - URL: https://api.example.com - - Rate Limit: X requests per minute - - Authentication: API key required - -## Development - -### Running Tests - -```bash -cd plugins/PLUGIN_ID -python test_PLUGIN_ID.py -``` - -### Testing with Emulator - -```bash -cd /path/to/LEDMatrix -python run.py --emulator -``` - -### Debugging - -Enable debug logging in `config/config.json`: - -```json -{ - "logging": { - "level": "DEBUG" - } -} -``` - -Check logs: - -```bash -# On Raspberry Pi (if running as service) -journalctl -u ledmatrix -f - -# Direct execution -python run.py -``` - -## Troubleshooting - -### Plugin Not Loading - -1. Check that `manifest.json` exists and is valid -2. Verify `entry_point` file exists -3. Check that `class_name` matches the class in manager.py -4. Review logs for import errors - -### Configuration Errors - -1. Validate config against `config_schema.json` -2. Check required fields are present -3. Verify data types match schema - -### API Errors - -1. Verify API key is correct -2. Check API rate limits -3. Review network connectivity -4. Check API service status - -## License - -[License information] - -## Author - -Your Name - -## Links - -- GitHub: https://github.com/username/ledmatrix-PLUGIN_ID -- Documentation: [Link to docs] -- Issues: https://github.com/username/ledmatrix-PLUGIN_ID/issues - diff --git a/.cursor/plugin_templates/config_schema.json.template b/.cursor/plugin_templates/config_schema.json.template deleted file mode 100644 index 8d512038..00000000 --- a/.cursor/plugin_templates/config_schema.json.template +++ /dev/null @@ -1,44 +0,0 @@ -{ - "$schema": "http://json-schema.org/draft-07/schema#", - "type": "object", - "title": "Plugin Configuration Schema", - "description": "Configuration schema for Plugin Name", - "properties": { - "enabled": { - "type": "boolean", - "default": true, - "description": "Enable or disable this plugin" - }, - "display_duration": { - "type": "number", - "default": 15, - "minimum": 1, - "maximum": 300, - "description": "How long to display this plugin in seconds" - }, - "live_priority": { - "type": "boolean", - "default": false, - "description": "Enable live priority takeover when plugin has live content" - }, - "refresh_interval": { - "type": "integer", - "default": 60, - "minimum": 1, - "description": "How often to refresh data in seconds" - }, - "api_key": { - "type": "string", - "description": "API key for external service (store in config_secrets.json)", - "default": "" - }, - "custom_setting": { - "type": "string", - "description": "Example custom setting - replace with your plugin's settings", - "default": "default_value" - } - }, - "required": ["enabled"], - "additionalProperties": false -} - diff --git a/.cursor/plugin_templates/manager.py.template b/.cursor/plugin_templates/manager.py.template deleted file mode 100644 index 6afe3e9b..00000000 --- a/.cursor/plugin_templates/manager.py.template +++ /dev/null @@ -1,226 +0,0 @@ -""" -Plugin Name - -Brief description of what this plugin does. - -API Version: 1.0.0 -""" - -from src.plugin_system.base_plugin import BasePlugin -from PIL import Image -from typing import Dict, Any, Optional -import logging -import time - - -class PluginClassName(BasePlugin): - """ - Plugin class that inherits from BasePlugin. - - This plugin demonstrates the basic structure and common patterns - for LEDMatrix plugins. - """ - - def __init__( - self, - plugin_id: str, - config: Dict[str, Any], - display_manager, - cache_manager, - plugin_manager, - ): - """Initialize the plugin.""" - super().__init__(plugin_id, config, display_manager, cache_manager, plugin_manager) - - # Initialize plugin-specific data - self.data = None - self.last_update_time = None - - # Load configuration values - self.api_key = config.get("api_key", "") - self.refresh_interval = config.get("refresh_interval", 60) - - self.logger.info(f"Plugin {plugin_id} initialized") - - def update(self) -> None: - """ - Fetch/update data for this plugin. - - This method is called periodically based on update_interval - specified in the manifest. Use cache_manager to avoid - excessive API calls. - """ - cache_key = f"{self.plugin_id}_data" - - # Check cache first - cached = self.cache_manager.get(cache_key, max_age=self.refresh_interval) - if cached: - self.data = cached - self.logger.debug("Using cached data") - return - - try: - # Fetch new data - self.data = self._fetch_data() - - # Cache the data - self.cache_manager.set(cache_key, self.data, ttl=self.refresh_interval) - self.last_update_time = time.time() - - self.logger.info("Data updated successfully") - - except Exception as e: - self.logger.error(f"Failed to update data: {e}") - # Use cached data if available, even if expired - # Use a very large max_age (1 year) to effectively bypass expiration for fallback - expired_cached = self.cache_manager.get(cache_key, max_age=31536000) - if expired_cached: - self.data = expired_cached - self.logger.warning("Using expired cache due to update failure") - - def display(self, force_clear: bool = False) -> None: - """ - Render this plugin's display. - - Args: - force_clear: If True, clear display before rendering - """ - if force_clear: - self.display_manager.clear() - - # Check if we have data to display - if not self.data: - self._display_error("No data available") - return - - try: - # Render plugin content - self._render_content() - - # Update the display - self.display_manager.update_display() - - except Exception as e: - self.logger.error(f"Display error: {e}") - self._display_error("Display error") - - def _fetch_data(self) -> Dict[str, Any]: - """ - Fetch data from external source. - - Returns: - Dictionary containing fetched data - """ - # TODO: Implement data fetching logic - # Example: - # import requests - # response = requests.get("https://api.example.com/data", - # headers={"Authorization": f"Bearer {self.api_key}"}) - # return response.json() - - # Placeholder - return { - "message": "Hello, World!", - "timestamp": time.time() - } - - def _render_content(self) -> None: - """Render the plugin content on the display.""" - # Get display dimensions - width = self.display_manager.width - height = self.display_manager.height - - # Example: Draw text - text = self.data.get("message", "No data") - x = 5 - y = height // 2 - - self.display_manager.draw_text( - text, - x=x, - y=y, - color=(255, 255, 255) # White - ) - - # Example: Draw image - # if hasattr(self, 'logo_image'): - # self.display_manager.draw_image( - # self.logo_image, - # x=0, - # y=0 - # ) - - def _display_error(self, message: str) -> None: - """Display an error message.""" - self.display_manager.clear() - width = self.display_manager.width - height = self.display_manager.height - - self.display_manager.draw_text( - message, - x=5, - y=height // 2, - color=(255, 0, 0) # Red - ) - self.display_manager.update_display() - - def validate_config(self) -> bool: - """ - Validate plugin configuration. - - Returns: - True if config is valid, False otherwise - """ - # Call parent validation first - if not super().validate_config(): - return False - - # Add custom validation - # Example: Check for required API key - # if self.config.get("require_api_key", True): - # if not self.api_key: - # self.logger.error("API key is required but not provided") - # return False - - return True - - def has_live_content(self) -> bool: - """ - Check if plugin has live content to display. - - Override this method to enable live priority features. - - Returns: - True if plugin has live content, False otherwise - """ - # Example: Check if there's live data - # return self.data and self.data.get("is_live", False) - return False - - def get_info(self) -> Dict[str, Any]: - """ - Return plugin info for display in web UI. - - Returns: - Dictionary with plugin information - """ - info = super().get_info() - - # Add plugin-specific info - info.update({ - "data_available": self.data is not None, - "last_update": self.last_update_time, - # Add more info as needed - }) - - return info - - def cleanup(self) -> None: - """Cleanup resources when plugin is unloaded.""" - # Clean up any resources (threads, connections, etc.) - # Example: - # if hasattr(self, 'api_client'): - # self.api_client.close() - - super().cleanup() - diff --git a/.cursor/plugin_templates/manifest.json.template b/.cursor/plugin_templates/manifest.json.template deleted file mode 100644 index ade5fa58..00000000 --- a/.cursor/plugin_templates/manifest.json.template +++ /dev/null @@ -1,55 +0,0 @@ -{ - "id": "PLUGIN_ID", - "name": "Plugin Name", - "version": "1.0.0", - "author": "Your Name", - "description": "Brief description of what this plugin does", - "homepage": "https://github.com/username/ledmatrix-PLUGIN_ID", - "entry_point": "manager.py", - "class_name": "PluginClassName", - "category": "custom", - "tags": ["custom", "example"], - "icon": "fas fa-icon-name", - "compatible_versions": [">=2.0.0"], - "min_ledmatrix_version": "2.0.0", - "max_ledmatrix_version": "3.0.0", - "requires": { - "python": ">=3.9", - "display_size": { - "min_width": 64, - "min_height": 32 - } - }, - "config_schema": "config_schema.json", - "assets": { - "logos": "Optional: Description of asset requirements" - }, - "update_interval": 60, - "default_duration": 15, - "display_modes": [ - "PLUGIN_ID" - ], - "api_requirements": [ - { - "name": "API Name", - "required": false, - "description": "Description of API requirements", - "url": "https://api.example.com", - "rate_limit": "Rate limit information" - } - ], - "download_url_template": "https://github.com/username/ledmatrix-PLUGIN_ID/archive/refs/tags/v{version}.zip", - "versions": [ - { - "released": "2025-01-01", - "version": "1.0.0", - "ledmatrix_min_version": "2.0.0" - } - ], - "last_updated": "2025-01-01", - "stars": 0, - "downloads": 0, - "verified": false, - "screenshot": "" -} - diff --git a/.cursor/plugin_templates/requirements.txt.template b/.cursor/plugin_templates/requirements.txt.template deleted file mode 100644 index a631c4f0..00000000 --- a/.cursor/plugin_templates/requirements.txt.template +++ /dev/null @@ -1,13 +0,0 @@ -# Plugin Dependencies -# Add your plugin's Python dependencies here - -# Example dependencies (uncomment and modify as needed): -# requests>=2.28.0 -# pillow>=9.0.0 -# python-dateutil>=2.8.0 - -# Note: Core LEDMatrix dependencies are already available: -# - PIL/Pillow (for image handling) -# - Core plugin system classes -# - Display manager, cache manager, config manager - diff --git a/.cursor/plugin_templates/test_manager.py.template b/.cursor/plugin_templates/test_manager.py.template deleted file mode 100644 index 0cff3425..00000000 --- a/.cursor/plugin_templates/test_manager.py.template +++ /dev/null @@ -1,136 +0,0 @@ -""" -Test file for Plugin Name plugin. - -This file provides example unit tests for your plugin. -Run tests with: python -m pytest test_manager.py -Or: python test_manager.py -""" - -import unittest -import sys -from pathlib import Path - -# Add project root to path -PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent -if str(PROJECT_ROOT) not in sys.path: - sys.path.insert(0, str(PROJECT_ROOT)) - -from src.plugin_system.testing import PluginTestCase -from manager import PluginClassName - - -class TestPluginClassName(PluginTestCase): - """Test cases for PluginClassName plugin.""" - - def setUp(self): - """Set up test fixtures.""" - super().setUp() - - # Update plugin_id to match the plugin being tested - self.plugin_id = 'PLUGIN_ID' - - # Create plugin instance - self.plugin = self.create_plugin_instance( - PluginClassName, - plugin_id='PLUGIN_ID', - config=self.get_mock_config() - ) - - def test_plugin_initialization(self): - """Test that plugin initializes correctly.""" - self.assert_plugin_initialized(self.plugin) - self.assertTrue(self.plugin.enabled) - - def test_config_validation(self): - """Test configuration validation.""" - # Valid config should pass - self.assertTrue(self.plugin.validate_config()) - - # Test with invalid config if applicable - # invalid_config = self.get_mock_config(enabled='not-a-boolean') - # invalid_plugin = self.create_plugin_instance( - # PluginClassName, - # config=invalid_config - # ) - # self.assertFalse(invalid_plugin.validate_config()) - - def test_update_method(self): - """Test the update() method.""" - # Reset mocks - self.cache_manager.reset() - - # Call update - self.plugin.update() - - # Assertions - # Example: Check that cache was used - # self.assert_cache_get('PLUGIN_ID_data') - - # Example: Check that data was fetched and cached - # self.assert_cache_set('PLUGIN_ID_data') - - def test_display_method(self): - """Test the display() method.""" - # Ensure plugin has data (call update first if needed) - # self.plugin.update() - - # Call display - self.plugin.display(force_clear=True) - - # Assertions - self.assert_display_cleared() - self.assert_display_updated() - - # Example: Check that text was drawn - # self.assert_text_drawn("Expected Text") - - # Example: Check that image was drawn - # self.assert_image_drawn() - - def test_display_without_data(self): - """Test display() behavior when no data is available.""" - # Clear any cached data - self.cache_manager.reset() - - # Call display - self.plugin.display() - - # Should handle gracefully (no exceptions) - # May show error message or fallback content - self.assert_display_updated() - - def test_get_display_duration(self): - """Test display duration configuration.""" - duration = self.plugin.get_display_duration() - self.assertIsInstance(duration, (int, float)) - self.assertGreater(duration, 0) - - # Test with custom duration - custom_config = self.get_mock_config(display_duration=30.0) - custom_plugin = self.create_plugin_instance( - PluginClassName, - config=custom_config - ) - self.assertEqual(custom_plugin.get_display_duration(), 30.0) - - def test_enable_disable(self): - """Test plugin enable/disable functionality.""" - self.assertTrue(self.plugin.enabled) - - self.plugin.on_disable() - self.assertFalse(self.plugin.enabled) - - self.plugin.on_enable() - self.assertTrue(self.plugin.enabled) - - def test_config_change(self): - """Test configuration change handling.""" - new_config = self.get_mock_config(display_duration=20.0) - self.plugin.on_config_change(new_config) - - self.assertEqual(self.plugin.config.get('display_duration'), 20.0) - - -if __name__ == '__main__': - unittest.main() - diff --git a/.cursor/plugins_guide.md b/.cursor/plugins_guide.md deleted file mode 100644 index 62ba63b6..00000000 --- a/.cursor/plugins_guide.md +++ /dev/null @@ -1,751 +0,0 @@ -# LEDMatrix Plugin Development Guide - -This guide provides comprehensive instructions for creating, running, and loading plugins in the LEDMatrix project. - -## Table of Contents - -1. [Plugin System Overview](#plugin-system-overview) -2. [Creating a New Plugin](#creating-a-new-plugin) -3. [Running Plugins](#running-plugins) -4. [Loading Plugins](#loading-plugins) -5. [Plugin Development Workflow](#plugin-development-workflow) -6. [Testing Plugins](#testing-plugins) -7. [Troubleshooting](#troubleshooting) - ---- - -## Plugin System Overview - -The LEDMatrix project uses a plugin-based architecture where all display functionality (except core calendar) is implemented as plugins. Plugins are dynamically loaded from the `plugins/` directory and integrated into the display rotation. - -### Plugin Architecture - -``` -LEDMatrix Core -├── Plugin Manager (discovers, loads, manages plugins) -├── Display Manager (handles LED matrix rendering) -├── Cache Manager (data persistence) -├── Config Manager (configuration management) -└── Plugins/ (plugin directory) - ├── plugin-1/ - ├── plugin-2/ - └── ... -``` - -### Plugin Lifecycle - -1. **Discovery**: PluginManager scans `plugins/` for directories with `manifest.json` -2. **Loading**: Plugin module is imported and class is instantiated -3. **Configuration**: Plugin config is loaded from `config/config.json` -4. **Validation**: `validate_config()` is called to verify configuration -5. **Registration**: Plugin is added to available display modes -6. **Execution**: `update()` is called periodically, `display()` is called during rotation - ---- - -## Creating a New Plugin - -### Method 1: Using dev_plugin_setup.sh (Recommended) - -This method is best for plugins stored in separate Git repositories. - -#### From GitHub Repository - -```bash -# Link a plugin from GitHub (auto-detects URL) -./scripts/dev/dev_plugin_setup.sh link-github - -# Example: Link hockey-scoreboard plugin -./scripts/dev/dev_plugin_setup.sh link-github hockey-scoreboard - -# With custom URL -./scripts/dev/dev_plugin_setup.sh link-github https://github.com/user/repo.git -``` - -The script will: -- Clone the repository to `~/.ledmatrix-dev-plugins/` (or configured directory) -- Create a symlink in `plugins//` pointing to the cloned repo -- Validate the plugin structure - -#### From Local Repository - -```bash -# Link a local plugin repository -./scripts/dev/dev_plugin_setup.sh link - -# Example: Link a local plugin -./scripts/dev/dev_plugin_setup.sh link my-plugin ../ledmatrix-my-plugin -``` - -### Method 2: Manual Plugin Creation - -1. **Create Plugin Directory** - -```bash -mkdir -p plugins/my-plugin -cd plugins/my-plugin -``` - -2. **Create manifest.json** - -```json -{ - "id": "my-plugin", - "name": "My Plugin", - "version": "1.0.0", - "author": "Your Name", - "description": "Description of what this plugin does", - "entry_point": "manager.py", - "class_name": "MyPlugin", - "category": "custom", - "tags": ["custom", "example"], - "display_modes": ["my_plugin"], - "update_interval": 60, - "default_duration": 15, - "requires": { - "python": ">=3.9" - }, - "config_schema": "config_schema.json" -} -``` - -3. **Create manager.py** - -```python -from src.plugin_system.base_plugin import BasePlugin -from PIL import Image -import logging - -class MyPlugin(BasePlugin): - """My custom plugin implementation.""" - - def update(self): - """Fetch/update data for this plugin.""" - # Fetch data from API, files, etc. - # Use self.cache_manager for caching - cache_key = f"{self.plugin_id}_data" - cached = self.cache_manager.get(cache_key, max_age=3600) - if cached: - self.data = cached - return - - # Fetch new data - self.data = self._fetch_data() - self.cache_manager.set(cache_key, self.data) - - def display(self, force_clear=False): - """Render this plugin's display.""" - if force_clear: - self.display_manager.clear() - - # Render content using display_manager - self.display_manager.draw_text( - "Hello, World!", - x=10, y=15, - color=(255, 255, 255) - ) - - self.display_manager.update_display() - - def _fetch_data(self): - """Fetch data from external source.""" - # Implement your data fetching logic - return {"message": "Hello, World!"} - - def validate_config(self): - """Validate plugin configuration.""" - # Check required config fields - if not super().validate_config(): - return False - - # Add custom validation - required_fields = ['api_key'] # Example - for field in required_fields: - if field not in self.config: - self.logger.error(f"Missing required field: {field}") - return False - - return True -``` - -4. **Create config_schema.json** - -```json -{ - "type": "object", - "properties": { - "enabled": { - "type": "boolean", - "default": true, - "description": "Enable or disable this plugin" - }, - "display_duration": { - "type": "number", - "default": 15, - "minimum": 1, - "description": "How long to display this plugin (seconds)" - }, - "api_key": { - "type": "string", - "description": "API key for external service" - } - }, - "required": ["enabled"] -} -``` - -5. **Create requirements.txt** (if needed) - -``` -requests>=2.28.0 -pillow>=9.0.0 -``` - -6. **Create README.md** - -Document your plugin's functionality, configuration options, and usage. - ---- - -## Running Plugins - -### Development Mode (Emulator) - -Run the LEDMatrix system with emulator for plugin testing: - -```bash -# Using run.py -python run.py --emulator - -# Using emulator script -./run_emulator.sh -``` - -The emulator will: -- Load all enabled plugins -- Display plugin content in a window (simulating LED matrix) -- Show logs for plugin loading and execution -- Allow testing without Raspberry Pi hardware - -### Production Mode (Raspberry Pi) - -Run on actual Raspberry Pi hardware: - -```bash -# Direct execution -python run.py - -# As systemd service -sudo systemctl start ledmatrix -sudo systemctl status ledmatrix -sudo journalctl -u ledmatrix -f # View logs -``` - -### Plugin-Specific Testing - -Test individual plugin loading: - -```python -# test_my_plugin.py -from src.plugin_system.plugin_manager import PluginManager -from src.config_manager import ConfigManager -from src.display_manager import DisplayManager -from src.cache_manager import CacheManager - -# Initialize managers -config_manager = ConfigManager() -config = config_manager.load_config() -display_manager = DisplayManager(config) -cache_manager = CacheManager() - -# Initialize plugin manager -plugin_manager = PluginManager( - plugins_dir="plugins", - config_manager=config_manager, - display_manager=display_manager, - cache_manager=cache_manager -) - -# Discover and load plugin -plugins = plugin_manager.discover_plugins() -print(f"Discovered plugins: {plugins}") - -if "my-plugin" in plugins: - if plugin_manager.load_plugin("my-plugin"): - plugin = plugin_manager.get_plugin("my-plugin") - plugin.update() - plugin.display() - print("Plugin loaded and displayed successfully!") - else: - print("Failed to load plugin") -``` - ---- - -## Loading Plugins - -### Enabling Plugins - -Plugins are enabled/disabled in `config/config.json`: - -```json -{ - "my-plugin": { - "enabled": true, - "display_duration": 15, - "api_key": "your-api-key-here" - } -} -``` - -### Plugin Configuration Structure - -Each plugin has its own section in `config/config.json`: - -```json -{ - "": { - "enabled": true, // Enable/disable plugin - "display_duration": 15, // Display duration in seconds - "live_priority": false, // Enable live priority takeover - "high_performance_transitions": false, // Use 120 FPS transitions - "transition": { // Transition configuration - "type": "redraw", // Transition type - "speed": 2, // Transition speed - "enabled": true // Enable transitions - }, - // ... plugin-specific configuration - } -} -``` - -### Secrets Management - -Store sensitive data (API keys, tokens) in `config/config_secrets.json` -under the same plugin id you use in `config/config.json`: - -```json -{ - "my-plugin": { - "api_key": "secret-api-key-here" - } -} -``` - -At load time, the config manager deep-merges `config_secrets.json` into -the main config (verified at `src/config_manager.py:162-172`). So in -your plugin's code: - -```python -class MyPlugin(BasePlugin): - def __init__(self, plugin_id, config, display_manager, cache_manager, plugin_manager): - super().__init__(plugin_id, config, display_manager, cache_manager, plugin_manager) - self.api_key = config.get("api_key") # already merged from secrets -``` - -There is no separate `config_secrets` reference field — just put the -secret value under the same plugin namespace and read it from the -merged config. - -### Plugin Discovery - -Plugins are automatically discovered when: -- Directory exists in `plugins/` -- Directory contains `manifest.json` -- Manifest has required fields (`id`, `entry_point`, `class_name`) - -Check discovered plugins: - -```bash -# Using dev_plugin_setup.sh -./scripts/dev/dev_plugin_setup.sh list - -# Output shows: -# ✓ plugin-name (symlink) -# → /path/to/repo -# ✓ Git repo is clean (branch: main) -``` - -### Plugin Status - -Check plugin status and git information: - -```bash -./scripts/dev/dev_plugin_setup.sh status - -# Output shows: -# ✓ plugin-name -# Path: /path/to/repo -# Branch: main -# Remote: https://github.com/user/repo.git -# Status: Clean and up to date -``` - ---- - -## Plugin Development Workflow - -### 1. Initial Setup - -```bash -# Create or clone plugin repository -git clone https://github.com/user/ledmatrix-my-plugin.git -cd ledmatrix-my-plugin - -# Link to LEDMatrix project -cd /path/to/LEDMatrix -./scripts/dev/dev_plugin_setup.sh link my-plugin ../ledmatrix-my-plugin -``` - -### 2. Development Cycle - -1. **Edit plugin code** in linked repository -2. **Test with the dev preview server**: - `python3 scripts/dev_server.py` (then open `http://localhost:5001`). - Or run the full display in emulator mode with - `python3 run.py --emulator` (or equivalently - `EMULATOR=true python3 run.py`). The `-e`/`--emulator` CLI flag is - defined in `run.py:19-20` and sets the same `EMULATOR` environment - variable internally. -3. **Check logs** for errors or warnings -4. **Update configuration** in `config/config.json` if needed -5. **Iterate** until plugin works correctly - -### 3. Testing on Hardware - -```bash -# Deploy to Raspberry Pi -rsync -avz plugins/my-plugin/ ledpi@your-pi-ip:/path/to/LEDMatrix/plugins/my-plugin/ - -# Or if using git, pull on Pi -ssh ledpi@your-pi-ip "cd /path/to/LEDMatrix/plugins/my-plugin && git pull" - -# Restart service -ssh ledpi@your-pi-ip "sudo systemctl restart ledmatrix" -``` - -### 4. Updating Plugins - -```bash -# Update single plugin from git -./scripts/dev/dev_plugin_setup.sh update my-plugin - -# Update all linked plugins -./scripts/dev/dev_plugin_setup.sh update -``` - -### 5. Unlinking Plugins - -```bash -# Remove symlink (preserves repository) -./scripts/dev/dev_plugin_setup.sh unlink my-plugin -``` - ---- - -## Testing Plugins - -### Unit Testing - -Create test files in plugin directory: - -```python -# plugins/my-plugin/test_my_plugin.py -import unittest -from unittest.mock import Mock, MagicMock -from manager import MyPlugin - -class TestMyPlugin(unittest.TestCase): - def setUp(self): - self.config = {"enabled": True} - self.display_manager = Mock() - self.cache_manager = Mock() - self.plugin_manager = Mock() - - self.plugin = MyPlugin( - plugin_id="my-plugin", - config=self.config, - display_manager=self.display_manager, - cache_manager=self.cache_manager, - plugin_manager=self.plugin_manager - ) - - def test_plugin_initialization(self): - self.assertEqual(self.plugin.plugin_id, "my-plugin") - self.assertTrue(self.plugin.enabled) - - def test_config_validation(self): - self.assertTrue(self.plugin.validate_config()) - - def test_update(self): - self.cache_manager.get.return_value = None - self.plugin.update() - # Assert data was fetched and cached - - def test_display(self): - self.plugin.display() - self.display_manager.draw_text.assert_called() - self.display_manager.update_display.assert_called() - -if __name__ == '__main__': - unittest.main() -``` - -Run tests: - -```bash -cd plugins/my-plugin -python -m pytest test_my_plugin.py -# or -python test_my_plugin.py -``` - -### Integration Testing - -Test plugin with actual managers: - -```python -# test_plugin_integration.py -from src.plugin_system.plugin_manager import PluginManager -from src.config_manager import ConfigManager -from src.display_manager import DisplayManager -from src.cache_manager import CacheManager - -def test_plugin_loading(): - config_manager = ConfigManager() - config = config_manager.load_config() - display_manager = DisplayManager(config) - cache_manager = CacheManager() - - plugin_manager = PluginManager( - plugins_dir="plugins", - config_manager=config_manager, - display_manager=display_manager, - cache_manager=cache_manager - ) - - plugins = plugin_manager.discover_plugins() - assert "my-plugin" in plugins - - assert plugin_manager.load_plugin("my-plugin") - plugin = plugin_manager.get_plugin("my-plugin") - assert plugin is not None - assert plugin.enabled - - plugin.update() - plugin.display() -``` - -### Emulator Testing - -Test plugin rendering visually: - -```bash -# Run with emulator -python run.py --emulator - -# Plugin should appear in display rotation -# Check logs for plugin loading and execution -``` - -### Hardware Testing - -1. Deploy plugin to Raspberry Pi -2. Enable in `config/config.json` -3. Restart LEDMatrix service -4. Observe LED matrix display -5. Check logs: `journalctl -u ledmatrix -f` - ---- - -## Troubleshooting - -### Plugin Not Loading - -**Symptoms**: Plugin doesn't appear in available modes, no logs about plugin - -**Solutions**: -1. Check plugin directory exists: `ls plugins/my-plugin/` -2. Verify `manifest.json` exists and is valid JSON -3. Check manifest has required fields: `id`, `entry_point`, `class_name` -4. Verify entry_point file exists: `ls plugins/my-plugin/manager.py` -5. Check class name matches: `grep "class.*Plugin" plugins/my-plugin/manager.py` -6. Review logs for import errors - -### Plugin Loading but Not Displaying - -**Symptoms**: Plugin loads successfully but doesn't appear in rotation - -**Solutions**: -1. Check plugin is enabled: `config/config.json` has `"enabled": true` -2. Verify display_modes in manifest match config -3. Check plugin is in rotation schedule -4. Review `display()` method for errors -5. Check logs for runtime errors - -### Configuration Errors - -**Symptoms**: Plugin fails to load, validation errors in logs - -**Solutions**: -1. Validate config against `config_schema.json` -2. Check required fields are present -3. Verify data types match schema -4. Check for typos in config keys -5. Review `validate_config()` method - -### Import Errors - -**Symptoms**: ModuleNotFoundError or ImportError in logs - -**Solutions**: -1. Install plugin dependencies: `pip install -r plugins/my-plugin/requirements.txt` -2. Check Python path includes plugin directory -3. Verify relative imports are correct -4. Check for circular import issues -5. Ensure all dependencies are in requirements.txt - -### Display Issues - -**Symptoms**: Plugin renders incorrectly or not at all - -**Solutions**: -1. Check display dimensions: `display_manager.width`, `display_manager.height` -2. Verify coordinates are within display bounds -3. Check color values are valid (0-255) -4. Ensure `update_display()` is called after rendering -5. Test with emulator first to debug rendering - -### Performance Issues - -**Symptoms**: Slow display updates, high CPU usage - -**Solutions**: -1. Use `cache_manager` to avoid excessive API calls -2. Implement background data fetching -3. Optimize rendering code -4. Consider using `high_performance_transitions` -5. Profile plugin code to identify bottlenecks - -### Git/Symlink Issues - -**Symptoms**: Plugin changes not appearing, broken symlinks - -**Solutions**: -1. Check symlink: `ls -la plugins/my-plugin` -2. Verify target exists: `readlink -f plugins/my-plugin` -3. Update plugin: `./scripts/dev/dev_plugin_setup.sh update my-plugin` -4. Re-link plugin if needed: `./scripts/dev/dev_plugin_setup.sh unlink my-plugin && ./scripts/dev/dev_plugin_setup.sh link my-plugin ` -5. Check git status: `cd plugins/my-plugin && git status` - ---- - -## Best Practices - -### Code Organization - -- Keep plugin code in `plugins//` directory -- Use descriptive class and method names -- Follow existing plugin patterns -- Place shared utilities in `src/common/` if reusable - -### Configuration - -- Always use `config_schema.json` for validation -- Store secrets in `config_secrets.json` -- Provide sensible defaults -- Document all configuration options in README - -### Error Handling - -- Use plugin logger for all logging -- Handle API failures gracefully -- Provide fallback displays when data unavailable -- Cache data to avoid excessive requests - -### Performance - -- Cache API responses appropriately -- Use background data fetching for long operations -- Optimize rendering for Pi's limited resources -- Test performance on actual hardware - -### Testing - -- Write unit tests for core logic -- Test with emulator before hardware -- Test on Raspberry Pi before deploying -- Test with other plugins enabled - -### Documentation - -- Document plugin functionality in README -- Include configuration examples -- Document API requirements and rate limits -- Provide usage examples - ---- - -## Resources - -- **Plugin System Documentation**: `docs/PLUGIN_ARCHITECTURE_SPEC.md` -- **Base Plugin Class**: `src/plugin_system/base_plugin.py` -- **Plugin Manager**: `src/plugin_system/plugin_manager.py` -- **Example Plugins**: - - `plugins/hockey-scoreboard/` - Sports scoreboard example - - `plugins/football-scoreboard/` - Complex multi-league example - - `plugins/ledmatrix-music/` - Real-time data example -- **Development Setup**: `dev_plugin_setup.sh` -- **Example Config**: `dev_plugins.json.example` - ---- - -## Quick Reference - -### Common Commands - -```bash -# Link plugin from GitHub -./scripts/dev/dev_plugin_setup.sh link-github - -# Link local plugin -./scripts/dev/dev_plugin_setup.sh link - -# List all plugins -./scripts/dev/dev_plugin_setup.sh list - -# Check plugin status -./scripts/dev/dev_plugin_setup.sh status - -# Update plugin(s) -./scripts/dev/dev_plugin_setup.sh update [name] - -# Unlink plugin -./scripts/dev/dev_plugin_setup.sh unlink - -# Run with emulator -python run.py --emulator - -# Run on Pi -python run.py -``` - -### Plugin File Structure - -``` -plugins/my-plugin/ -├── manifest.json # Required: Plugin metadata -├── manager.py # Required: Plugin class -├── config_schema.json # Required: Config validation -├── requirements.txt # Optional: Dependencies -├── README.md # Optional: Documentation -└── ... # Plugin-specific files -``` - -### Required Manifest Fields - -- `id`: Plugin identifier -- `entry_point`: Python file (usually "manager.py") -- `class_name`: Plugin class name -- `display_modes`: Array of mode names - diff --git a/.cursor/rules/coding-standards.mdc b/.cursor/rules/coding-standards.mdc deleted file mode 100644 index 5abeb94e..00000000 --- a/.cursor/rules/coding-standards.mdc +++ /dev/null @@ -1,38 +0,0 @@ ---- -globs: *.py ---- - -# Python Coding Standards - -## Code Quality Principles -- **Simplicity First**: Prefer clear, readable code over clever optimizations -- **Explicit over Implicit**: Make intentions clear through naming and structure -- **Fail Fast**: Validate inputs and handle errors early -- **Documentation**: Use docstrings for classes and complex functions - -## Naming Conventions -- **Classes**: PascalCase (e.g., `NHLRecentManager`) -- **Functions/Variables**: snake_case (e.g., `fetch_game_data`) -- **Constants**: UPPER_SNAKE_CASE (e.g., `ESPN_NHL_SCOREBOARD_URL`) -- **Private methods**: Leading underscore (e.g., `_fetch_data`) - -## Error Handling -- **Logging**: Use structured logging with context (e.g., `[NHL Recent]`) -- **Exceptions**: Catch specific exceptions, not bare `except:` -- **User-friendly messages**: Explain what went wrong and potential solutions -- **Graceful degradation**: Continue operation when non-critical features fail - -## Manager Pattern -All sports managers should follow this structure: -```python -class BaseManager: - def __init__(self, config, display_manager, cache_manager) - def update(self) # Fetch and process data - def display(self, force_clear=False) # Render to display -``` - -## Configuration Management -- **Type hints**: Use for function parameters and return values -- **Configuration validation**: Check required fields on initialization -- **Default values**: Provide sensible defaults in code, not config -- **Environment awareness**: Handle different deployment contexts \ No newline at end of file diff --git a/.cursor/rules/configuration-management.mdc b/.cursor/rules/configuration-management.mdc deleted file mode 100644 index 37f68300..00000000 --- a/.cursor/rules/configuration-management.mdc +++ /dev/null @@ -1,42 +0,0 @@ ---- -globs: config/*.json,src/*.py ---- - -# Configuration Management - -## Configuration Structure -- **Main config**: [config/config.json](mdc:config/config.json) - Primary configuration -- **Secrets**: [config/config_secrets.json](mdc:config/config_secrets.json) - API keys and sensitive data -- **Templates**: [config/config.template.json](mdc:config/config.template.json) - Default values - -## Configuration Principles -- **Validation**: Check required fields and data types on startup -- **Defaults**: Provide sensible defaults in code, not just config -- **Environment awareness**: Handle development vs production differences -- **Security**: Never commit secrets to version control - -## Manager Configuration Pattern -```python -def __init__(self, config, display_manager, cache_manager): - self.mode_config = config.get("sport_scoreboard", {}) - self.favorite_teams = self.mode_config.get("favorite_teams", []) - self.show_favorite_only = self.mode_config.get("show_favorite_teams_only", False) -``` - -## Required Configuration Sections -- **Display settings**: Update intervals, display durations -- **API settings**: Timeouts, retry logic, rate limiting -- **Background service**: Threading, caching, priority settings -- **Team preferences**: Favorite teams, filtering options - -## Configuration Validation -- **Type checking**: Ensure numeric values are numbers, lists are lists -- **Range validation**: Check that intervals are reasonable -- **Dependency checking**: Verify required services are available -- **Fallback values**: Provide defaults when config is missing or invalid - -## Best Practices -- **Documentation**: Comment complex configuration options -- **Examples**: Provide working examples in templates -- **Migration**: Handle configuration changes between versions -- **Testing**: Validate configuration in test environments \ No newline at end of file diff --git a/.cursor/rules/error-handling-logging.mdc b/.cursor/rules/error-handling-logging.mdc deleted file mode 100644 index 0bf404ee..00000000 --- a/.cursor/rules/error-handling-logging.mdc +++ /dev/null @@ -1,50 +0,0 @@ ---- -globs: src/*.py ---- - -# Error Handling and Logging - -## Logging Standards -- **Structured prefixes**: Use consistent tags like `[NHL Recent]`, `[NFL Live]` -- **Context information**: Include relevant details (team names, game status, dates) -- **Appropriate levels**: - - `info`: Normal operations and status updates - - `debug`: Detailed information for troubleshooting - - `warning`: Non-critical issues that should be noted - - `error`: Problems that need attention - -## Error Handling Patterns -```python -try: - data = self._fetch_data() - if not data or 'events' not in data: - self.logger.warning("[Manager] No events found in API response") - return -except requests.exceptions.RequestException as e: - self.logger.error(f"[Manager] API error: {e}") - return None -``` - -## User-Friendly Messages -- **Explain the situation**: "No games available during off-season" -- **Provide context**: "NHL season typically runs October-June" -- **Suggest solutions**: "Check back when season starts" -- **Distinguish issues**: API problems vs no data vs filtering results - -## Graceful Degradation -- **Fallback content**: Show alternative games when favorites unavailable -- **Cached data**: Use cached data when API fails -- **Service continuity**: Continue operation when non-critical features fail -- **Clear communication**: Explain what's happening to users - -## Debugging Support -- **Comprehensive logging**: Log API responses, filtering results, display updates -- **State tracking**: Log current state and transitions -- **Performance monitoring**: Track timing and resource usage -- **Error context**: Include stack traces for debugging - -## Off-Season Awareness -- **Seasonal messaging**: Different messages for different times of year -- **Helpful context**: Explain why no games are available -- **Future planning**: Mention when season starts -- **Realistic expectations**: Set appropriate expectations during off-season \ No newline at end of file diff --git a/.cursor/rules/git-workflow.mdc b/.cursor/rules/git-workflow.mdc deleted file mode 100644 index c28c603a..00000000 --- a/.cursor/rules/git-workflow.mdc +++ /dev/null @@ -1,51 +0,0 @@ ---- -alwaysApply: true ---- - -# Git Workflow and Branching - -## Branch Naming Conventions -- **Features**: `feature/description-of-feature` (e.g., `feature/weather-forecast-improvements`) -- **Bug fixes**: `fix/description-of-bug` (e.g., `fix/nhl-manager-improvements`) -- **Hotfixes**: `hotfix/critical-issue-description` -- **Refactoring**: `refactor/description-of-refactor` - -## Commit Message Format -``` -type(scope): description - -[optional body] - -[optional footer] -``` - -**Types**: feat, fix, docs, style, refactor, test, chore -**Examples**: -- `feat(nhl): Add enhanced logging for data visibility` -- `fix(display): Resolve rendering performance issue` -- `docs(api): Update ESPN API integration guide` - -## Pull Request Guidelines -- **Self-review**: Review your own PR before requesting review -- **Testing**: Test thoroughly on Raspberry Pi hardware -- **Documentation**: Update relevant documentation if needed -- **Clean history**: Squash commits if necessary for clean history - -## Code Review Checklist -- **Code Quality**: Proper error handling, logging, type hints -- **Architecture**: Follows project patterns, doesn't break existing functionality -- **Performance**: No negative impact on display performance -- **Testing**: Works on Raspberry Pi hardware -- **Documentation**: Comments added for complex logic - -## Merge Strategies -- **Squash and Merge**: Preferred for feature branches and bug fixes -- **Merge Commit**: For complex features with multiple logical commits -- **Rebase and Merge**: For simple, single-commit changes - -## Best Practices -- **Keep branches small and focused** -- **Commit frequently with meaningful messages** -- **Update branch regularly with main** -- **Test changes incrementally** -- **Delete feature branches after merge** \ No newline at end of file diff --git a/.cursor/rules/github-branches-rule.mdc b/.cursor/rules/github-branches-rule.mdc deleted file mode 100644 index 87b1ea5f..00000000 --- a/.cursor/rules/github-branches-rule.mdc +++ /dev/null @@ -1,213 +0,0 @@ ---- -description: GitHub branching and pull request best practices for LEDMatrix project -globs: ["**/*.py", "**/*.md", "**/*.json", "**/*.sh"] -alwaysApply: true ---- - -# GitHub Branching and Pull Request Guidelines - -## Branch Naming Conventions - -### Feature Branches -- **Format**: `feature/description-of-feature` -- **Examples**: - - `feature/weather-forecast-improvements` - - `feature/stock-api-integration` - - `feature/nba-live-scores` - -### Bug Fix Branches -- **Format**: `fix/description-of-bug` -- **Examples**: - - `fix/leaderboard-scrolling-performance` - - `fix/weather-api-timeout` - - `fix/display-rendering-issue` - -### Hotfix Branches -- **Format**: `hotfix/critical-issue-description` -- **Examples**: - - `hotfix/display-crash-fix` - - `hotfix/api-rate-limit-fix` - -### Refactoring Branches -- **Format**: `refactor/description-of-refactor` -- **Examples**: - - `refactor/sports-manager-architecture` - - `refactor/cache-management-system` - -## Branch Management Rules - -### Main Branch Protection -- **`main`** branch is protected and requires PR reviews -- Never commit directly to `main` -- All changes must go through pull requests - -### Branch Lifecycle -1. **Create** branch from `main` when starting work -2. **Keep** branch up-to-date with `main` regularly -3. **Test** thoroughly before creating PR -4. **Delete** branch after successful merge - -### Branch Updates -```bash -# Before starting new work -git checkout main -git pull origin main - -# Create new branch -git checkout -b feature/your-feature-name - -# Keep branch updated during development -git checkout main -git pull origin main -git checkout feature/your-feature-name -git merge main -``` - -## Pull Request Guidelines - -### PR Title Format -- **Feature**: `feat: Add weather forecast improvements` -- **Fix**: `fix: Resolve leaderboard scrolling performance issue` -- **Refactor**: `refactor: Improve sports manager architecture` -- **Docs**: `docs: Update API integration guide` -- **Test**: `test: Add unit tests for weather manager` - -### PR Description Template -```markdown -## Description -Brief description of changes and motivation. - -## Type of Change -- [ ] Bug fix (non-breaking change) -- [ ] New feature (non-breaking change) -- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) -- [ ] Documentation update -- [ ] Performance improvement -- [ ] Refactoring - -## Testing -- [ ] Tested on Raspberry Pi hardware -- [ ] Verified display rendering works correctly -- [ ] Checked API integration functionality -- [ ] Tested error handling scenarios - -## Screenshots/Videos -(If applicable, add screenshots or videos of the changes) - -## Checklist -- [ ] Code follows project style guidelines -- [ ] Self-review completed -- [ ] Comments added for complex logic -- [ ] No hardcoded values or API keys -- [ ] Error handling implemented -- [ ] Logging added where appropriate -``` - -### PR Review Requirements - -#### For Reviewers -- **Code Quality**: Check for proper error handling, logging, and type hints -- **Architecture**: Ensure changes follow project patterns and don't break existing functionality -- **Performance**: Verify changes don't negatively impact display performance -- **Testing**: Confirm changes work on Raspberry Pi hardware -- **Documentation**: Check if documentation needs updates - -#### For Authors -- **Self-Review**: Review your own PR before requesting review -- **Testing**: Test thoroughly on Pi hardware before submitting -- **Documentation**: Update relevant documentation if needed -- **Clean History**: Squash commits if necessary for clean history - -## Commit Message Guidelines - -### Format -``` -type(scope): description - -[optional body] - -[optional footer] -``` - -### Types -- **feat**: New feature -- **fix**: Bug fix -- **docs**: Documentation changes -- **style**: Code style changes (formatting, etc.) -- **refactor**: Code refactoring -- **test**: Adding or updating tests -- **chore**: Maintenance tasks - -### Examples -``` -feat(weather): Add hourly forecast display -fix(nba): Resolve live score update issue -docs(api): Update ESPN API integration guide -refactor(sports): Improve base class architecture -``` - -## Merge Strategies - -### Squash and Merge (Preferred) -- Use for feature branches and bug fixes -- Creates clean, linear history -- Combines all commits into single commit - -### Merge Commit -- Use for complex features with multiple logical commits -- Preserves commit history -- Use when commit messages are meaningful - -### Rebase and Merge -- Use sparingly for simple, single-commit changes -- Creates linear history without merge commits - -## Release Management - -### Version Tags -- Use semantic versioning: `v1.2.3` -- Tag releases on `main` branch -- Create release notes with technical details - -### Release Branches -- **Format**: `release/v1.2.3` -- Use for release preparation -- Include version bumps and final testing - -## Emergency Procedures - -### Hotfix Process -1. Create `hotfix/` branch from `main` -2. Make minimal fix -3. Test thoroughly -4. Create PR with expedited review -5. Merge to `main` and tag release -6. Cherry-pick to other branches if needed - -### Rollback Process -1. Identify last known good commit -2. Create revert PR if possible -3. Use `git revert` for clean rollback -4. Tag rollback release -5. Document issue and resolution - -## Best Practices - -### Before Creating PR -- [ ] Run all tests locally -- [ ] Test on Raspberry Pi hardware -- [ ] Check for linting errors -- [ ] Update documentation if needed -- [ ] Ensure commit messages are clear - -### During Development -- [ ] Keep branches small and focused -- [ ] Commit frequently with meaningful messages -- [ ] Update branch regularly with main -- [ ] Test changes incrementally - -### After PR Approval -- [ ] Delete feature branch after merge -- [ ] Update local main branch -- [ ] Verify changes work in production -- [ ] Update any related documentation diff --git a/.cursor/rules/project-structure.mdc b/.cursor/rules/project-structure.mdc deleted file mode 100644 index c9878a27..00000000 --- a/.cursor/rules/project-structure.mdc +++ /dev/null @@ -1,23 +0,0 @@ ---- -alwaysApply: true ---- - -# LEDMatrix Project Structure - -## Core Architecture -- **Main entry point**: [run.py](mdc:run.py) - Primary application launcher -- **Configuration**: [config/config.json](mdc:config/config.json) - Main configuration file -- **Display management**: [src/display_controller.py](mdc:src/display_controller.py) - Core display logic -- **Web interface**: [web_interface_v2.py](mdc:web_interface_v2.py) - Modern web UI - -## Source Code Organization -- **Managers**: [src/](mdc:src/) - All sports/weather/stock managers -- **Assets**: [assets/](mdc:assets/) - Logos, fonts, and static resources -- **Tests**: [test/](mdc:test/) - Unit and integration tests -- **Documentation**: [LEDMatrix.wiki/](mdc:LEDMatrix.wiki/) - Comprehensive guides - -## Key Design Principles -- **Single Responsibility**: Each manager handles one sport/domain -- **Consistent Patterns**: All managers follow similar structure -- **Configuration-Driven**: Behavior controlled via [config/config.json](mdc:config/config.json) -- **Raspberry Pi Focus**: Optimized for Pi hardware, not Windows development \ No newline at end of file diff --git a/.cursor/rules/raspberry-pi-development.mdc b/.cursor/rules/raspberry-pi-development.mdc deleted file mode 100644 index 829b7591..00000000 --- a/.cursor/rules/raspberry-pi-development.mdc +++ /dev/null @@ -1,41 +0,0 @@ ---- -alwaysApply: true ---- - -# Raspberry Pi Development Guidelines - -## Hardware Constraints -- **Pi-only execution**: Code must run on Raspberry Pi, not Windows development machine -- **LED matrix library**: Uses [rpi-rgb-led-matrix-master/](mdc:rpi-rgb-led-matrix-master/) for hardware control -- **Memory limitations**: Optimize for Pi's limited RAM -- **Performance**: Consider Pi's CPU capabilities in design - -## Development Workflow -- **Local development**: Write and test code on Windows -- **Pi deployment**: Deploy and test on actual Pi hardware -- **SSH access**: Use SSH for Pi-based testing and debugging -- **Service management**: Use systemd services for production deployment - -## Testing Strategy -- **Unit tests**: Test logic without hardware dependencies -- **Integration tests**: Test with mock display managers -- **Hardware tests**: Validate on actual Pi with LED matrix -- **Performance tests**: Monitor memory and CPU usage - -## Deployment Considerations -- **Service files**: [ledmatrix.service](mdc:ledmatrix.service), [ledmatrix-web.service](mdc:ledmatrix-web.service) -- **Installation scripts**: [first_time_install.sh](mdc:first_time_install.sh), [install_service.sh](mdc:install_service.sh) -- **Dependencies**: [requirements.txt](mdc:requirements.txt) for Pi environment -- **Permissions**: Handle file permissions for Pi user - -## Performance Optimization -- **Caching**: Use [src/cache_manager.py](mdc:src/cache_manager.py) for data persistence -- **Background services**: Non-blocking data fetching -- **Memory management**: Clean up resources regularly -- **Display optimization**: Minimize unnecessary redraws - -## Debugging on Pi -- **Logging**: Comprehensive logging for remote debugging -- **Error reporting**: Clear error messages for troubleshooting -- **Status monitoring**: Health checks and status reporting -- **Remote access**: Web interface for configuration and monitoring \ No newline at end of file diff --git a/.cursor/rules/sports-managers.mdc b/.cursor/rules/sports-managers.mdc deleted file mode 100644 index fe6c6fc8..00000000 --- a/.cursor/rules/sports-managers.mdc +++ /dev/null @@ -1,42 +0,0 @@ ---- -globs: src/*_managers.py ---- - -# Sports Manager Development - -## Manager Architecture -All sports managers inherit from base classes and follow consistent patterns: -- **Base classes**: [src/nhl_managers.py](mdc:src/nhl_managers.py), [src/nfl_managers.py](mdc:src/nfl_managers.py) -- **Common functionality**: Data fetching, caching, display rendering -- **Configuration-driven**: Behavior controlled via config sections - -## Required Methods -```python -def __init__(self, config, display_manager, cache_manager) -def update(self) # Fetch fresh data -def display(self, force_clear=False) # Render current data -``` - -## Data Flow Pattern -1. **Fetch**: Get data from API (with caching) -2. **Process**: Extract relevant game information -3. **Filter**: Apply favorite team preferences -4. **Display**: Render to LED matrix - -## Logging Standards -- **Structured prefixes**: `[NHL Recent]`, `[NFL Live]`, etc. -- **Context information**: Include team names, game status, dates -- **Debug levels**: Use appropriate log levels (info, debug, warning, error) -- **User-friendly messages**: Explain what's happening and why - -## Error Handling -- **API failures**: Log and continue with cached data if available -- **No data scenarios**: Distinguish between API issues vs no games available -- **Off-season awareness**: Provide helpful context during non-active periods -- **Fallback behavior**: Show alternative content when preferred content unavailable - -## Configuration Integration -- **Required settings**: Validate on initialization -- **Optional settings**: Provide sensible defaults -- **Background service**: Use for non-blocking data fetching -- **Caching strategy**: Implement intelligent cache management \ No newline at end of file diff --git a/.cursor/rules/testing-standards.mdc b/.cursor/rules/testing-standards.mdc deleted file mode 100644 index 274b76ef..00000000 --- a/.cursor/rules/testing-standards.mdc +++ /dev/null @@ -1,51 +0,0 @@ ---- -globs: test/*.py,src/*.py ---- - -# Testing Standards - -## Test Organization -- **Test directory**: [test/](mdc:test/) - All test files -- **Unit tests**: Test individual components in isolation -- **Integration tests**: Test component interactions -- **Hardware tests**: Validate on Raspberry Pi with actual LED matrix - -## Testing Principles -- **Test behavior, not implementation**: Focus on what the code does, not how -- **Mock external dependencies**: Use mocks for APIs, display managers, cache -- **Test edge cases**: Empty data, API failures, configuration errors -- **Pi-specific testing**: Validate hardware integration - -## Test Structure -```python -def test_manager_initialization(): - """Test that manager initializes with valid config""" - config = {"sport_scoreboard": {"enabled": True}} - manager = ManagerClass(config, mock_display, mock_cache) - assert manager.enabled == True - -def test_api_failure_handling(): - """Test graceful handling of API failures""" - # Test that system continues when API fails - # Verify fallback to cached data - # Check appropriate error logging -``` - -## Mock Patterns -- **Display Manager**: Mock for testing without hardware -- **Cache Manager**: Mock for testing data persistence -- **API responses**: Mock for consistent test data -- **Configuration**: Use test-specific configs - -## Test Categories -- **Unit tests**: Individual manager methods -- **Integration tests**: Manager interactions with services -- **Configuration tests**: Validate config loading and validation -- **Error handling tests**: API failures, invalid data, edge cases - -## Testing Best Practices -- **Descriptive names**: Test names should explain what they test -- **Single responsibility**: Each test should verify one thing -- **Independent tests**: Tests should not depend on each other -- **Clean setup/teardown**: Reset state between tests -- **Pi compatibility**: Ensure tests work in Pi environment \ No newline at end of file diff --git a/.cursorignore b/.cursorignore deleted file mode 100644 index 6f9f00ff..00000000 --- a/.cursorignore +++ /dev/null @@ -1 +0,0 @@ -# Add directories or file patterns to ignore during indexing (e.g. foo/ or *.csv) diff --git a/.cursorrules b/.cursorrules deleted file mode 100644 index 93b923fb..00000000 --- a/.cursorrules +++ /dev/null @@ -1,364 +0,0 @@ -# LEDMatrix Plugin Development Rules - -## Plugin System Overview - -The LEDMatrix project uses a plugin-based architecture. All display -functionality (except core calendar) is implemented as plugins that are -dynamically loaded from the directory configured by -`plugin_system.plugins_directory` in `config.json` — the default is -`plugin-repos/` (per `config/config.template.json:130`). - -> **Fallback note (scoped):** `PluginManager.discover_plugins()` -> (`src/plugin_system/plugin_manager.py:154`) only scans the -> configured directory — there is no fallback to `plugins/` in the -> main discovery path. A fallback to `plugins/` does exist in two -> narrower places: -> - `store_manager.py:1700-1718` — store operations (install/update/ -> uninstall) check `plugins/` if the plugin isn't found in the -> configured directory, so plugin-store flows work even when your -> dev symlinks live in `plugins/`. -> - `schema_manager.py:70-80` — `get_schema_path()` probes both -> `plugins/` and `plugin-repos/` for `config_schema.json` so the -> web UI form generation finds the schema regardless of where the -> plugin lives. -> -> The dev workflow in `scripts/dev/dev_plugin_setup.sh` creates -> symlinks under `plugins/`, which is why the store and schema -> fallbacks exist. For day-to-day development, set -> `plugin_system.plugins_directory` to `plugins` so the main -> discovery path picks up your symlinks. - -## Plugin Structure - -### Required Files -- **manifest.json**: Plugin metadata, entry point, class name, dependencies -- **manager.py**: Main plugin class (must inherit from `BasePlugin`) -- **config_schema.json**: JSON schema for plugin configuration validation -- **requirements.txt**: Python dependencies (if any) -- **README.md**: Plugin documentation - -### Plugin Class Requirements -- Must inherit from `src.plugin_system.base_plugin.BasePlugin` -- Must implement `update()` method for data fetching -- Must implement `display()` method for rendering -- Should implement `validate_config()` for configuration validation -- Optional: Override `has_live_content()` for live priority features - -## Plugin Development Workflow - -### 1. Creating a New Plugin - -**Option A: Use dev_plugin_setup.sh (Recommended)** -```bash -# Link from GitHub -./scripts/dev/dev_plugin_setup.sh link-github - -# Link local repository -./scripts/dev/dev_plugin_setup.sh link -``` - -**Option B: Manual Setup** -1. Create directory in `plugin-repos//` (or `plugins//` - if you're using the dev fallback location) -2. Add `manifest.json` with required fields -3. Create `manager.py` with plugin class -4. Add `config_schema.json` for configuration -5. Enable plugin in `config/config.json` under `"": {"enabled": true}` - -### 2. Plugin Configuration - -Plugins are configured in `config/config.json`: -```json -{ - "": { - "enabled": true, - "display_duration": 15, - "live_priority": false, - "high_performance_transitions": false, - "transition": { - "type": "redraw", - "speed": 2, - "enabled": true - }, - // ... plugin-specific config - } -} -``` - -### 3. Testing Plugins - -**On Development Machine:** -- Run the dev preview server: `python3 scripts/dev_server.py` (then - open `http://localhost:5001`) — renders plugins in the browser - without running the full display loop -- Or run the full display in emulator mode: - `python3 run.py --emulator` (or equivalently - `EMULATOR=true python3 run.py`, or `./scripts/dev/run_emulator.sh`). - The `-e`/`--emulator` CLI flag is defined in `run.py:19-20`. -- Test plugin loading: Check logs for plugin discovery and loading -- Validate configuration: Ensure config matches `config_schema.json` - -**On Raspberry Pi:** -- Deploy and test on actual hardware -- Monitor logs: `journalctl -u ledmatrix -f` (if running as service) -- Check plugin status in web interface - -### 4. Plugin Development Best Practices - -**Code Organization:** -- Keep plugin code in `plugin-repos//` (or its dev-time - symlink in `plugins//`) -- Use shared assets from `assets/` directory when possible -- Follow existing plugin patterns — canonical sources live in the - [`ledmatrix-plugins`](https://github.com/ChuckBuilds/ledmatrix-plugins) - repo (`plugins/hockey-scoreboard/`, `plugins/football-scoreboard/`, - `plugins/clock-simple/`, etc.) -- Place shared utilities in `src/common/` if reusable across plugins - -**Configuration Management:** -- Use `config_schema.json` for validation -- Store secrets in `config/config_secrets.json` under the same plugin - id namespace as the main config — they're deep-merged into the main - config at load time (`src/config_manager.py:162-172`), so plugin - code reads them directly from `config.get(...)` like any other key -- There is no separate `config_secrets` reference field -- Validate all required fields in `validate_config()` - -**Error Handling:** -- Use plugin's logger: `self.logger.info/error/warning()` -- Handle API failures gracefully -- Cache data to avoid excessive API calls -- Provide fallback displays when data unavailable - -**Performance:** -- Use `cache_manager` for API response caching -- Implement background data fetching if needed -- Use `high_performance_transitions` for smoother animations -- Optimize rendering for Pi's limited resources - -**Display Rendering:** -- Use `display_manager` for all drawing operations -- Support different display sizes (check `display_manager.width/height`) -- Use `apply_transition()` for smooth transitions between displays -- Clear display before rendering: `display_manager.clear()` -- Always call `display_manager.update_display()` after rendering - -## Plugin API Reference - -### BasePlugin Class -Located in: `src/plugin_system/base_plugin.py` - -**Required Methods:** -- `update()`: Fetch/update data (called based on `update_interval` in manifest) -- `display(force_clear=False)`: Render plugin content - -**Optional Methods:** -- `validate_config()`: Validate plugin configuration -- `has_live_content()`: Return True if plugin has live/urgent content -- `get_live_modes()`: Return list of modes for live priority -- `cleanup()`: Clean up resources on unload -- `on_config_change(new_config)`: Handle config updates -- `on_enable()`: Called when plugin enabled -- `on_disable()`: Called when plugin disabled - -**Available Properties:** -- `self.plugin_id`: Plugin identifier -- `self.config`: Plugin configuration dict -- `self.display_manager`: Display manager instance -- `self.cache_manager`: Cache manager instance -- `self.plugin_manager`: Plugin manager reference -- `self.logger`: Plugin-specific logger -- `self.enabled`: Boolean enabled status -- `self.transition_manager`: Transition system (if available) - -### Display Manager -Located in: `src/display_manager.py` - -**Key Methods:** -- `clear()`: Clear the display -- `draw_text(text, x, y, color, font, small_font, centered)`: Draw text -- `update_display()`: Push the buffer to the physical display -- `draw_weather_icon(condition, x, y, size)`: Draw a weather icon -- `width`, `height`: Display dimensions - -**Image rendering**: there is no `draw_image()` helper. Paste directly -onto the underlying PIL Image: -```python -self.display_manager.image.paste(pil_image, (x, y)) -self.display_manager.update_display() -``` -For transparency, paste with a mask: `image.paste(rgba, (x, y), rgba)`. - -### Cache Manager -Located in: `src/cache_manager.py` - -**Key Methods:** -- `get(key, max_age=300)`: Get cached value (returns None if missing/stale) -- `set(key, value, ttl=None)`: Cache a value -- `delete(key)` / `clear_cache(key=None)`: Remove a single cache entry, - or (for `clear_cache` with no argument) every cached entry. `delete` - is an alias for `clear_cache(key)`. -- `get_cached_data_with_strategy(key, data_type)`: Cache get with - data-type-aware TTL strategy -- `get_background_cached_data(key, sport_key)`: Cache get for the - background-fetch service path - -## Plugin Manifest Schema - -Required fields in `manifest.json`: -- `id`: Unique plugin identifier (matches directory name) -- `name`: Human-readable plugin name -- `version`: Semantic version (e.g., "1.0.0") -- `entry_point`: Python file (usually "manager.py") -- `class_name`: Plugin class name (must match class in entry_point) -- `display_modes`: Array of mode names this plugin provides - -Common optional fields: -- `description`: Plugin description -- `author`: Plugin author -- `homepage`: Plugin homepage URL -- `category`: Plugin category (e.g., "sports", "weather") -- `tags`: Array of tags -- `update_interval`: Seconds between update() calls (default: 60) -- `default_duration`: Default display duration (default: 15) -- `requires`: Python version, display size requirements -- `config_schema`: Path to config schema file -- `api_requirements`: API dependencies and rate limits - -## Plugin Loading Process - -1. **Discovery**: PluginManager scans `plugins/` directory for directories containing `manifest.json` -2. **Validation**: Validates manifest structure and required fields -3. **Loading**: Imports plugin module and instantiates plugin class -4. **Configuration**: Loads plugin config from `config/config.json` -5. **Validation**: Calls `validate_config()` on plugin instance -6. **Registration**: Adds plugin to available modes and stores instance -7. **Enablement**: Calls `on_enable()` if plugin is enabled - -## Common Plugin Patterns - -### Sports Scoreboard Plugin -- Use `background_data_service.py` pattern for API fetching -- Implement live/recent/upcoming game modes -- Use `scoreboard_renderer.py` for consistent rendering -- Support team filtering and game filtering -- Use shared sports logos from `assets/sports/` - -### Data Display Plugin -- Fetch data in `update()` method -- Cache API responses using `cache_manager` -- Render in `display()` method -- Handle API errors gracefully -- Provide configuration for refresh intervals - -### Real-time Content Plugin -- Implement `has_live_content()` for live priority -- Use `get_live_modes()` to specify which modes are live -- Set `live_priority: true` in config to enable live takeover -- Update data frequently when live content exists - -## Debugging Plugins - -**Check Plugin Loading:** -- Review logs for plugin discovery messages -- Verify manifest.json syntax is valid JSON -- Check that class_name matches actual class name -- Ensure entry_point file exists and is importable - -**Check Plugin Execution:** -- Add logging statements in `update()` and `display()` -- Use `self.logger` for plugin-specific logging -- Check cache_manager for cached data -- Verify display_manager is rendering correctly - -**Common Issues:** -- Import errors: Check Python path and dependencies -- Config errors: Validate against config_schema.json -- Display issues: Check display dimensions and coordinate calculations -- Performance: Monitor CPU/memory usage on Pi - -## Plugin Testing - -**Unit Tests:** -- Test plugin class instantiation -- Test `update()` data fetching logic -- Test `display()` rendering logic -- Test `validate_config()` with various configs -- Mock `display_manager` and `cache_manager` for testing - -**Integration Tests:** -- Test plugin loading via PluginManager -- Test plugin with actual config -- Test plugin with emulator display -- Test plugin with cache_manager - -**Hardware Tests:** -- Test on Raspberry Pi with LED matrix -- Verify display rendering on actual hardware -- Test performance under load -- Test with other plugins enabled - -## File Organization - -``` -plugins/ - / - manifest.json # Plugin metadata - manager.py # Main plugin class - config_schema.json # Config validation schema - requirements.txt # Python dependencies - README.md # Plugin documentation - # Plugin-specific files - data_manager.py - renderer.py - etc. -``` - -## Git Workflow for Plugins - -**Plugin Development:** -- Plugins are typically separate repositories -- Use `dev_plugin_setup.sh` to link plugins for development -- Symlinks are used to connect plugin repos to `plugins/` directory -- Plugin repos follow naming: `ledmatrix-` - -**Branching:** -- Develop plugins in feature branches -- Follow project branching conventions -- Test plugins before merging to main - -**Automatic Version Bumping:** -- **Automatic Version Management**: Version bumping is handled automatically via the pre-push git hook - no manual version bumping is required for normal development workflows -- **GitHub as Source of Truth**: Plugin store always fetches latest versions from GitHub (releases/tags/manifest/commit) -- **Pre-Push Hook**: Automatically bumps patch version and creates git tags when pushing code changes - - The hook is self-contained (no external dependencies) and works on any dev machine - - Installation: Copy the hook from LEDMatrix repo to your plugin repo: - ```bash - # From your plugin repository directory - cp /path/to/LEDMatrix/scripts/git-hooks/pre-push-plugin-version .git/hooks/pre-push - chmod +x .git/hooks/pre-push - ``` - - Or use the installer script from the main LEDMatrix repo (one-time setup) - - The hook automatically: - 1. Bumps the patch version (x.y.Z) in manifest.json when code changes are detected - 2. Creates a git tag (v{version}) for the new version - 3. Stages manifest.json for commit - - Skip auto-tagging: Set `SKIP_TAG=1` environment variable before pushing -- **Manual Version Bumping (Edge Cases Only)**: Manual version bumps are only needed in rare circumstances: - - CI/CD pipelines that bypass git hooks - - Forked repositories without the pre-push hook installed - - Major/minor version bumps (hook only handles patch versions) - - When skipping auto-tagging but still needing a version bump - - For manual bumps, use the standalone script: `scripts/bump_plugin_version.py` -- **Registry**: The plugin registry (plugins.json) stores only metadata (name, description, repo URL) - no versions -- **Version Priority**: Plugin store checks versions in this order: GitHub Releases → GitHub Tags → Manifest from branch → Git commit hash - -## Resources - -- Plugin System Docs: `docs/PLUGIN_ARCHITECTURE_SPEC.md` -- Plugin Examples: `plugins/hockey-scoreboard/`, `plugins/football-scoreboard/` -- Base Plugin: `src/plugin_system/base_plugin.py` -- Plugin Manager: `src/plugin_system/plugin_manager.py` -- Development Setup: `dev_plugin_setup.sh` -- Example Config: `dev_plugins.json.example` - diff --git a/.github/workflows/release-version-check.yml b/.github/workflows/release-version-check.yml new file mode 100644 index 00000000..d9c6320a --- /dev/null +++ b/.github/workflows/release-version-check.yml @@ -0,0 +1,40 @@ +name: Release version check + +# A release tag, the CHANGELOG, and src.__version__ must agree. They have not +# always: v3.1.0 was tagged while src/__init__.py still said "1.0.0", which +# silently exempted every device installed from that release from plugin +# compatibility warnings. See docs/SPORTS_UNIFICATION.md (phase B4). +on: + push: + tags: ["v*"] + release: + types: [published] + # Pre-flight: run this against the tag you are about to create. + workflow_dispatch: + inputs: + tag: + description: "Tag to check (e.g. v3.2.0)" + required: true + type: string + +permissions: + contents: read + +jobs: + version-matches-tag: + name: Tag matches src.__version__ + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 + with: + persist-credentials: false + + - uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5.3.0 + with: + python-version: "3.12" + + # No dependencies: the script reads src/__init__.py and CHANGELOG.md only. + - name: Assert the tag, CHANGELOG and src.__version__ agree + run: python scripts/check_release_version.py "${TAG}" + env: + TAG: ${{ inputs.tag || github.ref_name }} diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 1b3a7f71..7f91e726 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -4,11 +4,47 @@ on: pull_request: push: branches: [main] + # Manual runs against any branch — useful when a PR's automatic run + # needs a re-run or didn't get created. + workflow_dispatch: + +# Both jobs only check out the repo and run pytest. +permissions: + contents: read jobs: plugin-safety: name: Plugin safety harness + unit tests runs-on: ubuntu-latest + env: + # The bundled fixture plugin gives the harness at least one real plugin + # to render, and REQUIRE_PLUGINS turns "discovered zero plugins" into a + # hard failure instead of a silent all-skip green run. + LEDMATRIX_PLUGINS_DIR: test/fixtures/plugins + LEDMATRIX_REQUIRE_PLUGINS: "1" + steps: + - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 + with: + persist-credentials: false + + - uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5.3.0 + with: + python-version: "3.12" + cache: pip + + - name: Install dependencies + run: | + python -m pip install --upgrade pip + pip install -r requirements.txt -r requirements-test.txt + pip install RGBMatrixEmulator + + - name: Run plugin safety harness + run: | + pytest --no-cov test/plugins/ + + unit-tests: + name: Core unit tests + runs-on: ubuntu-latest steps: - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 with: @@ -25,9 +61,15 @@ jobs: pip install -r requirements.txt -r requirements-test.txt pip install RGBMatrixEmulator - - name: Run harness + visual rendering tests + # Run the ENTIRE test tree (except test/plugins, which the + # plugin-safety job owns). New test files are enrolled automatically; + # excluding anything requires a visible, commented --ignore here. + # Coverage is measured and enforced only in this step — pytest.ini + # deliberately carries no coverage flags so local runs stay fast. + - name: Run core unit suites run: | - pytest --no-cov \ - test/plugins/test_harness.py \ - test/plugins/test_visual_rendering.py \ - test/plugins/test_plugin_matrix.py + pytest -m "not hardware" test/ \ + --ignore=test/plugins \ + --cov=src --cov=web_interface \ + --cov-report=term \ + --cov-fail-under=52 diff --git a/.gitignore b/.gitignore index 8689da04..3c3c6a0b 100644 --- a/.gitignore +++ b/.gitignore @@ -48,3 +48,4 @@ config/backups/ # Starlark apps runtime storage (installed .star files and cached renders) /starlark-apps/ +skin_renders/ diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 00000000..db5ea778 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,176 @@ +# Changelog + +Notable changes to the LEDMatrix core. The version below is the value of +`src.__version__`, which the plugin loader reports to compatibility checks and +which plugin manifests reference via `ledmatrix_min_version`. + +**Why this file exists:** the plugin monorepo bundles fallback copies of several +core modules (see `docs/plugin-development/08-shared-sports-code.md` in +[ledmatrix-plugins](https://github.com/ChuckBuilds/ledmatrix-plugins)). A plugin +may delete its bundled copy only when its manifest floors on the first core +release that ships the module — which requires module additions to be recorded +here, against a version number. When you add a module plugins will import via +`src.*`, note it in the Unreleased section and bump `src/__init__.py` in the +release that ships it. + +**Use `ledmatrix_min_version` in manifests, not `ledmatrix_min`.** The loader +accepts both, but the store flags the old spelling as deprecated +(`store_manager.py`) and only the new one is in `schema/manifest_schema.json`. + +## 3.2.0 + +**The first release shipping the unified sports library.** This is the version +a sports plugin floors `ledmatrix_min_version` at before deleting its bundled +copy of `sports.py`, `scroll_display.py`, `data_sources.py` or +`base_odds_manager.py` — the sunset rule in +`docs/plugin-development/08-shared-sports-code.md` keys on exactly this number. + +Adoption is deliberately staged: the modules below ship here, plugins adopt them +behind guarded imports, and only then do the bundled copies go away. Nothing in +this release changes what an existing plugin loads. + +**This is also the first release that *enforces* `ledmatrix_min_version`.** +Before it, the floor was advisory — the loader logged a warning and continued, +and the plugin store never compared the core version at all, so an update could +deliver a plugin that could not run. From 3.2.0 the store refuses such an +install. That matters for the sunset rule: a plugin may only delete its bundled +fallback once the cores in the field actually enforce the floor, which means +waiting for 3.2.0 to be widely installed rather than merely released. See +`docs/SPORTS_UNIFICATION.md`, phase B6. + +One deliberate exception: a core reporting a version below `2.0.0` is treated as +*unknown* rather than old and is never blocked. The v3.1.0 release ships +`__version__ = "1.0.0"` (the tag was cut before the string was bumped), and +nearly every published manifest floors at `2.0.0` — so blocking on that number +would lock those users out of the plugin store entirely. + +### Added +- `src/element_style.py` — per-element style resolver backing the + `x-style-elements` config-schema extension. Already consumed (behind guarded + imports with classic fallbacks) by the `of-the-day`, `ledmatrix-music`, and + `football-scoreboard` plugins. +- Core unit-test CI job enrolling the previously unenrolled suites (skin + system, data sources, API extractors, scroll helper, adaptive layout, loader + compatibility warning) plus new characterization tests for + `src/base_classes/sports.py` ahead of the shared sports-code unification. +- `src/base_classes/sports/` — `sports.py` is now a package (`core.py` + + `modes.py`). The import path is unchanged: `from src.base_classes.sports + import SportsCore` still works. +- Nine methods promoted onto the sports base classes from the plugins' + bundled copies, plus the override points `_favorite_key`, + `_config_schema_path` and `_font_root` and the class attributes + `FINAL_PERIOD` / `CLOCK_COUNTS_DOWN`. See `docs/SPORTS_UNIFICATION.md`. + A plugin may start calling these once its manifest floors + `ledmatrix_min_version` at the release that ships them. + +- `src/base_classes/sports/capabilities/` — opt-in capabilities for the sports + scoreboards, composed by inheritance rather than gated by config branches + inside the base classes: + - `CelebrationMixin` — the score/win takeover, merging the goal and score + dialects behind the `score_phrase()` / `win_phrase()` hooks, the + `COALESCE_SCORING_SEQUENCE` class attribute and the `_favorite_key` seam. + Reads both the `celebrate_opponent_goals` and `celebrate_opponent_scores` + config spellings. Sports that do not mix it in have none of this code in + their MRO. + - `RotationStrategy` + a name registry (`swrr`, `weighted`, `simple`, + plus `register_rotation_strategy` for plugin-supplied orderings). Each + built-in is verified against a verbatim transcription of the plugin + implementation it replaces. An unknown name degrades to `simple`. + +- `src/common/sports_scroll.py` — `SportsScrollDisplay` and + `SportsScrollDisplayManager`, the shared scroll **orchestration** layer for + the sports scoreboards, plus native support for + `global_config['target_fps']` (the bundled plugin copies hardcode ~100 FPS + via `scroll_delay` and never consult the global target). Content building + (`prepare_scroll_content`, `_load_separator_icons`) is per-sport and stays an + override point — see `docs/SPORTS_UNIFICATION.md` for where the line falls + and why. + +- `src/plugin_system/compatibility.py` — the single place that answers "can this + plugin run on this core?", shared by the loader (advisory, at load time) and + the store (blocking, at install/update time) so the two cannot drift. Reads + every spelling published manifests use, including the deprecated + `versions[].ledmatrix_min`. It does **not** yet evaluate `compatible_versions`, + which is the schema-required field and can express upper bounds; closing that + is tracked in `docs/SPORTS_UNIFICATION.md` before B6. +- `scripts/check_release_version.py` and a `Release version check` workflow — + assert that a tag, the newest CHANGELOG heading and `src.__version__` agree, + on pushed `v*` tags and published releases. Runnable via `workflow_dispatch` + to check a tag *before* creating it. Added because `v3.1.0` was tagged six + weeks before `src/__init__.py` was bumped to match, which is why devices + installed from that release report `1.0.0`. + +### Changed +- `src/__init__.py` bumped to **3.2.0** — the number the sunset rule keys on. +- **The plugin store refuses an incompatible install.** + `StoreManager.install_plugin` now checks the downloaded manifest's declared + floor against `src.__version__` and refuses when the plugin needs a newer + core. The check sits in `install_plugin` because `_reinstall_with_rollback` + calls it, so a refused *update* restores the version the user already had. + Refusal requires evidence: an undeclared floor, an unparseable version on + either side, or an untrustworthy core version all allow the install. +- **A failed install no longer destroys the plugin it replaced.** + `install_plugin` previously deleted the existing plugin directory before + downloading, so any later failure — a dropped connection, a malformed + manifest, or the new compatibility refusal — left the user with nothing. The + existing copy is now set aside and restored if the install fails, matching + the protection `_reinstall_with_rollback` already gave the update path. +- `web_interface.__version__` re-exports `src.__version__` instead of carrying + its own hardcoded `"3.0.0"`, which had drifted two majors from the core. +- **Live games are no longer dropped when the feed omits a game clock.** + `SportsLive._is_game_really_over` previously (in the baseball and UFC + plugin lineages) coerced a missing or non-string clock to the literal + `"0:00"` and then treated the game as finished once `period >= 4`. Baseball + has no game clock and `period` is the inning, so live MLB games disappeared + from the scoreboard from the 5th inning onward; UFC was affected the same + way. The clock check is now skipped when the clock is unusable, and the + period threshold is the per-sport `FINAL_PERIOD` (hockey ends in P3). + Sports whose clocks count up — soccer, AFL, NRL — set + `CLOCK_COUNTS_DOWN = False` and never run the check at all, since `0:00` + there means kickoff rather than expiry. + +### Fixed +- **Plugin updates could hang the web request thread.** The per-plugin reinstall + locks were non-reentrant, and `_reinstall_with_rollback` holds one across its + call to `install_plugin` — which now takes the same lock to protect the + set-aside/restore above. That nesting deadlocked + `update_plugin → _reinstall_with_rollback → install_plugin`, the standard + path for every monorepo plugin update. The locks are now `RLock`s. +- `FontManager` resolves `assets/fonts` against the core install root instead + of the process working directory, so font loading works when the process + starts elsewhere (e.g. the plugin safety harness on CI). +- Hockey events whose competitors carry no `statistics` array are no longer + discarded. The extractor read `competitor["statistics"]` unguarded, so a + `KeyError` inside the generator dropped the entire event despite valid + scores and status; shot counts now fall back to `0`. +- Live baseball events that populate status only at the competition level are + no longer discarded. The extractor read the event top-level + `game_event["status"]` for the inning; real ESPN events duplicate it, but + MiLB events synthesized from the MLB Stats API do not, so the lookup raised + a bare `KeyError`. It now reads the already-validated competition-level + status. +- `SportsLive._is_game_really_over` no longer crashes the live-update pass when + a feed sends an explicit null `period`. `None >= FINAL_PERIOD` raised + `TypeError`, and the only caller (`_detect_stale_games`) has no `try/except` + — the same failure shape as the already-fixed null `period_text`. +- An expired clock spelled `"00:00"` now ends the game. The check compared the + colon-stripped clock against a hand-listed set of literals, which `"0000"` is + not a member of, so a finished game with a two-digit-minute clock stayed on + the scoreboard indefinitely. The comparison is now numeric. +- `SportsCore._load_fonts` resolves `assets/fonts` through the `_font_root()` + seam instead of the process working directory. Started outside the install + root, every scoreboard font silently degraded to PIL's default bitmap face. +- `SportsCore._should_log` no longer raises `AttributeError` on the first + warning of a run; `_last_warning_time` is initialized in `__init__` rather + than lazily by an unrelated method. +- `SportsCore._resolve_project_path` resolved relative logo directories + against `/src` instead of the repo root after `sports.py` became a + package — the class bodies moved byte-identically but `__file__` gained a + directory. Both it and `_font_root` now derive from one `_INSTALL_ROOT` + constant. + +## 3.1.0 + +Baseline for this changelog. Highlights already shipped at this version: +skin system for sports scoreboards (#419), Vegas continuous-scroll overhaul +(#423), plugin update surfacing (#421). diff --git a/CLAUDE.md b/CLAUDE.md index e5930fdd..b7e027c4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,12 +6,16 @@ - `config/config.json` — User plugin configuration (persists across plugin reinstalls) - `plugin-repos/` — **Default** plugin install directory used by the Plugin Store, set by `plugin_system.plugins_directory` in - `config.json` (default per `config/config.template.json:130`). + `config.json` (default per `config/config.template.json:167`). Not gitignored. - `plugins/` — Legacy/dev plugin location. Gitignored (`plugins/*`). Used by `scripts/dev/dev_plugin_setup.sh` for symlinks. The plugin - loader falls back to it when something isn't found in `plugin-repos/` - (`src/plugin_system/schema_manager.py:77`). + loader does NOT fall back to it — `PluginManager.discover_plugins()` + (`src/plugin_system/plugin_manager.py`) scans only the configured + directory. Fallbacks exist in two narrower places: store operations + (`StoreManager._find_plugin_path()` in `store_manager.py`) and schema + lookup (`SchemaManager.get_schema_path()` in `schema_manager.py`, + which probes `plugins/` *before* `plugin-repos/`). ## Plugin System - Plugins inherit from `BasePlugin` in `src/plugin_system/base_plugin.py` @@ -20,6 +24,16 @@ - Plugin instantiation args: `plugin_id, config, display_manager, cache_manager, plugin_manager` - Config schemas use JSON Schema Draft-7 - Display dimensions: always read dynamically from `self.display_manager.matrix.width/height` +- Secrets: namespaced by plugin id in `config/config_secrets.json`, declared + via `"x-secret": true` in the plugin's config schema, and deep-merged into + the plugin's config dict at load time — plugins read them with plain + `config.get(...)`, never a separate accessor + +## Dev Workflow +- Link a plugin for development: `./scripts/dev/dev_plugin_setup.sh link-github ` (or `link `); symlinks land in `plugins/` — set `plugin_system.plugins_directory` to `plugins` so discovery picks them up +- Browser preview without the display loop: `python3 scripts/dev_server.py` → http://localhost:5001 +- Full display in emulator mode: `python3 run.py -e` (or `EMULATOR=true python3 run.py`) +- Validate one plugin headlessly: `python3 scripts/check_plugin.py --plugin ` ## Plugin Store Architecture - Official plugins live in the `ledmatrix-plugins` monorepo (not individual repos) @@ -31,7 +45,18 @@ - Plugin configs stored in `config/config.json`, NOT in plugin directories — safe across reinstalls - Third-party plugins can use their own repo URL with empty `plugin_path` +## Skin System (visual overlays for sports scoreboards) +- Skins live in `skins//` (skin.json + skin.py), NOT in plugin dirs — plugin reinstall deletes plugin dirs +- Core: `src/skin_system/` (ScoreboardSkin, SkinContext, runtime); hook: `SportsCore._render_game()` in `src/base_classes/sports/core.py` +- Skins render onto `ctx.canvas` only; fallback to built-in renderer on `False`/exception (3 strikes disables for session) +- View-model guaranteed keys are frozen (see `test/test_skin_system.py::TestViewModelContract`) — renaming keys in `_extract_game_details_common` or sport extractors breaks published skins +- Validate skins headlessly: `python scripts/validate_skin.py --skin `; docs: `docs/SKIN_SYSTEM.md`, `docs/CREATING_SKINS.md` +- Skins are NOT monorepo plugins: no manifest bump / update_registry.py needed + ## Common Pitfalls - paho-mqtt 2.x needs `callback_api_version=mqtt.CallbackAPIVersion.VERSION1` for v1 compat - BasePlugin uses `get_logger()` from `src.logging_config`, not standard `logging.getLogger()` +- `DisplayManager` has no `draw_image()` — paste onto the PIL image directly: + `self.display_manager.image.paste(img, (x, y))` then `update_display()` + (use a mask for transparency: `image.paste(rgba, (x, y), rgba)`) - When modifying a plugin in the monorepo, you MUST bump `version` in its `manifest.json` and run `python update_registry.py` — otherwise users won't receive the update diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 613b51e1..a450585c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -40,7 +40,7 @@ improvements, and code changes. ## Running the tests ```bash -pip install -r requirements.txt +pip install -r requirements.txt -r requirements-test.txt pytest ``` @@ -57,9 +57,13 @@ integration tests. `docs/`. 3. **Keep PRs focused.** One conceptual change per PR. If you find adjacent bugs while working, fix them in a separate PR. -4. **Follow the existing code style.** Python code uses standard - `black`/`ruff` conventions; HTML/JS in `web_interface/` follows the - patterns already in `templates/v3/` and `static/v3/`. +4. **Follow the existing code style.** The pre-commit hooks run + `flake8` (E9, F63, F7, F82 plus bugbear `B` checks), `mypy` on + `src/`, `bandit`, and `gitleaks` — install the CLI with + `python -m pip install pre-commit`, then run + `pre-commit install` so they run on every commit; HTML/JS in + `web_interface/` follows the patterns already in `templates/v3/` + and `static/v3/`. 5. **Update documentation** alongside code changes. If you add a config key, document it in the relevant `*.md` file (or, for plugins, in `config_schema.json` so the form is auto-generated). diff --git a/README.md b/README.md index 7fdd4c99..681176ef 100644 --- a/README.md +++ b/README.md @@ -50,7 +50,15 @@ I'm trying to be open to constructive criticism and support, as long as it's a r
Core Features -The following plugins are available inside of the LEDMatrix project. These modular, rotating Displays that can be individually enabled or disabled per the user's needs with some configuration around display durations, teams, stocks, weather, timezones, and more. Displays include: +LEDMatrix is a plugin platform: the displays below are plugins installed +from the built-in Plugin Store (web interface → Plugins), where each can be +individually enabled, ordered, and configured — display durations, teams, +stocks, weather, timezones, and more. The core repo ships with just two +bundled plugins (`starlark-apps` and `web-ui-info`); the official plugins +live in the [ledmatrix-plugins](https://github.com/ChuckBuilds/ledmatrix-plugins) +monorepo and install with one click, and third-party plugins can be +installed from their own GitHub repositories. Displays available in the +store include: ### Time and Weather - Real-time clock display (2x 64x32 Displays 4mm Pixel Pitch) @@ -141,6 +149,7 @@ The system supports live, recent, and upcoming game information for multiple spo sudo RPI_RGB_FORCE_REBUILD=1 ./first_time_install.sh ``` - Pi 5 config: leave `rp1_rio` at `0` (PIO mode, default) and set `gpio_slowdown` to `1` or `2`. + - **1GB models (Pi 3B / 3B+) and other low-memory boards**: supported, but the `rpi-rgb-led-matrix` C++ build needs more memory than the Pi has. The installer detects this automatically, compiles with fewer parallel jobs, and adds a temporary swapfile for the build which it removes afterwards. Expect that step to take 15-25 minutes instead of 2-5, and leave at least **3GB free** on the SD card. If you manage swap yourself, opt out with `--skip-swap`. To pin the compiler down further, use `--build-jobs 1`. ### RGB Matrix Bonnet / HAT @@ -314,12 +323,12 @@ curl -fsSL https://raw.githubusercontent.com/ChuckBuilds/LEDMatrix/main/scripts/ ``` This one-shot installer will automatically: -- Check system prerequisites (network, disk space, sudo access) +- Check system prerequisites (network, disk space, memory, sudo access) - Install required system packages (git, python3, build tools, etc.) - Clone or update the LEDMatrix repository - Run the complete first-time installation script -The installation process typically takes 10-30 minutes depending on your internet connection and Pi model. All errors are reported explicitly with actionable fixes. +The installation process typically takes 10-30 minutes depending on your internet connection and Pi model. Pi 3B/3B+ and other 1GB boards land at the top of that range, because the C++ library is compiled serially to stay within available memory. All errors are reported explicitly with actionable fixes. **Note:** The script is safe to run multiple times and will handle existing installations gracefully. @@ -356,6 +365,12 @@ sudo bash ./first_time_install.sh This single script installs services, dependencies, configures permissions and sudoers, and validates the setup. +It finishes by asking whether to reboot. If you run it non-interactively — piped, over a script, or with `-y` — there is no one to ask, so **it reboots immediately without prompting**. Pass `--no-reboot-prompt` to install without rebooting: + +```bash +sudo bash ./first_time_install.sh -y --no-reboot-prompt +``` +
@@ -371,6 +386,10 @@ This single script installs services, dependencies, configures permissions and s ### Initial Setup +For a complete list of every key in `config.json` and +`config_secrets.json`, see +[docs/CONFIG_REFERENCE.md](docs/CONFIG_REFERENCE.md). + For most settings I recommend using the web interface: Edit the project via the web interface at http://[IP ADDRESS or HOSTNAME]:5000 or http://ledpi:5000 . @@ -416,7 +435,7 @@ I recommend using the web-ui "Quick Actions" to control the Display. ## Plugins
-LEDMatrix uses a plugin-based architecture where all display functionality (except the core calendar) is implemented as plugins. All managers that were previously built into the core system are now available as plugins through the Plugin Store. +LEDMatrix uses a plugin-based architecture where all display functionality is implemented as plugins. All managers that were previously built into the core system are now available as plugins through the Plugin Store. ### Plugin Store See the [Plugin Store documentation](https://github.com/ChuckBuilds/ledmatrix-plugins) for detailed installation instructions. @@ -440,6 +459,16 @@ See the [Plugin Store documentation](https://github.com/ChuckBuilds/ledmatrix-pl For plugin development, check out the [Hello World Plugin](https://github.com/ChuckBuilds/ledmatrix-hello-world) repository as a starter template. +### Visual Skins for Scoreboards + +Want a different look for a sports scoreboard without forking the plugin? +**Skins** restyle the live/recent/upcoming screens while the plugin keeps +handling data, scheduling, caching, and vegas mode. Install one with +`git clone skins/`, select it in the plugin's config, +and you're done — see [docs/SKIN_SYSTEM.md](docs/SKIN_SYSTEM.md) (how it +works) and [docs/CREATING_SKINS.md](docs/CREATING_SKINS.md) (build your own, +including a ready-made Claude Code prompt). + 2. **Built-in Managers Deprecated**: The built-in managers (hockey, football, stocks, etc.) are now deprecated and have been moved to the plugin system. **You must install replacement plugins from the Plugin Store** in the web interface instead. The plugin system provides the same functionality with better maintainability and extensibility.
@@ -571,6 +600,14 @@ These settings are typically only needed for non-standard panels or custom confi - Leave empty unless you need custom mapping - See rpi-rgb-led-matrix documentation for full options +- **`orientation`** (string, default: "normal") + - Rotates the rendered image to match how the panel is physically mounted + - Set to `"180"` (or use the "Upside Down" option in the web UI's Display + settings) if the panel is mounted upside down — useful for optimizing + where the Raspberry Pi and wiring sit relative to the mounting location + - Applied independently of `pixel_mapper_config` (appended as a trailing + `Rotate:180` mapper), so custom mapper configs keep working alongside it + - **`row_address_type`** (integer, default: 0) - How rows are addressed on the panel - Most panels use 0 (direct addressing) @@ -599,12 +636,7 @@ These settings control runtime behavior and GPIO timing: ### Display Durations (`display.display_durations`) -Controls how long each display module stays visible in seconds before switching to the next one. - -- **`calendar`** (integer, default: 30) - - Duration in seconds for the calendar display - - Increase for more time to read dates/events - - Decrease to cycle through other displays faster +Controls how long each installed plugin stays visible in seconds before switching to the next one, keyed by plugin id. - **Plugin-specific durations** - Each plugin can have its own duration setting diff --git a/assets/README.md b/assets/README.md new file mode 100644 index 00000000..e88d7f0d --- /dev/null +++ b/assets/README.md @@ -0,0 +1,22 @@ +# assets/ + +Static assets bundled with LEDMatrix. **Do not delete these directories** — +several look unused from core code alone but are resolved at runtime by +installed store plugins. + +| Directory | Used by | +|---|---| +| `fonts/` | Core (`FontManager`, `DisplayManager`) and most plugins | +| `sports/` | Core logo tooling (`src/logo_downloader.py`) and the sports scoreboard plugins; team logos are downloaded here on demand | +| `stocks/` | `ledmatrix-stocks` plugin (`crypto_icons/`, `ticker_icons/`) | +| `weather/` | `ledmatrix-weather` plugin (weather icons) | +| `news_logos/` | `news` plugin | +| `broadcast_logos/` | `news` and `odds-ticker` plugins | +| `static_images/` | Legacy examples referenced in the `static-image` plugin's docs; the plugin itself stores uploads under `assets/plugins//uploads/` | +| `plugins/` | Per-plugin uploaded files (`assets/plugins//uploads/`), served by the web interface | + +Plugins resolve these paths relative to the LEDMatrix install directory, so +the directories are part of the de-facto plugin API even where no file in +this repo references them. New plugins should bundle their own assets or +use the per-plugin upload directory instead of adding top-level +directories here. diff --git a/assets/sports/ncaa_logos/COLGATE.png b/assets/sports/ncaa_logos/COLGATE.png new file mode 100644 index 00000000..9d17a9d0 Binary files /dev/null and b/assets/sports/ncaa_logos/COLGATE.png differ diff --git a/assets/sports/ncaa_logos/COR.png b/assets/sports/ncaa_logos/COR.png new file mode 100644 index 00000000..d39027fa Binary files /dev/null and b/assets/sports/ncaa_logos/COR.png differ diff --git a/assets/sports/ncaa_logos/LEHIGH.png b/assets/sports/ncaa_logos/LEHIGH.png new file mode 100644 index 00000000..cbe8f4e2 Binary files /dev/null and b/assets/sports/ncaa_logos/LEHIGH.png differ diff --git a/assets/sports/ncaa_logos/MICHIGAN.png b/assets/sports/ncaa_logos/MICHIGAN.png new file mode 100644 index 00000000..3929cafe Binary files /dev/null and b/assets/sports/ncaa_logos/MICHIGAN.png differ diff --git a/assets/sports/ncaa_logos/RUTGERS.png b/assets/sports/ncaa_logos/RUTGERS.png new file mode 100644 index 00000000..19e826dc Binary files /dev/null and b/assets/sports/ncaa_logos/RUTGERS.png differ diff --git a/bandit.yaml b/bandit.yaml new file mode 100644 index 00000000..14a6772d --- /dev/null +++ b/bandit.yaml @@ -0,0 +1,29 @@ +# bandit.yaml — LEDMatrix bandit configuration +# https://bandit.readthedocs.io/en/latest/config.html +# +# Skips are justified by the specific codebase context documented below. +# Do not remove skips without updating the justification comment. + +skips: + # B104: Binding to all interfaces (0.0.0.0) + # Intentional — the Flask server binds 0.0.0.0 for LAN access on a Raspberry Pi. + # This is not internet-facing and is documented in web_interface/app.py. + - B104 + + # B603: subprocess call without shell=True + # All subprocess.run() calls in this codebase use list arguments (confirmed by + # grep — zero uses of shell=True in src/ or web_interface/). List args prevent + # shell injection. See src/common/permission_utils.py for the primary usage. + - B603 + + # B607: Starting a process with a partial executable path + # The subprocess calls invoke system utilities (systemctl, sudo, git) by name. + # These are fixed-list invocations, not user-controlled, and rely on PATH. + - B607 + +exclude_dirs: + - tests + - test + - venv + - .venv + - rpi-rgb-led-matrix-master diff --git a/config/config.template.json b/config/config.template.json index 07a72b60..95ea0863 100644 --- a/config/config.template.json +++ b/config/config.template.json @@ -88,6 +88,7 @@ } }, "timezone": "America/New_York", + "target_fps": 100, "location": { "city": "Tampa", "state": "Florida", @@ -109,7 +110,12 @@ "inverse_colors": false, "show_refresh_rate": false, "led_rgb_sequence": "RGB", - "limit_refresh_rate_hz": 100 + "limit_refresh_rate_hz": 100, + "pixel_mapper_config": "", + "orientation": "normal", + "row_address_type": 0, + "multiplexing": 0, + "panel_type": "" }, "runtime": { "gpio_slowdown": 3, @@ -121,15 +127,39 @@ "axis": "horizontal" }, "display_durations": {}, + "plugin_rotation_order": [], "use_short_date_format": true, "vegas_scroll": { + "live_in_ticker": false, + "live_weight": 3, + "favorite_live_weight": 5, "enabled": false, "scroll_speed": 50, "separator_width": 32, "plugin_order": [], "excluded_plugins": [], "target_fps": 125, - "buffer_ahead": 2 + "buffer_ahead": 2, + "intra_plugin_gap": 8, + "render_width_pct": 100, + "min_content_separation": 24, + "min_cut_gap": 6, + "continuous_scroll": true, + "smooth_scroll": true, + "extend_threshold_screens": 2.0, + "auto_trim": true, + "trim_threshold": 10, + "content_padding": 8, + "min_plugin_width": 8, + "lead_in_width": 0, + "plugins_per_cycle": 6, + "max_plugin_width_ratio": 0.0, + "overflow_mode": "rotate", + "dynamic_duration_enabled": true, + "min_cycle_duration": 60, + "max_cycle_duration": 240, + "frame_based_scrolling": true, + "scroll_delay": 0.02 } }, "sync": { @@ -140,7 +170,8 @@ "plugin_system": { "plugins_directory": "plugin-repos", "auto_discover": true, - "auto_load_enabled": true + "auto_load_enabled": true, + "development_mode": false }, "web-ui-info": { "enabled": true, diff --git a/config/config_secrets.template.json b/config/config_secrets.template.json index 61909876..cbf60336 100644 --- a/config/config_secrets.template.json +++ b/config/config_secrets.template.json @@ -1,9 +1,5 @@ { - "youtube": { - "api_key": "YOUR_YOUTUBE_API_KEY", - "channel_id": "YOUR_YOUTUBE_CHANNEL_ID" - }, "github": { "api_token": "YOUR_GITHUB_PERSONAL_ACCESS_TOKEN" } -} \ No newline at end of file +} diff --git a/docs/ADVANCED_FEATURES.md b/docs/ADVANCED_FEATURES.md index ec94df6c..d620fef7 100644 --- a/docs/ADVANCED_FEATURES.md +++ b/docs/ADVANCED_FEATURES.md @@ -47,6 +47,11 @@ Enable Vegas mode in `config/config.json`: } ``` +Vegas mode can also be configured entirely from the web UI — the +**Display** tab has a Vegas Scroll Mode section (enable toggle, scroll +speed, separator width, dynamic duration, and more), so hand-editing +JSON is optional. + **Configuration Options:** | Setting | Default | Description | @@ -57,7 +62,99 @@ Enable Vegas mode in `config/config.json`: | `plugin_order` | `[]` | Plugin display order (empty = auto) | | `excluded_plugins` | `[]` | Plugins to exclude from Vegas mode | | `target_fps` | `125` | Target frame rate | -| `buffer_ahead` | `2` | Number of panels to render ahead | +| `buffer_ahead` | `2` | Number of plugins buffered ahead | + +This table is a subset — `display.vegas_scroll` supports 30 keys in +total. See the full list in +[CONFIG_REFERENCE.md](CONFIG_REFERENCE.md#displayvegas_scroll--continuous-scroll-mode). + +### Live Content in the Ticker + +By default, live content **preempts** Vegas mode: while any plugin reports +live priority, the display controller refuses to run the ticker and shows +that plugin's full-screen display instead. You get a big readable scoreboard, +but the marquee stops entirely for the duration of the game. + +Set `live_in_ticker` to keep the ticker running and let live content take +**extra turns inside it** instead: + +```json +"vegas_scroll": { + "live_in_ticker": true, + "live_weight": 3, + "favorite_live_weight": 5 +} +``` + +#### Why weights exist + +The rotation is otherwise a strict round robin — every plugin appears exactly +once per cycle. With a dozen plugins enabled, a live score comes round once a +lap and can be minutes old by the time you see it. A weight of *N* gives a +plugin *N* slots per cycle. + +The slots are placed by **Smooth Weighted Round-Robin**, the same scheduler +the sports plugins use internally to rotate their own games. The important +property is that repeats are *spread through the cycle* rather than clumped: +three appearances in a row followed by a long silence would be worse than not +boosting at all. + +Twelve plugins, with a favorite's baseball game and an ordinary live hockey +game (`live_weight: 3`, `favorite_live_weight: 5`): + +``` +baseball > hockey > weather > clock > baseball +stocks > news > flights > baseball > hockey +calendar > f1 > music > baseball > tides +birds > hockey > baseball +``` + +18 slots for 12 plugins. Baseball appears 5 times, hockey 3, everything else +once, and no plugin ever appears twice in a row — **including across the seam** +where the cycle loops back on itself. Smooth Weighted Round-Robin schedules the +heaviest item first and usually last as well, so the strip would otherwise show +it twice running at exactly the one join a within-cycle check cannot see. The +trailing repeat is moved into the widest remaining gap. Where a double is +unavoidable — a plugin holding most of the slots has to neighbour itself — the +schedule is left as it is. + +#### Where the weight comes from + +For each plugin in the rotation, in order: + +1. **The plugin's own answer.** If it implements + `get_vegas_priority_weight()` and returns a number, that wins. This is the + only route for favorite-team awareness — the core can see *that* a game is + live, but not *whose*, so a scoreboard has to say so itself. +2. **The core's default.** When the plugin returns `None` (the base-class + default), a plugin where both `has_live_priority()` and `has_live_content()` + are true gets `live_weight`. +3. **Everything else** gets 1. + +Because of step 2, **existing plugins need no changes** — any scoreboard with +`live_priority` enabled already gets extra turns. Step 1 is opt-in, for +plugins that want to distinguish a favorite's game from any other live game. + +Weights are clamped to 1–10. A weight of 1 is no boost; a weight below 1 would +drop the plugin from the rotation entirely, which is never what is meant. + +#### Things worth knowing + +- **Weights are per plugin, not per game.** A scoreboard showing four live + games still occupies one slot at a time, rotating its own games within that + slot using its own `favorite_live_boost`. This controls how often the + *plugin* comes round. +- **The ticker is zero-sum.** Giving baseball 5 slots does not make the cycle + faster; it makes the cycle *longer* and everything else proportionally + rarer. If you want live scores sooner in wall-clock terms, pair this with a + smaller `plugins_per_cycle`. +- **Frequency is not freshness.** Each appearance redraws from the plugin's + current data (`refresh_updated_plugins()` drops cached content when a + plugin's data changes), but how current that data is depends on the + plugin's own `live_update_interval`. Showing a stale score five times a lap + is no better than showing it once. +- **Everything still appears.** A boost never starves another plugin out of + the cycle; low-weight plugins keep their single slot. ### Per-Plugin Configuration @@ -79,9 +176,13 @@ Override Vegas behavior for specific plugins: | Setting | Values | Description | |---------|--------|-------------| | `vegas_mode` | `scroll`, `fixed`, `static` | Display mode for this plugin | -| `vegas_panel_count` | `1-10` | Width in panels (1 panel = display width) | +| `vegas_panel_count` | any positive integer | Width in panels (1 panel = display width) | | `display_duration` | seconds | Pause duration for STATIC mode | +Plugins may also set `vegas_overflow` and `vegas_max_width_screens` in +their config section to control how oversized content is handled (see +`PluginManager` in `src/plugin_system/plugin_manager.py`). + ### Plugin Integration (Developer Guide) **1. Implement Content Method:** @@ -451,7 +552,7 @@ time when something is active. ### REST API Reference -The API is mounted at `/api/v3` (`web_interface/app.py:144`). +The API is mounted at `/api/v3` (`web_interface/app.py:199`). #### Start On-Demand Display @@ -518,13 +619,15 @@ curl http://localhost:5000/api/v3/display/on-demand/status > There is no public Python on-demand API. The display controller's > on-demand machinery is internal — drive it through the REST endpoints -> above (or the web UI buttons), which write a request into the cache -> manager under the `display_on_demand_request` key -> (`web_interface/blueprints/api_v3.py:1622,1687`) that the controller -> polls at `src/display_controller.py:921`. A separate +> above (or the web UI buttons). The API handlers +> (`start_on_demand_display()` / `stop_on_demand_display()` in +> `web_interface/blueprints/api_v3.py`) write a request into the cache +> manager under the `display_on_demand_request` key, which +> `DisplayController._poll_on_demand_requests()` +> (`src/display_controller.py`) picks up. A separate > `display_on_demand_config` key is used by the controller itself -> during activation to track what's currently running (written at -> `display_controller.py:1195`, cleared at `:1221`). +> during activation (`_activate_on_demand()`) to track what's +> currently running, and is cleared by `_clear_on_demand()`. ### Duration Modes @@ -646,13 +749,13 @@ keys helps troubleshoot stuck states. **When Set:** Every display loop iteration **Auto-Cleared:** Never (continuously updated) -**4. display_on_demand_processed_id** (TTL: 5 minutes) -``` +**4. display_on_demand_processed_id** (TTL: 1 hour) +```text "uuid-string-of-last-processed-request" ``` **Purpose:** Prevents duplicate request processing **When Set:** After processing request -**Auto-Cleared:** After 5 minutes TTL +**Auto-Cleared:** After 1 hour TTL ### When Manual Clearing is Needed @@ -685,9 +788,9 @@ keys helps troubleshoot stuck states. The cache is stored as JSON files under one of: - `/var/cache/ledmatrix/` (preferred when the service has permission) -- `~/.cache/ledmatrix/` +- `~/.ledmatrix_cache/` - `/opt/ledmatrix/cache/` -- `/tmp/ledmatrix-cache/` (fallback) +- `$TMPDIR/ledmatrix_cache/` (fallback) ```bash # Find the cache dir actually in use @@ -711,8 +814,9 @@ cache.clear_cache('display_on_demand_request') cache.clear_cache('display_on_demand_processed_id') ``` -> The actual public method is `clear_cache(key=None)` — there is no -> `delete()` method on `CacheManager`. +> `CacheManager` also has a `delete(key)` method — a thin wrapper over +> `clear_cache(key)` — so `cache.delete('display_on_demand_config')` +> works equally well. ### Cache Impact on Running Service @@ -730,7 +834,7 @@ The display controller automatically handles cleanup: - **Config key**: Cleared when on-demand stops - **State key**: Updated every display loop iteration - **Request key**: Expires after 1 hour TTL (or after processing) -- **Processed ID**: Expires after 5 minutes TTL +- **Processed ID**: Expires after 1 hour TTL --- @@ -821,9 +925,6 @@ same shape as the example above. ### Testing ```bash -# Run background service test -python test_background_service.py - # Check logs for background operations sudo journalctl -u ledmatrix -f | grep "background" ``` @@ -832,9 +933,10 @@ sudo journalctl -u ledmatrix -f | grep "background" **View Statistics:** ```python -from src.background_data_service import BackgroundDataService +from src.background_data_service import get_background_service +from src.cache_manager import CacheManager -service = BackgroundDataService() +service = get_background_service(CacheManager()) stats = service.get_statistics() print(f"Active tasks: {stats['active_tasks']}") print(f"Completed: {stats['completed']}") @@ -875,6 +977,7 @@ from src.common.permission_utils import ( ensure_file_permissions, get_config_file_mode, get_assets_file_mode, + get_assets_dir_mode, get_plugin_file_mode, get_cache_dir_mode ) @@ -883,7 +986,10 @@ from src.common.permission_utils import ( ensure_directory_permissions(Path("assets/sports"), get_assets_dir_mode()) # Set file permissions after writing -ensure_file_permissions(Path("config/config.json"), get_config_file_mode()) +# (get_config_file_mode requires the file path — secrets files get a +# stricter mode than the main config) +config_path = Path("config/config.json") +ensure_file_permissions(config_path, get_config_file_mode(config_path)) ``` ### When to Use Utilities @@ -938,7 +1044,7 @@ from src.common.permission_utils import ensure_file_permissions, get_config_file config_path = Path("config/config.json") with open(config_path, 'w') as f: json.dump(data, f) -ensure_file_permissions(config_path, get_config_file_mode()) +ensure_file_permissions(config_path, get_config_file_mode(config_path)) ``` **Pattern 3: Downloading Logo** @@ -984,8 +1090,11 @@ These core utilities **already handle permissions** - you don't need to call per If you encounter permission issues: ```bash -# Fix all permissions at once -sudo ./scripts/fix_permissions.sh +# Targeted permission fixes (see scripts/fix_perms/README.md) +sudo ./scripts/fix_perms/fix_assets_permissions.sh # assets/ tree (logos, fonts) +sudo ./scripts/fix_perms/fix_cache_permissions.sh # all cache directories +sudo ./scripts/fix_perms/fix_plugin_permissions.sh # plugin directories +sudo ./scripts/fix_perms/fix_web_permissions.sh # web interface files # Fix specific directory sudo chown -R ledpi:ledpi /home/ledpi/LEDMatrix/config @@ -1017,7 +1126,7 @@ stat -c "%a %n" config/config.json ## Related Documentation -- [PLUGIN_DEVELOPMENT.md](PLUGIN_DEVELOPMENT.md) - Creating plugins with Vegas/on-demand support +- [PLUGIN_DEVELOPMENT_GUIDE.md](PLUGIN_DEVELOPMENT_GUIDE.md) - Creating plugins with Vegas/on-demand support - [WEB_INTERFACE_GUIDE.md](WEB_INTERFACE_GUIDE.md) - Using on-demand controls in web UI - [PLUGIN_API_REFERENCE.md](PLUGIN_API_REFERENCE.md) - Complete API documentation - [DEVELOPMENT.md](DEVELOPMENT.md) - Development environment and testing diff --git a/docs/CONFIG_REFERENCE.md b/docs/CONFIG_REFERENCE.md new file mode 100644 index 00000000..4af3c0fb --- /dev/null +++ b/docs/CONFIG_REFERENCE.md @@ -0,0 +1,177 @@ +# Configuration Reference + +Every key in `config/config.json`, what it does, its default, and where the +code reads it. The file is created from `config/config.template.json` on +first run, and `ConfigManager._migrate_config()` merges any template keys +added by later releases into your existing config (your values are never +overwritten). Secrets live in `config/config_secrets.json` and are merged +into the config at load time. + +Most settings are editable from the web interface; this page documents the +underlying keys for people editing `config.json` directly or writing +tooling against it. + +## Top level + +| Key | Type / default | Meaning | Read by | +|---|---|---|---| +| `web_display_autostart` | bool, `true` | Whether the web interface service starts with the system | `scripts/utils/start_web_conditionally.py` | +| `timezone` | string, `"America/New_York"` | IANA timezone for schedules and displays | `ConfigManager.get_timezone()` | +| `target_fps` | int, `100` | Frame-rate ceiling for plugin rendering | `src/plugin_system/base_plugin.py`, `src/common/sports_scroll.py` | +| `location` | object | `city` / `state` / `country`. Supplies the **default** for a plugin's own `location_city` / `location_state` / `location_country` setting, so weather, radar and friends follow this device without being configured twice. A value saved on the plugin itself still overrides it. | `SchemaManager.apply_device_location()`, then plugins via merged config | + +## `schedule` — display on/off hours + +| Key | Type / default | Meaning | +|---|---|---| +| `enabled` | bool, `false` | Master switch for scheduled display on/off | +| `mode` | `"global"` or `"per-day"`, template uses `"per-day"` | Whether one time range applies to all days or each day has its own | +| `start_time` / `end_time` | `"HH:MM"`, `07:00`–`23:00` | Global-mode on/off times | +| `days..{enabled,start_time,end_time}` | per-day objects | Per-day-mode overrides | + +Read by `DisplayController` (`src/display_controller.py`, `_check_schedule` +around line 603). Managed in the web UI under Schedule. + +## `dim_schedule` — scheduled brightness dimming + +Same shape as `schedule`, plus: + +| Key | Type / default | Meaning | +|---|---|---| +| `dim_brightness` | int, `30` | Brightness percentage applied while the dim window is active | + +Read by `DisplayController` (`src/display_controller.py` around line 770; +saved via `POST /api/v3/config/dim-schedule`). The display returns to +`display.hardware.brightness` outside the window. + +## `display.hardware` — matrix panel hardware + +All keys map to the corresponding `rpi-rgb-led-matrix` options and are read +in `DisplayManager` (`src/display_manager.py`, ~lines 270–295). + +| Key | Type / default | +|---|---| +| `rows` / `cols` | int, `32` / `64` | +| `chain_length` | int, `2` | +| `parallel` | int, `1` | +| `brightness` | int, `90` | +| `hardware_mapping` | string, `"adafruit-hat"` (code default `"adafruit-hat-pwm"`) | +| `scan_mode` | int, `0` | +| `pwm_bits` | int, `9` (code default 10) | +| `pwm_dither_bits` | int, `1` | +| `pwm_lsb_nanoseconds` | int, `130` (code default 150) | +| `disable_hardware_pulsing` | bool, `false` | +| `inverse_colors` | bool, `false` | +| `show_refresh_rate` | bool, `false` | +| `led_rgb_sequence` | string, `"RGB"` | +| `limit_refresh_rate_hz` | int, `100` (code default 90) | +| `pixel_mapper_config` | string, `""` — e.g. `"U-mapper"` / `"Rotate:90"` | +| `orientation` | string, `"normal"` — `"180"` rotates the rendered image 180° for panels physically mounted upside down (e.g. to move the Pi/wiring to a more convenient side); composed onto `pixel_mapper_config` as a trailing `Rotate:180` mapper, so it stays independent of any custom `pixel_mapper_config` value | +| `row_address_type` | int, `0` — non-standard panel row addressing | +| `multiplexing` | int, `0` — panel multiplexing scheme | +| `panel_type` | string, `""` — set to `"FM6126A"` or `"FM6127"` for panels needing init | + +Where "code default" differs from the template value, the code default only +applies if the key is missing entirely from your config. + +## `display.runtime` + +| Key | Type / default | Meaning | +|---|---|---| +| `gpio_slowdown` | int, `3` | GPIO timing slowdown for faster Pis | +| `rp1_rio` | int, `0` | RP1 RIO mode on Pi 5 (applied only if the installed matrix library supports it) | + +## `display.double_sided` + +Drives `_LogicalMatrix` in `src/display_manager.py` — renders the same +logical image to multiple chained physical panels. + +| Key | Type / default | Meaning | +|---|---|---| +| `enabled` | bool, `false` | Mirror output across panel copies | +| `copies` | int, `2` | Number of physical copies in the chain | +| `axis` | `"horizontal"`, default | Axis along which panels are chained | + +## `display` — other keys + +| Key | Type / default | Meaning | Read by | +|---|---|---|---| +| `display_durations` | object, `{}` | Per-plugin display duration in seconds, keyed by plugin id (e.g. `"clock": 15`) | `src/display_controller.py:1030` | +| `plugin_rotation_order` | array, `[]` | Explicit rotation order of plugin ids; empty = all enabled plugins in discovery order | `src/display_controller.py:2894` | +| `use_short_date_format` | bool, `true` | Compact date rendering in sports scoreboards | `src/base_classes/sports/core.py` | +| `dynamic_duration.max_duration_seconds` | int, optional | Cap for plugins that request dynamic display time | `src/display_controller.py:405` | + +## `display.vegas_scroll` — continuous scroll mode + +Read by `src/vegas_mode/config.py` (`VegasScrollConfig.from_config`). See +[ADVANCED_FEATURES.md](ADVANCED_FEATURES.md) for behavior details, including +[live content in the ticker](ADVANCED_FEATURES.md#live-content-in-the-ticker). + +| Key | Type / default | +|---|---| +| `enabled` | bool, `false` | +| `scroll_speed` | int, `50` (px/s) | +| `separator_width` | int, `32` | +| `plugin_order` | array, `[]` | +| `excluded_plugins` | array, `[]` | +| `target_fps` | int, `125` | +| `buffer_ahead` | int, `2` | +| `intra_plugin_gap` | int, `8` | +| `render_width_pct` | int, `100` | +| `min_content_separation` | int, `24` | +| `min_cut_gap` | int, `6` | +| `continuous_scroll` | bool, `true` | +| `smooth_scroll` | bool, `true` | +| `extend_threshold_screens` | float, `2.0` | +| `auto_trim` | bool, `true` | +| `trim_threshold` | int, `10` | +| `content_padding` | int, `8` | +| `min_plugin_width` | int, `8` | +| `lead_in_width` | int, `0` | +| `plugins_per_cycle` | int, `6` | +| `max_plugin_width_ratio` | float, `0.0` | +| `overflow_mode` | string, `"rotate"` | +| `dynamic_duration_enabled` | bool, `true` | +| `min_cycle_duration` | int, `60` | +| `max_cycle_duration` | int, `240` | +| `frame_based_scrolling` | bool, `true` — frame-count-based scroll stepping | +| `scroll_delay` | float, `0.02` — seconds between scroll updates (~50 FPS) | +| `live_in_ticker` | bool, `false` — keep scrolling during live games instead of handing the display to a full-screen scoreboard | +| `live_weight` | int, `3` (1–10) — slots per cycle for a plugin with live content | +| `favorite_live_weight` | int, `5` (1–10) — slots per cycle when a plugin reports a favorite team is live | + +## `sync` — multi-display synchronization + +Read by `src/common/sync_manager.py` and `src/display_controller.py`. + +| Key | Type / default | Meaning | +|---|---|---| +| `role` | `"standalone"` (default), `"leader"`, or `"follower"` | This device's role in a synced pair | +| `port` | int, `5765` | TCP port used for sync traffic | +| `follower_position` | `"left"` (default) or `"right"` | Which half of the combined image this follower renders (`src/display_controller.py:522`) | + +## `plugin_system` + +Read by the plugin loader/manager (`src/plugin_system/`). + +| Key | Type / default | Meaning | +|---|---|---| +| `plugins_directory` | string, `"plugin-repos"` | Where the Plugin Store installs plugins | +| `auto_discover` | bool, `true` | Scan the plugins directory at startup | +| `auto_load_enabled` | bool, `true` | Load discovered plugins automatically | +| `development_mode` | bool, `false` | Development conveniences in the web UI (editable under General settings) | + +## Plugin config blocks + +Every installed plugin stores its settings under a top-level key equal to +its plugin id (the template ships one for the bundled `web-ui-info` +plugin). The shape of each block is defined by that plugin's +`config_schema.json`; common keys are `enabled` and `display_duration`. +See [PLUGIN_CONFIG_CORE_PROPERTIES.md](PLUGIN_CONFIG_CORE_PROPERTIES.md). + +## `config/config_secrets.json` + +| Key | Meaning | +|---|---| +| `github.api_token` | Optional GitHub token the Plugin Store uses to avoid API rate limits (`src/plugin_system/store_manager.py:348`) | +| `.*` | Secrets a plugin declares with `"x-secret": true` in its config schema; merged into that plugin's config at load time | diff --git a/docs/CREATING_SKINS.md b/docs/CREATING_SKINS.md new file mode 100644 index 00000000..501c1510 --- /dev/null +++ b/docs/CREATING_SKINS.md @@ -0,0 +1,242 @@ +# Creating Skins + +A skin restyles a sports scoreboard (live / recent / upcoming) without +forking the plugin: the plugin keeps fetching data, scheduling, caching, and +doing vegas mode; your skin only draws. Architecture background: +[SKIN_SYSTEM.md](SKIN_SYSTEM.md). + +## Quick start + +```bash +cp -r skins/example-classic-baseball skins/my-skin +# edit skins/my-skin/skin.json -> set id ("my-skin"), name, author, class_name +# edit skins/my-skin/skin.py -> rename the class, start restyling +python scripts/validate_skin.py --skin my-skin +``` + +The validator renders your skin against bundled fixture games at several +panel sizes with **no hardware, no network, no running service**, saves PNGs +(plus 4x previews) to `skin_renders/`, and fails loudly on errors. Iterate: +edit → validate → look at the PNGs. + +To see it on your matrix, add to your plugin's section in `config/config.json`: + +```json +"baseball-scoreboard": { + "skin": "my-skin", + "skin_options": { } +} +``` + +or pick it from the **Visual Skin** dropdown in the web UI (it appears once a +matching skin is installed). `"skin"` also accepts a per-mode mapping: +`{"live": "my-skin", "recent": "built-in"}`. + +## The manifest (`skin.json`) + +```json +{ + "id": "my-skin", + "name": "My Skin", + "version": "1.0.0", + "author": "you", + "description": "What it looks like", + "skin_api_version": "1.0.0", + "targets": { + "sports": ["baseball"], + "sport_keys": ["mlb", "milb"], + "plugins": [] + }, + "entry_point": "skin.py", + "class_name": "MySkin", + "modes": ["live", "recent", "upcoming"], + "preview": "preview.png" +} +``` + +Field notes: `id` must equal the directory name; `skin_api_version`'s major +version must match the host's `SKIN_API_VERSION` or the skin is refused at +load; `targets` takes sport families (`sports`), exact sport keys +(`sport_keys`), and/or exact plugin ids (`plugins`) — any match applies. + +## The renderer (`skin.py`) + +```python +from src.skin_system.skin_base import ScoreboardSkin, SkinContext + +class MySkin(ScoreboardSkin): + def render_live(self, ctx: SkinContext, game: dict) -> bool: + score = f"{game.get('away_score', '0')}-{game.get('home_score', '0')}" + fit = ctx.layout.fit_text(score, ctx.layout.bounds) + ctx.draw_fit(fit, ctx.layout.bounds) + return True # True = "I drew it"; False = use the built-in layout +``` + +Implement only the modes you care about — anything else falls back to the +plugin's built-in rendering. Return `False` to decline a specific game (e.g. +a layout that only makes sense while a game is live). + +### The rules (they keep your skin from breaking the display) + +1. **Draw only onto `ctx.canvas`** (via the helpers or `ctx.draw`). Never + reassign `ctx.canvas`, never touch the display or call any update method. +2. **No I/O in render paths.** No network, no file loads per frame — + `render_live` runs every display pass, and a slow render stalls the whole + matrix (the host warns at >150 ms). Use `ctx.load_logo` (cached) and + `cache_key=` for images. +3. **Derive everything from `(ctx, game)`.** Skins must be stateless: the + live/recent/upcoming modes each get their own instance. +4. **Always `.get()` optional keys.** Only the guaranteed keys below are + promised to exist. +5. **Never hardcode pixel positions for the panel.** Use `ctx.width`/ + `ctx.height`, `ctx.layout` regions and `fit_text` — your skin will be run + at sizes you didn't test (64x32, 128x64, vegas cards). +6. **No third-party dependencies.** Stdlib + PIL + what `ctx` provides. + +A skin that raises 3 renders in a row is disabled until the service restarts +(the built-in layout takes over), so a bug is cosmetic — but check your logs. + +## SkinContext reference + +| Member | What it is | +|---|---| +| `ctx.canvas` / `ctx.draw` | Fresh RGB `PIL.Image` at display size + its `ImageDraw` (raw-PIL escape hatch) | +| `ctx.width`, `ctx.height` | Canvas size — the only size truth | +| `ctx.layout` | `LayoutContext` (see [ADAPTIVE_LAYOUT.md](ADAPTIVE_LAYOUT.md)): `bounds`, `fit_text`, `fit_text_proportional`, `fit_image`, `px`, `by_tier` | +| `ctx.draw_fit(fit, box, color, align, valign)` | Draw a `fit_text` result aligned in a `Region` (handles BDF fonts) | +| `ctx.draw_text(text, x, y, color, font)` | Positioned text (handles BDF fonts) | +| `ctx.draw_image(img, box, mode, align, valign, cache_key)` | Fit + paste an image with alpha; no-ops on `None` | +| `ctx.load_logo("home" \| "away")` | Team logo as RGBA, or `None` (always handle `None`). Cached after first use; see note below | +| `ctx.draw_text_outlined(text, (x, y), font, fill, outline_color)` | The classic scorebug outlined text (TTF fonts only) | +| `ctx.fonts` | The host's font dict — keys `score`, `time`, `team`, `status`, `detail`, `rank` | +| `ctx.options` | Your user's `skin_options` from config | +| `ctx.sport`, `ctx.view_model_version`, `ctx.logger` | Context metadata + logger | + +**A note on `ctx.load_logo` vs the no-I/O rule:** `load_logo` is the one +sanctioned exception. It goes through the host's logo cache — after the +first call per team it's a pure in-memory lookup. If a logo file is missing +on disk, the *first* call may download it, exactly like the built-in +renderer does for the same game (a skin is never worse than built-in here). +Always pass a stable `cache_key` when drawing it, never load image files +yourself in a render path, and always handle `None`. + +The default layout idiom — carve regions, then fit text into them: + +```python +from src.adaptive_layout import scoreboard_regions + +regions = scoreboard_regions(ctx.layout.bounds, ctx=ctx.layout) +ctx.draw_image(ctx.load_logo("away"), regions.away_slot, cache_key=f"logo:{game.get('away_abbr')}") +ctx.draw_image(ctx.load_logo("home"), regions.home_slot, cache_key=f"logo:{game.get('home_abbr')}") +fit = ctx.layout.fit_text("3-5", regions.score_area) +ctx.draw_fit(fit, regions.score_area) +``` + +`Region` supports `split_h`/`split_v`/`inset`/`top_band`/`bottom_band`/ +`left_col`/`right_col` for custom carves. Raw `ctx.draw.rectangle/polygon/ +ellipse/...` is always available for custom marks (see the bases diamond in +the example skin). + +## The game view model + +Guaranteed for every sport (view model v1.0 — renaming these breaks skins and +is treated as a breaking change upstream): + +| Key | Notes | +|---|---| +| `id` | Event id (string) | +| `status_text` | Display-ready status, e.g. `"Final"`, `"7:30 PM"`, `"Bot 7th"` | +| `is_live`, `is_final`, `is_upcoming`, `is_halftime` | Booleans | +| `game_date`, `game_time` | Pre-formatted local date/time strings | +| `start_time_utc` | UTC `datetime` | +| `home_abbr`, `away_abbr` | Team abbreviations (can be 2–5 chars — fit, don't assume) | +| `home_id`, `away_id` | Team ids | +| `home_score`, `away_score` | **Strings**, not ints | +| `home_record`, `away_record` | `"58-33"` or `""` (0-0 records are blanked) | +| `home_logo_path`, `away_logo_path` | Prefer `ctx.load_logo` over touching these | + +Sport extras (present for that sport, still `.get()` defensively): + +- **baseball**: `inning` (int), `inning_half` (`"top"`/`"bottom"`), `balls`, + `strikes`, `outs` (ints), `bases_occupied` (`[first, second, third]` + booleans), `series_summary` (str) +- **football**: `period`, `period_text`, `clock`, `home_timeouts`, + `away_timeouts`, `down_distance_text`, `down_distance_text_long`, + `is_redzone`, `possession`, `possession_indicator` (`"home"`/`"away"`), + `scoring_event` +- **basketball**: `period`, `period_text`, `clock` +- **hockey**: `period`, `period_text`, `clock`, `power_play`, `penalties`, + `home_shots`, `away_shots` + +Optional everywhere (only when the user enabled the feature): `odds` (dict), +`series_summary`, rankings-related fields. + +Fixture copies of these dicts live in `src/skin_system/fixtures/` — that's +exactly what the validator feeds your skin. + +## Vegas mode + +You get vegas support for free: vegas captures the normal display output, +which is already your skin's rendering. Optionally implement +`render_vegas_card(ctx, game)` to return a purpose-built card at +`ctx.width x ctx.height` (sizes vary — never assume 128x32). + +## Building a skin with Claude Code + +Skins are ideal Claude Code projects: small, isolated, and verifiable with +one command. Paste this to start: + +> You are building a **display skin** for LEDMatrix — a visual overlay for a +> sports scoreboard on a small LED matrix (commonly 128x32 or 64x32 pixels). +> First read `docs/CREATING_SKINS.md` and the reference skin in +> `skins/example-classic-baseball/`. +> +> Rules: +> - Create/modify files ONLY under `skins//`. Do NOT modify +> anything in `src/`, `scripts/`, the plugins, or any other skin. +> - Render only from the `game` dict and `ctx` helpers. No network calls, no +> per-frame file I/O, no new pip dependencies, no touching the display — +> draw onto `ctx.canvas` and return True. +> - Use `ctx.layout` regions and `fit_text` for positioning so the skin works +> at any panel size; use `.get()` for every optional game key. +> - After every change run +> `python scripts/validate_skin.py --skin ` and LOOK at the +> PNGs it writes to `skin_renders/` (the `_x4.png` files are easiest to +> read). Iterate until it passes and looks right at both 128x32 and 64x32. +> +> What I want it to look like: status go; colors; what shows during live vs upcoming vs final> + +Tips that keep Claude (and you) out of trouble: + +- One mode at a time: get `render_live` right before touching the others — + unimplemented modes automatically use the built-in look. +- Ask for edge-case renders: long team abbreviations, missing logos + (`ctx.load_logo` returning `None`), 0-0 records, extra innings/OT. +- If the render looks cramped at 64x32, ask Claude to use + `ctx.layout.by_tier(...)` to drop elements on small panels rather than + shrinking everything. +- Never let it "fix" a problem by editing `src/` — if the skin can't do + something within its directory, that's a feature request, not a workaround. + +## Pre-publish checklist + +- [ ] `python scripts/validate_skin.py --skin --size 128x32 --size 64x32 --size 128x64` passes +- [ ] Looked at every PNG in `skin_renders/` — nothing clipped or overlapping +- [ ] Handles a missing logo (`None`) without crashing — temporarily point a + fixture's logo path at a nonexistent file to test +- [ ] Long abbreviations (`"TA&M"`, 4–5 chars) don't overflow +- [ ] No render warning above the time budget +- [ ] `skin.json`: `id` matches the directory, `version` set, + `skin_api_version` matches the host, targets correct +- [ ] `preview.png` added (grab your favorite `_x4` render) +- [ ] Tested on real hardware if you have it — a Pi is much slower than your + dev machine + +Distribute by publishing the directory as a git repo (users +`git clone skins/`), or submit it to the plugin registry as an +entry with `"type": "skin"` (see [SKIN_SYSTEM.md](SKIN_SYSTEM.md) §Distribution). + +**Trust note:** a skin is Python running inside the display service — the +same trust level as a plugin. Review code before installing skins from +others. diff --git a/docs/DEVELOPER_QUICK_REFERENCE.md b/docs/DEVELOPER_QUICK_REFERENCE.md index 93ca8e16..7eefb0c9 100644 --- a/docs/DEVELOPER_QUICK_REFERENCE.md +++ b/docs/DEVELOPER_QUICK_REFERENCE.md @@ -31,7 +31,7 @@ POST /api/v3/system/action **Base URL**: `http://your-pi-ip:5000/api/v3` -See [API_REFERENCE.md](API_REFERENCE.md) for complete documentation. +See [REST_API_REFERENCE.md](REST_API_REFERENCE.md) for complete documentation. ## Display Manager Quick Methods @@ -190,12 +190,13 @@ def display(self, force_clear=False): ``` LEDMatrix/ -├── plugins/ # Installed plugins +├── plugin-repos/ # Installed plugins (default; plugins/ is only +│ # for dev symlinks via scripts/dev/dev_plugin_setup.sh) ├── config/ │ ├── config.json # Main configuration │ └── config_secrets.json # API keys and secrets ├── docs/ # Documentation -│ ├── API_REFERENCE.md +│ ├── REST_API_REFERENCE.md │ ├── PLUGIN_API_REFERENCE.md │ └── ... └── src/ @@ -207,7 +208,7 @@ LEDMatrix/ ## Quick Links -- [Complete API Reference](API_REFERENCE.md) +- [Complete REST API Reference](REST_API_REFERENCE.md) - [Plugin API Reference](PLUGIN_API_REFERENCE.md) - [Plugin Development Guide](PLUGIN_DEVELOPMENT_GUIDE.md) - [Advanced Patterns](ADVANCED_PLUGIN_DEVELOPMENT.md) diff --git a/docs/EMULATOR_SETUP_GUIDE.md b/docs/EMULATOR_SETUP_GUIDE.md index e30c3063..d5130ccf 100644 --- a/docs/EMULATOR_SETUP_GUIDE.md +++ b/docs/EMULATOR_SETUP_GUIDE.md @@ -69,23 +69,24 @@ default configuration as it ships in the repo: ```json { "pixel_outline": 0, - "pixel_size": 5, + "pixel_size": 16, "pixel_style": "square", "pixel_glow": 6, - "display_adapter": "pygame", + "display_adapter": "browser", + "allow_adapter_fallback": true, "icon_path": null, "emulator_title": null, "suppress_font_warnings": false, - "suppress_adapter_load_errors": false, "browser": { "_comment": "For use with the browser adapter only.", "port": 8888, - "target_fps": 24, + "target_fps": 60, "fps_display": false, "quality": 70, "image_border": true, "debug_text": false, - "image_format": "JPEG" + "image_format": "JPEG", + "open_immediately": false }, "log_level": "info" } @@ -96,13 +97,13 @@ default configuration as it ships in the repo: | Option | Description | Default | Values | |--------|-------------|---------|--------| | `pixel_outline` | Pixel border thickness | 0 | 0-5 | -| `pixel_size` | Size of each pixel | 5 | 1-64 (8–16 is typical for testing) | +| `pixel_size` | Size of each pixel | 16 | 1-64 (8–16 is typical for testing) | | `pixel_style` | Pixel shape | "square" | "square", "circle" | | `pixel_glow` | Glow effect intensity | 6 | 0-20 | -| `display_adapter` | Display backend | "pygame" | "pygame", "browser" | +| `display_adapter` | Display backend | "browser" | "browser", "pygame" | +| `allow_adapter_fallback` | Fall back to another adapter if the configured one fails to load | true | true/false | | `emulator_title` | Window title | null | Any string | | `suppress_font_warnings` | Hide font warnings | false | true/false | -| `suppress_adapter_load_errors` | Hide adapter errors | false | true/false | ### 3. Browser Adapter Configuration @@ -111,18 +112,32 @@ When using the browser adapter, additional options are available: | Option | Description | Default | |--------|-------------|---------| | `port` | Web server port | 8888 | -| `target_fps` | Target frames per second | 24 | +| `target_fps` | Target frames per second | 60 | | `fps_display` | Show FPS counter | false | | `quality` | Image compression quality | 70 | | `image_border` | Show image border | true | | `debug_text` | Show debug information | false | | `image_format` | Image format | "JPEG" | +| `open_immediately` | Open the browser page automatically on start | false | ## Running the Emulator -### 1. Set Environment Variable +### 1. Use the `-e` Flag (Recommended) -Enable emulator mode by setting the `EMULATOR` environment variable: +`run.py` accepts exactly two flags: `-e`/`--emulator` and +`-d`/`--debug`. + +```bash +python3 run.py -e + +# With verbose logging +python3 run.py -e -d +``` + +### 2. Alternative: Set the Environment Variable + +You can also enable emulator mode via the `EMULATOR` environment +variable: **Windows (Command Prompt):** ```cmd @@ -137,15 +152,6 @@ python run.py ``` **Linux/macOS:** -```bash -export EMULATOR=true -python3 run.py -``` - -### 2. Alternative: Direct Python Execution - -You can also run the emulator directly: - ```bash EMULATOR=true python3 run.py ``` @@ -153,7 +159,8 @@ EMULATOR=true python3 run.py ### 3. Verify Emulator Mode When running in emulator mode, you should see: -- A window displaying the LED matrix simulation +- The emulated matrix — a web page at `http://localhost:8888` with the + default browser adapter, or a desktop window with the pygame adapter - Console output indicating emulator mode - No hardware initialization errors @@ -161,7 +168,36 @@ When running in emulator mode, you should see: LEDMatrix supports two display adapters for the emulator: -### 1. Pygame Adapter (Default) +### 1. Browser Adapter (Default) + +The browser adapter runs a web server and displays the matrix as a web +page at `http://localhost:8888`. This is the adapter the shipped +`emulator_config.json` uses. + +**Features:** +- Web-based interface +- Remote access capability +- Mobile-friendly +- Screenshot capture + +**Configuration:** +```json +{ + "display_adapter": "browser", + "browser": { + "port": 8888, + "target_fps": 60, + "quality": 70 + } +} +``` + +**Usage:** +1. Start the emulator (`python3 run.py -e`) +2. Open browser to `http://localhost:8888` +3. View the LED matrix display + +### 2. Pygame Adapter (Alternative) The pygame adapter provides a native desktop window with real-time display. @@ -186,33 +222,6 @@ The pygame adapter provides a native desktop window with real-time display. - `+/-` - Zoom in/out - `R` - Reset zoom -### 2. Browser Adapter - -The browser adapter runs a web server and displays the matrix in a web browser. - -**Features:** -- Web-based interface -- Remote access capability -- Mobile-friendly -- Screenshot capture - -**Configuration:** -```json -{ - "display_adapter": "browser", - "browser": { - "port": 8888, - "target_fps": 24, - "quality": 70 - } -} -``` - -**Usage:** -1. Start the emulator with browser adapter -2. Open browser to `http://localhost:8888` -3. View the LED matrix display - ## Troubleshooting ### Common Issues @@ -274,8 +283,7 @@ Enable debug logging: ```json { "log_level": "debug", - "suppress_font_warnings": false, - "suppress_adapter_load_errors": false + "suppress_font_warnings": false } ``` @@ -299,17 +307,18 @@ Modify the display dimensions in your main config: ### 2. Plugin Development -For plugin development with the emulator: +`run.py` always runs the full rotation — it has no single-plugin flag. +To preview or check one plugin in isolation, use the dev tools: ```bash -# Enable emulator mode -export EMULATOR=true +# Run the full display in emulator mode (optionally with debug logging) +python3 run.py -e -d -# Run with specific plugin -python run.py --plugin my-plugin +# Live single-plugin preview in the browser (port 5001) +python3 scripts/dev_server.py -# Debug mode -python run.py --debug +# Headless render/validation of one plugin +python3 scripts/check_plugin.py --plugin my-plugin ``` ### 3. Performance Tuning @@ -344,11 +353,10 @@ The emulator can work alongside the web interface: ```bash # Terminal 1: Start emulator -export EMULATOR=true -python run.py +python3 run.py -e -# Terminal 2: Start web interface -python web_interface/app.py +# Terminal 2: Start web interface (supported entry point) +python3 web_interface/start.py ``` Access the web interface at `http://localhost:5000` while the emulator runs. @@ -365,13 +373,14 @@ Access the web interface at `http://localhost:5000` while the emulator runs. ### 2. Plugin Testing ```bash -# Test specific plugin -export EMULATOR=true -python run.py --plugin clock-simple +# Test a specific plugin (headless check) +python3 scripts/check_plugin.py --plugin clock-simple -# Test all plugins -export EMULATOR=true -python run.py --test-plugins +# Preview a single plugin live in the browser (port 5001) +python3 scripts/dev_server.py + +# Test the full rotation in the emulator +python3 run.py -e ``` ### 3. Configuration Management @@ -385,9 +394,8 @@ python run.py --test-plugins ### Basic Clock Display ```bash -# Start emulator with clock -export EMULATOR=true -python run.py +# Start emulator with clock enabled in config.json +python3 run.py -e ``` ### Sports Scores @@ -395,16 +403,16 @@ python run.py ```bash # Configure for sports display # Edit config/config.json to enable sports plugins -export EMULATOR=true -python run.py +python3 run.py -e ``` ### Custom Text Display ```bash -# Use text display plugin -export EMULATOR=true -python run.py --plugin text-display --text "Hello World" +# Preview the text display plugin on its own +python3 scripts/check_plugin.py --plugin text-display +# or use the live dev preview server +python3 scripts/dev_server.py ``` ## Support diff --git a/docs/GETTING_STARTED.md b/docs/GETTING_STARTED.md index de6e15b6..d0102422 100644 --- a/docs/GETTING_STARTED.md +++ b/docs/GETTING_STARTED.md @@ -21,18 +21,30 @@ This guide will help you set up your LEDMatrix display for the first time and ge --- -## Quick Start (5 Minutes) +## Quick Start -### 1. First Boot +### 1. Install LEDMatrix -1. Insert the MicroSD card with LEDMatrix installed -2. Connect the LED matrix to your Raspberry Pi -3. Plug in the power supply -4. Wait for the Pi to boot (about 60 seconds) +There is no prebuilt SD card image — you install LEDMatrix onto stock +Raspberry Pi OS Lite yourself: -**Expected Behavior:** +1. Flash Raspberry Pi OS Lite to the MicroSD card (Raspberry Pi Imager) +2. Connect the LED matrix to your Raspberry Pi, insert the card, and + power on +3. SSH into the Pi and run the one-shot installer: + ```bash + curl -fsSL https://raw.githubusercontent.com/ChuckBuilds/LEDMatrix/main/scripts/install/one-shot-install.sh | bash + ``` + or clone the repo and run `sudo ./first_time_install.sh` — see the + [README Installation Steps / Quick Install](../README.md#installation-steps) + for full details + +**Expected Behavior after install:** - LED matrix will light up -- Display will show default plugins (clock, weather, etc.) +- A fresh install ships only the bundled `starlark-apps` and + `web-ui-info` plugins — clock, weather, sports, etc. must be + installed from the Plugin Store (web UI → Plugin Manager) before + anything else displays - Pi creates WiFi network "LEDMatrix-Setup" if not connected ### 2. Connect to WiFi @@ -73,7 +85,7 @@ You should see: 2. Set your matrix configuration: - **Rows**: 32 or 64 (match your hardware) - **Columns**: commonly 64 or 96; the web UI accepts any integer - in the 16–128 range, but 64 and 96 are the values the bundled + in the 1–128 range, but 64 and 96 are the values the bundled panel hardware ships with - **Chain Length**: Number of panels chained horizontally - **Hardware Mapping**: usually `adafruit-hat-pwm` (with the PWM jumper @@ -115,11 +127,16 @@ You can also install community plugins straight from a GitHub URL using the 1. Each installed plugin gets its own tab in the second navigation row 2. Open that plugin's tab to edit its settings (favorite teams, API keys, - update intervals, display duration, etc.) + update intervals, etc.) 3. Click **Save** 4. Restart the display service from **Overview** so the new settings take effect +**Note:** how long each plugin stays on screen is not set in the +plugin's own tab — use the **Rotation** tab's **Screen Durations** +section instead (saved to `display.display_durations` in +`config.json`). + **Example: Weather Plugin** - Set your location (city, state, country) - Add an API key from OpenWeatherMap (free signup) to @@ -208,12 +225,14 @@ The fastest way to verify a plugin works without waiting for the rotation: ### Customize Your Display **Adjust display durations:** -- Each plugin's tab has a **Display Duration (seconds)** field — set how - long that plugin stays on screen each rotation. +- Open the **Rotation** tab and use the **Screen Durations** section to + set how long each plugin stays on screen per rotation (saved to + `display.display_durations`). **Organize plugin order:** -- Use the **Plugin Manager** tab to enable/disable plugins. The display - cycles through enabled plugins in the order they appear. +- The **Rotation** tab also has a drag-and-drop **Rotation Order** list + (saved to `display.plugin_rotation_order`). Enable/disable plugins + from the **Plugin Manager** tab. **Add more plugins:** - Check the **Plugin Store** section of **Plugin Manager** for new plugins. @@ -280,10 +299,14 @@ sudo journalctl -u ledmatrix-web -f │ ├── config_secrets.json # API keys and secrets │ └── wifi_config.json # WiFi settings ├── plugin-repos/ # Installed plugins (default location) -├── cache/ # Cached data └── web_interface/ # Web interface files ``` +> Cached data does not live in the project directory — the cache manager +> uses the first writable location among `/var/cache/ledmatrix`, +> `~/.ledmatrix_cache`, `/opt/ledmatrix/cache`, and +> `$TMPDIR/ledmatrix_cache`. +> > The plugin install location is configurable via > `plugin_system.plugins_directory` in `config.json`. The default is > `plugin-repos/`. Plugin discovery (`PluginManager.discover_plugins()`) @@ -303,11 +326,14 @@ System tabs: - WiFi Network selection and AP-mode setup - Schedule Power and dim schedules - Display Matrix hardware configuration +- Rotation Rotation order (drag-and-drop) and screen durations - Config Editor Raw config.json editor +- Backup & Restore Config backup and restore - Fonts Upload and manage fonts - Logs Real-time log viewing - Cache Cached data inspection and cleanup - Operation History Recent service operations +- Tools System diagnostics, updates, dependencies, maintenance Plugin tabs (second row): - Plugin Manager Browse the Plugin Store, install/enable plugins diff --git a/docs/HOW_TO_RUN_TESTS.md b/docs/HOW_TO_RUN_TESTS.md index 828f52cc..db1cef12 100644 --- a/docs/HOW_TO_RUN_TESTS.md +++ b/docs/HOW_TO_RUN_TESTS.md @@ -10,10 +10,7 @@ Make sure you have the testing packages installed: ```bash # Install all dependencies including test packages -pip install -r requirements.txt - -# Or install just the test dependencies -pip install pytest pytest-cov pytest-mock +pip install -r requirements.txt -r requirements-test.txt ``` ### 2. Set Environment Variables @@ -248,13 +245,11 @@ test/ ├── test_config_service.py # Config service tests ├── test_config_validation_edge_cases.py # Config edge cases ├── test_font_manager.py # Font manager tests -├── test_layout_manager.py # Layout manager tests ├── test_text_helper.py # Text helper tests ├── test_error_handling.py # Error handling tests ├── test_error_aggregator.py # Error aggregation tests ├── test_schema_manager.py # Schema manager tests ├── test_web_api.py # Web API tests -├── test_nba_*.py # NBA-specific test suites ├── plugins/ # Per-plugin test suites │ ├── test_clock_simple.py │ ├── test_calendar.py @@ -304,7 +299,7 @@ If tests fail due to missing packages: ```bash # Install all dependencies -pip install -r requirements.txt +pip install -r requirements.txt -r requirements-test.txt # Or install specific missing package pip install @@ -336,15 +331,15 @@ pytest --cov=src --cov-report=html ## Continuous Integration -The repo runs -[`.github/workflows/security-audit.yml`](../.github/workflows/security-audit.yml) -(bandit + semgrep) on every push. A pytest CI workflow at -`.github/workflows/tests.yml` is queued to land alongside this -PR ([ChuckBuilds/LEDMatrix#307](https://github.com/ChuckBuilds/LEDMatrix/pull/307)); -the workflow file itself was held back from that PR because the -push token lacked the GitHub `workflow` scope, so it needs to be -committed separately by a maintainer. Once it's in, this section -will be updated to describe what the job runs. +The repo runs the pytest suite via +[`.github/workflows/test.yml`](../.github/workflows/test.yml) on every +push and pull request: a plugin-safety job (harness, visual rendering +and plugin-matrix tests) plus a unit-test job that runs an explicit +allowlist of suites — new test files must be added to that list to run +in CI. Release version consistency is checked by +[`.github/workflows/release-version-check.yml`](../.github/workflows/release-version-check.yml). +Bandit, flake8, mypy and gitleaks run as pre-commit hooks (see +`.pre-commit-config.yaml`), not in CI. ## Best Practices diff --git a/docs/LOW_MEMORY_BOARDS.md b/docs/LOW_MEMORY_BOARDS.md new file mode 100644 index 00000000..186e8a4a --- /dev/null +++ b/docs/LOW_MEMORY_BOARDS.md @@ -0,0 +1,115 @@ +# Running on Low-Memory Boards + +Applies to the Pi Zero 2 W (512 MB), Pi 3 / 3B+ (1 GB), and the 1 GB Pi 4. +If your board has 2 GB or more you can skip this document. + +## The failure this prevents + +The display process is the largest thing on the board. On a 1 GB Pi 3B+ with +around 20 plugins enabled it settles near **600 MB of 905 MB usable**, leaving +under 200 MB of headroom for everything else. + +When that headroom runs out, the board does not crash cleanly. `fork()` starts +failing, and because a new process is needed to do almost anything, the +symptoms look nothing like "out of memory": + +| What you see | Why | +|---|---| +| SSH accepts the connection then closes it instantly, before any banner | `sshd` forks a session per connection; the fork fails | +| The web UI still responds quickly | Already running, serves from existing threads, forks nothing | +| Ping is perfect, 0% loss | Handled entirely in the kernel | +| The panel is dark | The display process was killed and cannot be respawned | +| The clock is wrong after the next boot | `fake-hwclock`'s periodic save is a scheduled job, and it cannot fork either | + +The board looks healthy from the outside and cannot be logged into. Only a +power cycle clears it. If you are here because SSH stopped working, also see +[SSH_UNAVAILABLE_AFTER_INSTALL.md](SSH_UNAVAILABLE_AFTER_INSTALL.md), which +covers the more common cause (AP mode). + +## Check your headroom + +```bash +free -m +ps -eo rss,comm --sort=-rss | head -5 +``` + +If `MemAvailable` is under ~150 MB while the display is running, you are close +to the edge. To watch it over time: + +```bash +watch -n 30 'free -m | head -2' +``` + +Available memory that falls steadily rather than holding flat means you will +reach the wall; it is a question of when. + +## What to do + +**1. Enable the memory cgroup controller.** Without it, the `MemoryMax=85%` in +`systemd/ledmatrix.service` is accepted by systemd and silently ignored, so the +service has no ceiling and a runaway takes the whole board down instead of just +restarting. Raspberry Pi firmware disables this controller by default. + +`first_time_install.sh` does this for you. To check it took effect: + +```bash +grep memory /sys/fs/cgroup/cgroup.controllers +``` + +If that prints nothing, add `cgroup_enable=memory cgroup_memory=1` to the +kernel command line and reboot. Edit whichever file your image uses — +`/boot/firmware/cmdline.txt` on current Raspberry Pi OS, `/boot/cmdline.txt` on +older layouts (the installer checks the first and falls back to the second). +Everything must stay on a single line. + +This changes the failure mode from "the board becomes unreachable" to "the +display service restarts". It is a safety net, not a fix. + +**2. Run fewer plugins.** This is the actual remedy. Every enabled plugin costs +memory permanently — its module, its parsed config, and its cached API +responses. On a 512 MB or 1 GB board, keep the enabled set small and prefer +plugins that poll infrequently. + +**3. Lower the cache ceiling.** The in-memory cache is sized from total RAM +(150 entries at 1 GB and below, up to 1500 at 8 GB). To go lower still: + +```ini +# /etc/systemd/system/ledmatrix.service.d/override.conf +[Service] +Environment=LEDMATRIX_CACHE_MAX_ENTRIES=75 +``` + +Writing the file does not change the running service. Reload systemd and +restart it: + +```bash +sudo systemctl daemon-reload +sudo systemctl restart ledmatrix +``` + +Fewer entries means more API calls, so lower this only while you are actually +short of memory. + +**4. Consider `MemoryHigh`.** `MemoryMax` kills and restarts. `MemoryHigh` +throttles and reclaims instead, which is gentler — but on a board where the +process genuinely wants more than the limit, sustained reclaim can stall the +render loop and show as visible stutter on the panel. Add it only if you prefer +degraded output to a restart: + +```ini +[Service] +MemoryHigh=70% +``` + +## Keep your logs + +These images default to volatile journald storage, so every reboot destroys the +logs — including the ones explaining why the board rebooted. `first_time_install.sh` +enables persistent storage capped at 64 MB. To confirm: + +```bash +journalctl --list-boots +``` + +More than one boot listed means logs are surviving reboots. If only one is +listed, journald is still writing to `/run` (tmpfs). diff --git a/docs/MIGRATION_GUIDE.md b/docs/MIGRATION_GUIDE.md index 422cd064..b64269b9 100644 --- a/docs/MIGRATION_GUIDE.md +++ b/docs/MIGRATION_GUIDE.md @@ -86,7 +86,7 @@ The plugin system has been enhanced but remains backward compatible with existin If you encounter issues during migration: -1. Check the [README.md](README.md) for current installation and usage instructions +1. Check the [project root README](../README.md) for current installation and usage instructions 2. Review script README files: - [`scripts/install/README.md`](../scripts/install/README.md) - Installation scripts documentation - [`scripts/fix_perms/README.md`](../scripts/fix_perms/README.md) - Permission scripts documentation diff --git a/docs/PLUGIN_API_REFERENCE.md b/docs/PLUGIN_API_REFERENCE.md index 751d5609..4045c50b 100644 --- a/docs/PLUGIN_API_REFERENCE.md +++ b/docs/PLUGIN_API_REFERENCE.md @@ -170,6 +170,47 @@ Default returns `False`. List of display modes to show during a live takeover. Default returns the plugin's `display_modes` from its manifest. +#### `get_vegas_priority_weight() -> Optional[int]` + +How many slots per Vegas cycle this plugin should get. Default returns +`None`, which defers to the core. + +The Vegas ticker is otherwise a strict round robin — every plugin appears +exactly once per cycle — so with a dozen plugins enabled a live score can be +minutes stale by the time it comes round. A weight of *N* gives the plugin +*N* slots per cycle, spread evenly through it rather than clumped. + +**You usually do not need this.** When the hook returns `None`, the core +already gives a plugin `vegas_scroll.live_weight` whenever +`has_live_priority()` and `has_live_content()` are both true. Live sports get +extra turns with no code at all. + +Implement it only when the plugin knows something the core cannot. The +motivating case is favorite teams — the core can see *that* a game is live, +but not *whose*: + +```python +def get_vegas_priority_weight(self): + if not (self.has_live_priority() and self.has_live_content()): + return None # let the core decide + vegas = self.global_config.get('display', {}).get('vegas_scroll', {}) + if self._favorite_is_live(): + return vegas.get('favorite_live_weight', 5) + return vegas.get('live_weight', 3) +``` + +The weight is per *plugin*, not per game: a scoreboard showing four live games +still occupies one slot at a time and rotates its own games within it. Values +are clamped to 1–10 by the caller. An exception here is caught and logged, and +the core then falls back to its own live-content check — so a plugin whose +weight calculation is broken still gets `live_weight` for a game that really +is live, rather than being demoted to 1. + +Only consulted when the user has set `vegas_scroll.live_in_ticker`. With the +default (`false`) live content preempts Vegas entirely and there is no ticker +to be weighted within. See +[ADVANCED_FEATURES.md](ADVANCED_FEATURES.md#live-content-in-the-ticker). + ### Vegas scroll hooks Vegas mode shows multiple plugins as a single continuous scroll instead of @@ -201,8 +242,9 @@ the mode selector for this plugin. #### `get_vegas_segment_width() -> Optional[int]` -For `FIXED_SEGMENT` plugins, the width in pixels of the segment they -occupy in the scroll. `None` lets the controller pick a default. +For `FIXED_SEGMENT` plugins, the number of *panels* the segment +occupies in the scroll (pixel width = panels × `single_panel_width`, +from `display.hardware.cols`). `None` uses the default of 1 panel. > The full source for `BasePlugin` lives in > `src/plugin_system/base_plugin.py`. If a method here disagrees with the diff --git a/docs/PLUGIN_ARCHITECTURE_SPEC.md b/docs/PLUGIN_ARCHITECTURE_SPEC.md index fbd45b34..00a2914d 100644 --- a/docs/PLUGIN_ARCHITECTURE_SPEC.md +++ b/docs/PLUGIN_ARCHITECTURE_SPEC.md @@ -8,9 +8,12 @@ > - Code paths reference `web_interface_v2.py`; the current web UI is > `web_interface/app.py` with v3 Blueprint-based templates. > - The example Flask routes use `/api/plugins/*`; the real API -> blueprint is mounted at `/api/v3` (`web_interface/app.py:144`). +> blueprint is mounted at `/api/v3` (`web_interface/app.py:199`). > - The default plugin location is `plugin-repos/` (configurable via > `plugin_system.plugins_directory`), not `./plugins/`. +> - Example imports use `src/plugin_system/base_classes/*_plugin.py`; +> the shipped base classes live in `src/base_classes/` (e.g. +> `src.base_classes.sports.SportsCore`, `src.base_classes.hockey.Hockey`). > - The "Migration Strategy" and "Implementation Roadmap" sections > describe work that has now shipped. > diff --git a/docs/PLUGIN_CONFIG_ARCHITECTURE.md b/docs/PLUGIN_CONFIG_ARCHITECTURE.md index 28fa1ba3..823bb4db 100644 --- a/docs/PLUGIN_CONFIG_ARCHITECTURE.md +++ b/docs/PLUGIN_CONFIG_ARCHITECTURE.md @@ -1,5 +1,11 @@ # Plugin Configuration Tabs - Architecture +> This page covers internals (how the config system works under the +> hood). For designing a plugin's config schema, the canonical guide is +> [PLUGIN_CONFIGURATION_GUIDE.md](PLUGIN_CONFIGURATION_GUIDE.md); for +> the user-facing tabs feature, see +> [PLUGIN_CONFIGURATION_TABS.md](PLUGIN_CONFIGURATION_TABS.md). + ## System Architecture ### Component Overview diff --git a/docs/PLUGIN_CUSTOM_ICONS.md b/docs/PLUGIN_CUSTOM_ICONS.md index da9db63c..79cabc5b 100644 --- a/docs/PLUGIN_CUSTOM_ICONS.md +++ b/docs/PLUGIN_CUSTOM_ICONS.md @@ -296,7 +296,7 @@ Want to change icons programmatically? While not officially supported, you could ## Related Documentation - [Plugin Configuration Tabs](PLUGIN_CONFIGURATION_TABS.md) - Main plugin tabs documentation -- [Plugin Development Guide](plugin_docs/) - How to create plugins +- [Plugin Development Guide](PLUGIN_DEVELOPMENT_GUIDE.md) - How to create plugins - [Font Awesome Icons](https://fontawesome.com/icons) - Browse all available icons - [Emoji Reference](https://unicode.org/emoji/charts/full-emoji-list.html) - All emoji options diff --git a/docs/PLUGIN_DEPENDENCY_TROUBLESHOOTING.md b/docs/PLUGIN_DEPENDENCY_TROUBLESHOOTING.md index 755760c9..1f33cb88 100644 --- a/docs/PLUGIN_DEPENDENCY_TROUBLESHOOTING.md +++ b/docs/PLUGIN_DEPENDENCY_TROUBLESHOOTING.md @@ -169,6 +169,6 @@ If you continue to experience issues: ## Related Documentation - [Plugin Dependency Guide](PLUGIN_DEPENDENCY_GUIDE.md) -- [Plugin Development Guide](docs/plugin_development.md) -- [Troubleshooting Quick Start](TROUBLESHOOTING_QUICK_START.md) +- [Plugin Development Guide](PLUGIN_DEVELOPMENT_GUIDE.md) +- [Troubleshooting](TROUBLESHOOTING.md) diff --git a/docs/PLUGIN_DEVELOPMENT_GUIDE.md b/docs/PLUGIN_DEVELOPMENT_GUIDE.md index 01ef6029..e2a0c240 100644 --- a/docs/PLUGIN_DEVELOPMENT_GUIDE.md +++ b/docs/PLUGIN_DEVELOPMENT_GUIDE.md @@ -10,6 +10,12 @@ This guide explains how to set up a development workflow for plugins that are ma > scale. Existing plugins keep their classic rendering unless they adopt > those APIs; nothing migrates automatically. +> **Just want a different look for an existing sports scoreboard?** You may +> not need a plugin at all — a **skin** restyles the live/recent/upcoming +> rendering while the plugin keeps handling data, scheduling, caching, and +> vegas mode, in ~100 lines of drawing code. See +> [CREATING_SKINS.md](CREATING_SKINS.md). + ## Overview When developing plugins in separate repositories, you need a way to: @@ -583,11 +589,24 @@ Your plugin must: ### Versioning Best Practices - **Use semantic versioning**: `MAJOR.MINOR.PATCH` (e.g., `1.2.3`) -- **Automatic version bumping**: Use the pre-push git hook for automatic patch version bumps -- **Manual versioning**: Only needed for major/minor bumps or special cases -- **GitHub as source of truth**: Plugin store fetches versions from GitHub releases/tags/manifest +- **GitHub as source of truth**: the plugin store resolves versions in this + order: GitHub Releases → GitHub Tags → manifest from branch → git commit hash +- **Automatic version bumping**: install the self-contained pre-push hook in + your plugin repo and patch versions bump themselves on push (a git tag + `v{version}` is created and `manifest.json` staged automatically): -See the [Git Workflow rules](../.cursorrules) for version management details. + ```bash + # From your plugin repository directory + cp /path/to/LEDMatrix/scripts/git-hooks/pre-push-plugin-version .git/hooks/pre-push + chmod +x .git/hooks/pre-push + ``` + + Set `SKIP_TAG=1` in the environment to skip auto-tagging for one push. +- **Manual versioning**: only needed for major/minor bumps, CI pipelines that + bypass hooks, or forks without the hook — use + `scripts/bump_plugin_version.py`. +- **Registry stores no versions**: `plugins.json` holds only metadata (name, + description, repo URL). ### Submitting to Official Registry @@ -661,5 +680,5 @@ For your plugin to work well in the plugin store: - [Advanced Plugin Development](ADVANCED_PLUGIN_DEVELOPMENT.md) - Advanced patterns and examples - [Plugin Quick Reference](PLUGIN_QUICK_REFERENCE.md) - Quick development reference - [Plugin Configuration Guide](PLUGIN_CONFIGURATION_GUIDE.md) - Configuration setup -- [Plugin Store User Guide](PLUGIN_STORE_USER_GUIDE.md) - Using the plugin store +- [Plugin Store Guide](PLUGIN_STORE_GUIDE.md) - Using the plugin store diff --git a/docs/PLUGIN_QUICK_REFERENCE.md b/docs/PLUGIN_QUICK_REFERENCE.md index 7f8c0bf0..850fbb22 100644 --- a/docs/PLUGIN_QUICK_REFERENCE.md +++ b/docs/PLUGIN_QUICK_REFERENCE.md @@ -14,8 +14,10 @@ and [PLUGIN_DEVELOPMENT_GUIDE.md](PLUGIN_DEVELOPMENT_GUIDE.md). ✅ **GitHub Store**: Discovery from `ledmatrix-plugins` registry plus any GitHub URL ✅ **Plugin Location**: configured by `plugin_system.plugins_directory` - in `config.json` (default `plugin-repos/`; the loader also searches - `plugins/` as a fallback) + in `config.json` (default `plugin-repos/`). Plugin discovery scans + only this directory — there is no loader fallback to `plugins/` + (only Plugin Store operations and schema lookup additionally probe + `plugins/`) ## File Structure @@ -109,7 +111,7 @@ git push -u origin main git tag v1.0.0 git push origin v1.0.0 -# Submit to registry (PR to ChuckBuilds/ledmatrix-plugin-registry) +# Submit to registry (PR to ChuckBuilds/ledmatrix-plugins) ``` ## Using Plugins @@ -120,12 +122,12 @@ git push origin v1.0.0 2. **Install**: Click **Install** in the plugin's row 3. **Configure**: open the plugin's tab in the second nav row 4. **Enable/Disable**: toggle switch in the **Installed Plugins** list -5. **Reorder**: order is set by the position in `display_modes` / - plugin order; rearranging via drag-and-drop is not yet supported +5. **Reorder**: use the drag-and-drop **Rotation Order** list in the + **Rotation** tab (saved to `display.plugin_rotation_order`) ### REST API -The API is mounted at `/api/v3` (`web_interface/app.py:144`). +The API is mounted at `/api/v3` (`web_interface/app.py:199`). ```bash # Install plugin from the registry diff --git a/docs/PLUGIN_REGISTRY_SETUP_GUIDE.md b/docs/PLUGIN_REGISTRY_SETUP_GUIDE.md index 7c1569b8..d1426979 100644 --- a/docs/PLUGIN_REGISTRY_SETUP_GUIDE.md +++ b/docs/PLUGIN_REGISTRY_SETUP_GUIDE.md @@ -323,16 +323,22 @@ curl -X POST http://pi:5000/api/v3/plugins/install-from-url \ ### Regular Updates ```bash -# Update stars/downloads counts -python3 scripts/update_stats.py +# Refresh local clones of all plugin repos +python3 scripts/update_plugin_repos.py -# Validate all plugin entries -python3 scripts/validate_registry.py +# (Re-)create local plugin repo checkouts from the registry +python3 scripts/setup_plugin_repos.py -# Check for plugin updates -python3 scripts/check_updates.py +# Audit installed plugins for manifest/schema problems +python3 scripts/audit_plugins.py + +# Validate a single plugin +python3 scripts/check_plugin.py --plugin ``` +Registry regeneration (`update_registry.py`) lives in the +`ledmatrix-plugins` monorepo, not in this repo. + ## Converting Existing Plugins To convert your existing plugins (hello-world, clock-simple) to this system: @@ -400,7 +406,7 @@ print(f'Found {len(registry[\"plugins\"])} plugins') ## References -- Plugin Store Implementation: See `PLUGIN_STORE_IMPLEMENTATION_SUMMARY.md` -- User Guide: See `PLUGIN_STORE_USER_GUIDE.md` +- Plugin Store Implementation: See `PLUGIN_IMPLEMENTATION_SUMMARY.md` +- User Guide: See `PLUGIN_STORE_GUIDE.md` - Architecture: See `PLUGIN_ARCHITECTURE_SPEC.md` diff --git a/docs/PLUGIN_STORE_GUIDE.md b/docs/PLUGIN_STORE_GUIDE.md index 4482d5b9..421c99de 100644 --- a/docs/PLUGIN_STORE_GUIDE.md +++ b/docs/PLUGIN_STORE_GUIDE.md @@ -481,13 +481,13 @@ A: Yes, if a plugin needs API keys, it can access them like core managers do. A: Most plugins are small (1-5MB). Check individual plugin documentation for specific requirements. **Q: Can I create my own plugin?** -A: Yes! See [PLUGIN_DEVELOPMENT.md](PLUGIN_DEVELOPMENT.md) for instructions. +A: Yes! See [PLUGIN_DEVELOPMENT_GUIDE.md](PLUGIN_DEVELOPMENT_GUIDE.md) for instructions. --- ## Related Documentation -- [PLUGIN_DEVELOPMENT.md](PLUGIN_DEVELOPMENT.md) - Create your own plugins +- [PLUGIN_DEVELOPMENT_GUIDE.md](PLUGIN_DEVELOPMENT_GUIDE.md) - Create your own plugins - [PLUGIN_API_REFERENCE.md](PLUGIN_API_REFERENCE.md) - Plugin API documentation -- [PLUGIN_ARCHITECTURE.md](PLUGIN_ARCHITECTURE.md) - Plugin system architecture +- [PLUGIN_ARCHITECTURE_SPEC.md](PLUGIN_ARCHITECTURE_SPEC.md) - Plugin system architecture (historical) - [REST_API_REFERENCE.md](REST_API_REFERENCE.md) - Complete REST API reference diff --git a/docs/README.md b/docs/README.md index 24345c52..63128da6 100644 --- a/docs/README.md +++ b/docs/README.md @@ -14,6 +14,7 @@ the one-shot installer. The pages here go deeper. 5. [TROUBLESHOOTING.md](TROUBLESHOOTING.md) — common issues and fixes 6. [SSH_UNAVAILABLE_AFTER_INSTALL.md](SSH_UNAVAILABLE_AFTER_INSTALL.md) — recovering SSH after install 7. [CONFIG_DEBUGGING.md](CONFIG_DEBUGGING.md) — diagnosing config problems +8. [LOW_MEMORY_BOARDS.md](LOW_MEMORY_BOARDS.md) — Pi Zero 2 W / 3B+ / 1GB Pi 4 memory limits ## I want to write a plugin @@ -29,15 +30,16 @@ Start here: Going deeper: - [ADVANCED_PLUGIN_DEVELOPMENT.md](ADVANCED_PLUGIN_DEVELOPMENT.md) — advanced patterns -- [PLUGIN_ARCHITECTURE_SPEC.md](PLUGIN_ARCHITECTURE_SPEC.md) — full plugin-system spec +- [PLUGIN_ARCHITECTURE_SPEC.md](PLUGIN_ARCHITECTURE_SPEC.md) — original plugin-system design spec (historical; see its banner for what has drifted) - [PLUGIN_DEPENDENCY_GUIDE.md](PLUGIN_DEPENDENCY_GUIDE.md) / [PLUGIN_DEPENDENCY_TROUBLESHOOTING.md](PLUGIN_DEPENDENCY_TROUBLESHOOTING.md) - [PLUGIN_WEB_UI_ACTIONS.md](PLUGIN_WEB_UI_ACTIONS.md) (+ [example JSON](PLUGIN_WEB_UI_ACTIONS_EXAMPLE.json)) -- [PLUGIN_CUSTOM_ICONS.md](PLUGIN_CUSTOM_ICONS.md) / - [PLUGIN_CUSTOM_ICONS_FEATURE.md](PLUGIN_CUSTOM_ICONS_FEATURE.md) +- [PLUGIN_CUSTOM_ICONS.md](PLUGIN_CUSTOM_ICONS.md) - [PLUGIN_REGISTRY_SETUP_GUIDE.md](PLUGIN_REGISTRY_SETUP_GUIDE.md) (+ [registry template](plugin_registry_template.json)) - [STARLARK_APPS_GUIDE.md](STARLARK_APPS_GUIDE.md) — Starlark-based mini-apps - [widget-guide.md](widget-guide.md) — widget development +- [ADAPTIVE_LAYOUT.md](ADAPTIVE_LAYOUT.md) — render legibly on any panel size (opt-in font/layout scaling) +- [plugin-safety-harness.md](plugin-safety-harness.md) — test a plugin across every screen and matrix size ## Configuring plugins @@ -52,9 +54,12 @@ Going deeper: - [ADVANCED_FEATURES.md](ADVANCED_FEATURES.md) — Vegas scroll, on-demand display, cache management, background services, permissions - [FONT_MANAGER.md](FONT_MANAGER.md) — font system +- [SKIN_SYSTEM.md](SKIN_SYSTEM.md) — skin architecture for sports scoreboards +- [CREATING_SKINS.md](CREATING_SKINS.md) — writing and validating a skin ## Reference +- [CONFIG_REFERENCE.md](CONFIG_REFERENCE.md) — every key in config.json and config_secrets.json - [REST_API_REFERENCE.md](REST_API_REFERENCE.md) — all web-interface HTTP endpoints - [PLUGIN_API_REFERENCE.md](PLUGIN_API_REFERENCE.md) — Python APIs available to plugins - [DEVELOPER_QUICK_REFERENCE.md](DEVELOPER_QUICK_REFERENCE.md) — common dev tasks @@ -66,6 +71,7 @@ Going deeper: - [HOW_TO_RUN_TESTS.md](HOW_TO_RUN_TESTS.md) — running the test suite - [MULTI_ROOT_WORKSPACE_SETUP.md](MULTI_ROOT_WORKSPACE_SETUP.md) — multi-repo workspace - [MIGRATION_GUIDE.md](MIGRATION_GUIDE.md) — breaking changes between releases +- [SPORTS_UNIFICATION.md](SPORTS_UNIFICATION.md) — how the sports scoreboard base classes are organized ## Archive diff --git a/docs/REST_API_REFERENCE.md b/docs/REST_API_REFERENCE.md index 132547b0..f95cb84c 100644 --- a/docs/REST_API_REFERENCE.md +++ b/docs/REST_API_REFERENCE.md @@ -31,9 +31,9 @@ All endpoints return JSON responses with a standard format: - [Plugin-specific endpoints](#plugin-specific-endpoints) - [Starlark Apps](#starlark-apps) -> The API blueprint is mounted at `/api/v3` (`web_interface/app.py:144`). +> The API blueprint is mounted at `/api/v3` (`web_interface/app.py:199`). > SSE stream endpoints (`/api/v3/stream/*`) are defined directly on the -> Flask app at `app.py:607-615`. There are about 92 routes total — see +> Flask app at `app.py:799-809`. There are 94 routes total — see > `web_interface/blueprints/api_v3.py` for the canonical list. --- diff --git a/docs/SKIN_SYSTEM.md b/docs/SKIN_SYSTEM.md new file mode 100644 index 00000000..f7f4861f --- /dev/null +++ b/docs/SKIN_SYSTEM.md @@ -0,0 +1,170 @@ +# Skin System Architecture + +Skins are user-installable **visual overlays** for the sports scoreboards. +A skin replaces only the *look* of a scoreboard — the host plugin keeps doing +data fetching, scheduling, caching, dedup, live-priority takeover, and vegas +mode. If you only want to **build** a skin, read +[CREATING_SKINS.md](CREATING_SKINS.md); this document explains how the system +works and why it is shaped this way. + +## Why skins instead of forks + +Before skins, changing a scoreboard's layout meant forking the whole plugin +(e.g. the community MLB scoreboard fork). The fork gets the new look but loses +everything the maintained plugin keeps earning: duration/scheduling behavior, +vegas mode support, caching and background-fetch improvements, bug fixes. It +also silently drifts: every upstream improvement now has to be re-ported by +hand. + +A skin inverts that trade. The plugin remains stock and keeps updating through +the store; the skin is ~100 lines of pure rendering code that receives the +plugin's already-fetched data each frame. Uninstalling the skin (or the skin +crashing) simply restores the built-in look. + +```text + (unchanged) (the skin seam) + ESPN API ──► update() ──► game view model ──► _render_game() ──► display + fetching (a dict) │ │ + caching │ └─ built-in + scheduling └─ skin.render_(ctx, game) + live priority draws onto ctx.canvas +``` + +## The render funnel + +Every sports scoreboard (baseball, football, basketball, hockey — anything +built on the `src/base_classes/sports/` package, `core.py`) renders through exactly one seam: +`SportsCore._render_game(game, force_clear)`. + +1. The mode class's `display()` (live, `SportsUpcoming`, `SportsRecent`) + picks `self.current_game` and calls `_render_game`. +2. `_render_game` lazily loads the configured skin (once, on first render — + a broken skin can never block plugin startup). +3. If a skin is active, the host builds a `SkinContext` — a fresh black + canvas at the current display size plus layout/font/logo helpers — and + calls the skin's `render_live` / `render_recent` / `render_upcoming` + with a **copy** of the game dict. +4. If the skin returns `True`, the canvas is composited onto the display. + If it returns `False`, isn't implemented for that mode, or raises, the + built-in `_draw_scorebug_layout` runs instead. + +Key properties that fall out of this design: + +- **Per-mode fallback.** A skin that only implements `render_live` gets the + stock recent/upcoming screens for free. +- **Three strikes.** A skin that raises 3 times in a row is disabled for the + rest of the session (one loud error log per failure); the display never + goes dark. Restarting the service re-arms it. +- **Copies, not references.** Skins receive a shallow copy of the game dict, + so a buggy skin cannot corrupt the plugin's scheduling state. +- **Vegas mode works untouched.** Vegas capture falls back to grabbing the + regular `display()` output, which is already skin-rendered. Skins can + additionally implement `render_vegas_card` for purpose-built scroll cards, + and hosts can call `SportsCore.render_skin_card(game, size)` to use it. +- **Hot-loop caution.** `render_live` runs every display-loop pass during a + live game. The host logs a warning when a skin render exceeds 150 ms, and + `scripts/validate_skin.py` enforces a budget at development time — but + Python cannot forcibly time-out a stuck render, so a skin that blocks + (network I/O, giant image ops) stalls the display. This is why the rules + in CREATING_SKINS.md ban I/O in render paths. + +## The view model contract + +The `game` dict a skin receives is the plugin's already-extracted view model +(`SportsCore._extract_game_details_common` plus per-sport extras from +`src/base_classes/{baseball,basketball,football,hockey}.py`). + +- **Guaranteed keys (view model v1.0)** — always present for every sport: + `id`, `game_time`, `game_date`, `start_time_utc` (a UTC `datetime`), + `status_text`, `is_live`, `is_final`, `is_upcoming`, `is_halftime`, + `home_abbr`/`away_abbr`, `home_id`/`away_id`, `home_score`/`away_score` + (**strings**), `home_logo_path`/`away_logo_path`, `home_record`/`away_record`. +- **Sport extras** — documented per sport in CREATING_SKINS.md (e.g. baseball + adds `inning`, `inning_half`, `balls`, `strikes`, `outs`, `bases_occupied`). +- **Optional keys** (`odds`, rankings, `series_summary`, …) are present only + when the feature is enabled — skins must always use `.get()`. + +Versioning policy: additive changes bump the minor version +(`VIEW_MODEL_VERSION` in `src/skin_system/skin_base.py`, surfaced to skins as +`ctx.view_model_version`); renaming or removing a guaranteed key requires a +major bump plus a compat shim. `test/test_skin_system.py::TestViewModelContract` +fails CI if a guaranteed key disappears from the extractor. + +Separately, `SKIN_API_VERSION` versions the Python API (`ScoreboardSkin`, +`SkinContext`). The loader refuses a skin whose manifest declares a different +major version and falls back to the built-in renderer with a clear +"skin needs an update" log line. + +## Package layout and lifecycle + +```text +skins// + skin.json # manifest (required) + skin.py # ScoreboardSkin subclass (required) + preview.png # optional, shown by the web UI + assets/ # optional skin-local images + helpers.py ... # optional extra modules (namespaced per skin at import) +``` + +Skins live in the central `skins/` directory — deliberately **not** inside the +plugin's directory, because plugin reinstall/update deletes the whole plugin +directory and a skin must survive that. One skin can also target several +plugins (mlb + milb). + +Lifecycle: discovered lazily on first render → manifest validated → API major +version gated → module imported under a namespaced `sys.modules` key (two +skins can both ship a `helpers.py`, same scheme plugins use) → instantiated +with `(manifest, options)`. Every failure logs and falls back to built-in. + +Skins should be **stateless**: the live, recent, and upcoming mode classes +each hold their own skin instance, so derive everything from `(ctx, game)`. + +## Selection and configuration + +Inside the plugin's own config section in `config/config.json`: + +```json +"baseball-scoreboard": { + "skin": "retro-baseball", + "skin_options": { "accent_color": [255, 80, 0] } +} +``` + +`"skin"` is either one id for all modes or a per-mode mapping +(`{"live": "retro-baseball", "recent": "built-in"}`). Absent, empty, or +`"built-in"` means the stock renderer. Because this rides the plugin's config +section, it persists across plugin reinstalls like every other setting. + +The web UI shows a **Visual Skin** dropdown for plugins that have matching +skins installed: `SchemaManager.inject_skin_selector` adds an enum to the +*served* schema only. Validation never sees the enum — so a config that +references an uninstalled skin stays valid (rendering just falls back), and +the currently-configured value is always kept selectable. `GET /api/v3/skins` +lists installed skins (optionally filtered by `?plugin_id=`). + +## Distribution + +- **Manual:** `git clone skins/` — that's the whole + install. No manifest bumps, no `update_registry.py`; skins are not monorepo + plugins. +- **Store:** registry entries with `"type": "skin"` install through the same + `plugins.json` pipeline; `PluginStoreManager` routes them to `skins/`, + validates `skin.json` (including the API major version) instead of + `manifest.json`, and never installs dependencies — skins are render-only + (stdlib + PIL + the provided context, no third-party packages in v1). + +## Trust model + +A skin is Python executing inside the display service — **exactly the same +trust level as a plugin**, even though "skin" sounds cosmetic. Only install +skins from sources you'd be willing to install a plugin from. + +## v2 directions (not in v1) + +- A generic `BasePlugin` opt-in (`render_with_skin()`) so non-sports plugins + (weather, music) can offer skinnable layouts; `skin_runtime` is already + sports-agnostic in anticipation. +- Store UI: preview gallery, one-click install from the skin browser. +- An update path for git-cloned skins (today: re-clone or store reinstall). +- Animation support in skins (today the API is one frame per render call; + stateful tricks work but are at-your-own-risk). diff --git a/docs/SPORTS_UNIFICATION.md b/docs/SPORTS_UNIFICATION.md new file mode 100644 index 00000000..4e731278 --- /dev/null +++ b/docs/SPORTS_UNIFICATION.md @@ -0,0 +1,387 @@ +# Sports Code Unification — Architecture + +How the nine sports scoreboard plugins converge onto shared core code **without** +becoming nine clients of a god class. + +## The problem + +Nine plugins (`afl`, `baseball`, `basketball`, `football`, `hockey`, `lacrosse`, +`nrl`, `soccer`, `ufc`) each ship a ~3,000-line `sports.py` descended from this +repo's `src/base_classes/sports.py`. They have drifted into three lineages, and +only 28 of the 66 methods appearing across them are present in all nine. One +logical fix (the UTC start-time bug) cost 75 files. + +Merging everything into one base class would fix the duplication and create a +worse problem: a single 2,500-line class that all nine plugins inherit, where any +change has a nine-plugin blast radius and per-sport behavior survives only as +`if self.sport == "hockey"` branches. + +## Three properties, three mechanisms + +These are independent concerns. Conflating them is what produces god classes. + +### Upgradability — a plugin keeps working across core versions + +| Rule | Mechanism | +|---|---| +| Plugin loads on a core that predates a module | Guarded import with a bundled fallback (`try: from src.X import Y / except ModuleNotFoundError: from y import Y`) | +| Plugin loads on a core that predates a *method* | Capability probing — `hasattr(SportsCore, "_detect_stale_games")` — never a version comparison. The loader's compat check is advisory-only (it logs and continues), so probing is the real protection. | +| Core changes never break a plugin's rendering | The **view-model contract**: `_extract_game_details_common` returns a dict whose `GUARANTEED_KEYS` are frozen by `test/test_skin_system.py::TestViewModelContract`. Keys may be added, never renamed or removed. | +| A plugin can drop its bundled copy safely | The **sunset rule**: its manifest must floor `ledmatrix_min_version` at the first core release shipping the module (recorded in `CHANGELOG.md`) — *necessary but not sufficient*. Nothing enforces that floor today, so the copy also waits for the B6 gate below. | + +The core API is **additive-only**. A method the plugins call is never removed or +given a new required parameter; new behavior arrives as new methods with +defaults, or as capabilities they opt into. + +### Reusability — write once, nine plugins benefit + +Only code that is **identical in intent across all nine** moves into the base +class. That set is small and knowable — it is exactly the methods present in every +copy today (phase B1 below). Everything else stays where it is until it earns +promotion. + +### Modularity — a change to one feature cannot reach a plugin that doesn't use it + +This is the property the naive merge destroys, and it is enforced structurally: + +1. **Capabilities are separate modules composed by inheritance, not config + branches inside the base class.** Hockey has no celebrations, so + `HockeyLive` does not inherit `CelebrationMixin` — the celebration code is not + merely disabled for hockey, it is *not in hockey's MRO at all*. No shared + state, no dead branches, no risk. Contrast with + `if self.celebrations_enabled:` inside `SportsLive`, where a bug in + celebration code can still crash a plugin that never wanted the feature. + +2. **Variant behavior is a strategy object chosen by name, not a branch.** + Live rotation exists in three dialects across the lineages; core ships all + three behind `rotation_strategy: "swrr" | "weighted" | "simple"` and a plugin + may register its own. Core never learns sport names. + +3. **Sport-specific behavior is a documented override point.** The base class + declares the seam; the plugin fills it. Basketball's tournament-round parsing + and baseball's BDF sizing stay in their plugins forever — they are not + candidates for promotion, and core must never grow a branch for them. + +4. **Files bound the blast radius.** Capabilities live in their own modules so a + diff shows at a glance which plugins a change can reach. + +## Layering + +``` +src/base_classes/sports/ + __init__.py re-exports the public API (import path unchanged) + core.py SportsCore — fetch, cache, config, logos, fonts, odds, + view-model extraction, the skin seam + modes.py SportsUpcoming / SportsRecent / SportsLive + capabilities/ + celebrations.py CelebrationMixin (opt-in: 4 of 9 plugins) + rotation.py RotationStrategy + registry + +src/common/ + sports_scroll.py SportsScrollDisplay / …Manager — scroll orchestration + (content building stays in the plugins) +``` + +`from src.base_classes.sports import SportsCore` keeps working — the package +`__init__` re-exports, so the conversion is invisible to every existing importer. + +## Override points (the plugin-facing seam) + +The base class calls these; plugins implement or override them. This table is the +contract — additions require a default implementation, removals require a +deprecation cycle. + +| Hook | Purpose | Default | +|---|---|---| +| `_fetch_data()` | Sport's schedule source | abstract | +| `_extract_game_details(event)` | Sport-specific view-model fields on top of the common ones | delegates to `_extract_game_details_common` | +| `_draw_scorebug_layout(game, force_clear)` | Sport's card rendering | base layout | +| `_custom_scorebug_layout(game, draw)` | Per-sport overlay on the base layout | no-op | +| `render_skin_card(game, size)` | Skin-system entry point | built-in fallback | +| `score_phrase(points, team_abbr)` | Celebration wording (`"GOOOOAAALLL!"` vs `"TOUCHDOWN!"`). `points` is the score delta, which sports with variable-value scores use to name the play | `" SCORES!"` — only consulted when `CelebrationMixin` is present | +| `win_phrase(team_abbr)` | Win-celebration wording | `" WINS!"` — mixin only | +| `_favorite_key(game, side)` | Which view-model field identifies a team for favorites matching | `game["_abbr"]` | +| `_config_schema_path()` | Plugin's `config_schema.json` — returning it routes `_get_layout_offset` through the `src.element_style` resolver (and gives it the defaults to compare against) | `None`, i.e. the classic inline `customization.layout` read | +| `_font_root()` | Directory to resolve `assets/fonts` against | core install root | + +Two class attributes serve the same purpose for values that are per-sport +constants rather than behavior: + +| Attribute | Meaning | Default | +|---|---|---| +| `FINAL_PERIOD` | Period at/after which a zero clock can mean "over" | `4` (hockey overrides to `3`) | +| `CLOCK_COUNTS_DOWN` | Whether `0:00` means "expired" | `True` (soccer/afl/nrl override to `False` — their clocks count up, so `0:00` is kickoff) | +| `COALESCE_SCORING_SEQUENCE` | Fold score increments arriving during an active celebration into that one celebration | `False` (football overrides to `True` — a touchdown lands as +6, then +1 for the extra point) | + +### Why these are seams and not branches + +`_favorite_key` exists because NRL abbreviations are **not unique** — "NEW" is both +Newcastle Knights and New Zealand Warriors, "CAN" both Canberra and Canterbury — +so NRL matches favorites on team ID. Flattening every plugin to abbreviations +would silently select the wrong club for NRL users. The base declares the seam, +NRL fills it, and core never learns the string `"nrl"`. + +`CLOCK_COUNTS_DOWN` exists for the same reason in the opposite direction: a +soccer clock reading `0:00` means the match has not kicked off, so running the +clock-expiry branch there would evict live games. + +`COALESCE_SCORING_SEQUENCE` is the third of the same kind. In football one +scoring play arrives as two score updates, so the follow-up must be folded into +the first celebration; in soccer two increments a few seconds apart are two real +goals, and folding them would swallow one. Neither default is "right" — which is +precisely why it is a declared per-sport constant rather than a hidden +assumption baked into the shared body. + +## Capabilities + +``` +capabilities/ + celebrations.py CelebrationMixin opt-in: afl, nrl, soccer, football + rotation.py RotationStrategy + registry +``` + +**`CelebrationMixin`** merges the two dialects the lineages grew +(`_check_for_goal`/`celebrate_opponent_goals` vs +`_check_for_score`/`celebrate_opponent_scores`). Their bodies were identical +apart from three things, each now a seam: wording (`score_phrase`), follow-up +suppression (`COALESCE_SCORING_SEQUENCE`), and team identity (`_favorite_key`, +so NRL matches on id). Both config spellings are read, so a plugin adopting the +mixin keeps working with the keys already in its published schema. + +Mix it in **before** the mode class — `class SoccerLive(CelebrationMixin, +SportsLive)` — so the celebration `display()` runs first and falls through to +the scorebug via `super()`. + +**Rotation strategies.** The three "dialects" turned out to be one algorithm +(Smooth Weighted Round-Robin) in two shapes: an incremental picker holding state +across calls (afl/nrl/soccer) and a precomputed per-cycle list +(football/baseball/basketball, and hockey with a different loop shape). They +agree within a cycle and differ only at the boundary — the incremental form has +no restart seam — so core ships both rather than declaring a winner: + +```python +self.rotation = get_rotation_strategy("swrr", weight_for=self._live_weight) +``` + +`weight_for` is supplied by the host, so the *favorites* policy stays with the +plugin and `rotation.py` never learns what a favorite is. An unknown strategy +name degrades to `simple` rather than raising: the name comes from user config, +and a typo should cost the boost, not the scoreboard. When a plugin needs an +ordering that core does not ship, it calls `register_rotation_strategy` to add +its own — rather than core growing a branch for it. + +`test_sports_capabilities.py` checks each strategy against a **verbatim +transcription** of the plugin code it replaces, over every live-game shape up to +four games. That differential is what B5 deletes the bundled copies on the +strength of. + +## Scroll display — where the promotion line falls + +`src/common/sports_scroll.py` is deliberately *not* a superset of the ten +`scroll_display.py` copies. A method-level comparison of the eight that share a +shape (f1 and ufc are genuine forks) found a sharp split: + +| Layer | Evidence | Outcome | +|---|---|---| +| Orchestration — `get_all_vegas_content_items`, `clear_all`, `get_scroll_info`, `get_dynamic_duration`, `is_complete`, `display_frame` | identical to 96–100% similar across all eight | **promoted** | +| Settings — `_get_scroll_settings` | one algorithm; the copies differ *only* in which league keys they walk | **promoted**, with the ladder as data (`SCROLL_LEAGUE_KEYS`) | +| Content — `prepare_scroll_content`, `_load_separator_icons` | 8 distinct bodies across 8 plugins (145 lines, 53% similar at worst); icons 6% | **override point, permanently** | + +Same name, different job: `prepare_scroll_content` draws *this sport's* game +card. Merging the eight bodies would be the exact mistake the promotion rule +exists to prevent, so the base class raises `NotImplementedError` rather than +rendering something plausible — a base that rendered *something* would let a +plugin ship a silently blank scroll. + +The one behavior the upstreamed version adds is native +`global_config['target_fps']` support. The bundled copies hardcode ~100 FPS via +`scroll_delay = 0.01` and never consult the global smooth-scrolling target; +Part A threaded it through each copy by hand, and this makes that threading +legacy compatibility rather than the mechanism. + +## Phases + +B0–B3 are merged and shipping in core 3.2.0. Everything that remains is +**rollout**, and it splits into three phases with very different risk profiles. +The original plan folded the last two together; they are separated here because +one of them is safe by construction and the other is not. + +| Phase | Scope | Status | Gate | +|---|---|---|---| +| **B0** | Characterization tests, CI unit job, `element_style`, font cwd fix, CHANGELOG discipline | ✅ | — | +| **B1** | Promote the nine universal methods; convert `sports.py` → package | ✅ | Characterization suite green; no behavior change intended | +| **B2** | `CelebrationMixin` + rotation strategies as opt-in capabilities | ✅ | Non-adopters have zero new code in their MRO; strategies checked against verbatim plugin transcriptions | +| **B3** | Upstream the scroll **orchestration** layer as `src/common/sports_scroll.py`, reading `global_config['target_fps']` natively | ✅ | Content building stays per-sport | +| **B4** | Ship 3.2.0 *and* make version reporting trustworthy | ⏳ **next** | Tag, release, and `src.__version__` agree; compatibility gate merged | +| **B5** | Adoption — guarded core imports: three pilots, then the remaining six. **Bundled copies stay.** | after B4 | Per plugin: harness + goldens byte-identical, then a device soak | +| **B6** | Sunset — delete the bundled copies | **blocked** | B4's gate shipped *and* in users' hands (see below) | + +### B4 — what "ship 3.2.0" actually requires + +Cutting the tag is the small part. The version *number* has to become something +a floor can be trusted against, and today it is not: + +- **The tag and `src.__version__` have never agreed.** `v3.1.0` was tagged + 2026-05-31; `__version__` only became `"3.1.0"` on 2026-07-12 (`7f7f0d64`). + The v3.1.0 release therefore reports `__version__ = "1.0.0"`. +- **Which silences the compatibility warning entirely for that population.** + `PluginLoader._warn_if_incompatible` skips the check when the parsed core + version is below `(2, 0, 0)` — an anti-spam guard that, given the above, + matches exactly the users most likely to be behind. +- **Nothing enforces a floor anyway.** The check is advisory (it logs and + continues), and neither `StoreManager.install_plugin` nor + `StoreManager.update_plugin` compares the core version at all — `update_plugin` + compares the plugin's manifest version against the registry's + `latest_version` and nothing else. + +So B4 is: tag and release 3.2.0; make the tag, the release, and `__version__` +agree, and keep them agreeing; reconsider the `< 2.0.0` skip; migrate manifests +from `ledmatrix_min` to `ledmatrix_min_version`; and add the install/update +compatibility gate that B6 depends on. + +#### Two fields express compatibility, and the gate only reads one + +`compatible_versions` is the canonical contract: `schema/manifest_schema.json` +**requires** it, all 42 published manifests carry it, and it holds semver +*ranges* — `[">=2.0.0"]` in 41 of them, `[">=1.0.0"]` in `7-segment-clock`. +`ledmatrix_min_version` is the optional per-release floor inside `versions[]`. + +The gate as merged reads only the floor. Today that is harmless: no manifest +uses an upper bound, and the two fields agree everywhere except +`7-segment-clock` (`>=1.0.0` against a `2.0.0` floor). But the fields *can* +disagree, and the range syntax the schema already permits includes upper bounds +— a plugin declaring `["2.0.0 - 2.9.9"]` means "not compatible with 3.x" and +the gate would install it on 3.2.0 regardless. + +**Before B6, the gate must evaluate `compatible_versions` as well**, and the +manifest migration must reconcile the two fields rather than only renaming the +floor. Deciding which wins when they disagree is part of that work; the safe +default is the more restrictive. + +(The schema also deprecates a top-level `ledmatrix_version` in favour of +`compatible_versions`. No manifest still carries it, so there is nothing to +migrate there.) + +### B5 — adoption is safe by construction + +A plugin adopting core imports keeps its bundled copy and reaches it through the +guarded import (see the Upgradability table above). On a core that ships the +module the plugin uses core code; on one that doesn't it falls back and behaves +exactly as it does today. There is no version of this step that breaks a user, +which is why it does not wait for B6's gate. + +The hockey scroll-display pilot is **already validated**: adopted against a core +carrying 3.2.0, `scroll_display.py` went from 691 to 289 lines and all 16 harness +renders (8 sizes × 2 screens) came out byte-for-byte identical to the +pre-adoption run. That byte-comparison is the acceptance gate for every +adoption. The recipe and its two gotchas are in the plugins repo's +`docs/plugin-development/08-shared-sports-code.md`. + +### B6 — why the sunset needs more than a version floor + +Deleting a bundled copy removes the fallback, so the guarded import becomes a +hard dependency. On a core without the module the plugin raises +`ModuleNotFoundError` at load; `PluginManager.load_plugin` catches it, records +`PluginState.ERROR`, logs one line, and continues. Nothing crashes — the user +simply loses that scoreboard, with no visible explanation. + +Verified against a `v3.1.0` worktree: `src/common/sports_scroll.py`, +`src/element_style.py` and the `src/base_classes/sports/` package are all absent +there, and the import fails with `exc.name == 'src.common.sports_scroll'`. Guard +sets must name that exact dotted path — `{"src"}` alone does not match it. + +Combined with the B4 findings, a plugin that deletes its copy today reaches an +un-updated user through a normal store update, fails to load, and warns nobody. +**B6 therefore waits for B4's compatibility gate to have shipped and to have +been in users' hands long enough that the population running a core without it +is small.** The bundled copies cost disk space; deleting them early costs +scoreboards, silently. That trade is not close. + +Before the first sunset, add a **compatibility regression test**. It has to +cover four cases, not one — B5's safety claim and B6's failure mode are +different propositions and only the second is obvious: + +| | bundled copy present | bundled copy removed | +|---|---|---| +| **pinned old core** | **loads** — this is B5's whole guarantee, that the guarded import falls back | `PluginState.ERROR`, and the recorded error names the exact missing module | +| **current core** | loads, using core code | loads, using core code | + +The top-left cell is the one worth writing first: nothing in the suite currently +proves that an adopted plugin still works on a core that predates the module, +which is the entire basis for saying B5 is safe to run ahead of the gate. + +Assert the old-core/removed-copy case as `PluginState.ERROR` **plus the missing +module path**, not as an uncaught exception. `PluginManager.load_plugin` catches +`ModuleNotFoundError`, so nothing propagates — a test expecting a raise would +pass for the wrong reason on a core where the module is merely broken rather +than absent. "Fails loudly" is aspirational, not what the code does today: it +fails into `ERROR` state with one log line, which is precisely why B6 needs the +gate rather than trusting the failure to be noticed. + +The same suite should exercise the install/update gate, since it is the other +half of the guarantee. + +## What's next + +In order. Each step is independently useful and independently revertible. + +1. **Tag and publish v3.2.0.** The code is already on `main` (`21825cbf`). + Nothing else blocks this, and it is what makes `ledmatrix_min_version: + "3.2.0"` refer to something real. +2. **Make the version number honest.** Have the release process assert that the + tag, the GitHub release, and `src.__version__` agree — a check in CI is + cheaper than the confusion of the last two releases. Then revisit the + `< 2.0.0` skip in `_warn_if_incompatible`, which currently silences the + warning for the users who most need it. +3. **Add the compatibility gate** to `StoreManager.install_plugin` and + `.update_plugin`: refuse a plugin whose declared floor exceeds + `src.__version__`, and surface the reason in the store UI rather than only + the log. This is the single change that turns the floor from documentation + into a guarantee, and B6 depends on it. +4. **Migrate the manifests** to `ledmatrix_min_version`, and reconcile them with + `compatible_versions` (see above — that field is the required, canonical one, + and the gate does not read it yet). Currently 28 plugins spell the floor both + ways across their `versions[]` entries, 12 use only the old spelling, and 2 + only the new. Scope the sweep to the nine sports plugins if a 42-plugin + version-bump wave isn't worth it — but the `compatible_versions` half has to + cover every manifest the gate can refuse, or define explicit legacy handling, + before the gate is allowed to block anything. +5. **Run B5 adoption** — hockey, soccer, football, then the remaining six. + Bundled copies stay. Byte-identical harness output per plugin, then a soak. +6. **Only then plan B6**, with the compatibility regression test described above + in CI first. + +## How to keep this project healthy + +Lessons this migration paid for, worth applying beyond it: + +- **A version number is a promise; keep it in one place.** Three different + answers to "what version am I on" (tag, release, `__version__`) is what made + the floor untrustworthy. Assert their agreement mechanically. +- **Advisory checks protect nobody.** If a rule matters, enforce it where the + action happens — the install path, not a log line the user will never read. + If it doesn't matter enough to enforce, don't write the rule. +- **Prefer failures that are loud and early.** A plugin that dies at load with + one journal line is indistinguishable, to a user, from a plugin that was never + installed. Surface plugin health in the UI. +- **Keep the two repos' rules in sync deliberately.** The sunset rule lives in + both this file and the plugins repo's + `docs/plugin-development/08-shared-sports-code.md`. When one changes, change + the other in the same PR — drift between them is how a contributor ends up + following a rule that was superseded. +- **Measure before and after, on real hardware.** Byte-identical harness renders + and a device soak caught what unit tests could not. Reserve "it should be + fine" for things you have actually looked at. + +## Rules for contributors + +- **Promote on evidence, not intuition.** A method moves to core when every copy + has it and they agree on intent. Otherwise it stays in the plugins. +- **Never add a sport name to core.** If core needs to know which sport it is, + the design is wrong — add an override point instead. +- **A capability that is not opted into must not execute.** If you find yourself + writing `if self._enabled` inside a base class, it belongs in a + mixin. +- **Touch the view-model keys only additively.** Published skins depend on them. +- **Every promotion lands with the characterization suite green**, and every + pilot adoption lands with that plugin's harness and golden suites green. diff --git a/docs/SSH_UNAVAILABLE_AFTER_INSTALL.md b/docs/SSH_UNAVAILABLE_AFTER_INSTALL.md index d9830165..bc065eb1 100644 --- a/docs/SSH_UNAVAILABLE_AFTER_INSTALL.md +++ b/docs/SSH_UNAVAILABLE_AFTER_INSTALL.md @@ -20,7 +20,22 @@ The installation script: - Installs and configures `dnsmasq` (DHCP server for AP mode) - These services can interfere with normal WiFi client mode -### 3. Reboot After Installation +### 3. The Board Ran Out of Memory + +On a 512MB or 1GB board, memory exhaustion stops `sshd` being able to fork a +session process. The connection is accepted and then closed immediately, before +any banner: + +```text +kex_exchange_identification: Connection closed by remote host +``` + +The giveaway is that the board is otherwise healthy — ping is clean and the web +UI still responds — but nothing that needs to start a new process works, and +the panel is usually dark. Only a power cycle clears it. See +[LOW_MEMORY_BOARDS.md](LOW_MEMORY_BOARDS.md). + +### 4. Reboot After Installation If the script reboots the Pi (which it recommends), network services may restart in a different state, potentially triggering AP mode. @@ -190,11 +205,23 @@ The web interface allows you to: ## Summary -**SSH becomes unavailable because**: +**SSH becomes unavailable because** — two unrelated causes, and they need +different responses: + +*AP mode (most common):* - WiFi monitor service enables AP mode when WiFi disconnects - AP mode switches WiFi from client to access point mode - Pi loses connection to your original network +*Memory exhaustion (low-memory boards):* +- The board runs out of memory, so `sshd` cannot fork a session process +- The connection is accepted and closed before any banner +- Ping still answers and the web UI still responds, so it looks healthy +- The panel is usually dark and the service cannot restart +- **Only a power cycle clears this** — there is no remote recovery, because + every remote route needs a new process +- Prevention and tuning: [LOW_MEMORY_BOARDS.md](LOW_MEMORY_BOARDS.md) + **To regain SSH**: 1. Connect to **LEDMatrix-Setup** AP network (password: `ledmatrix123`) 2. SSH to `192.168.4.1` diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md index e173e720..fe3a1a74 100644 --- a/docs/TROUBLESHOOTING.md +++ b/docs/TROUBLESHOOTING.md @@ -82,6 +82,70 @@ python3 web_interface/start.py ## Common Issues by Category +### Installation & Build Issues + +#### Step 6 fails: "Failed building wheel for rgbmatrix" + +**Symptoms:** + +``` +note: This error originates from a subprocess, and is likely not a problem with pip. + ERROR: Failed building wheel for rgbmatrix +Failed to build rgbmatrix +✗ Failed to install rpi-rgb-led-matrix Python package +``` + +**Cause:** + +Almost always the kernel's out-of-memory killer, not missing build tools. The +`rpi-rgb-led-matrix` library compiles roughly 45 C++ translation units, two of +them Cython-generated — a single `cc1plus` on those can peak near 800MB. The +build system defaults to running several of those at once, which exceeds RAM on +512MB and 1GB boards. Because the OOM killer writes nothing to pip's output, the +failure looks like a toolchain problem, and `sudo apt install -y +python-dev-is-python3 cmake build-essential` will report everything is already +up to date. + +**How to confirm:** + +```bash +dmesg -T | grep -i "out of memory" # look for "Killed process ... (cc1plus)" +free -h # total RAM and swap +``` + +**Fix:** + +Current versions of the installer handle this automatically: they cap build +parallelism based on available RAM and add a temporary swapfile for the build, +removing it when the build finishes. If you are on an older checkout, or the +temporary swapfile could not be created, either force a serial compile: + +```bash +sudo ./first_time_install.sh --build-jobs 1 +``` + +or add permanent swap and re-run the installer, which resumes at Step 6: + +```bash +sudo apt install -y dphys-swapfile +sudo sed -i 's/^#\?CONF_SWAPSIZE=.*/CONF_SWAPSIZE=2048/' /etc/dphys-swapfile +sudo sed -i 's/^#\?CONF_MAXSWAP=.*/CONF_MAXSWAP=2048/' /etc/dphys-swapfile +sudo dphys-swapfile swapoff && sudo dphys-swapfile setup && sudo dphys-swapfile swapon +sudo ./first_time_install.sh +``` + +`CONF_MAXSWAP` matters: it defaults to 2048 and silently clamps `CONF_SWAPSIZE`, +so setting only `CONF_SWAPSIZE` to a larger value has no effect. + +**Related:** + +- The installer needs roughly 3GB free on the card to place the swapfile. If + disk is tight it will say so and skip the swapfile: `sudo apt clean` first. +- `sudo bash scripts/check_system_compatibility.sh` reports RAM and disk. +- `sudo bash scripts/diagnose_dependencies.sh` dumps build-dependency state. + +--- + ### Web Interface & Service Issues #### Service Not Running/Starting @@ -266,8 +330,8 @@ sudo systemctl cat ledmatrix-web | grep User 6. **Manually enable AP mode:** ```bash - # Via API - curl -X POST http://localhost:5000/api/wifi/ap/enable + # Via API (the WiFi blueprint is mounted under /api/v3) + curl -X POST http://localhost:5000/api/v3/wifi/ap/enable # Via Python python3 -c " @@ -418,19 +482,19 @@ sudo systemctl cat ledmatrix-web | grep User 1. **Check plugin directory exists:** ```bash - ls -ld plugins/plugin-id/ + ls -ld plugin-repos/plugin-id/ ``` 2. **Verify manifest.json:** ```bash - cat plugins/plugin-id/manifest.json + cat plugin-repos/plugin-id/manifest.json # Verify all required fields present ``` 3. **Check dependencies installed:** ```bash - if [ -f plugins/plugin-id/requirements.txt ]; then - pip3 install --break-system-packages -r plugins/plugin-id/requirements.txt + if [ -f plugin-repos/plugin-id/requirements.txt ]; then + pip3 install --break-system-packages -r plugin-repos/plugin-id/requirements.txt fi ``` @@ -443,7 +507,7 @@ sudo systemctl cat ledmatrix-web | grep User ```bash python3 -c " import sys - sys.path.insert(0, 'plugins/plugin-id') + sys.path.insert(0, 'plugin-repos/plugin-id') from manager import PluginClass print('Plugin imports successfully') " @@ -459,12 +523,18 @@ sudo systemctl cat ledmatrix-web | grep User **Solutions:** 1. **Manual cache clearing:** - ```bash - # Remove plugin-specific cache - rm -rf cache/plugin-id* - # Or remove all cache - rm -rf cache/* + The cache does not live in the project directory. The cache manager + uses the first writable location among `/var/cache/ledmatrix`, + `~/.ledmatrix_cache`, `/opt/ledmatrix/cache`, and + `$TMPDIR/ledmatrix_cache`. The easiest option is the helper script: + + ```bash + # Clear the cache with the helper script + sudo python3 scripts/utils/clear_cache.py + + # Or remove files manually from the cache dir in use, e.g.: + sudo rm -rf /var/cache/ledmatrix/* # Restart display sudo systemctl restart ledmatrix @@ -472,8 +542,8 @@ sudo systemctl cat ledmatrix-web | grep User 2. **Check cache permissions:** ```bash - ls -ld cache/ - sudo chown -R ledpi:ledpi cache/ + ls -ld /var/cache/ledmatrix + sudo ./scripts/fix_perms/fix_cache_permissions.sh ``` --- @@ -708,11 +778,11 @@ nmcli device status ```bash # Check file exists ls -l config/config.json -ls -l plugins/plugin-id/manifest.json +ls -l plugin-repos/plugin-id/manifest.json # Check directory structure ls -la web_interface/ -ls -la plugins/ +ls -la plugin-repos/ # Check file permissions ls -l config/config_secrets.json @@ -740,7 +810,7 @@ python3 -c "from src.wifi_manager import WiFiManager; print('OK')" # Test plugin import python3 -c " import sys -sys.path.insert(0, 'plugins/plugin-id') +sys.path.insert(0, 'plugin-repos/plugin-id') from manager import PluginClass print('Plugin imports OK') " @@ -748,40 +818,29 @@ print('Plugin imports OK') --- -## Service File Template +## Reinstalling Service Files -If your systemd service file is corrupted or missing, use this template: - -```ini -[Unit] -Description=LEDMatrix Web Interface -After=network.target - -[Service] -Type=simple -User=ledpi -Group=ledpi -WorkingDirectory=/home/ledpi/LEDMatrix -Environment="PYTHONUNBUFFERED=1" -ExecStart=/usr/bin/python3 /home/ledpi/LEDMatrix/web_interface/start.py -Restart=on-failure -RestartSec=5s -StandardOutput=journal -StandardError=journal -SyslogIdentifier=ledmatrix-web - -[Install] -WantedBy=multi-user.target -``` - -Save to `/etc/systemd/system/ledmatrix-web.service` and run: +If a systemd service file is corrupted or missing, do NOT hand-write +one. The real unit files live in the repo's `systemd/` directory +(`ledmatrix.service`, `ledmatrix-web.service`, +`ledmatrix-wifi-monitor.service`) and contain a +`__PROJECT_ROOT_DIR__` placeholder that the install scripts substitute +with your actual checkout path: ```bash -sudo systemctl daemon-reload -sudo systemctl enable ledmatrix-web -sudo systemctl start ledmatrix-web +# Reinstall the display service unit +sudo ./scripts/install/install_service.sh + +# Reinstall the web interface service unit +sudo ./scripts/install/install_web_service.sh ``` +Note that `ledmatrix-web.service` runs as root via +`scripts/utils/start_web_conditionally.py` — root is needed for +system operations (service control, WiFi management), and the wrapper +honors the `web_display_autostart` config flag before actually +starting the web server. + --- ## Complete Diagnostic Script @@ -814,7 +873,7 @@ echo "" echo "5. File Structure:" ls -la web_interface/ | head -10 -ls -la plugins/ | head -10 +ls -la plugin-repos/ | head -10 echo "" echo "6. Python Imports:" @@ -890,12 +949,11 @@ sudo systemctl restart ledmatrix-web # Reinstall WiFi monitor sudo ./scripts/install/install_wifi_monitor.sh -# Recreate service files from templates -sudo cp templates/ledmatrix.service /etc/systemd/system/ -sudo cp templates/ledmatrix-web.service /etc/systemd/system/ +# Recreate service files (substitutes __PROJECT_ROOT_DIR__ in systemd/ units) +sudo ./scripts/install/install_service.sh +sudo ./scripts/install/install_web_service.sh -# Reload and restart -sudo systemctl daemon-reload +# Restart sudo systemctl restart ledmatrix ledmatrix-web ``` diff --git a/docs/WEB_INTERFACE_GUIDE.md b/docs/WEB_INTERFACE_GUIDE.md index 09afc494..c57691b9 100644 --- a/docs/WEB_INTERFACE_GUIDE.md +++ b/docs/WEB_INTERFACE_GUIDE.md @@ -39,12 +39,18 @@ present: - **WiFi** — Network selection and AP-mode setup - **Schedule** — Power and dim schedules - **Display** — Matrix hardware configuration (rows, cols, hardware - mapping, GPIO slowdown, brightness, PWM) + mapping, GPIO slowdown, brightness, PWM) and Vegas Scroll Mode + settings +- **Rotation** — drag-and-drop **Rotation Order** list and per-plugin + **Screen Durations** - **Config Editor** — Raw `config.json` editor with validation +- **Backup & Restore** — back up and restore your configuration - **Fonts** — Upload and manage fonts - **Logs** — Real-time log streaming - **Cache** — Cached data inspection and cleanup - **Operation History** — Recent service operations +- **Tools** — system diagnostics, git & updates, Python dependencies, + maintenance, power supply, network radio, services, and plugin health A second nav row holds plugin tabs: @@ -111,6 +117,12 @@ Configure your LED matrix hardware: - Dynamic Duration — global cap for plugins that extend their display time based on content +**Vegas Scroll Mode:** the Display tab also has a full Vegas Scroll +Mode section — enable toggle, scroll speed, separator width, dynamic +duration, and related settings — so you can configure Vegas mode +entirely from the web UI without hand-editing JSON. See +[ADVANCED_FEATURES.md](ADVANCED_FEATURES.md) for what the options do. + Changes require **Restart Display Service** from the Overview tab. ### Plugin Manager Tab @@ -159,9 +171,10 @@ Manage fonts for your display: - See font previews - Check font sizes and styles -**Plugin Font Overrides:** -- Set custom fonts for specific plugins -- Override default font choices +**Font Overrides:** +- Overrides are set per display *element* (e.g. a specific score or + clock text element), not per plugin +- Override default font choices for individual elements - Preview font changes **Delete Fonts:** @@ -183,9 +196,11 @@ View real-time system logs: - Filter by plugin or component **Actions:** +- **Refresh**: Reload the log view - **Clear**: Clear the current view - **Download**: Download logs for offline analysis -- **Pause**: Pause auto-scrolling +- **Auto-scroll** checkbox: toggle automatic scrolling to the latest + entries --- @@ -248,7 +263,8 @@ The web interface uses Server-Sent Events (SSE) for real-time updates: **Performance:** - Minimal bandwidth usage - Server-side rendering for fast load times -- Progressive enhancement - works without JavaScript +- The UI is built on Alpine.js and HTMX, so JavaScript must be enabled + in the browser --- @@ -267,17 +283,6 @@ The interface is fully responsive and works on mobile devices: --- -## Keyboard Shortcuts - -Use keyboard shortcuts for faster navigation: - -- **Tab**: Navigate between form fields -- **Enter**: Submit forms -- **Esc**: Close modals -- **Ctrl+F**: Search in logs - ---- - ## API Access The web interface is built on a REST API that you can access programmatically: @@ -288,7 +293,7 @@ http://your-pi-ip:5000/api/v3 ``` The API blueprint mounts at `/api/v3` (see -`web_interface/app.py:144`). All endpoints below are relative to that +`web_interface/app.py:199`). All endpoints below are relative to that base. **Common Endpoints:** diff --git a/.cursor/plans/implement_audit_fixes_plan.md b/docs/archive/CURSOR_PLUGIN_SCHEMA_AUDIT_PLAN.md similarity index 100% rename from .cursor/plans/implement_audit_fixes_plan.md rename to docs/archive/CURSOR_PLUGIN_SCHEMA_AUDIT_PLAN.md diff --git a/docs/PLUGIN_CUSTOM_ICONS_FEATURE.md b/docs/archive/PLUGIN_CUSTOM_ICONS_FEATURE.md similarity index 100% rename from docs/PLUGIN_CUSTOM_ICONS_FEATURE.md rename to docs/archive/PLUGIN_CUSTOM_ICONS_FEATURE.md diff --git a/docs/widget-guide.md b/docs/widget-guide.md index dadda4cf..9bc271eb 100644 --- a/docs/widget-guide.md +++ b/docs/widget-guide.md @@ -206,6 +206,81 @@ To use an existing widget in your plugin's `config_schema.json`, simply add the The widget will be automatically rendered when the plugin configuration form is loaded. +## Labelling Enum Options (`x-options.labels`) + +A plain `enum` renders as a dropdown whose option text is the value with +underscores replaced and title case applied — `day_first` becomes "Day First". +That is fine for values that read as their own label, and wrong for values that +do not: `vs` becomes "Vs", and `abbrev` says nothing about the `Sep 19` it +actually produces. + +Supply `x-options.labels` to set the visible text. This is the same convention +the `checkbox-group` widget uses: + +```json +{ + "date_format": { + "type": "string", + "enum": ["abbrev", "numeric", "day_first"], + "default": "abbrev", + "x-options": { + "labels": { + "abbrev": "Sep 19", + "numeric": "9/19", + "day_first": "19 Sep" + } + } + } +} +``` + +Labels are **display only** — the stored value is still the enum value, so +adding them never changes a saved config. The map may be partial: any value +without a label keeps the humanised fallback. Older cores that predate this +support ignore `x-options` and render the fallback for every option, so a +plugin can ship labels without requiring a core upgrade. + +Array-table columns (`x-widget: array-table`) accept the same +`x-options.labels` on a column definition, but their fallback is the **raw +value** rather than the humanised one, because those columns hold values such +as ticker symbols where `aapl` → "Aapl" would be wrong. Rows added in the +browser use the labels too (`array-table.js`), so a column reads the same +before and after a page reload. + +## Marking Fields as Advanced (`x-advanced`) + +Add `"x-advanced": true` to any top-level, non-object property to move it out +of the main form and into a single collapsed **Advanced Settings** section at +the bottom of the plugin's configuration page: + +```json +{ + "properties": { + "city": { + "type": "string", + "title": "City" + }, + "request_timeout": { + "type": "integer", + "default": 10, + "description": "HTTP timeout in seconds", + "x-advanced": true + } + } +} +``` + +Guidelines: + +- Use it for fine-tuning knobs most users never touch (timeouts, retry + behavior, cache TTLs, styling overrides). Anything a first-time user must + set to get the plugin working should stay basic. +- Nothing is hidden permanently — the section expands on click, and the + settings search finds and auto-expands advanced fields like any others. +- The flag is ignored on `object`-type properties (they already render as + their own collapsible sections) and is safely ignored by older cores, so + adding it never breaks compatibility. + ## Creating Custom Widgets ### Step 1: Create Widget File diff --git a/first_time_install.sh b/first_time_install.sh old mode 100644 new mode 100755 index bfc3a631..c7ec7cf1 --- a/first_time_install.sh +++ b/first_time_install.sh @@ -152,6 +152,8 @@ ASSUME_YES=${LEDMATRIX_ASSUME_YES:-0} SKIP_SOUND=${LEDMATRIX_SKIP_SOUND:-0} SKIP_PERF=${LEDMATRIX_SKIP_PERF:-0} SKIP_REBOOT_PROMPT=${LEDMATRIX_SKIP_REBOOT_PROMPT:-0} +SKIP_SWAP=${LEDMATRIX_SKIP_SWAP:-0} +BUILD_JOBS_OVERRIDE=${LEDMATRIX_BUILD_JOBS:-} usage() { cat </dev/null 2>&1 && CPU_CORES=$(nproc) || CPU_CORES=1 + + # Validated up front rather than trusted: a non-numeric value would other- + # wise survive as far as an arithmetic test in Step 6 and fail there with a + # generic error. This must precede the fallback return below, which also + # honours the override. + if [ -n "$BUILD_JOBS_OVERRIDE" ]; then + if ! echo "$BUILD_JOBS_OVERRIDE" | grep -qE '^[1-9][0-9]*$'; then + echo "✗ Invalid build job count: '$BUILD_JOBS_OVERRIDE' (expected a positive integer)" + exit 1 + fi + fi + + if [ "$LOWMEM_AVAILABLE" != "1" ]; then + TOTAL_RAM_MB=0 + TOTAL_SWAP_MB=0 + LOW_RAM=0 + BUILD_JOBS=${BUILD_JOBS_OVERRIDE:-$CPU_CORES} + return 0 + fi + + TOTAL_RAM_MB=$(lm_total_ram_mb) + TOTAL_SWAP_MB=$(lm_total_swap_mb) + + # Test hook: exercise the low-memory path on a machine that has plenty. + if [ -n "${LEDMATRIX_FORCE_LOW_RAM:-}" ] && [ "${LEDMATRIX_FORCE_LOW_RAM}" != "0" ]; then + TOTAL_RAM_MB="${LEDMATRIX_FORCE_LOW_RAM}" + echo "⚠ LEDMATRIX_FORCE_LOW_RAM set: pretending this device has ${TOTAL_RAM_MB}MB of RAM" + fi + + LOW_RAM=0 + if [ "$TOTAL_RAM_MB" -gt 0 ] && [ "$TOTAL_RAM_MB" -lt 2048 ]; then + LOW_RAM=1 + fi + + if [ -n "$BUILD_JOBS_OVERRIDE" ]; then + BUILD_JOBS="$BUILD_JOBS_OVERRIDE" + else + BUILD_JOBS=$(lm_build_jobs "$TOTAL_RAM_MB" "$CPU_CORES") + fi + + echo "System memory: ${TOTAL_RAM_MB}MB RAM, ${TOTAL_SWAP_MB}MB swap, ${CPU_CORES} core(s)" + if [ "$LOW_RAM" = "1" ]; then + echo "⚠ Low-memory device detected." + echo " The rpi-rgb-led-matrix C++ build in Step 6 will use ${BUILD_JOBS} parallel job(s)" + echo " instead of all cores, and a temporary swapfile will be added for the build" + echo " and removed afterwards. Without this the compiler is killed by the kernel" + echo " out-of-memory killer. Expect Step 6 to take 15-25 minutes." + if [ "$SKIP_SWAP" = "1" ]; then + echo " Temporary swap is disabled (--skip-swap); the build may still run out of memory." + fi + else + echo "✓ Memory sufficient for the rpi-rgb-led-matrix build (${BUILD_JOBS} parallel job(s))" + fi +} + +# Compile and install the rgbmatrix Python package. +# +# CMAKE_BUILD_PARALLEL_LEVEL is the setting that actually caps the compile: +# upstream's pyproject.toml declares no [tool.scikit-build] options, so +# scikit-build-core drives Ninja through `cmake --build`, which reads this +# variable. Ninja's own default is nproc+2, i.e. six concurrent cc1plus +# processes on a 4-core Pi. MAKEFLAGS is ignored by Ninja and is set only to +# cover the Makefile-generator fallback if ninja-build is somehow absent. +# +# BUILD_TMPDIR redirects pip's build tree off tmpfs where applicable — see +# where it is computed in Step 6. +run_rgbmatrix_build() { + local jobs="$1" out="$2" + local pid elapsed=0 + + TMPDIR="${BUILD_TMPDIR:-${TMPDIR:-/tmp}}" \ + CMAKE_BUILD_PARALLEL_LEVEL="$jobs" \ + MAKEFLAGS="-j${jobs}" \ + python3 -m pip install --break-system-packages . > "$out" 2>&1 & + pid=$! + + # The build's output is captured to a file, so without a heartbeat a serial + # compile on a 1GB Pi looks like a 20-minute hang and invites a Ctrl-C. + # + # Polled at a short interval but reported every 30s: polling at the report + # interval instead would add most of that interval to the wall time of + # every build, including fast ones on a Pi 4/5. + while kill -0 "$pid" 2>/dev/null; do + sleep 2 + elapsed=$((elapsed + 2)) + if [ "$((elapsed % 30))" -eq 0 ] && kill -0 "$pid" 2>/dev/null; then + printf ' ... still compiling (%dm%02ds elapsed)\n' "$((elapsed / 60))" "$((elapsed % 60))" + fi + done + + wait "$pid" +} + +# Explain a failed rgbmatrix build. The kernel OOM killer writes nothing to the +# build's own output, which is why this used to be reported as a missing +# build-tools problem and sent users chasing packages they already had. +print_rgbmatrix_build_failure() { + local out="$1" + + if [ "$LOWMEM_AVAILABLE" = "1" ] && lm_build_failed_on_oom "$out"; then + echo "✗ The rpi-rgb-led-matrix build was killed: the system ran out of memory." + echo " This is NOT a missing build-tools problem — the C++ compiler ran out of RAM." + echo " RAM: ${TOTAL_RAM_MB}MB Swap: $(lm_total_swap_mb)MB Parallel jobs used: ${BUILD_JOBS}" + if [ -n "${LM_SWAP_SKIP_REASON:-}" ]; then + echo " No temporary swap was added: ${LM_SWAP_SKIP_REASON}" + fi + echo "" + echo " Try one of these, then re-run this script (it resumes at Step 6):" + echo " 1. Force a single compile job:" + echo " sudo ./first_time_install.sh --build-jobs 1" + echo " 2. Add permanent swap, if the temporary swapfile could not be created:" + echo " sudo apt install -y dphys-swapfile" + echo " sudo sed -i 's/^#\\?CONF_SWAPSIZE=.*/CONF_SWAPSIZE=2048/' /etc/dphys-swapfile" + echo " sudo sed -i 's/^#\\?CONF_MAXSWAP=.*/CONF_MAXSWAP=2048/' /etc/dphys-swapfile" + echo " sudo dphys-swapfile swapoff && sudo dphys-swapfile setup && sudo dphys-swapfile swapon" + echo " 3. Free up disk space so a larger swapfile fits: sudo apt clean" + else + echo "✗ Failed to install rpi-rgb-led-matrix Python package" + echo " Ensure build tools are installed:" + echo " sudo apt install -y python-dev-is-python3 cmake build-essential" + fi +} + echo "" echo "This script will perform the following steps:" -echo "1. Install system dependencies" +echo "1. Check prerequisites (network, disk, memory) and install system dependencies" echo "2. Fix cache permissions" echo "3. Fix assets directory permissions" echo "3.1. Fix plugin directory permissions" echo "4. Ensure configuration files exist" echo "5. Install Python project dependencies (requirements.txt)" echo "6. Build and install rpi-rgb-led-matrix and test import" +echo " (compiles C++; low-memory Pis get temporary swap and a serial build)" echo "7. Install web interface dependencies" echo "7.5. Install main LED Matrix service" echo "8. Install web interface service" @@ -315,9 +482,16 @@ echo "----------------------------------------" # Pre-flight checks before APT operations check_network check_disk_space +check_memory -# Update package list -apt_update +# Update package list. The one-shot installer refreshes the lists moments +# before invoking this script and exports LEDMATRIX_APT_UPDATED=1, so skip the +# duplicate refresh on that path. +if [ "${LEDMATRIX_APT_UPDATED:-0}" = "1" ]; then + echo "Package lists already refreshed by the one-shot installer; skipping apt update." +else + apt_update +fi # Install required system packages echo "Installing Python packages and dependencies..." @@ -647,10 +821,6 @@ if [ ! -f "$PROJECT_ROOT_DIR/config/config_secrets.json" ]; then echo "⚠ Template config/config_secrets.template.json not found; creating a minimal secrets file" cat > "$PROJECT_ROOT_DIR/config/config_secrets.json" <<'EOF' { - "youtube": { - "api_key": "YOUR_YOUTUBE_API_KEY", - "channel_id": "YOUR_YOUTUBE_CHANNEL_ID" - }, "github": { "api_token": "YOUR_GITHUB_PERSONAL_ACCESS_TOKEN" } @@ -902,29 +1072,66 @@ else fi fi + # Add temporary swap on low-memory devices so the compiler survives. + CURRENT_STEP="Prepare the low-memory build environment" + if [ "$LOWMEM_AVAILABLE" = "1" ] && [ "$SKIP_SWAP" != "1" ]; then + lm_ensure_build_swap "$(lm_swap_needed_mb "$TOTAL_RAM_MB" "$TOTAL_SWAP_MB")" + elif [ "$SKIP_SWAP" = "1" ]; then + LM_SWAP_SKIP_REASON="disabled with --skip-swap" + fi + + # pip builds in $TMPDIR. Debian 13 mounts /tmp as tmpfs, so the default + # would hold the entire C++ build tree in RAM — competing with the very + # compiler we are trying to keep under the memory limit. + BUILD_TMPDIR="" + if [ "$LOWMEM_AVAILABLE" = "1" ]; then + _disk_tmp=$(lm_disk_backed_tmpdir) + if [ -n "$_disk_tmp" ]; then + BUILD_TMPDIR="$_disk_tmp/ledmatrix-build" + # If this fails (a nearly-full disk being the likely cause on + # exactly the devices this targets), fall back to the default + # rather than pointing the build at a path that does not exist. + if mkdir -p "$BUILD_TMPDIR" 2>/dev/null; then + echo "Building in $BUILD_TMPDIR (TMPDIR is memory-backed; keeping the build tree on disk)" + else + echo "⚠ Could not create $BUILD_TMPDIR; falling back to the default TMPDIR" + BUILD_TMPDIR="" + fi + fi + fi + + CURRENT_STEP="Build and install rpi-rgb-led-matrix" pushd "$PROJECT_ROOT_DIR/rpi-rgb-led-matrix-master" >/dev/null echo "Installing rpi-rgb-led-matrix Python package (scikit-build-core + cmake)..." echo " Build deps required: python-dev-is-python3 cmake" - echo " This compiles C++ — may take 2-5 minutes on Pi 4/5..." + echo " Compiling C++ with ${BUILD_JOBS} parallel job(s)..." + if [ "$BUILD_JOBS" -le 1 ]; then + echo " Deliberately serial to stay within this device's memory — expect 15-25 minutes." + else + echo " This may take 2-5 minutes on a Pi 4/5..." + fi BUILD_OUTPUT=$(mktemp) BUILD_SUCCESS=false - if python3 -m pip install --break-system-packages . > "$BUILD_OUTPUT" 2>&1; then + if run_rgbmatrix_build "$BUILD_JOBS" "$BUILD_OUTPUT"; then BUILD_SUCCESS=true fi cat "$BUILD_OUTPUT" >> "$LOG_FILE" if [ "$BUILD_SUCCESS" != true ]; then - echo "✗ Failed to install rpi-rgb-led-matrix Python package" - echo " Ensure build tools are installed:" - echo " sudo apt install -y python-dev-is-python3 cmake build-essential" + print_rgbmatrix_build_failure "$BUILD_OUTPUT" echo "" echo "-- Last 50 lines of build output --" tail -n 50 "$BUILD_OUTPUT" rm -f "$BUILD_OUTPUT" + if [ -n "$BUILD_TMPDIR" ]; then rm -rf "$BUILD_TMPDIR"; fi popd >/dev/null + lm_remove_build_swap exit 1 fi rm -f "$BUILD_OUTPUT" + if [ -n "$BUILD_TMPDIR" ]; then rm -rf "$BUILD_TMPDIR"; fi popd >/dev/null + # Hand the memory back well before Step 14's reboot. + lm_remove_build_swap else echo "✗ rpi-rgb-led-matrix-master directory not found at $PROJECT_ROOT_DIR" echo "Failed to initialize submodule or clone repository" @@ -976,25 +1183,26 @@ else cd "$PROJECT_ROOT_DIR" # Try to install dependencies using the smart installer if available + WEB_DEPS_OK=true if [ -f "$PROJECT_ROOT_DIR/scripts/install_dependencies_apt.py" ]; then echo "Using smart dependency installer..." # -u: unbuffered stdout/stderr so output is captured in $LOG_FILE in # real time and in order relative to this script's own echo statements - python3 -u "$PROJECT_ROOT_DIR/scripts/install_dependencies_apt.py" - else - echo "Using pip to install dependencies..." - if [ -f "$PROJECT_ROOT_DIR/requirements_web_v2.txt" ]; then - # --ignore-installed: see the Step 5 web_interface/requirements.txt - # install above — same apt/pip RECORD-file conflict applies here. - python3 -m pip install --break-system-packages --prefer-binary --ignore-installed -r requirements_web_v2.txt - else - echo "⚠ requirements_web_v2.txt not found; skipping web dependency install" + if ! python3 -u "$PROJECT_ROOT_DIR/scripts/install_dependencies_apt.py"; then + WEB_DEPS_OK=false fi + else + echo "Web dependencies already installed from web_interface/requirements.txt in Step 5" fi - # Create marker file to indicate dependencies are installed - touch "$PROJECT_ROOT_DIR/.web_deps_installed" - echo "✓ Web interface dependencies installed" + # Create the marker only when installation actually succeeded, so a + # re-run retries instead of silently skipping missing dependencies. + if [ "$WEB_DEPS_OK" = true ]; then + touch "$PROJECT_ROOT_DIR/.web_deps_installed" + echo "✓ Web interface dependencies installed" + else + echo "⚠ Web interface dependency install reported errors; not creating .web_deps_installed (will retry on next run)" + fi fi echo "" @@ -1211,9 +1419,16 @@ $ACTUAL_USER ALL=(ALL) NOPASSWD: $BASH_PATH $PROJECT_ROOT_DIR/scripts/fix_perms/ EOF if [ -n "$JOURNALCTL_PATH" ]; then cat >> /tmp/ledmatrix_web_sudoers << EOF -$ACTUAL_USER ALL=(ALL) NOPASSWD: $JOURNALCTL_PATH -u ledmatrix.service * -$ACTUAL_USER ALL=(ALL) NOPASSWD: $JOURNALCTL_PATH -u ledmatrix * -$ACTUAL_USER ALL=(ALL) NOPASSWD: $JOURNALCTL_PATH -t ledmatrix * +# NOEXEC, because these rules end in a wildcard and journalctl starts a pager +# when its output is a terminal. From that pager (less) a "!sh" is a root +# shell -- the standard journalctl escalation. The web interface always passes +# --no-pager, so nothing here needs it, but the rule cannot require a flag that +# sits in the middle of the command line. NOEXEC stops the command executing +# another program at all, which closes the hole without depending on wildcard +# matching subtleties. +$ACTUAL_USER ALL=(ALL) NOPASSWD:NOEXEC: $JOURNALCTL_PATH -u ledmatrix.service * +$ACTUAL_USER ALL=(ALL) NOPASSWD:NOEXEC: $JOURNALCTL_PATH -u ledmatrix * +$ACTUAL_USER ALL=(ALL) NOPASSWD:NOEXEC: $JOURNALCTL_PATH -t ledmatrix * EOF fi @@ -1272,27 +1487,54 @@ if [ -f "$PROJECT_ROOT_DIR/config/config.json" ]; then fi # Set proper permissions for secrets file (restrictive: owner rw, group r) -# If service runs as root, set ownership to root so it can read as owner -# Otherwise, use ACTUAL_USER and rely on group membership +# Owned by whoever WRITES the file, which is the web interface. +# +# This used to read the User= of ledmatrix.service — the display service — +# and, finding root, hand the file to root:ledmatrix 640. But the display +# service only ever reads secrets, and root can read any file regardless of +# mode. The account that *writes* them is the web interface: it saves config +# edits and performs backup restores, and it deliberately does not run as root +# (a web server should not). So a root-owned, group-read-only file left the web +# UI unable to write its own secrets, and restoring a backup failed with +# "Permission denied: config_secrets.json" while every other file in the same +# backup restored fine. +# +# Owning by the writer keeps the tighter 640 rather than loosening to +# group-writable, and root still reads it as superuser. if [ -f "$PROJECT_ROOT_DIR/config/config_secrets.json" ]; then - # Check if service runs as root (from service file or template) - SERVICE_USER="root" - if [ -f "/etc/systemd/system/ledmatrix.service" ]; then - SERVICE_USER=$(grep "^User=" /etc/systemd/system/ledmatrix.service | cut -d'=' -f2 || echo "root") - elif [ -f "$PROJECT_ROOT_DIR/systemd/ledmatrix.service" ]; then - SERVICE_USER=$(grep "^User=" "$PROJECT_ROOT_DIR/systemd/ledmatrix.service" | cut -d'=' -f2 || echo "root") + # The web service is the writer; fall back to the display service, then to + # the installing user, so an unusual layout still lands somewhere sensible. + SECRETS_OWNER="" + for unit in "/etc/systemd/system/ledmatrix-web.service" \ + "$PROJECT_ROOT_DIR/systemd/ledmatrix-web.service"; do + if [ -f "$unit" ]; then + SECRETS_OWNER=$(grep -m1 "^User=" "$unit" | cut -d'=' -f2) + [ -n "$SECRETS_OWNER" ] && break + fi + done + if [ -z "$SECRETS_OWNER" ]; then + SECRETS_OWNER="$ACTUAL_USER" fi - - if [ "$SERVICE_USER" = "root" ]; then - # Service runs as root - set ownership to root so it can read as owner - chown "root:$LEDMATRIX_GROUP" "$PROJECT_ROOT_DIR/config/config_secrets.json" || true - echo "✓ Secrets file permissions set (root:ledmatrix for root service)" - else - # Service runs as regular user - use ACTUAL_USER and rely on group membership - chown "$ACTUAL_USER:$LEDMATRIX_GROUP" "$PROJECT_ROOT_DIR/config/config_secrets.json" || true - echo "✓ Secrets file permissions set ($ACTUAL_USER:ledmatrix)" + SECRETS_FILE="$PROJECT_ROOT_DIR/config/config_secrets.json" + # A root-owned file is only correct when the writer really is root. + if ! chown "$SECRETS_OWNER:$LEDMATRIX_GROUP" "$SECRETS_FILE"; then + echo "✗ ERROR: Failed to set ownership on $SECRETS_FILE to $SECRETS_OWNER:$LEDMATRIX_GROUP" >&2 + echo " Try: sudo chown $SECRETS_OWNER:$LEDMATRIX_GROUP $SECRETS_FILE" >&2 + exit 1 fi - chmod 640 "$PROJECT_ROOT_DIR/config/config_secrets.json" + if ! chmod 640 "$SECRETS_FILE"; then + echo "✗ ERROR: Failed to set permissions on $SECRETS_FILE to 640" >&2 + echo " Try: sudo chmod 640 $SECRETS_FILE" >&2 + exit 1 + fi + ACTUAL_OWNERSHIP=$(stat -c '%U:%G' "$SECRETS_FILE" 2>/dev/null || echo "unknown") + ACTUAL_MODE=$(stat -c '%a' "$SECRETS_FILE" 2>/dev/null || echo "unknown") + if [ "$ACTUAL_OWNERSHIP" != "$SECRETS_OWNER:$LEDMATRIX_GROUP" ] || [ "$ACTUAL_MODE" != "640" ]; then + echo "✗ ERROR: $SECRETS_FILE ended up as $ACTUAL_OWNERSHIP mode $ACTUAL_MODE, expected $SECRETS_OWNER:$LEDMATRIX_GROUP mode 640" >&2 + echo " The web interface may be unable to read or write config_secrets.json." >&2 + exit 1 + fi + echo "✓ Secrets file owned by the web service user ($SECRETS_OWNER:$LEDMATRIX_GROUP, mode 640)" fi # Set proper permissions for YTM auth file (readable by all users including root service) @@ -1453,6 +1695,94 @@ else echo "✗ $CMDLINE_FILE not found; skipping isolcpus optimization" fi +# Enable the memory cgroup controller (idempotent). +# The Pi firmware boots with cgroup_disable=memory, so systemd's MemoryMax= is +# accepted and silently ignored — the display service then has no ceiling, and +# a runaway takes the whole board down (sshd can no longer fork, the panel goes +# dark) rather than just restarting the one service. +if [ "$SKIP_PERF" != "1" ] && [ -f "$CMDLINE_FILE" ]; then + # Both parameters are required for the memory controller, and they can get + # separated -- an image, another tool or a half-applied earlier run can + # leave one without the other. Checking only cgroup_enable=memory would + # report success while MemoryMax= silently does nothing, so each is checked + # and appended independently. + cgroup_missing="" + for cgroup_param in cgroup_enable=memory cgroup_memory=1; do + if ! grep -qw "$cgroup_param" "$CMDLINE_FILE"; then + cgroup_missing="$cgroup_missing $cgroup_param" + fi + done + if [ -z "$cgroup_missing" ]; then + echo "cgroup memory parameters already present in $CMDLINE_FILE" + else + echo "Adding${cgroup_missing} to $CMDLINE_FILE..." + cp "$CMDLINE_FILE" "$CMDLINE_FILE.bak" 2>/dev/null || true + # The kernel command line must stay on one line. + sed -i "1 s|\$|${cgroup_missing}|" "$CMDLINE_FILE" + echo " Takes effect after reboot. Verify with:" + echo " grep memory /sys/fs/cgroup/cgroup.controllers" + fi +fi + +# Persist the journal (idempotent). +# These images default to volatile storage: journald keeps everything in /run +# (tmpfs), so every reboot destroys the logs — including the ones that would +# explain why the board rebooted. Capped so an SD card is not worn out by logs. +# A non-empty /var/log/journal does not prove journald is configured the way +# this needs: the directory survives a switch back to volatile storage, and it +# says nothing about whether a size cap is set. Read the effective +# configuration instead, and only write the keys the user has not set +# themselves so an explicit local limit is preserved. +journald_effective() { + # systemd-analyze merges journald.conf with every drop-in; grep is the + # fallback for images that ship without it. + if command -v systemd-analyze >/dev/null 2>&1 && + systemd-analyze cat-config systemd/journald.conf >/dev/null 2>&1; then + systemd-analyze cat-config systemd/journald.conf 2>/dev/null + else + cat /etc/systemd/journald.conf /etc/systemd/journald.conf.d/*.conf 2>/dev/null + fi +} +journald_conf="$(journald_effective)" +journald_storage="$(printf '%s\n' "$journald_conf" | grep -E '^[[:space:]]*Storage=' | tail -n1 | cut -d= -f2 | tr -d '[:space:]')" +journald_cap="$(printf '%s\n' "$journald_conf" | grep -E '^[[:space:]]*SystemMaxUse=' | tail -n1 | cut -d= -f2 | tr -d '[:space:]')" + +if [ "$journald_storage" = "persistent" ] && [ -n "$journald_cap" ]; then + echo "Persistent journald storage already configured (SystemMaxUse=$journald_cap)" +else + echo "Enabling persistent journald storage..." + mkdir -p /etc/systemd/journald.conf.d + { + echo "# Installed by LEDMatrix first_time_install.sh" + echo "[Journal]" + echo "Storage=persistent" + if [ -n "$journald_cap" ]; then + echo "# SystemMaxUse left to your existing setting ($journald_cap)" + else + # Capped so logs cannot wear out or fill an SD card. + echo "SystemMaxUse=64M" + fi + } > /etc/systemd/journald.conf.d/ledmatrix-persistent.conf + mkdir -p /var/log/journal + systemd-tmpfiles --create --prefix /var/log/journal >/dev/null 2>&1 || true + systemctl restart systemd-journald >/dev/null 2>&1 || true + + # Drop-ins are applied in lexical order, so a locally added file that sorts + # after ledmatrix-persistent.conf (zz-local.conf and friends) still wins. + # Writing the file is not evidence it took effect -- re-read and say so + # plainly rather than reporting success we cannot confirm. + journald_now="$(journald_effective | grep -E '^[[:space:]]*Storage=' | tail -n1 | cut -d= -f2 | tr -d '[:space:]')" + if [ "$journald_now" = "persistent" ]; then + echo " Persistent journald storage active" + else + echo " WARNING: journald storage is still '${journald_now:-unset}' after" + echo " writing /etc/systemd/journald.conf.d/ledmatrix-persistent.conf." + echo " Another drop-in that sorts later is overriding it. Check:" + echo " systemd-analyze cat-config systemd/journald.conf | grep -n Storage=" + echo " Logs will not survive a reboot until that is resolved." + fi +fi + # Ensure dtparam=audio=off in config.txt (idempotent) if [ "$SKIP_PERF" = "1" ]; then : # skipped diff --git a/pytest.ini b/pytest.ini index a13cc6cb..aaffa360 100644 --- a/pytest.ini +++ b/pytest.ini @@ -10,16 +10,13 @@ python_functions = test_* testpaths = test # Output options -# Note: Coverage options require pytest-cov to be installed -# Run: pip install pytest-cov -addopts = +# Coverage is deliberately NOT configured here: a bare local `pytest` should +# be fast and dependency-light. Coverage is measured and enforced in exactly +# one place — the unit-tests job in .github/workflows/test.yml. +addopts = -v --strict-markers --tb=short - --cov=src - --cov-report=term-missing - --cov-report=html - --cov-fail-under=30 # Markers markers = diff --git a/requirements-test.txt b/requirements-test.txt index a11b33a4..60e26951 100644 --- a/requirements-test.txt +++ b/requirements-test.txt @@ -1,9 +1,9 @@ -# Test-only dependencies for the plugin safety harness and pytest suite. +# Test/dev-only dependencies (not needed on a running display). # Install alongside requirements.txt: pip install -r requirements.txt -r requirements-test.txt -# -# pytest, pytest-cov, pytest-mock, and jsonschema are already pinned (with -# major-version caps) in requirements.txt, so they are intentionally NOT -# repeated here — re-pinning pytest to <9 collided with requirements.txt's -# pytest>=9.0.3,<10 and made the two files impossible to install together. -# Only declare what requirements.txt doesn't already provide. +pytest>=9.0.3,<10.0.0 +pytest-cov>=4.1.0,<5.0.0 +pytest-mock>=3.11.0,<4.0.0 freezegun>=1.2,<2 # deterministic time for golden-image tests +psutil>=6.0.0,<8.0.0 # optional at runtime; installed for tests so the + # /system/status endpoint's real path is exercised +mypy>=1.5.0,<2.0.0 # static type checking (also pinned in .pre-commit-config.yaml) diff --git a/requirements.txt b/requirements.txt index 5e1bf365..760dc596 100644 --- a/requirements.txt +++ b/requirements.txt @@ -8,37 +8,25 @@ numpy>=1.24.0 # For fast array operations in ScrollHelper (compatible with 2.x) # Timezone handling pytz>=2024.2,<2025.0 # Updated for latest timezone data -timezonefinder>=6.5.0,<7.0.0 # Updated for better performance and accuracy -geopy>=2.4.1,<3.0.0 # HTTP requests requests>=2.33.0,<3.0.0 +urllib3>=2.7.0,<3.0.0 # requests transitive, but imported directly (urllib3.util.retry.Retry); floor is a security floor, not the API floor — 1.26.x carries ~10 CVEs # Google API integration -google-auth-oauthlib>=1.2.0,<2.0.0 -google-auth-httplib2>=0.2.0,<1.0.0 -google-api-python-client>=2.147.0,<3.0.0 # Font rendering freetype-py>=2.5.1,<3.0.0 -# Spotify integration +# Spotify integration (used by web_interface/blueprints/api_v3.py OAuth endpoints) spotipy>=2.25.2,<3.0.0 # Flask web framework Flask>=3.1.3,<4.0.0 -# Text processing -unidecode>=1.3.8,<2.0.0 - -# Calendar integration -icalevents>=0.1.27,<1.0.0 - -# WebSocket support -python-socketio>=5.14.0,<6.0.0 -python-engineio>=4.9.0,<5.0.0 -websockets>=12.0,<14.0 -websocket-client>=1.8.0,<2.0.0 +# WebSocket support: intentionally NOT declared here. Plugins that need +# it (e.g. ledmatrix-music's Socket.IO client) declare it in their own +# requirements.txt, which the plugin store installs. # JSON Schema validation jsonschema>=4.20.0,<5.0.0 @@ -46,11 +34,8 @@ jsonschema>=4.20.0,<5.0.0 # Requirement specifier parsing (plugin dependency satisfaction checks) packaging>=23.0,<27.0 -# Testing dependencies -pytest>=9.0.3,<10.0.0 -pytest-cov>=4.1.0,<5.0.0 -pytest-mock>=3.11.0,<4.0.0 -mypy>=1.5.0,<2.0.0 +# Testing dependencies live in requirements-test.txt: +# pip install -r requirements.txt -r requirements-test.txt # ─────────────────────────────────────────────────────────────────────── # Optional dependencies — the code imports these inside try/except @@ -66,7 +51,9 @@ mypy>=1.5.0,<2.0.0 # psutil — per-plugin resource monitoring in # src/plugin_system/resource_monitor.py. The monitor # silently no-ops when missing (PSUTIL_AVAILABLE = False). -# pip install 'psutil>=5.9.0,<6.0.0' +# Note: web_interface/requirements.txt requires this +# range as a hard dependency — keep the two in sync. +# pip install 'psutil>=6.0.0,<7.0.0' # # Flask-Limiter — request rate limiting in web_interface/app.py # (accidental-abuse protection, not security). The diff --git a/scripts/add_defaults_to_schemas.py b/scripts/add_defaults_to_schemas.py index c0474530..3ef33e94 100755 --- a/scripts/add_defaults_to_schemas.py +++ b/scripts/add_defaults_to_schemas.py @@ -201,7 +201,7 @@ def process_schema_file(schema_path: Path) -> bool: def main(): """Main entry point.""" project_root = Path(__file__).parent.parent - plugins_dir = project_root / 'plugins' + plugins_dir = project_root / 'plugin-repos' if not plugins_dir.exists(): print(f"Error: Plugins directory not found: {plugins_dir}") diff --git a/scripts/analyze_plugin_schemas.py b/scripts/analyze_plugin_schemas.py index d10a308f..e05f20eb 100755 --- a/scripts/analyze_plugin_schemas.py +++ b/scripts/analyze_plugin_schemas.py @@ -193,7 +193,7 @@ def analyze_schema(schema_path: Path) -> Dict[str, Any]: def main(): """Main analysis function.""" project_root = Path(__file__).parent.parent - plugins_dir = project_root / "plugins" + plugins_dir = project_root / "plugin-repos" if not plugins_dir.exists(): print(f"Plugins directory not found: {plugins_dir}") diff --git a/scripts/audit_plugins.py b/scripts/audit_plugins.py new file mode 100644 index 00000000..79a61acf --- /dev/null +++ b/scripts/audit_plugins.py @@ -0,0 +1,344 @@ +#!/usr/bin/env python3 +""" +LEDMatrix Plugin Security Auditor + +Performs AST-based security analysis of all Python files in plugin directories. +Designed to run in CI — exits non-zero on CRITICAL findings only. + +Usage: + python scripts/audit_plugins.py + python scripts/audit_plugins.py --verbose + python scripts/audit_plugins.py --plugin hello-world + python scripts/audit_plugins.py --output results.json +""" + +import ast +import argparse +import json +import sys +from dataclasses import dataclass, asdict +from pathlib import Path +from datetime import datetime, timezone + +PROJECT_ROOT = Path(__file__).resolve().parent.parent + +PLUGIN_BASE_DIRS = [ + PROJECT_ROOT / "plugins", + PROJECT_ROOT / "plugin-repos", +] + + +# ───────────────────────────────────────────────────────────────────────────── +# Finding dataclass +# ───────────────────────────────────────────────────────────────────────────── + +@dataclass +class Finding: + plugin_id: str + file: str + line: int + severity: str # CRITICAL | WARNING | INFO + rule: str + message: str + + def to_dict(self) -> dict: + return asdict(self) + + +# ───────────────────────────────────────────────────────────────────────────── +# AST visitor +# ───────────────────────────────────────────────────────────────────────────── + +class _PluginVisitor(ast.NodeVisitor): + """Collect security findings from a single plugin Python file.""" + + def __init__(self, filepath: Path, plugin_id: str): + self.filepath = filepath + self.plugin_id = plugin_id + self.findings: list[Finding] = [] + # Local name -> real dotted path, so aliased imports and from-imports + # of dangerous APIs (import subprocess as sp; from builtins import + # eval as e) are still recognized in visit_Call below. + self._aliases: dict[str, str] = {} + + def _add(self, node: ast.AST, severity: str, rule: str, message: str) -> None: + self.findings.append(Finding( + plugin_id=self.plugin_id, + file=str(self.filepath.relative_to(PROJECT_ROOT)), + line=getattr(node, "lineno", 0), + severity=severity, + rule=rule, + message=message, + )) + + def _resolve(self, local_name: str) -> str: + """Resolve a local name through recorded import aliases to its real + dotted path (e.g. "sp" -> "subprocess"); unresolved names pass through + unchanged.""" + return self._aliases.get(local_name, local_name) + + def _resolve_call_target(self, func: ast.expr) -> str | None: + """Resolve a Call's func node to a fully-qualified dotted target, + covering a direct name (bare builtin, aliased import, or + from-import: from builtins import eval as e; from subprocess + import run; from os import system as s) and module-attribute + access (subprocess.run, sp.run, os.system, o.system) uniformly. + Returns None for call shapes this doesn't attempt to resolve.""" + if isinstance(func, ast.Name): + return self._resolve(func.id) + if isinstance(func, ast.Attribute) and isinstance(func.value, ast.Name): + base = self._resolve(func.value.id) + return f"{base}.{func.attr}" + return None + + def visit_Call(self, node: ast.Call) -> None: + target = self._resolve_call_target(node.func) + if target is None: + self.generic_visit(node) + return + + leaf = target.rsplit(".", 1)[-1] + + # eval() / exec() / compile() — arbitrary code execution, whether a + # bare call, an aliased import, or a from-import + # (from builtins import eval as e; e(...)) + if leaf == "eval": + self._add(node, "CRITICAL", "PLUGIN-001", + "eval() call — arbitrary code execution risk") + elif leaf == "exec": + self._add(node, "CRITICAL", "PLUGIN-002", + "exec() call — arbitrary code execution risk") + elif leaf == "compile": + self._add(node, "WARNING", "PLUGIN-003", + "compile() call — dynamic code compilation") + + # subprocess.*(shell=True), whether subprocess.run(...), sp.run(...), + # or a from-import (from subprocess import run; run(..., shell=True)) + if target in { + "subprocess.run", "subprocess.call", "subprocess.Popen", + "subprocess.check_call", "subprocess.check_output", + }: + for kw in node.keywords: + if (kw.arg == "shell" and + isinstance(kw.value, ast.Constant) and + kw.value.value is True): + self._add(node, "WARNING", "PLUGIN-004", + f"subprocess.{leaf}(shell=True) — " + f"shell injection risk if args include user input") + + # os.system(), whether os.system(...), o.system(...), or a + # from-import (from os import system as s; s(...)) + if target == "os.system": + self._add(node, "WARNING", "PLUGIN-005", + "os.system() call — prefer subprocess with list args") + + self.generic_visit(node) + + def visit_Import(self, node: ast.Import) -> None: + for alias in node.names: + if alias.asname: + local, real = alias.asname, alias.name + else: + # `import os.path` binds the top-level name `os`, not `os.path` + local = real = alias.name.split(".")[0] + self._aliases[local] = real + self._check_import(node, alias.name) + self.generic_visit(node) + + def visit_ImportFrom(self, node: ast.ImportFrom) -> None: + if node.module: + for alias in node.names: + local = alias.asname or alias.name + self._aliases[local] = f"{node.module}.{alias.name}" + self._check_import(node, node.module) + self.generic_visit(node) + + def _check_import(self, node: ast.AST, module_name: str) -> None: + dangerous = { + "ctypes": ("WARNING", "PLUGIN-010", "ctypes import — native code execution"), + "cffi": ("WARNING", "PLUGIN-011", "cffi import — native code execution"), + "pickle": ("WARNING", "PLUGIN-012", + "pickle import — deserialization can execute arbitrary code"), + "marshal": ("WARNING", "PLUGIN-013", + "marshal import — deserialization risk"), + } + for mod, (severity, rule, msg) in dangerous.items(): + if module_name == mod or module_name.startswith(mod + "."): + self._add(node, severity, rule, msg) + + +# ───────────────────────────────────────────────────────────────────────────── +# Per-plugin audit +# ───────────────────────────────────────────────────────────────────────────── + +def audit_plugin(plugin_dir: Path) -> list[Finding]: + """Audit a single plugin directory. Returns all findings.""" + findings: list[Finding] = [] + plugin_id = plugin_dir.name + + # Check for required files + for required_file, rule, msg in [ + ("manifest.json", "PLUGIN-020", + "manifest.json missing — plugin may be incomplete"), + ("config_schema.json", "PLUGIN-021", + "config_schema.json missing — no input validation schema declared"), + ]: + if not (plugin_dir / required_file).exists(): + findings.append(Finding( + plugin_id=plugin_id, + file=str((plugin_dir / required_file).relative_to(PROJECT_ROOT)), + line=0, + severity="WARNING", + rule=rule, + message=msg, + )) + + # AST analysis of all Python files + for py_file in sorted(plugin_dir.rglob("*.py")): + try: + source = py_file.read_text(encoding="utf-8") + tree = ast.parse(source, filename=str(py_file)) + visitor = _PluginVisitor(py_file, plugin_id) + visitor.visit(tree) + findings.extend(visitor.findings) + except SyntaxError as exc: + # A file the visitor can't even parse is a file we can't verify + # is safe -- this must block the audit, not just warn. + findings.append(Finding( + plugin_id=plugin_id, + file=str(py_file.relative_to(PROJECT_ROOT)), + line=getattr(exc, "lineno", 0) or 0, + severity="CRITICAL", + rule="PLUGIN-030", + message=f"Python syntax error — cannot be parsed: {exc}", + )) + except OSError as exc: + # Same reasoning as SyntaxError: an unreadable file was never + # actually scanned, so it must block rather than pass silently. + findings.append(Finding( + plugin_id=plugin_id, + file=str(py_file.relative_to(PROJECT_ROOT)), + line=0, + severity="CRITICAL", + rule="PLUGIN-031", + message=f"Could not read file: {exc}", + )) + + return findings + + +# ───────────────────────────────────────────────────────────────────────────── +# Main +# ───────────────────────────────────────────────────────────────────────────── + +def main() -> int: + parser = argparse.ArgumentParser( + description="LEDMatrix plugin security auditor", + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + parser.add_argument("--plugin", "-p", default=None, + help="Audit a specific plugin ID only") + parser.add_argument("--output", "-o", default=None, + help="Write JSON results to this file") + parser.add_argument("--verbose", "-v", action="store_true", + help="Show all findings, not just summary") + args = parser.parse_args() + + print("=" * 60) + print("LEDMatrix Plugin Security Audit") + print(f"Project root: {PROJECT_ROOT}") + print("=" * 60) + + all_findings: list[Finding] = [] + plugins_scanned = 0 + plugin_found = args.plugin is None + + for base_dir in PLUGIN_BASE_DIRS: + if not base_dir.exists(): + if args.verbose: + print(f" ⏭️ Skipping {base_dir.name}/ (directory not found)") + continue + + base_label = base_dir.relative_to(PROJECT_ROOT) + print(f"\n Scanning {base_label}/") + + for plugin_dir in sorted(base_dir.iterdir()): + if not plugin_dir.is_dir(): + continue + if plugin_dir.name.startswith((".", "_")): + continue + if args.plugin and plugin_dir.name != args.plugin: + continue + if args.plugin: + plugin_found = True + + findings = audit_plugin(plugin_dir) + all_findings.extend(findings) + plugins_scanned += 1 + + critical = [f for f in findings if f.severity == "CRITICAL"] + warnings = [f for f in findings if f.severity == "WARNING"] + + if critical: + icon, label = "🚨", "CRITICAL" + elif warnings: + icon, label = "⚠️ ", "WARN " + else: + icon, label = "✅", "PASS " + + print(f" {icon} [{label}] {plugin_dir.name}" + f" — {len(critical)} critical, {len(warnings)} warnings") + + if args.verbose: + for f in findings: + severity_icon = {"CRITICAL": "🚨", "WARNING": "⚠️ ", "INFO": "ℹ️ "}.get( + f.severity, " " + ) + print(f" {severity_icon} {f.rule} {f.file}:{f.line} — {f.message}") + + if args.plugin and not plugin_found: + print(f"\n 🚨 Plugin '{args.plugin}' not found in any of " + f"{[str(d.relative_to(PROJECT_ROOT)) for d in PLUGIN_BASE_DIRS]} — " + f"nothing was audited") + return 1 + + # Summary + critical_findings = [f for f in all_findings if f.severity == "CRITICAL"] + warning_findings = [f for f in all_findings if f.severity == "WARNING"] + + print(f"\n{'=' * 60}") + print(f" Plugins scanned : {plugins_scanned}") + print(f" CRITICAL : {len(critical_findings)}") + print(f" WARNING : {len(warning_findings)}") + + if critical_findings: + print("\n 🚨 CRITICAL findings:") + for f in critical_findings: + print(f" {f.plugin_id} | {Path(f.file).name}:{f.line} | {f.message}") + + # Write JSON output + if args.output: + output_data = { + "timestamp": datetime.now(timezone.utc).isoformat(), + "plugins_scanned": plugins_scanned, + "summary": { + "critical": len(critical_findings), + "warnings": len(warning_findings), + }, + "findings": [f.to_dict() for f in all_findings], + } + Path(args.output).write_text( + json.dumps(output_data, indent=2), encoding="utf-8" + ) + print(f"\n Results written to: {args.output}") + + if critical_findings: + print("\n 🚨 Blocking — CRITICAL issues must be resolved") + return 1 + + print("\n ✅ No critical issues found") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/check_plugin.py b/scripts/check_plugin.py index 997bf6d8..3f111d4d 100644 --- a/scripts/check_plugin.py +++ b/scripts/check_plugin.py @@ -41,6 +41,7 @@ from src.plugin_system.testing.loading import ( # noqa: E402 ) from src.plugin_system.testing.harness import ( # noqa: E402 RenderResult, render_plugin_matrix, compare_to_goldens, write_goldens, + check_empty_claimed, check_scale_up, ) from src.plugin_system.testing.sizes import ( # noqa: E402 @@ -115,6 +116,11 @@ def check_one(plugin_id: str, search_dirs: List[str], sizes, mock_data: Dict, declared = load_manifest(plugin_dir).get("display", {}).get("design_size", {}) design_size = (int(declared.get("width", 128)), int(declared.get("height", 32))) fill_strict = spec.get("fill_check") == "strict" + # A mode that renders nothing without returning False is never skipped by + # the display controller, so it holds a blank panel for its whole duration. + # Warn-only by default: a scroll mode's first frame is legitimately its + # blank scroll-in buffer. + empty_strict = spec.get("empty_check") == "strict" # Every run: the base config, plus one per harness.json "variant" — # a config overlay with its own golden dir (e.g. adaptive layout mode @@ -142,6 +148,7 @@ def check_one(plugin_id: str, search_dirs: List[str], sizes, mock_data: Dict, compare_to_goldens(results, golden_dir) check_scale_up(results, design_size=design_size, strict=fill_strict) + check_empty_claimed(results, strict=empty_strict) # Tag variant runs so the report and PNG dumps stay distinguishable. if variant_name: @@ -178,6 +185,9 @@ def print_report(all_results: Dict[str, List[RenderResult]]) -> bool: # warn-only underfill: big panel left mostly empty ex, ey = r.fill_extent detail += f" (fill warn: extent {ex:.0%}x{ey:.0%})" + if r.empty_claimed and r.empty_ok is None: + detail += (f" (empty warn: drew nothing but display() returned" + f" {r.display_returned!r}, so the mode is not skipped)") else: everything_ok = False if r.error is not None: @@ -191,6 +201,11 @@ def print_report(all_results: Dict[str, List[RenderResult]]) -> bool: ex, ey = r.fill_extent or (0.0, 0.0) status = "FAIL" detail = f" fill: extent {ex:.0%}x{ey:.0%} below required coverage" + elif r.empty_ok is False: + status = "FAIL" + detail = (f" drew nothing but display() returned" + f" {r.display_returned!r}; return False so the" + f" controller skips the mode") else: status, detail = "FAIL", "" print(f" [{status}] {r.size_label:>7} {r.mode}{detail}") diff --git a/scripts/check_release_version.py b/scripts/check_release_version.py new file mode 100644 index 00000000..45dc0ead --- /dev/null +++ b/scripts/check_release_version.py @@ -0,0 +1,122 @@ +#!/usr/bin/env python3 +"""Assert that a release tag, the CHANGELOG, and `src.__version__` all agree. + +Run it *before* creating a tag to check yourself: + + python scripts/check_release_version.py v3.2.0 + +Wiring it into CI (on pushed `v*` tags and published releases) is a follow-up +PR, so for now it is a manual pre-flight: run it before creating the tag and a +mismatch shows up here rather than as a silent wrong answer on user devices. + +Why this exists: `v3.1.0` was tagged 2026-05-31 while `src/__init__.py` still +said `"1.0.0"`; the bump to `"3.1.0"` did not land until 2026-07-12. Devices +installed from that release report `1.0.0`, which is below the `(2, 0, 0)` floor +in `PluginLoader._warn_if_incompatible`, so they are silently exempt from every +plugin compatibility warning. Plugin `ledmatrix_min_version` floors are only as +trustworthy as this agreement. See `docs/SPORTS_UNIFICATION.md`, phase B4. +""" + +from __future__ import annotations + +import argparse +import re +import sys +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(REPO_ROOT)) + +# [0-9] rather than \d, and [ \t] rather than \s: \d also matches non-ASCII +# decimal digits (which int() parses), and \s matches newlines, so "##\n3.2.0" +# would otherwise read as a version heading. Keep these in step with +# test/test_version_consistency.py. +SEMVER = re.compile(r"^[0-9]+\.[0-9]+\.[0-9]+$") +HEADING = re.compile( + r"^##[ \t]+(?P[0-9]+\.[0-9]+\.[0-9]+)[ \t]*$", re.MULTILINE) + + +def normalize(tag: str) -> str: + """`v3.2.0` and `3.2.0` are the same release; tags here carry the `v`.""" + return tag[1:] if tag.startswith("v") else tag + + +def newest_changelog_version(changelog: Path) -> str | None: + """Newest version heading, or None when there is none. + + Raises OSError if the file cannot be read; main() turns that into a clear + message rather than a traceback, because this runs as a release gate and a + traceback there reads as "the tooling is broken", not "your CHANGELOG is + missing". + """ + headings = HEADING.findall(changelog.read_text(encoding="utf-8")) + return headings[0] if headings else None + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "tag", + help="Release tag to check, with or without the leading 'v' (e.g. v3.2.0)", + ) + args = parser.parse_args() + + from src import __version__ as core_version + + tag_version = normalize(args.tag) + changelog_path = REPO_ROOT / "CHANGELOG.md" + + problems: list[str] = [] + + try: + changelog_version = newest_changelog_version(changelog_path) + except OSError as e: + print( + f"Release version check FAILED for tag {args.tag}:\n" + f" - could not read {changelog_path}: {e}\n" + f" Restore the file (git checkout -- CHANGELOG.md) and re-run.", + file=sys.stderr, + ) + return 1 + + if not SEMVER.match(tag_version): + problems.append( + f"tag {args.tag!r} is not vX.Y.Z. Older tags (v2.5) predate this " + "check; new releases must be full semver so floors can parse them." + ) + + if not SEMVER.match(core_version): + problems.append(f"src.__version__ is {core_version!r}, which is not X.Y.Z") + + if tag_version != core_version: + problems.append( + f"tag says {tag_version} but src.__version__ says {core_version}. " + "Bump src/__init__.py to match the tag before releasing — devices " + "report __version__, not the tag, and plugin floors compare " + "against it." + ) + + if changelog_version is None: + problems.append("CHANGELOG.md has no '## X.Y.Z' version heading") + elif changelog_version != core_version: + problems.append( + f"CHANGELOG.md's newest heading is {changelog_version} but " + f"src.__version__ is {core_version}. Plugin authors read the " + "CHANGELOG to pick a ledmatrix_min_version floor." + ) + + if problems: + print(f"Release version check FAILED for tag {args.tag}:", file=sys.stderr) + for problem in problems: + print(f" - {problem}", file=sys.stderr) + return 1 + + print( + f"OK: tag {args.tag}, src.__version__ {core_version}, and the CHANGELOG " + "all agree." + ) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/debug/check_imports.py b/scripts/debug/check_imports.py deleted file mode 100644 index a1cefae0..00000000 --- a/scripts/debug/check_imports.py +++ /dev/null @@ -1,28 +0,0 @@ -#!/usr/bin/env python3 -""" -Check what imports are actually in the app.py file on the Pi -""" - -from pathlib import Path - -# Read the app.py file and check the import lines -app_py_path = Path.home() / 'LEDMatrix' / 'web_interface' / 'app.py' - -print(f"🔍 Checking imports in: {app_py_path}") -print(f"📁 File exists: {app_py_path.exists()}") - -if app_py_path.exists(): - with open(app_py_path, 'r') as f: - lines = f.readlines() - - print("\n🔍 Import lines in app.py:") - for i, line in enumerate(lines, 1): - if 'from' in line and 'blueprints' in line and 'import' in line: - print(f" Line {i}: {line.strip()}") - - print("\n🔍 Blueprint registration lines:") - for i, line in enumerate(lines, 1): - if 'register_blueprint' in line: - print(f" Line {i}: {line.strip()}") -else: - print("❌ app.py file not found!") diff --git a/scripts/debug/debug_web_manual.py b/scripts/debug/debug_web_manual.py index 36f64e1a..62620946 100644 --- a/scripts/debug/debug_web_manual.py +++ b/scripts/debug/debug_web_manual.py @@ -13,8 +13,8 @@ def main(): print("🔍 LED Matrix Web Interface Debug Tool") print("=" * 50) - # Change to project root (where this script is located) - project_root = Path(__file__).parent.resolve() + # Change to project root (two levels up from scripts/debug/) + project_root = Path(__file__).parent.parent.parent.resolve() os.chdir(project_root) print(f"📁 Working directory: {os.getcwd()}") diff --git a/scripts/debug/direct_fix_imports.py b/scripts/debug/direct_fix_imports.py deleted file mode 100644 index 79d451a9..00000000 --- a/scripts/debug/direct_fix_imports.py +++ /dev/null @@ -1,58 +0,0 @@ -#!/usr/bin/env python3 -""" -Direct fix for import issues - manually edit the app.py file -""" - -import os -from pathlib import Path - -def fix_imports(): - app_py_path = Path.home() / 'LEDMatrix' / 'web_interface' / 'app.py' - - print(f"🔧 Directly fixing imports in: {app_py_path}") - - # Read the file - with open(app_py_path, 'r') as f: - lines = f.readlines() - - # Find and fix the import lines - fixed = False - for i, line in enumerate(lines, 1): - if 'from blueprints.pages_v3 import' in line: - lines[i-1] = "from web_interface.blueprints.pages_v3 import pages_v3\n" - print(f"✅ Fixed line {i}: from blueprints.pages_v3 import → from web_interface.blueprints.pages_v3 import") - fixed = True - elif 'from blueprints.api_v3 import' in line: - lines[i-1] = "from web_interface.blueprints.api_v3 import api_v3\n" - print(f"✅ Fixed line {i}: from blueprints.api_v3 import → from web_interface.blueprints.api_v3 import") - fixed = True - - if not fixed: - print("❌ No import lines found to fix") - return False - - # Write the fixed file back - with open(app_py_path, 'w') as f: - f.writelines(lines) - - print("✅ File updated successfully") - return True - -def verify_fix(): - print("\n🔍 Verifying the fix...") - os.system("python3 check_imports.py") - -if __name__ == "__main__": - if fix_imports(): - print("\n🧹 Clearing Python cache...") - os.system("find ~/LEDMatrix -type d -name '__pycache__' -exec rm -rf {} + 2>/dev/null || true") - os.system("find ~/LEDMatrix -name '*.pyc' -delete 2>/dev/null || true") - - print("\n✅ Imports fixed and cache cleared!") - verify_fix() - - print("\n🚀 Now try running the web interface:") - print("cd ~/LEDMatrix") - print("python3 web_interface/start.py") - else: - print("\n❌ Fix failed") diff --git a/scripts/dev/test_pillow_compat.py b/scripts/dev/test_pillow_compat.py index 590d632c..679b8c3a 100755 --- a/scripts/dev/test_pillow_compat.py +++ b/scripts/dev/test_pillow_compat.py @@ -55,7 +55,7 @@ def main(): failures += not check("draw.textbbox", lambda: draw.textbbox((0, 0), "Test", font=font)) - print("\nResampling (used in logo_helper, image_utils, sports base):") + print("\nResampling (used in logo_helper, sports base):") logo = Image.new('RGBA', (200, 200), (255, 128, 0, 200)) failures += not check("Image.Resampling.LANCZOS exists", lambda: str(Image.Resampling.LANCZOS)) diff --git a/scripts/dev/vegas_audit.py b/scripts/dev/vegas_audit.py new file mode 100644 index 00000000..e3fdffa7 --- /dev/null +++ b/scripts/dev/vegas_audit.py @@ -0,0 +1,384 @@ +#!/usr/bin/env python3 +""" +Vegas Mode Density Audit + +Reports how much of the Vegas ticker is actually showing something. Loads the +real enabled plugins, pulls each one's content through the real +``PluginAdapter``, composes the strip through the real ``ScrollHelper``, then +measures the result. + +The headline number is the **dead-frame ratio**: the fraction of viewport +positions across a full cycle that are effectively blank. Because the panel +only ever shows ``display_width`` columns at a time, a blank stretch wider than +the viewport is a stretch where the display looks switched off — so this ratio +tracks perceived dead time rather than just counting unlit pixels. + +Runs entirely off-hardware, so it is safe to run alongside a live display. + +Usage: + # Audit every enabled plugin at the display size from config.json + python scripts/dev/vegas_audit.py + + # Specific plugins, dump each segment as a PNG for eyeballing + python scripts/dev/vegas_audit.py -p of-the-day,youtube-stats --dump-dir /tmp/vg + + # Machine-readable, for before/after comparison + python scripts/dev/vegas_audit.py --json > after.json +""" + +import argparse +import json +import logging +import os +import sys +import time +from pathlib import Path +from typing import Any, Dict, List + +PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent +sys.path.insert(0, str(PROJECT_ROOT)) + +# Must precede any src import that may reach for hardware. +os.environ.setdefault('EMULATOR', 'true') + +from PIL import Image # noqa: E402 + +from src.common.scroll_helper import ScrollHelper # noqa: E402 +from src.plugin_system.testing.loading import ( # noqa: E402 + build_full_config, + find_plugin_dir, + load_manifest, +) +from src.vegas_mode.config import VegasModeConfig # noqa: E402 +from src.vegas_mode.geometry import ( # noqa: E402 + DEFAULT_INK_THRESHOLD, + column_has_ink, + content_bounds, + dead_window_stats, + window_coverage_stats, +) +from src.vegas_mode.plugin_adapter import PluginAdapter # noqa: E402 + +# Sampling stride for the dead-window scan. A full cycle can be 30,000px wide; +# 4px granularity keeps the scan instant while staying well under the ~10px a +# single scroll step ever covers, so no dead stretch is missed. +DEAD_SCAN_STEP = 4 + + +def load_main_config(path: Path) -> Dict[str, Any]: + with open(path, 'r') as fh: + return json.load(fh) + + +def display_size_from_config(config: Dict[str, Any]) -> tuple: + """Derive the logical ticker size the way DisplayManager does.""" + hw = config.get('display', {}).get('hardware', {}) + cols = int(hw.get('cols', 64)) + chain = int(hw.get('chain_length', 1)) + rows = int(hw.get('rows', 32)) + parallel = int(hw.get('parallel', 1)) + return cols * chain, rows * parallel + + +def enabled_plugin_ids(config: Dict[str, Any]) -> List[str]: + """Plugin IDs that are enabled in config, excluding non-plugin sections.""" + ids = [] + for key, value in config.items(): + if isinstance(value, dict) and value.get('enabled') is True: + ids.append(key) + return ids + + +def instantiate(plugin_id: str, display_manager, cache_manager, plugin_manager): + """Load one plugin offline. Returns the instance or None.""" + from src.plugin_system.plugin_loader import PluginLoader + + search_dirs = [ + str(PROJECT_ROOT / 'plugin-repos'), + str(PROJECT_ROOT / 'plugins'), + ] + plugin_dir = find_plugin_dir(plugin_id, search_dirs) + if not plugin_dir: + return None + + try: + manifest = load_manifest(Path(plugin_dir)) + cfg = build_full_config(Path(plugin_dir)) + instance, _ = PluginLoader().load_plugin( + plugin_id=plugin_id, + manifest=manifest, + plugin_dir=Path(plugin_dir), + config=cfg, + display_manager=display_manager, + cache_manager=cache_manager, + plugin_manager=plugin_manager, + install_deps=False, + ) + return instance + except Exception as exc: # noqa: BLE001 - audit tool must survive any plugin + print(f" ! {plugin_id}: load failed ({type(exc).__name__}: {exc})", + file=sys.stderr) + return None + + +def join_rows(images: List[Image.Image], gap: int) -> Image.Image: + """Concatenate one plugin's rows, matching RenderPipeline._join_plugin_rows.""" + if len(images) == 1: + return images[0] + gap = max(0, gap) + width = sum(img.width for img in images) + gap * (len(images) - 1) + height = max(img.height for img in images) + block = Image.new('RGB', (width, height), (0, 0, 0)) + x = 0 + for img in images: + block.paste(img, (x, 0)) + x += img.width + gap + return block + + +def measure_segment(images: List[Image.Image], display_width: int, + scroll_speed: float, threshold: int) -> Dict[str, Any]: + """Geometry of one plugin's contribution to the ticker.""" + total_width = sum(img.width for img in images) + combined = Image.new('RGB', (max(1, total_width), images[0].height)) + x = 0 + for img in images: + combined.paste(img, (x, 0)) + x += img.width + + ink = column_has_ink(combined, threshold) + bounds = content_bounds(combined, threshold) + ink_cols = int(ink.sum()) + + return { + 'images': len(images), + 'width_px': total_width, + 'ink_cols': ink_cols, + 'ink_pct': round(100.0 * ink_cols / total_width, 1) if total_width else 0.0, + 'lead_black_px': bounds[0] if bounds else total_width, + 'trail_black_px': (total_width - 1 - bounds[1]) if bounds else 0, + 'seconds_on_screen': round(total_width / scroll_speed, 1) if scroll_speed else 0.0, + 'widths': [img.width for img in images], + } + + +def main() -> int: + parser = argparse.ArgumentParser( + description='Audit Vegas mode content density') + parser.add_argument('--config', default=str(PROJECT_ROOT / 'config' / 'config.json'), + help='Path to main config.json') + parser.add_argument('-p', '--plugins', default=None, + help='Comma-separated plugin IDs (default: all enabled)') + parser.add_argument('--width', type=int, default=None, + help='Override display width (default: from config hardware)') + parser.add_argument('--height', type=int, default=None, + help='Override display height (default: from config hardware)') + parser.add_argument('--dump-dir', default=None, + help='Write each segment and the composed strip as PNGs here') + parser.add_argument('--threshold', type=int, default=DEFAULT_INK_THRESHOLD, + help=f'Ink threshold (default: {DEFAULT_INK_THRESHOLD})') + parser.add_argument('--per-cycle', type=int, default=None, + help='Plugins composed per cycle ' + '(default: buffer_ahead + 1, matching production)') + parser.add_argument('--json', action='store_true', + help='Emit JSON instead of a text report') + args = parser.parse_args() + + config = load_main_config(Path(args.config)) + vegas = VegasModeConfig.from_config(config) + + cfg_w, cfg_h = display_size_from_config(config) + width = args.width or cfg_w + height = args.height or cfg_h + speed = vegas.scroll_speed + + if args.plugins: + plugin_ids = [p.strip() for p in args.plugins.split(',') if p.strip()] + else: + plugin_ids = vegas.get_ordered_plugins(enabled_plugin_ids(config)) + + dump_dir = Path(args.dump_dir) if args.dump_dir else None + if dump_dir: + dump_dir.mkdir(parents=True, exist_ok=True) + + from src.plugin_system.testing import ( + MockCacheManager, MockPluginManager, VisualTestDisplayManager, + ) + + display_manager = VisualTestDisplayManager(width=width, height=height) + cache_manager = MockCacheManager() + plugin_manager = MockPluginManager() + # Pass the loaded config, exactly as VegasModeCoordinator does. Omitting it + # makes PluginAdapter fall back to VegasModeConfig() defaults, so the audit + # would silently report trimming and width-budget behaviour that differs + # from the user's config.json — the same drift the lead_gap and grouping + # arguments below exist to avoid. + adapter = PluginAdapter(display_manager, vegas) + + if not args.json: + print(f"Vegas audit — display {width}x{height}, scroll {speed:g}px/s, " + f"separator {vegas.separator_width}px") + print(f"One display width = {width / speed:.1f}s of screen time\n") + + results: List[Dict[str, Any]] = [] + segments: List[Image.Image] = [] + + for plugin_id in plugin_ids: + started = time.time() + instance = instantiate(plugin_id, display_manager, cache_manager, plugin_manager) + if instance is None: + results.append({'plugin': plugin_id, 'status': 'load_failed'}) + continue + + plugin_manager.plugins[plugin_id] = instance + adapter.invalidate_cache(plugin_id) + + try: + images = adapter.get_content(instance, plugin_id) + except Exception as exc: # noqa: BLE001 + results.append({'plugin': plugin_id, 'status': 'fetch_error', + 'error': f'{type(exc).__name__}: {exc}'}) + continue + + fetch_ms = round((time.time() - started) * 1000) + + if not images: + results.append({'plugin': plugin_id, 'status': 'no_content', + 'fetch_ms': fetch_ms}) + if not args.json: + print(f" {plugin_id:28s} NO CONTENT ({fetch_ms}ms)") + continue + + entry = {'plugin': plugin_id, 'status': 'ok', 'fetch_ms': fetch_ms} + entry.update(measure_segment(images, width, speed, args.threshold)) + results.append(entry) + segments.extend(images) + + if dump_dir: + for idx, img in enumerate(images): + img.save(dump_dir / f"{plugin_id}__{idx:02d}.png") + + if not args.json: + print(f" {plugin_id:28s} {entry['width_px']:>6d}px " + f"{entry['images']:>2d} img ink {entry['ink_pct']:>5.1f}% " + f"lead {entry['lead_black_px']:>4d} tail {entry['trail_black_px']:>4d} " + f"{entry['seconds_on_screen']:>6.1f}s ({fetch_ms}ms)") + + summary: Dict[str, Any] = { + 'display_width': width, + 'display_height': height, + 'scroll_speed': speed, + 'separator_width': vegas.separator_width, + 'plugins_audited': len(plugin_ids), + 'plugins_with_content': sum(1 for r in results if r.get('status') == 'ok'), + } + + # Production composes only the plugins sitting in the active buffer, so + # measuring one giant strip of every plugin would hide the per-cycle costs + # (most importantly the leading gap, which is charged once per cycle). + # Group the segments the way the running service does. + per_cycle = max(1, args.per_cycle or vegas.plugins_per_cycle) + + cycles: List[Dict[str, Any]] = [] + with_content = [r for r in results if r.get('status') == 'ok'] + + if segments: + logger = logging.getLogger('vegas_audit') + seg_index = 0 + for start in range(0, len(with_content), per_cycle): + group = with_content[start:start + per_cycle] + + # Mirror RenderPipeline: each plugin's rows are joined by + # intra_plugin_gap into one block, and separator_width is applied + # only between blocks. Measuring a flat list here would report gaps + # the service does not emit. + blocks: List[Image.Image] = [] + for entry in group: + count = entry['images'] + rows = segments[seg_index:seg_index + count] + seg_index += count + if rows: + blocks.append(join_rows(rows, vegas.intra_plugin_gap)) + if not blocks: + continue + + # ScrollHelper logs unconditionally, so it needs a real logger. + helper = ScrollHelper(width, height, logger) + helper.create_scrolling_image( + content_items=blocks, + item_gap=vegas.separator_width, + element_gap=0, + # Must match RenderPipeline. Omitting this made the audit + # measure a full-display-width leading gap the service no + # longer emits, overstating dead space by 512px per cycle. + lead_gap=vegas.lead_in_width, + ) + composed = helper.cached_image + if composed is None: + continue + + dead = dead_window_stats(composed, width, args.threshold, step=DEAD_SCAN_STEP) + cover = window_coverage_stats( + composed, width, args.threshold, step=DEAD_SCAN_STEP) + + if dump_dir: + composed.save(dump_dir / f"_cycle{len(cycles):02d}.png") + + cycles.append({ + 'plugins': [e['plugin'] for e in group], + 'width_px': composed.width, + 'seconds': round(composed.width / speed, 1) if speed else 0.0, + 'dead_pct': round(100 * dead.dead_ratio, 1), + 'longest_dead_seconds': round( + dead.longest_dead_run * DEAD_SCAN_STEP / speed, 1) if speed else 0.0, + 'mean_ink_pct': round(100 * cover.mean_ink_ratio, 1), + 'sparse_pct': round(100 * cover.sparse_ratio, 1), + 'longest_sparse_seconds': round( + cover.longest_sparse_run * DEAD_SCAN_STEP / speed, 1) if speed else 0.0, + }) + + if cycles: + total_px = sum(c['width_px'] for c in cycles) + # Weight each cycle by its width so a long cycle counts proportionally. + summary.update({ + 'cycles': len(cycles), + 'total_px': total_px, + 'full_rotation_seconds': round(total_px / speed, 1) if speed else 0.0, + 'dead_pct': round( + sum(c['dead_pct'] * c['width_px'] for c in cycles) / total_px, 1), + 'mean_ink_pct': round( + sum(c['mean_ink_pct'] * c['width_px'] for c in cycles) / total_px, 1), + 'sparse_pct': round( + sum(c['sparse_pct'] * c['width_px'] for c in cycles) / total_px, 1), + 'worst_dead_seconds': max(c['longest_dead_seconds'] for c in cycles), + 'worst_sparse_seconds': max(c['longest_sparse_seconds'] for c in cycles), + }) + + if args.json: + print(json.dumps({'summary': summary, 'cycles': cycles, 'plugins': results}, + indent=2)) + else: + print(f"\n Cycles ({per_cycle} plugins each, as production composes them):") + for idx, cyc in enumerate(cycles): + print(f" [{idx}] {cyc['width_px']:>6d}px {cyc['seconds']:>6.1f}s " + f"ink {cyc['mean_ink_pct']:>5.1f}% blank {cyc['dead_pct']:>5.1f}% " + f"worst blank {cyc['longest_dead_seconds']:>5.1f}s " + f"| {', '.join(cyc['plugins'])}") + + print(f"\n {'-' * 66}") + print(f" full rotation {summary.get('full_rotation_seconds', 0):>7.1f}s " + f"over {summary.get('cycles', 0)} cycles") + print(f" mean ink coverage {summary.get('mean_ink_pct', 0):>7.1f}% " + f"(higher is better; target >25%)") + print(f" fully blank {summary.get('dead_pct', 0):>7.1f}% (target <2%)") + print(f" reads as empty {summary.get('sparse_pct', 0):>7.1f}% (target <15%)") + print(f" worst blank stretch {summary.get('worst_dead_seconds', 0):>7.1f}s " + f"(target <1.5s)") + print(f" plugins w/ content {summary.get('plugins_with_content', 0):>7d}" + f" of {summary['plugins_audited']}") + + return 0 + + +if __name__ == '__main__': + raise SystemExit(main()) diff --git a/scripts/diagnose_dependencies.sh b/scripts/diagnose_dependencies.sh index d83f508d..520dd1d7 100755 --- a/scripts/diagnose_dependencies.sh +++ b/scripts/diagnose_dependencies.sh @@ -156,9 +156,13 @@ echo "" echo "6. Check disk space - building packages requires temporary space" echo " df -h" echo "" -echo "7. For slow builds, increase swap space:" +echo "7. For slow builds or out-of-memory kills, increase swap space." +echo " first_time_install.sh already adds temporary swap on low-memory devices;" +echo " this makes it permanent. Set CONF_MAXSWAP too - it defaults to 2048 and" +echo " silently clamps CONF_SWAPSIZE, so raising CONF_SWAPSIZE alone does nothing." echo " sudo dphys-swapfile swapoff" -echo " sudo nano /etc/dphys-swapfile # Set CONF_SWAPSIZE=2048" +echo " sudo sed -i 's/^#\\?CONF_SWAPSIZE=.*/CONF_SWAPSIZE=2048/' /etc/dphys-swapfile" +echo " sudo sed -i 's/^#\\?CONF_MAXSWAP=.*/CONF_MAXSWAP=2048/' /etc/dphys-swapfile" echo " sudo dphys-swapfile setup" echo " sudo dphys-swapfile swapon" echo "" diff --git a/scripts/download_nba_logos.py b/scripts/download_nba_logos.py index c0bfdb89..99bb5b4b 100644 --- a/scripts/download_nba_logos.py +++ b/scripts/download_nba_logos.py @@ -7,8 +7,8 @@ import os import logging from typing import Tuple -# Add the src directory to Python path so we can import the logo downloader -sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'src')) +# Add the project root to Python path so we can import the logo downloader +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) # Set up logging logging.basicConfig( @@ -28,7 +28,7 @@ def download_nba_logos(force_download: bool = False) -> Tuple[int, int]: Tuple of (downloaded_count, failed_count) """ try: - from logo_downloader import download_all_logos_for_league + from src.logo_downloader import download_all_logos_for_league logger.info("🏀 Starting NBA logo download...") logger.info(f"Target directory: assets/sports/nba_logos/") diff --git a/scripts/download_pixlet.sh b/scripts/download_pixlet.sh index 53a38d22..80caeb74 100755 --- a/scripts/download_pixlet.sh +++ b/scripts/download_pixlet.sh @@ -24,9 +24,29 @@ echo "========================================" # Auto-detect latest version if needed if [ "$PIXLET_VERSION" = "latest" ]; then echo "Detecting latest version..." - PIXLET_VERSION=$(curl -s "https://api.github.com/repos/${REPO}/releases/latest" | grep '"tag_name"' | sed -E 's/.*"([^"]+)".*/\1/') - if [ -z "$PIXLET_VERSION" ]; then - echo "Failed to detect latest version, using fallback" + # When this response arrives on a single line -- as it did on the device + # where Starlark apps were failing -- `grep '"tag_name"'` matches the whole + # document and a greedy `sed 's/.*"([^"]+)".*/\1/'` captures the LAST + # quoted token in it rather than the tag. That resolved to + # "mentions_count", which built a download URL for a release that does not + # exist. (The API is pretty-printed by default, which is why the old + # command looks correct when you try it by hand -- but the formatting is + # not something to depend on.) Match the field itself and take the value + # after it, which is right for either shape. + PIXLET_VERSION=$(curl -fsSL "https://api.github.com/repos/${REPO}/releases/latest" \ + | grep -o '"tag_name"[[:space:]]*:[[:space:]]*"[^"]*"' \ + | head -n1 \ + | sed -E 's/.*:[[:space:]]*"([^"]*)".*/\1/') + + # A wrong-but-non-empty value is what made the old bug silent, so check the + # shape rather than just that something came back. Anchored at both ends: a + # partial match would accept "v0.53garbage" or "0.53" and build a URL for a + # release that cannot exist, which is the failure this check is here to + # stop. Every tronbyt/pixlet release to date is vX.Y.Z; the optional suffix + # leaves room for a future -rc.1 or +build tag. + if ! printf '%s' "$PIXLET_VERSION" \ + | grep -qE '^v[0-9]+\.[0-9]+\.[0-9]+([-+][0-9A-Za-z.-]+)?$'; then + echo "Could not detect the latest version (got: '${PIXLET_VERSION:-}'), using fallback" PIXLET_VERSION="v0.50.2" fi fi @@ -67,8 +87,26 @@ download_binary() { temp_dir=$(mktemp -d -p "$PROJECT_ROOT" -t pixlet_download.XXXXXXXXXX) local temp_file="$temp_dir/$archive_name" - if ! curl -L -o "$temp_file" "$url" 2>/dev/null; then - echo "✗ Failed to download $arch" + # -f so an HTTP error is a failure. Without it curl writes the 404 body + # to the file and exits 0, and the first sign of trouble is tar saying + # "not in gzip format" about what is actually a page of HTML. + if ! curl -fL -o "$temp_file" "$url" 2>/dev/null; then + echo "✗ Failed to download $arch from $url" + rm -rf "$temp_dir" + return 1 + fi + + # Belt and braces: a mirror or proxy can return 200 with an error page. + if ! gzip -t "$temp_file" 2>/dev/null; then + echo "✗ Downloaded file is not a gzip archive: $url" + # These bytes come from whatever answered the request, so strip + # everything non-printable before echoing them: an error page carrying + # terminal escapes would otherwise be able to rewrite this output or + # bury it in a CI log. Printable characters are kept rather than + # hex-encoding the lot, because "" is the diagnostic. + local first_bytes + first_bytes=$(head -c 60 "$temp_file" | tr -cd '[:print:]') + printf ' (first bytes: %s)\n' "$first_bytes" rm -rf "$temp_dir" return 1 fi diff --git a/scripts/fix_perms/README.md b/scripts/fix_perms/README.md index 96f293d2..5616e51a 100644 --- a/scripts/fix_perms/README.md +++ b/scripts/fix_perms/README.md @@ -31,9 +31,6 @@ owned by the `ledmatrix` service user or by `root`. systemd journal access, and the sudoers entries the web interface needs to control the display service. -- **`fix_nhl_cache.sh`** — Targeted fix for NHL plugin cache issues - (clears the NHL cache and restarts the display service). - - **`safe_plugin_rm.sh`** — Validates that a plugin removal path is inside an allowed base directory before deleting it. Used by the web interface (via sudo) when a user clicks **Uninstall** on a plugin — diff --git a/scripts/fix_perms/fix_assets_permissions.sh b/scripts/fix_perms/fix_assets_permissions.sh old mode 100644 new mode 100755 diff --git a/scripts/fix_perms/fix_cache_permissions.sh b/scripts/fix_perms/fix_cache_permissions.sh old mode 100644 new mode 100755 diff --git a/scripts/fix_perms/fix_nhl_cache.sh b/scripts/fix_perms/fix_nhl_cache.sh deleted file mode 100644 index ce84514a..00000000 --- a/scripts/fix_perms/fix_nhl_cache.sh +++ /dev/null @@ -1,21 +0,0 @@ -#!/bin/bash -""" -Script to fix NHL cache issues on Raspberry Pi. -This will clear the NHL cache and restart the display service. -""" - -echo "==========================================" -echo "Fixing NHL Cache Issues" -echo "==========================================" - -# Clear NHL cache -echo "Clearing NHL cache..." -python3 clear_nhl_cache.py - -# Restart the display service to force fresh data fetch -echo "Restarting display service..." -sudo systemctl restart ledmatrix.service - -echo "NHL cache cleared and service restarted!" -echo "NHL managers should now fetch fresh data from ESPN API." -echo "Check the logs to see if NHL games are now being displayed." diff --git a/scripts/fix_perms/fix_plugin_permissions.sh b/scripts/fix_perms/fix_plugin_permissions.sh old mode 100644 new mode 100755 diff --git a/scripts/fix_perms/fix_web_permissions.sh b/scripts/fix_perms/fix_web_permissions.sh old mode 100644 new mode 100755 diff --git a/scripts/install/configure_web_sudo.sh b/scripts/install/configure_web_sudo.sh index feccc3e7..bbcc14ed 100644 --- a/scripts/install/configure_web_sudo.sh +++ b/scripts/install/configure_web_sudo.sh @@ -100,10 +100,15 @@ TEMP_SUDOERS="/tmp/ledmatrix_web_sudoers_$$" echo "$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH restart ledmatrix-web.service" # Optional: journalctl (non-critical — skip if not found) + # + # NOEXEC, matching first_time_install.sh. These rules end in a wildcard and + # journalctl starts a pager, so without it the caller can reach a shell: + # less runs "!command" as the user the pager belongs to, which here is + # root. NOEXEC stops the granted command executing anything of its own. if [ -n "$JOURNALCTL_PATH" ]; then - echo "$WEB_USER ALL=(ALL) NOPASSWD: $JOURNALCTL_PATH -u ledmatrix.service *" - echo "$WEB_USER ALL=(ALL) NOPASSWD: $JOURNALCTL_PATH -u ledmatrix *" - echo "$WEB_USER ALL=(ALL) NOPASSWD: $JOURNALCTL_PATH -t ledmatrix *" + echo "$WEB_USER ALL=(ALL) NOPASSWD:NOEXEC: $JOURNALCTL_PATH -u ledmatrix.service *" + echo "$WEB_USER ALL=(ALL) NOPASSWD:NOEXEC: $JOURNALCTL_PATH -u ledmatrix *" + echo "$WEB_USER ALL=(ALL) NOPASSWD:NOEXEC: $JOURNALCTL_PATH -t ledmatrix *" fi # Required: python3, bash diff --git a/scripts/install/configure_wifi_permissions.sh b/scripts/install/configure_wifi_permissions.sh index cfba4bc4..37e2c7db 100755 --- a/scripts/install/configure_wifi_permissions.sh +++ b/scripts/install/configure_wifi_permissions.sh @@ -25,9 +25,44 @@ if [ "$EUID" -eq 0 ]; then exit 1 fi +# Resolve command paths against a fixed PATH, and check what we resolved. +# +# Every path found here is written into a sudoers file as a NOPASSWD grant, so +# whoever controls the binary at that path controls root. first_time_install.sh +# re-execs itself with `sudo -E`, which preserves the invoking user's +# environment -- PATH included -- so without pinning it, `which nmcli` can +# resolve to anything on that PATH: a writable directory early in it turns a +# compromise of the low-privilege web user into permanent root. +PATH=/usr/sbin:/usr/bin:/sbin:/bin +export PATH + +# A binary named in a sudoers rule must be root-owned and writable by nobody +# else, or the grant hands root to whoever can rewrite it. +require_trusted_binary() { + local label="$1" path="$2" + if [ ! -x "$path" ]; then + echo "✗ $label: $path is not an executable file" + exit 1 + fi + local owner perms + owner=$(stat -c '%u' "$path") || exit 1 + perms=$(stat -c '%a' "$path") || exit 1 + if [ "$owner" != "0" ]; then + echo "✗ $label: $path is not owned by root (uid $owner); refusing to" + echo " grant it NOPASSWD sudo." + exit 1 + fi + # Group- or world-writable means someone other than root can replace it. + case "$perms" in + *[2367]) echo "✗ $label: $path is writable by group or other ($perms);" + echo " refusing to grant it NOPASSWD sudo." + exit 1 ;; + esac +} + # Get the full paths to commands -NMCLI_PATH=$(which nmcli || echo "/usr/bin/nmcli") -SYSTEMCTL_PATH=$(which systemctl) +NMCLI_PATH=$(command -v nmcli || echo "/usr/bin/nmcli") +SYSTEMCTL_PATH=$(command -v systemctl) echo "Command paths:" echo " nmcli: $NMCLI_PATH" @@ -37,6 +72,18 @@ echo " systemctl: $SYSTEMCTL_PATH" echo "" echo "Step 1: Configuring sudo permissions for nmcli..." SUDOERS_FILE="/etc/sudoers.d/ledmatrix_wifi" +SYSCTL_PATH=$(command -v sysctl || echo /usr/sbin/sysctl) +NFT_PATH=$(command -v nft || echo /usr/sbin/nft) +RFKILL_PATH=$(command -v rfkill || echo /usr/sbin/rfkill) +MKDIR_PATH=$(command -v mkdir || echo /usr/bin/mkdir) + +# Checked before any of them reaches the sudoers file. +require_trusted_binary "nmcli" "$NMCLI_PATH" +require_trusted_binary "systemctl" "$SYSTEMCTL_PATH" +require_trusted_binary "sysctl" "$SYSCTL_PATH" +require_trusted_binary "nft" "$NFT_PATH" +require_trusted_binary "rfkill" "$RFKILL_PATH" +require_trusted_binary "mkdir" "$MKDIR_PATH" # Create a temporary sudoers file using mktemp (handles permissions better) TEMP_SUDOERS=$(mktemp) || { @@ -62,6 +109,36 @@ $WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH start dnsmasq $WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH stop dnsmasq $WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH restart dnsmasq $WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH restart NetworkManager +# The captive portal turns IP forwarding on while the access point is up and +# restores the previous value when it comes down (wifi_manager._setup_iptables_ +# redirect / _teardown_iptables_redirect). Without this rule that sudo call +# needs a password, so forwarding stays off and clients associate to the AP but +# cannot route. It goes unnoticed on a stock Raspberry Pi image, where +# /etc/sudoers.d/010_pi-nopasswd grants the default user blanket NOPASSWD and +# masks every gap in this file -- it only bites once that blanket rule is +# removed. +$WEB_USER ALL=(ALL) NOPASSWD: $SYSCTL_PATH -w net.ipv4.ip_forward=0 +$WEB_USER ALL=(ALL) NOPASSWD: $SYSCTL_PATH -w net.ipv4.ip_forward=1 +# The portal's redirect lives in its own nftables table, created when the AP +# comes up and deleted when it goes down, and the radio has to be unblocked +# before the AP can start at all. Same story as the sysctl rules above: called +# with sudo, never granted here, and invisible on a stock Pi image. +$WEB_USER ALL=(ALL) NOPASSWD: $NFT_PATH add table ip ledmatrix +$WEB_USER ALL=(ALL) NOPASSWD: $NFT_PATH delete table ip ledmatrix +$WEB_USER ALL=(ALL) NOPASSWD: $RFKILL_PATH unblock wifi +# NetworkManager's dnsmasq drop-in directory, exact path. +$WEB_USER ALL=(ALL) NOPASSWD: $MKDIR_PATH -p /etc/NetworkManager/dnsmasq-shared.d +# +# iptables is deliberately NOT granted here. Its rules are built from the live +# interface name and port, so a rule covering them needs a trailing wildcard -- +# and `iptables --modprobe=/path/to/anything` runs that path as root, so +# `NOPASSWD: iptables *` is a root shell for the web user by another name. That +# is a worse outcome than the gap it would close, which today is masked anyway +# by the blanket NOPASSWD rule on stock Pi images. +# +# Closing it safely means a wrapper script that builds the rules itself and +# takes only an interface and a port, granted the way safe_plugin_rm.sh already +# is. That belongs in its own change rather than being smuggled into this one. # Allow copying hostapd and dnsmasq config files into place $WEB_USER ALL=(ALL) NOPASSWD: /usr/bin/cp /tmp/hostapd.conf /etc/hostapd/hostapd.conf diff --git a/scripts/install/install_service.sh b/scripts/install/install_service.sh old mode 100644 new mode 100755 diff --git a/scripts/install/install_web_service.sh b/scripts/install/install_web_service.sh old mode 100644 new mode 100755 diff --git a/scripts/install/lib_lowmem.sh b/scripts/install/lib_lowmem.sh new file mode 100644 index 00000000..f443ac72 --- /dev/null +++ b/scripts/install/lib_lowmem.sh @@ -0,0 +1,283 @@ +#!/bin/bash +# +# Low-memory build helpers for the LED Matrix installer. +# +# Sourced by first_time_install.sh. These live in a separate, sourceable file +# so the pure sizing/detection functions can be unit-tested +# (test/test_install_lowmem.py); first_time_install.sh itself is not sourceable +# because it self-elevates and runs top to bottom. +# +# Why this exists: the rgbmatrix build compiles ~45 C++ translation units, two +# of them Cython-generated (a single cc1plus on those peaks around 400-800MB at +# -O3). Upstream's pyproject.toml sets no [tool.scikit-build] options, so +# scikit-build-core uses Ninja at its default of nproc+2 jobs -- six concurrent +# compiles on a 4-core Pi. On a 512MB-1GB Pi the OOM killer reaps cc1plus and +# pip reports only "Failed building wheel for rgbmatrix". +# +# The caller runs under `set -Eeuo pipefail` with an ERR trap, and these are +# invoked from the middle of numbered steps, so nothing here may call exit and +# the swap helpers must always return 0. + +# Overridable so tests can point at fixture files instead of /proc. +LM_MEMINFO="${LM_MEMINFO:-/proc/meminfo}" +LM_SWAPS="${LM_SWAPS:-/proc/swaps}" + +# Temporary swapfile created for the build and removed afterwards. Deliberately +# never added to /etc/fstab: a malformed fstab can leave a novice with an +# unbootable Pi, and this swap only needs to outlive the compile. +LM_SWAPFILE="${LM_SWAPFILE:-/var/swap.ledmatrix-install}" + +# Bring RAM + real swap up to this much before compiling, capped per swapfile. +LM_SWAP_TARGET_MB="${LM_SWAP_TARGET_MB:-3072}" +LM_SWAP_MAX_MB="${LM_SWAP_MAX_MB:-2048}" + +# Worst-case cc1plus footprint on the Cython translation unit, used to size +# build parallelism against available RAM. +LM_MB_PER_JOB="${LM_MB_PER_JOB:-768}" + +# Set to 1 once swap is live, so lm_remove_build_swap (wired up as an EXIT +# trap) knows whether there is anything to undo. +LM_TEMP_SWAP_ACTIVE=0 + +# Human-readable reason no swapfile was created, quoted back in the failure +# message so a user who still OOMs is told why the safety net was absent. +LM_SWAP_SKIP_REASON="" + +# --------------------------------------------------------------------------- +# Pure helpers (no side effects; unit-tested) +# --------------------------------------------------------------------------- + +# Total physical RAM in MB, or 0 if it cannot be determined. +lm_total_ram_mb() { + # Defaults are re-resolved here as well as at source time so the function + # stays safe under the installer's `set -u`. + awk '/^MemTotal:/ {printf "%d\n", $2 / 1024; found = 1; exit} END {if (!found) print 0}' \ + "${LM_MEMINFO:-/proc/meminfo}" 2>/dev/null || echo 0 +} + +# Total swap in MB, EXCLUDING zram devices. +# +# zram swap is compressed RAM: it consumes the very resource that is already +# exhausted and does nothing for a build OOM. Counting it would let a +# zram-enabled image decide it has enough swap and then fail exactly as before. +lm_total_swap_mb() { + awk 'NR > 1 && $1 !~ /^\/dev\/zram/ {total += $3} END {printf "%d\n", total / 1024}' \ + "${LM_SWAPS:-/proc/swaps}" 2>/dev/null || echo 0 +} + +# lm_build_jobs -> max(1, min(cores, ram_mb / LM_MB_PER_JOB)) +# +# Computed from RAM alone and never RAM+swap: handing out extra jobs because +# swap exists just guarantees SD-card thrash, which is far slower than +# compiling serially. +lm_build_jobs() { + local ram_mb="${1:-0}" cores="${2:-1}" jobs + local per_job="${LM_MB_PER_JOB:-768}" + if [ "$cores" -lt 1 ]; then + cores=1 + fi + jobs=$(( ram_mb / per_job )) + if [ "$jobs" -lt 1 ]; then + jobs=1 + fi + if [ "$jobs" -gt "$cores" ]; then + jobs="$cores" + fi + echo "$jobs" +} + +# lm_swap_needed_mb -> swapfile size in MB, or 0. +# +# Brings RAM + real swap up to LM_SWAP_TARGET_MB, capped at LM_SWAP_MAX_MB and +# rounded up to a 256MB multiple. Machines with enough memory get 0 and are +# left completely untouched. +lm_swap_needed_mb() { + local ram_mb="${1:-0}" swap_mb="${2:-0}" needed + local target="${LM_SWAP_TARGET_MB:-3072}" max="${LM_SWAP_MAX_MB:-2048}" + needed=$(( target - ram_mb - swap_mb )) + if [ "$needed" -le 0 ]; then + echo 0 + return 0 + fi + if [ "$needed" -gt "$max" ]; then + needed="$max" + fi + echo $(( ( (needed + 255) / 256 ) * 256 )) +} + +# lm_build_failed_on_oom -> 0 if the build was OOM-killed. +# +# Two independent evidence sources, because neither alone is reliable: the +# compiler sometimes reports its own allocation failure, but when the kernel +# OOM killer fires it writes nothing to the build's stdout. That silence is +# exactly why the old handler misdiagnosed this as missing build tools. +lm_build_failed_on_oom() { + local build_output="${1:-}" kernel_log="" + + if [ -n "$build_output" ] && [ -f "$build_output" ]; then + if grep -qiE 'cc1plus: out of memory|virtual memory exhausted|Cannot allocate memory|MemoryError|fatal error: Killed signal terminated program|signal 9' \ + "$build_output"; then + return 0 + fi + fi + + # LM_KERNEL_LOG_FILE lets tests supply a fixture instead of the real kernel + # ring buffer, which on a shared CI machine may hold unrelated OOM events. + if [ -n "${LM_KERNEL_LOG_FILE:-}" ]; then + if [ -f "$LM_KERNEL_LOG_FILE" ]; then + kernel_log=$(cat "$LM_KERNEL_LOG_FILE" 2>/dev/null || true) + fi + elif command -v dmesg >/dev/null 2>&1; then + kernel_log=$(dmesg -T 2>/dev/null || dmesg 2>/dev/null || true) + fi + if [ -z "$kernel_log" ] && [ -z "${LM_KERNEL_LOG_FILE:-}" ] && command -v journalctl >/dev/null 2>&1; then + kernel_log=$(journalctl -k --since "30 min ago" --no-pager 2>/dev/null || true) + fi + + if [ -n "$kernel_log" ]; then + if printf '%s\n' "$kernel_log" | tail -n 300 | \ + grep -qiE 'Out of memory: Kill|oom_kill|oom-kill|Killed process'; then + return 0 + fi + fi + + return 1 +} + +# lm_disk_backed_tmpdir [candidate] -> a disk-backed temp dir, or nothing. +# +# pip builds in $TMPDIR. Debian 13 mounts /tmp as tmpfs, so the default puts the +# whole C++ build tree in RAM, competing with the compiler we are already trying +# to keep under the limit. Prints a replacement only when the current TMPDIR is +# memory-backed and the candidate is not; otherwise prints nothing and the +# caller keeps its default. +lm_disk_backed_tmpdir() { + local candidate="${1:-/var/tmp}" + local current="${TMPDIR:-/tmp}" + local current_fs="" candidate_fs="" + + current_fs=$(lm_fstype_of "$current") + case "$current_fs" in + tmpfs|ramfs) ;; + *) return 0 ;; + esac + + candidate_fs=$(lm_fstype_of "$candidate") + case "$candidate_fs" in + tmpfs|ramfs|"") return 0 ;; + esac + + echo "$candidate" +} + +# Filesystem type backing a path, or empty if it cannot be determined. +lm_fstype_of() { + local path="${1:-/}" + if command -v findmnt >/dev/null 2>&1; then + findmnt -no FSTYPE --target "$path" 2>/dev/null | head -n 1 + return 0 + fi + if command -v stat >/dev/null 2>&1; then + stat -f -c %T "$path" 2>/dev/null | head -n 1 + return 0 + fi + return 0 +} + +# --------------------------------------------------------------------------- +# Swap management (requires root; not unit-tested) +# --------------------------------------------------------------------------- + +# lm_ensure_build_swap +# +# Always returns 0. On any refusal it sets LM_SWAP_SKIP_REASON and leaves the +# system untouched -- swap is a safety net for the build, never a precondition. +lm_ensure_build_swap() { + local needed_mb="${1:-0}" + local swap_dir free_mb budget + + LM_SWAP_SKIP_REASON="" + + if [ "$needed_mb" -le 0 ]; then + LM_SWAP_SKIP_REASON="not needed (RAM and existing swap are sufficient)" + return 0 + fi + + if ! command -v mkswap >/dev/null 2>&1 || ! command -v swapon >/dev/null 2>&1; then + LM_SWAP_SKIP_REASON="mkswap/swapon are not available on this system" + echo "⚠ Cannot add build swap: $LM_SWAP_SKIP_REASON" + return 0 + fi + + # Clear a stale swapfile left by a run that was killed before its cleanup + # ran, so this is safe to call repeatedly. + if [ -e "$LM_SWAPFILE" ]; then + echo "Removing a leftover swapfile from a previous run: $LM_SWAPFILE" + swapoff "$LM_SWAPFILE" >/dev/null 2>&1 || true + rm -f "$LM_SWAPFILE" || true + fi + + # Keep a working margin for the build tree itself; never eat the last GB. + swap_dir=$(dirname "$LM_SWAPFILE") + free_mb=$(df -m "$swap_dir" 2>/dev/null | awk 'NR==2{print $4}') + free_mb=${free_mb:-0} + budget=$(( free_mb - 1024 )) + if [ "$budget" -lt 256 ]; then + LM_SWAP_SKIP_REASON="only ${free_mb}MB free on ${swap_dir}, need about $(( needed_mb + 1024 ))MB" + echo "⚠ Skipping the build swapfile: $LM_SWAP_SKIP_REASON" + return 0 + fi + if [ "$needed_mb" -gt "$budget" ]; then + echo "⚠ Trimming the build swapfile from ${needed_mb}MB to leave 1GB free on ${swap_dir}" + needed_mb=$(( ( budget / 256 ) * 256 )) + fi + + echo "Adding a temporary ${needed_mb}MB swapfile for the build: $LM_SWAPFILE" + echo " This is removed automatically once the build finishes." + + # fallocate can produce a sparse file that mkswap rejects, and is not + # supported on every filesystem; dd always yields a usable file. + if ! fallocate -l "${needed_mb}M" "$LM_SWAPFILE" 2>/dev/null; then + if ! dd if=/dev/zero of="$LM_SWAPFILE" bs=1M count="$needed_mb" status=none 2>/dev/null; then + LM_SWAP_SKIP_REASON="could not allocate ${needed_mb}MB at $LM_SWAPFILE" + echo "⚠ $LM_SWAP_SKIP_REASON" + rm -f "$LM_SWAPFILE" || true + return 0 + fi + fi + + chmod 600 "$LM_SWAPFILE" || true + + if ! mkswap "$LM_SWAPFILE" >/dev/null 2>&1; then + LM_SWAP_SKIP_REASON="mkswap failed on $LM_SWAPFILE" + echo "⚠ $LM_SWAP_SKIP_REASON" + rm -f "$LM_SWAPFILE" || true + return 0 + fi + + if ! swapon "$LM_SWAPFILE" >/dev/null 2>&1; then + LM_SWAP_SKIP_REASON="swapon failed on $LM_SWAPFILE" + echo "⚠ $LM_SWAP_SKIP_REASON" + rm -f "$LM_SWAPFILE" || true + return 0 + fi + + LM_TEMP_SWAP_ACTIVE=1 + echo "✓ Temporary build swap active (${needed_mb}MB; total swap is now $(lm_total_swap_mb)MB)" + return 0 +} + +# Remove the temporary swapfile. Safe to call unconditionally and repeatedly. +# +# Wired up as an EXIT trap, so it must never return non-zero -- a failing trap +# would surface as a spurious installer error. +lm_remove_build_swap() { + if [ "${LM_TEMP_SWAP_ACTIVE:-0}" != "1" ]; then + return 0 + fi + LM_TEMP_SWAP_ACTIVE=0 + echo "Removing the temporary build swapfile: $LM_SWAPFILE" + swapoff "$LM_SWAPFILE" >/dev/null 2>&1 || true + rm -f "$LM_SWAPFILE" || true + return 0 +} diff --git a/scripts/install/one-shot-install.sh b/scripts/install/one-shot-install.sh index 472b687f..5c29d191 100755 --- a/scripts/install/one-shot-install.sh +++ b/scripts/install/one-shot-install.sh @@ -145,6 +145,34 @@ check_disk_space() { fi } +# Report available memory so the user knows what to expect before the wait. +# +# Informational only — first_time_install.sh does the real work of capping +# build parallelism and adding temporary swap. Never fatal: a low-RAM Pi is +# supported, it is just slower. +check_memory() { + CURRENT_STEP="Memory check" + if [ ! -r /proc/meminfo ]; then + print_warning "Cannot read /proc/meminfo, skipping memory check" + return 0 + fi + + TOTAL_RAM_MB=$(awk '/^MemTotal:/ {printf "%d\n", $2 / 1024; exit}' /proc/meminfo 2>/dev/null || echo 0) + TOTAL_RAM_MB=${TOTAL_RAM_MB:-0} + + if [ "$TOTAL_RAM_MB" -eq 0 ]; then + print_warning "Could not determine system memory, continuing" + elif [ "$TOTAL_RAM_MB" -lt 2048 ]; then + print_warning "Low memory: ${TOTAL_RAM_MB}MB RAM" + echo " The rpi-rgb-led-matrix C++ build needs more memory than this Pi has." + echo " The installer will compile with fewer parallel jobs and add a temporary" + echo " swapfile for the build, removing it afterwards. That step will take" + echo " 15-25 minutes rather than the usual 2-5." + else + print_success "Memory sufficient: ${TOTAL_RAM_MB}MB RAM" + fi +} + # Ensure sudo access check_sudo() { CURRENT_STEP="Sudo access check" @@ -204,7 +232,7 @@ main() { print_step "LED Matrix One-Shot Installation" echo "This script will:" - echo " 1. Check prerequisites (network, disk space, sudo)" + echo " 1. Check prerequisites (network, disk space, memory, sudo)" echo " 2. Install system dependencies (git, python3, build tools)" echo " 3. Clone the LEDMatrix repository" echo " 4. Run the first-time installation script" @@ -213,6 +241,7 @@ main() { # Check prerequisites check_network check_disk_space + check_memory check_sudo # Note: /tmp permissions are checked and fixed inline before running first_time_install.sh # (only if actually wrong, not preemptively) @@ -228,12 +257,14 @@ main() { exit 1 fi - # Update package list first + # Update package list first. first_time_install.sh is told the lists are + # already fresh so it does not repeat this a minute later. if [ "$EUID" -eq 0 ]; then retry apt-get update -qq else retry sudo apt-get update -qq fi + export LEDMATRIX_APT_UPDATED=1 # Install git and curl (needed for cloning and the script itself) if ! command -v git >/dev/null 2>&1 || ! command -v curl >/dev/null 2>&1; then @@ -372,7 +403,12 @@ main() { # Pass both -y flag AND environment variable for non-interactive mode # This ensures it works even if the script re-executes itself with sudo # Also ensure stdin is properly handled for non-interactive mode - sudo -E env TMPDIR=/tmp LEDMATRIX_ASSUME_YES=1 bash ./first_time_install.sh -y Tuple[bool, str]: apt_package_map = { 'flask': 'python3-flask', 'PIL': 'python3-pil', - 'freetype': 'python3-freetype', + 'freetype-py': 'python3-freetype', 'psutil': 'python3-psutil', 'werkzeug': 'python3-werkzeug', 'numpy': 'python3-numpy', 'requests': 'python3-requests', - 'python-dateutil': 'python3-dateutil', - 'pytz': 'python3-tz', - 'geopy': 'python3-geopy', - 'unidecode': 'python3-unidecode', - 'websockets': 'python3-websockets', - 'websocket-client': 'python3-websocket-client' + 'pytz': 'python3-tz' } apt_package = apt_package_map.get(package_name, f'python3-{package_name}') @@ -81,8 +76,8 @@ def install_via_pip(package_name: str) -> Tuple[bool, str]: pip RECORD file, so an uninstall attempt fails with "uninstall-no-record-file" and aborts the whole install. With --ignore-installed, pip lays the new version down in /usr/local where it shadows the apt copy instead of removing - it. This matters when a pip dependency (google-api-python-client pulls a - newer requests) needs to upgrade an apt-managed package. + it. This matters when a pip dependency needs to upgrade an apt-managed + package (e.g. a package that pulls a newer requests). Returns (success, output). """ @@ -101,13 +96,35 @@ def install_via_pip(package_name: str) -> Tuple[bool, str]: # Distribution (pip/apt) names whose importable module name differs. IMPORT_NAME_MAP = { - 'python-dateutil': 'dateutil', - 'websocket-client': 'websocket', + 'freetype-py': 'freetype', +} + +# Minimum versions that must be met for an already-installed package to count +# as satisfied. Debian Bookworm's python3-freetype is 2.3.0, below the +# freetype-py>=2.5.1 pin in requirements.txt, so an import-only check would +# wrongly skip the pip upgrade. +MIN_VERSIONS = { + 'freetype-py': (2, 5, 1), } +def _installed_version_tuple(dist_name: str) -> tuple: + """Return the installed distribution version as an int tuple, or () if unknown.""" + try: + from importlib.metadata import version + parts = [] + for part in version(dist_name).split('.'): + digits = ''.join(ch for ch in part if ch.isdigit()) + if not digits: + break + parts.append(int(digits)) + return tuple(parts) + except Exception: + return () + + def check_package_installed(package_name: str) -> bool: - """Check if a package is already installed.""" + """Check if a package is already installed (and meets any minimum version).""" import_name = IMPORT_NAME_MAP.get(package_name, package_name) # Suppress deprecation warnings when checking if packages are installed # (we're just checking, not using them) @@ -115,9 +132,16 @@ def check_package_installed(package_name: str) -> bool: warnings.filterwarnings('ignore', category=DeprecationWarning) try: __import__(import_name) - return True except ImportError: return False + minimum = MIN_VERSIONS.get(package_name) + if minimum: + installed = _installed_version_tuple(package_name) + if not installed or installed < minimum: + print(f"{package_name} is installed but below the required " + f"{'.'.join(map(str, minimum))}; will upgrade via pip") + return False + return True def print_failure_summary(failed_packages: List[str], failure_details: dict) -> None: @@ -147,17 +171,12 @@ def main(): required_packages = [ 'flask', 'PIL', - 'freetype', + 'freetype-py', 'psutil', 'werkzeug', 'numpy', 'requests', - 'python-dateutil', - 'pytz', - 'geopy', - 'unidecode', - 'websockets', - 'websocket-client' + 'pytz' ] failed_packages = [] @@ -168,8 +187,13 @@ def main(): print(f"{package} is already installed") continue - # Try apt first, then pip + # Try apt first, then pip. An apt install only counts if it also + # satisfies any minimum version (Debian's python3-freetype can be + # older than the freetype-py pin), otherwise fall through to pip. ok, apt_output = install_via_apt(package) + if ok and package in MIN_VERSIONS and not check_package_installed(package): + ok = False + apt_output = f"apt version of {package} is below the required minimum" if not ok: ok, pip_output = install_via_pip(package) if not ok: @@ -177,15 +201,12 @@ def main(): failure_details[package] = pip_output or apt_output # Install packages that don't have apt equivalents + # Packages without apt equivalents. Plugin-specific dependencies + # (timezonefinder, google-api stack, icalevents, socketio, ...) are + # no longer installed here — store plugins declare their own + # requirements.txt, which the plugin store installs. special_packages = [ - 'timezonefinder>=6.5.0,<7.0.0', - 'google-auth-oauthlib>=1.2.0,<2.0.0', - 'google-auth-httplib2>=0.2.0,<1.0.0', - 'google-api-python-client>=2.147.0,<3.0.0', 'spotipy', - 'icalevents', - 'python-socketio>=5.11.0,<6.0.0', - 'python-engineio>=4.9.0,<5.0.0' ] for package in special_packages: diff --git a/scripts/prove_security.py b/scripts/prove_security.py new file mode 100644 index 00000000..9a5fbc88 --- /dev/null +++ b/scripts/prove_security.py @@ -0,0 +1,593 @@ +#!/usr/bin/env python3 +""" +LEDMatrix Security Proof Tests + +Automated proofs that run in CI to verify security properties hold on every +commit. Inspired by the Huntarr security review approach of using standard +tooling to confirm specific vulnerability classes are absent. + +Usage: + python scripts/prove_security.py + python scripts/prove_security.py --verbose + python scripts/prove_security.py --output results.json + +Exit code: 1 only if CRITICAL findings are detected. Warnings are reported +but do not block CI. +""" + +import ast +import argparse +import hashlib +import json +import re +import sys +from dataclasses import dataclass, asdict +from pathlib import Path + +PROJECT_ROOT = Path(__file__).resolve().parent.parent + + +# ───────────────────────────────────────────────────────────────────────────── +# Result dataclass +# ───────────────────────────────────────────────────────────────────────────── + +@dataclass +class TestResult: + test_id: str + severity: str # PASS | INFO | WARNING | CRITICAL | SKIP + message: str + details: str = "" + + def to_dict(self) -> dict: + return asdict(self) + + @property + def icon(self) -> str: + return { + "PASS": "✅", # nosec B105 - severity label, not a credential + "INFO": "ℹ️ ", + "WARNING": "⚠️ ", + "CRITICAL": "🚨", + "SKIP": "⏭️ ", + }.get(self.severity, "❓") + + +# ───────────────────────────────────────────────────────────────────────────── +# T1: Plugin Loading / Zip Slip +# ───────────────────────────────────────────────────────────────────────────── + +def test_t1a_zip_slip_protection() -> TestResult: + """ + Verify that zip-slip protection actually guards zip extraction in + store_manager.py. + + A whole-file substring check for "is_relative_to"/"Zip-slip detected" + would pass even if the guard existed somewhere unrelated, or covered + only one of several extract()/extractall() call sites. Instead, this + walks the AST: for every extract()/extractall() call, it confirms an + is_relative_to() check (and the "Zip-slip detected" log) appears + earlier in that same enclosing function -- validate-then-bulk-extract + (validate every member, then call extractall() only after all passed) + counts as protecting the call, since it covers the same member list. + """ + store_manager = PROJECT_ROOT / "src" / "plugin_system" / "store_manager.py" + if not store_manager.exists(): + return TestResult("T1a", "CRITICAL", + "store_manager.py not found", + f"Expected at {store_manager}") + + content = store_manager.read_text(encoding="utf-8") + try: + tree = ast.parse(content, filename=str(store_manager)) + except SyntaxError as exc: + return TestResult("T1a", "CRITICAL", + "store_manager.py could not be parsed", + str(exc)) + + extraction_sites = 0 + unprotected: list[str] = [] + + for func in ast.walk(tree): + if not isinstance(func, (ast.FunctionDef, ast.AsyncFunctionDef)): + continue + + extract_calls = [ + node for node in ast.walk(func) + if isinstance(node, ast.Call) and isinstance(node.func, ast.Attribute) + and node.func.attr in ("extract", "extractall") + ] + if not extract_calls: + continue + extraction_sites += len(extract_calls) + + guard_lines = [ + n.lineno for n in ast.walk(func) + if isinstance(n, ast.Attribute) and n.attr == "is_relative_to" + ] + has_zip_slip_log = any( + isinstance(n, ast.Constant) and isinstance(n.value, str) + and "Zip-slip detected" in n.value + for n in ast.walk(func) + ) + + for call in extract_calls: + guarded = has_zip_slip_log and any(g < call.lineno for g in guard_lines) + if not guarded: + unprotected.append( + f"{func.name}() line {call.lineno}: {call.func.attr}() call not " + f"clearly preceded by an is_relative_to() guard + Zip-slip log " + f"in the same function" + ) + + if extraction_sites == 0: + return TestResult("T1a", "WARNING", + "No zipfile extract()/extractall() calls found in store_manager.py", + "Verify plugin installation no longer extracts zip archives, " + "or that this check still targets the right file") + + if unprotected: + return TestResult("T1a", "CRITICAL", + f"{len(unprotected)} of {extraction_sites} zip extraction " + f"call(s) not clearly guarded", + "; ".join(unprotected)) + + return TestResult("T1a", "PASS", + "Zip-slip protection verified", + f"All {extraction_sites} extract()/extractall() call(s) in " + f"store_manager.py are preceded by an is_relative_to() guard " + f"with a Zip-slip log in the same function") + + +def test_t1b_dangerous_plugin_calls() -> list[TestResult]: + """ + Scan plugin directories for dangerous function calls (eval, exec). + These represent arbitrary code execution risks in plugin code. + """ + results = [] + plugin_dirs = [ + PROJECT_ROOT / "plugins", + PROJECT_ROOT / "plugin-repos", + ] + + violations: list[str] = [] + files_scanned = 0 + + scan_errors: list[str] = [] + + for base in plugin_dirs: + if not base.exists(): + continue + for plugin_dir in sorted(base.iterdir()): + if not plugin_dir.is_dir() or plugin_dir.name.startswith(('.', '_')): + continue + for py_file in plugin_dir.rglob("*.py"): + files_scanned += 1 + try: + source = py_file.read_text(encoding="utf-8") + tree = ast.parse(source, filename=str(py_file)) + for node in ast.walk(tree): + if isinstance(node, ast.Call) and isinstance(node.func, ast.Name): + if node.func.id in ("eval", "exec"): + rel = py_file.relative_to(PROJECT_ROOT) + violations.append( + f"{rel}:{node.lineno} — {node.func.id}() call") + except (SyntaxError, OSError) as exc: + # A file we couldn't parse/read was never actually + # scanned for eval()/exec() -- that must block this + # test, not silently pass as if it were clean. + rel = py_file.relative_to(PROJECT_ROOT) + scan_errors.append(f"{rel} — {type(exc).__name__}: {exc}") + + if scan_errors: + results.append(TestResult( + "T1b", "CRITICAL", + f"{len(scan_errors)} plugin file(s) could not be scanned for eval()/exec()", + "; ".join(scan_errors[:10]) + )) + + if violations: + results.append(TestResult( + "T1b", "CRITICAL", + f"Dangerous function calls found in plugins ({len(violations)} instance(s))", + "; ".join(violations[:10]) + )) + elif not scan_errors: + results.append(TestResult( + "T1b", "PASS", + "No eval()/exec() calls found in plugins", + f"{files_scanned} plugin Python files scanned" + )) + + return results + + +# ───────────────────────────────────────────────────────────────────────────── +# T2: API Surface Inventory +# ───────────────────────────────────────────────────────────────────────────── + +def test_t2a_api_surface_inventory() -> TestResult: + """ + Document the API surface area. + + This app intentionally has no authentication (local-only Raspberry Pi + design, documented in web_interface/app.py). This test produces an + inventory for audit purposes and warns only if the design-intent comment + is removed from app.py (which would indicate someone deleted the rationale + without adding auth, rather than a deliberate undocumented change). + """ + api_file = PROJECT_ROOT / "web_interface" / "blueprints" / "api_v3.py" + app_file = PROJECT_ROOT / "web_interface" / "app.py" + + if not api_file.exists(): + return TestResult("T2a", "WARNING", "api_v3.py not found", str(api_file)) + + api_content = api_file.read_text(encoding="utf-8") + routes = re.findall(r"@api_v3\.route\('([^']+)'", api_content) + + csrf_documented = False + if app_file.exists(): + app_content = app_file.read_text(encoding="utf-8") + csrf_documented = "CSRF protection disabled for local-only" in app_content + + summary = ( + f"{len(routes)} API routes in api_v3.py. " + f"No auth decorators (intentional local-only design). " + f"CSRF disabled: {'YES — design intent documented in app.py' if csrf_documented else 'YES — but design intent comment NOT found in app.py'}. " + f"Rate limiting: 1000/min." + ) + + if not csrf_documented: + return TestResult( + "T2a", "WARNING", + "CSRF is disabled but the design-intent comment is missing from app.py", + "Add the rationale comment back, or add proper CSRF protection if " + "the app is now internet-facing" + ) + + # There is currently no config mechanism that actually enforces the + # local-only boundary the design-intent comment describes -- app.py + # hardcodes host='0.0.0.0' unconditionally, so nothing here can confirm + # this deployment is in fact LAN-only. Reporting this as mere INFO + # understates that: an unauthenticated, CSRF-disabled API surface is a + # real risk the moment this ever runs somewhere other than a home LAN, + # documented rationale or not. + return TestResult( + "T2a", "WARNING", + "API surface has no auth and CSRF disabled; enforcement of the " + "documented local-only boundary cannot be confirmed", + summary + ) + + +# ───────────────────────────────────────────────────────────────────────────── +# T3: Secrets & Credential Handling +# ───────────────────────────────────────────────────────────────────────────── + +# Patterns that suggest real credentials (must be >8 chars, not placeholders) +_SECRET_PATTERNS = [ + (r'(?i)password\s*=\s*["\'](?!none|empty|placeholder|example|test|default|""|'')[^"\']{8,}["\']', "WARNING", "password"), + (r'(?i)api[_-]?key\s*=\s*["\'](?!none|empty|placeholder|YOUR_|example|test)[^"\']{16,}["\']', "WARNING", "api_key"), + (r'(?i)secret\s*=\s*["\'](?!none|empty|placeholder|YOUR_|example|test)[^"\']{16,}["\']', "WARNING", "secret"), + # Real GitHub token pattern + (r'ghp_[a-zA-Z0-9]{36}', "CRITICAL", "github_token"), + # Generic long bearer tokens + (r'Bearer\s+[a-zA-Z0-9\-_\.]{32,}', "WARNING", "bearer_token"), +] + +_TEMPLATE_SKIP_STRINGS = [ + "YOUR_", "PLACEHOLDER", "_HERE", "example.com", "config_secrets.template", + "prove_security", # this file itself +] + +_SCAN_DIRS = ["src", "web_interface", "scripts"] + + +def test_t3a_hardcoded_secrets() -> TestResult: + """Scan source code for hardcoded credentials.""" + violations: list[str] = [] + + for dir_name in _SCAN_DIRS: + scan_dir = PROJECT_ROOT / dir_name + if not scan_dir.exists(): + continue + for py_file in scan_dir.rglob("*.py"): + # Skip test files and this script + if "test" in str(py_file).lower() or "prove_security" in str(py_file): + continue + try: + content = py_file.read_text(encoding="utf-8") + except OSError: + continue + + for pattern, severity, pattern_type in _SECRET_PATTERNS: + for match in re.finditer(pattern, content): + line_content = match.group(0) + # Skip lines containing template placeholder strings. + # line_content is only used for this in-memory check -- + # it must never be stored or included in output below. + if any(skip in line_content for skip in _TEMPLATE_SKIP_STRINGS): + continue + rel = py_file.relative_to(PROJECT_ROOT) + line_no = content[: match.start()].count("\n") + 1 + # Redacted fingerprint lets the same finding be recognized + # across scans without ever reporting the matched + # credential itself (which would otherwise get published + # into CI logs, JSON artifacts, and PR comments -- wider + # exposure than the original leak). + fingerprint = hashlib.sha256(line_content.encode()).hexdigest()[:12] + violations.append( + f"[{severity}] {rel}:{line_no} — {pattern_type} " + f"(fingerprint {fingerprint})" + ) + + critical_violations = [v for v in violations if "[CRITICAL]" in v] + if critical_violations: + return TestResult( + "T3a", "CRITICAL", + f"Hardcoded secrets found ({len(critical_violations)} critical)", + "; ".join(critical_violations[:5]) + ) + if violations: + return TestResult( + "T3a", "WARNING", + f"Potential hardcoded secrets found ({len(violations)} instance(s))", + "; ".join(violations[:5]) + ) + + return TestResult("T3a", "PASS", "No hardcoded secrets detected", + f"Scanned {', '.join(_SCAN_DIRS)}") + + +def test_t3b_plaintext_password_storage() -> TestResult: + """ + Check for user account password storage without hashing. + + The LEDMatrix app has no user account system, so this should produce INFO. + It would only CRITICAL if someone added user auth and stored passwords without hashing. + + We require all three of: a password *variable assignment or DB operation*, + a clear storage call (INSERT / db commit / ORM save), and no hashing lib present + — to avoid false positives from files that contain 'password' for WiFi handling + and '.save()' for image/file saving in unrelated functions. + """ + hashing_libs = ["bcrypt", "argon2", "pbkdf2", "scrypt", + "generate_password_hash", "hashpw", "make_password"] + # Patterns that indicate password being stored in a database / ORM context. + # Must be specific enough to avoid matching set.add(), file.save(), etc. + db_storage_patterns = ["INSERT INTO", "db.session", "session.add(", "session.commit(", "orm.save"] + + password_storage_found = False + + for dir_name in _SCAN_DIRS: + scan_dir = PROJECT_ROOT / dir_name + if not scan_dir.exists(): + continue + for py_file in scan_dir.rglob("*.py"): + try: + content = py_file.read_text(encoding="utf-8") + except OSError: + continue + # Require DB/ORM context specifically — not just any .save() call + if ("password" in content.lower() and + any(store in content for store in db_storage_patterns) and + not any(h in content for h in hashing_libs)): + password_storage_found = True + + if password_storage_found: + return TestResult( + "T3b", "CRITICAL", + "Potential plaintext password storage in database/ORM detected", + "Found password + database storage operations without a recognized hashing library" + ) + + return TestResult("T3b", "INFO", + "No plaintext password storage detected", + "App has no user account system — expected result") + + +# ───────────────────────────────────────────────────────────────────────────── +# T4: Path Traversal +# ───────────────────────────────────────────────────────────────────────────── + +def test_t4a_path_traversal() -> TestResult: + """ + Verify static file serving uses send_from_directory (safe) rather than + open() with user-supplied paths. Also checks for extractall() calls that + lack the is_relative_to() guard. + """ + issues: list[str] = [] + + app_file = PROJECT_ROOT / "web_interface" / "app.py" + if app_file.exists(): + content = app_file.read_text(encoding="utf-8") + # The file-serve route should use send_from_directory or commonpath + if "send_from_directory" not in content and "commonpath" not in content: + issues.append("app.py: file-serve routes may not use send_from_directory/commonpath") + + # Check all extractall() calls have a preceding is_relative_to guard + for py_file in (PROJECT_ROOT / "src").rglob("*.py"): + try: + content = py_file.read_text(encoding="utf-8") + except OSError: + continue + if "extractall(" in content and "is_relative_to" not in content: + rel = py_file.relative_to(PROJECT_ROOT) + issues.append(f"{rel}: extractall() without is_relative_to() guard") + + if issues: + return TestResult( + "T4a", "WARNING", + f"Potential path traversal patterns found ({len(issues)})", + "; ".join(issues) + ) + + return TestResult("T4a", "PASS", + "Path traversal mitigations verified", + "send_from_directory/commonpath used for file serving; " + "extractall() calls have is_relative_to() guards") + + +# ───────────────────────────────────────────────────────────────────────────── +# T5: Auth Bypass Patterns +# ───────────────────────────────────────────────────────────────────────────── + +def test_t5a_auth_bypass_patterns() -> TestResult: + """ + Look for broken auth bypass patterns — not the intentional no-auth design + (T2a covers that), but patterns that suggest auth was INTENDED to exist + but has an exploitable bypass: broad substring matching, debug-mode skips, + or if-True conditions. + """ + bypass_signals = [ + (r'if\s+True\s*:', "if True: bypass"), + (r'if\s+debug\s*:', "debug-mode auth skip"), + (r'request\.path\s+in\s+', "substring path matching in auth (Huntarr pattern)"), + (r'EXEMPT_ROUTES\s*=', "exempt routes list"), + ] + + findings: list[str] = [] + + for dir_name in ["src", "web_interface"]: + scan_dir = PROJECT_ROOT / dir_name + if not scan_dir.exists(): + continue + for py_file in scan_dir.rglob("*.py"): + try: + content = py_file.read_text(encoding="utf-8") + except OSError: + continue + for pattern, label in bypass_signals: + if re.search(pattern, content): + # Only flag if the file also contains auth-related terms + if any(auth in content.lower() for auth in + ["auth", "login", "authenticate", "token", "permission"]): + rel = py_file.relative_to(PROJECT_ROOT) + findings.append(f"{rel}: {label}") + + if findings: + return TestResult( + "T5a", "WARNING", + f"Potential auth bypass patterns found ({len(findings)})", + "; ".join(findings[:5]) + ) + + return TestResult("T5a", "PASS", + "No auth bypass patterns detected", + "Checked src/ and web_interface/ for bypass signals") + + +# ───────────────────────────────────────────────────────────────────────────── +# T6: Docker / Container Hardening +# ───────────────────────────────────────────────────────────────────────────── + +def test_t6_docker_hardening() -> TestResult: + """Container security — skipped if no Dockerfile exists.""" + dockerfile = PROJECT_ROOT / "Dockerfile" + if not dockerfile.exists(): + return TestResult("T6", "SKIP", + "No Dockerfile found — container security scan not applicable", + "If Docker support is added in future, enable hadolint/trivy scanning " + "in .github/workflows/security-audit.yml") + + content = dockerfile.read_text(encoding="utf-8") + issues: list[str] = [] + + # Check for non-root USER directive + user_lines = [l for l in content.splitlines() if l.strip().startswith("USER")] + if not user_lines or user_lines[-1].strip() == "USER root": + issues.append("Container runs as root — use USER directive to drop privileges") + + # Check for pinned base image tags. A tag (even a specific version, not + # just :latest) is mutable -- the same tag can point to a different + # image later. Only a @sha256 digest is truly immutable/reproducible. + from_lines = [line for line in content.splitlines() if line.strip().startswith("FROM")] + for from_line in from_lines: + parts = from_line.split() + # FROM [--platform=] [AS ] -- skip an + # optional --platform= flag so it's never mistaken for the image + # token itself (which would falsely report it as unpinned). + image_parts = [p for p in parts[1:] if not p.startswith("--platform=")] + if image_parts: + image = image_parts[0] + if "@sha256:" not in image: + issues.append(f"Base image not pinned to a digest: {image}") + + if issues: + return TestResult("T6", "WARNING", + f"Dockerfile hardening issues ({len(issues)})", + "; ".join(issues)) + + return TestResult("T6", "PASS", "Dockerfile hardening checks passed", "") + + +# ───────────────────────────────────────────────────────────────────────────── +# Runner +# ───────────────────────────────────────────────────────────────────────────── + +def main() -> int: + parser = argparse.ArgumentParser( + description="LEDMatrix security proof tests", + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + parser.add_argument("--output", "-o", default=None, + help="Write JSON results to this file") + parser.add_argument("--verbose", "-v", action="store_true", + help="Show details for each check") + args = parser.parse_args() + + print("=" * 60) + print("LEDMatrix Security Proof Tests") + print(f"Project root: {PROJECT_ROOT}") + print("=" * 60) + + all_results: list[TestResult] = [] + + # Run all test groups + all_results.append(test_t1a_zip_slip_protection()) + all_results.extend(test_t1b_dangerous_plugin_calls()) + all_results.append(test_t2a_api_surface_inventory()) + all_results.append(test_t3a_hardcoded_secrets()) + all_results.append(test_t3b_plaintext_password_storage()) + all_results.append(test_t4a_path_traversal()) + all_results.append(test_t5a_auth_bypass_patterns()) + all_results.append(test_t6_docker_hardening()) + + # Print results + print() + for r in all_results: + line = f" {r.icon} [{r.severity:<8}] {r.test_id}: {r.message}" + print(line) + if args.verbose and r.details: + print(f" {r.details}") + + # Tally + critical = [r for r in all_results if r.severity == "CRITICAL"] + warnings = [r for r in all_results if r.severity == "WARNING"] + passed = [r for r in all_results if r.severity == "PASS"] + skipped = [r for r in all_results if r.severity == "SKIP"] + + print() + print(f" Results: {len(passed)} PASS {len(warnings)} WARN " + f"{len(critical)} CRITICAL {len(skipped)} SKIP") + + # Write JSON output + if args.output: + output_data = [r.to_dict() for r in all_results] + Path(args.output).write_text( + json.dumps(output_data, indent=2), encoding="utf-8" + ) + print(f" Results written to: {args.output}") + + if critical: + print(f"\n 🚨 {len(critical)} CRITICAL issue(s) found — blocking") + return 1 + + if warnings: + print(f"\n ⚠️ {len(warnings)} warning(s) found — non-blocking") + + print("\n ✅ All checks passed (warnings are non-blocking)") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/remove_plugin_backups.sh b/scripts/remove_plugin_backups.sh deleted file mode 100755 index ab8146b4..00000000 --- a/scripts/remove_plugin_backups.sh +++ /dev/null @@ -1,117 +0,0 @@ -#!/bin/bash - -# Script to safely remove plugin backup directories -# These were created during the plugin-to-submodule conversion - -set -euo pipefail - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" -PLUGINS_DIR="$PROJECT_ROOT/plugins" - -# Colors -GREEN='\033[0;32m' -YELLOW='\033[1;33m' -RED='\033[0;31m' -NC='\033[0m' - -log_info() { - echo -e "${GREEN}[INFO]${NC} $1" -} - -log_warn() { - echo -e "${YELLOW}[WARN]${NC} $1" -} - -log_error() { - echo -e "${RED}[ERROR]${NC} $1" -} - -# Verify submodules are working -verify_submodules() { - log_info "Verifying submodules are working..." - local issues=0 - - for submod in football-scoreboard hockey-scoreboard ledmatrix-flights \ - ledmatrix-leaderboard ledmatrix-stocks ledmatrix-weather \ - mqtt-notifications; do - if [ ! -d "$PLUGINS_DIR/$submod" ]; then - log_error "Submodule directory missing: $submod" - issues=$((issues + 1)) - elif [ ! -f "$PLUGINS_DIR/$submod/.git" ]; then - log_error "Submodule .git file missing: $submod" - issues=$((issues + 1)) - elif [ ! -f "$PLUGINS_DIR/$submod/manifest.json" ]; then - log_warn "Submodule manifest missing: $submod (may be OK)" - fi - done - - if [ $issues -eq 0 ]; then - log_info "All submodules verified ✓" - return 0 - else - log_error "Found $issues issues with submodules" - return 1 - fi -} - -# Remove backup directories -remove_backups() { - log_info "Removing backup directories..." - - local removed=0 - local total_size=0 - - for backup in "$PLUGINS_DIR"/*.backup*; do - if [ -d "$backup" ]; then - local name=$(basename "$backup") - local size=$(du -sb "$backup" 2>/dev/null | awk '{print $1}') - total_size=$((total_size + size)) - - log_info "Removing: $name" - rm -rf "$backup" - removed=$((removed + 1)) - fi - done - - if [ $removed -gt 0 ]; then - log_info "Removed $removed backup directory(ies)" - log_info "Freed approximately $(numfmt --to=iec-i --suffix=B $total_size 2>/dev/null || echo "$total_size bytes")" - else - log_info "No backup directories found" - fi -} - -# Main -main() { - cd "$PROJECT_ROOT" - - echo "=== Plugin Backup Removal Script ===" - echo - - # Verify submodules first - if ! verify_submodules; then - log_error "Submodule verification failed. Not removing backups." - log_warn "Please fix submodule issues before removing backups." - exit 1 - fi - - echo - log_warn "This will permanently delete backup directories:" - ls -1d "$PLUGINS_DIR"/*.backup* 2>/dev/null | sed 's|.*/| - |' || echo " (none found)" - echo - - read -p "Continue? (y/N): " -n 1 -r - echo - if [[ ! $REPLY =~ ^[Yy]$ ]]; then - log_info "Aborted" - exit 0 - fi - - remove_backups - - log_info "Done!" -} - -main "$@" - diff --git a/scripts/utils/clear_cache.py b/scripts/utils/clear_cache.py index 6d4d7b0d..c490435b 100644 --- a/scripts/utils/clear_cache.py +++ b/scripts/utils/clear_cache.py @@ -8,10 +8,10 @@ import os import sys import argparse -# Add the src directory to the path so we can import our modules -sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'src')) +# Add the project root to the path so we can import our modules +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))) -from cache_manager import CacheManager +from src.cache_manager import CacheManager def list_cache_keys(cache_manager): """List all available cache keys.""" diff --git a/scripts/validate_skin.py b/scripts/validate_skin.py new file mode 100644 index 00000000..52d7d61d --- /dev/null +++ b/scripts/validate_skin.py @@ -0,0 +1,248 @@ +#!/usr/bin/env python3 +""" +Headless skin validator — render a skin against bundled fixture games at +multiple panel sizes without hardware, a network, or a running service. + + python scripts/validate_skin.py --skin my-skin + python scripts/validate_skin.py --skin my-skin --sport baseball \ + --size 128x32 --size 64x32 --output-dir /tmp/skin_renders + +For each (mode x size) it checks: the manifest loads and its API version +matches, the render raises no exception, the canvas isn't blank, and the +render finishes inside a time budget (warn — the live renderer runs every +display-loop pass, and a Pi is far slower than your dev machine). PNGs are +saved (native plus 4x nearest-neighbor previews) so you can eyeball the +result. Exit code is non-zero when any check fails. +""" + +import argparse +import json +import logging +import sys +import time +from pathlib import Path + +PROJECT_ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(PROJECT_ROOT)) + +from PIL import Image, ImageDraw, ImageFont # noqa: E402 + +FIXTURES_DIR = PROJECT_ROOT / "src" / "skin_system" / "fixtures" +MODES = ("live", "recent", "upcoming") +SPORTS = ("baseball", "basketball", "football", "hockey") +RENDER_BUDGET_S = 0.100 + + +class FixtureHost: + """Stands in for a SportsCore instance: fonts, logger, logo loading, + outlined text — everything build_context needs, no network.""" + + def __init__(self, sport: str, skin_options: dict) -> None: + self.sport = sport + self.sport_key = sport + self.skin_options = skin_options + self.logger = logging.getLogger(f"validate_skin.{sport}") + self.fonts = self._load_fonts() + self._logo_cache = {} + self.display_manager = None # build_context is always given a size + + def _load_fonts(self) -> dict: + """Load the SportsCore font set (TTF, with PIL default fallback).""" + fonts = {} + try: + press = str(PROJECT_ROOT / "assets/fonts/PressStart2P-Regular.ttf") + small = str(PROJECT_ROOT / "assets/fonts/4x6-font.ttf") + fonts['score'] = ImageFont.truetype(press, 10) + fonts['time'] = ImageFont.truetype(press, 8) + fonts['team'] = ImageFont.truetype(press, 8) + fonts['status'] = ImageFont.truetype(small, 6) + fonts['detail'] = ImageFont.truetype(small, 6) + fonts['rank'] = ImageFont.truetype(press, 10) + except IOError: + default = ImageFont.load_default() + for key in ('score', 'time', 'team', 'status', 'detail', 'rank'): + fonts[key] = default + return fonts + + def _load_and_resize_logo(self, team_id: str, team_abbrev: str, + logo_path, logo_url) -> "Image.Image | None": + """Load a fixture logo from disk (no downloads), cached per team.""" + if team_abbrev in self._logo_cache: + return self._logo_cache[team_abbrev] + path = Path(logo_path) + if not path.is_absolute(): + path = PROJECT_ROOT / path + if not path.exists(): + return None + logo = Image.open(path).convert('RGBA') + self._logo_cache[team_abbrev] = logo + return logo + + def _draw_text_with_outline(self, draw: "ImageDraw.ImageDraw", text: str, + position: tuple, font, + fill: tuple = (255, 255, 255), + outline_color: tuple = (0, 0, 0)) -> None: + """Classic outlined scorebug text, same as SportsCore's helper.""" + x, y = position + for dx, dy in [(-1, -1), (-1, 0), (-1, 1), (0, -1), (0, 1), + (1, -1), (1, 0), (1, 1)]: + draw.text((x + dx, y + dy), text, font=font, fill=outline_color) + draw.text((x, y), text, font=font, fill=fill) + + +def load_fixture(sport: str, mode: str) -> dict: + with open(FIXTURES_DIR / f"{sport}_{mode}.json", encoding="utf-8") as f: + game = json.load(f) + # Real view models carry start_time_utc as a UTC datetime, not a string. + if isinstance(game.get("start_time_utc"), str): + from datetime import datetime + game["start_time_utc"] = datetime.fromisoformat(game["start_time_utc"]) + return game + + +def parse_size(value: str) -> "tuple[int, int]": + try: + w_text, h_text = value.lower().split("x") + w, h = int(w_text), int(h_text) + except ValueError as exc: + raise argparse.ArgumentTypeError(f"size must look like 128x32, got {value!r}") from exc + if w <= 0 or h <= 0: + raise argparse.ArgumentTypeError(f"size dimensions must be positive, got {value!r}") + return w, h + + +def parse_options(value: str) -> dict: + try: + options = json.loads(value) + except json.JSONDecodeError as exc: + raise argparse.ArgumentTypeError(f"options must be valid JSON: {exc.msg}") from exc + if not isinstance(options, dict): + raise argparse.ArgumentTypeError("options must be a JSON object") + return options + + +def display_path(path: Path) -> str: + """Repo-relative when inside the repo, absolute otherwise (--output-dir + may point anywhere, e.g. /tmp/skin_renders).""" + try: + return str(path.relative_to(PROJECT_ROOT)) + except ValueError: + return str(path) + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument("--skin", required=True, help="skin id (directory name under skins/)") + parser.add_argument("--sport", choices=SPORTS, + help="fixture sport (default: first sport the skin targets, else baseball)") + parser.add_argument("--size", action="append", type=parse_size, dest="sizes", + metavar="WxH", help="panel size to render at (repeatable; default 128x32 and 64x32)") + parser.add_argument("--output-dir", type=Path, + default=PROJECT_ROOT / "skin_renders", + help="where rendered PNGs are written") + parser.add_argument("--options", type=parse_options, default={}, + help="skin_options JSON to pass the skin") + args = parser.parse_args() + sizes = args.sizes or [(128, 32), (64, 32)] + + logging.basicConfig(level=logging.INFO, format="%(levelname)s %(name)s: %(message)s") + + from src.skin_system import skin_runtime + from src.skin_system.skin_base import SKIN_API_VERSION + + skins = skin_runtime.discover_skins() + manifest = skins.get(args.skin) + if manifest is None: + print(f"FAIL: skin '{args.skin}' not found under {skin_runtime.get_skins_directory()}") + if skins: + print(f" installed skins: {', '.join(sorted(skins))}") + return 1 + + sport = args.sport + if sport is None: + declared = skin_runtime.skin_targets(manifest)[0] + sport = next((s for s in declared if s in SPORTS), "baseball") + + skin = skin_runtime.load_skin(args.skin, sport=sport, sport_key=sport, + options=args.options) + if skin is None: + print(f"FAIL: skin '{args.skin}' did not load " + f"(see log above; host API is {SKIN_API_VERSION})") + return 1 + + host = FixtureHost(sport, args.options) + args.output_dir.mkdir(parents=True, exist_ok=True) + failures = 0 + rendered = 0 + + for mode in MODES: + game = load_fixture(sport, mode) + render = getattr(skin, f"render_{mode}") + for width, height in sizes: + label = f"{mode}@{width}x{height}" + try: + # Warm-up render absorbs one-time font/image loads, second + # render is the one timed against the budget. + ctx = skin_runtime.build_context(host, game, size=(width, height)) + handled = render(ctx, dict(game)) + if handled: + ctx = skin_runtime.build_context(host, game, size=(width, height)) + started = time.monotonic() + handled = render(ctx, dict(game)) + elapsed = time.monotonic() - started + else: + elapsed = 0.0 + except Exception as e: + print(f"FAIL {label}: render raised {type(e).__name__}: {e}") + import traceback + traceback.print_exc() + failures += 1 + continue + + if not handled: + print(f"skip {label}: render_{mode} returned False (built-in renderer would be used)") + continue + + if ctx.canvas.size != (width, height): + print(f"FAIL {label}: canvas was replaced/resized to {ctx.canvas.size} — draw onto ctx.canvas, never reassign it") + failures += 1 + continue + if ctx.canvas.convert("L").getbbox() is None: + print(f"FAIL {label}: canvas is blank — render returned True but drew nothing") + failures += 1 + continue + if elapsed > RENDER_BUDGET_S: + print(f"WARN {label}: render took {elapsed * 1000:.0f}ms " + f"(budget {RENDER_BUDGET_S * 1000:.0f}ms; a Pi is much slower than this machine)") + + out = args.output_dir / f"{args.skin}_{sport}_{mode}_{width}x{height}.png" + ctx.canvas.save(out) + preview = ctx.canvas.resize((width * 4, height * 4), Image.NEAREST) + preview.save(out.with_name(out.stem + "_x4.png")) + print(f"ok {label}: {elapsed * 1000:.0f}ms -> {display_path(out)}") + rendered += 1 + + # Vegas card, once per mode at the first size (optional API) + try: + width, height = sizes[0] + ctx = skin_runtime.build_context(host, game, size=(width, height)) + card = skin.render_vegas_card(ctx, dict(game)) + if card is not None: + out = args.output_dir / f"{args.skin}_{sport}_{mode}_vegas.png" + card.save(out) + print(f"ok {mode} vegas card -> {display_path(out)}") + except Exception as e: + print(f"FAIL {mode} vegas card: {type(e).__name__}: {e}") + failures += 1 + + if rendered == 0 and failures == 0: + print(f"FAIL: skin '{args.skin}' rendered nothing — no render_ returned True") + return 1 + print(f"\n{'FAILED' if failures else 'PASSED'}: {rendered} renders, {failures} failures " + f"(PNGs in {args.output_dir})") + return 1 if failures else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/skins/README.md b/skins/README.md new file mode 100644 index 00000000..adf62491 --- /dev/null +++ b/skins/README.md @@ -0,0 +1,23 @@ +# skins/ + +User-installable **visual skins** for the sports scoreboards. Each +subdirectory is one skin: + +```text +skins// + skin.json # manifest + skin.py # renderer (a ScoreboardSkin subclass) + preview.png # optional +``` + +- Install a skin: `git clone skins/` (or via the Plugin + Store for registry entries with `"type": "skin"`). +- Select it: set `"skin": ""` in the plugin's section of + `config/config.json`, or use the web UI's Visual Skin dropdown. +- Build one: start from `example-classic-baseball/` and read + [docs/CREATING_SKINS.md](../docs/CREATING_SKINS.md). Validate with + `python scripts/validate_skin.py --skin `. + +Skins survive plugin reinstalls/updates (that's why they live here and not in +the plugin's directory). A skin is Python at the same trust level as a +plugin — review before installing. diff --git a/skins/example-classic-baseball/preview.png b/skins/example-classic-baseball/preview.png new file mode 100644 index 00000000..37ad2dad Binary files /dev/null and b/skins/example-classic-baseball/preview.png differ diff --git a/skins/example-classic-baseball/skin.json b/skins/example-classic-baseball/skin.json new file mode 100644 index 00000000..ccec3071 --- /dev/null +++ b/skins/example-classic-baseball/skin.json @@ -0,0 +1,25 @@ +{ + "id": "example-classic-baseball", + "name": "Example: Classic Baseball", + "version": "1.0.0", + "author": "LEDMatrix", + "description": "Reference skin: a restyled baseball scorebug demonstrating the skin API. Copy this directory to start your own skin.", + "skin_api_version": "1.0.0", + "targets": { + "sports": [ + "baseball" + ], + "sport_keys": [ + "mlb", + "milb" + ] + }, + "entry_point": "skin.py", + "class_name": "ClassicBaseballSkin", + "modes": [ + "live", + "recent", + "upcoming" + ], + "preview": "preview.png" +} diff --git a/skins/example-classic-baseball/skin.py b/skins/example-classic-baseball/skin.py new file mode 100644 index 00000000..6882a52c --- /dev/null +++ b/skins/example-classic-baseball/skin.py @@ -0,0 +1,131 @@ +""" +Example: Classic Baseball — the reference skin. + +Shows the whole skin API surface on purpose: adaptive regions +(scoreboard_regions), fitted text (ctx.layout.fit_text + ctx.draw_fit), +logos (ctx.load_logo + ctx.draw_image), raw PIL (ctx.draw for the bases +diamond), and per-user options (ctx.options). Everything is derived from +ctx and the game dict — a skin holds no state, does no I/O, and never +touches the display. + +Copy this directory to skins//, rename the class and the +manifest fields, and run: + + python scripts/validate_skin.py --skin +""" + +from src.adaptive_layout import LADDER_GRID, scoreboard_regions +from src.skin_system.skin_base import ScoreboardSkin, SkinContext + +DEFAULT_ACCENT = (255, 200, 0) + + +class ClassicBaseballSkin(ScoreboardSkin): + """Reference baseball skin: classic scorebug with bases/outs/count.""" + + def __init__(self, manifest: dict, options: dict): + super().__init__(manifest, options) + # Validate user options once at load time (fail fast, fall back + # gracefully) rather than surprising every render. + accent = self.options.get("accent_color", DEFAULT_ACCENT) + if (isinstance(accent, (list, tuple)) and len(accent) == 3 + and all(isinstance(c, int) and 0 <= c <= 255 for c in accent)): + self._accent_color = tuple(accent) + else: + import logging + logging.getLogger(__name__).error( + "accent_color must be three 0-255 integers, got %r; using default", accent) + self._accent_color = DEFAULT_ACCENT + + # -- shared pieces ---------------------------------------------------- + + def _accent(self, ctx: SkinContext) -> tuple: + """Users can recolor the skin from config via skin_options.""" + return self._accent_color + + def _draw_card(self, ctx: SkinContext, game: dict, status: str, + center_lines: list, detail: str) -> None: + """The common card: logos left/right, status on top, the given + center content, detail along the bottom.""" + regions = scoreboard_regions(ctx.layout.bounds, ctx=ctx.layout) + + ctx.draw_image(ctx.load_logo("away"), regions.away_slot, + cache_key=f"logo:{game.get('away_abbr')}") + ctx.draw_image(ctx.load_logo("home"), regions.home_slot, + cache_key=f"logo:{game.get('home_abbr')}") + + if status: + fit = ctx.layout.fit_text(status, regions.status_band, LADDER_GRID) + ctx.draw_fit(fit, regions.status_band, color=self._accent(ctx)) + + if center_lines: + rows = regions.score_area.split_v(*[1] * len(center_lines)) + for line, row in zip(center_lines, rows): + if line: + fit = ctx.layout.fit_text(line, row, LADDER_GRID) + ctx.draw_fit(fit, row) + + if detail: + fit = ctx.layout.fit_text(detail, regions.detail_band, LADDER_GRID) + ctx.draw_fit(fit, regions.detail_band, color=(160, 160, 160)) + + def _draw_bases_and_outs(self, ctx: SkinContext, game: dict) -> None: + """Raw-PIL escape hatch: a bases diamond + out dots in the bottom + band, sized from the layout scale so it works on any panel.""" + size = ctx.layout.px(3, minimum=2) # half-diagonal of one base + gap = ctx.layout.px(1) + cx = ctx.width // 2 + cy = ctx.height - (size * 2) - 1 + + bases = game.get("bases_occupied") or [False, False, False] + # (dx, dy) per base: first (right), second (top), third (left) + offsets = [(size + gap, 0), (0, -(size + gap)), (-(size + gap), 0)] + for occupied, (dx, dy) in zip(bases, offsets): + x, y = cx + dx, cy + dy + diamond = [(x, y - size), (x + size, y), (x, y + size), (x - size, y)] + if occupied: + ctx.draw.polygon(diamond, fill=self._accent(ctx)) + else: + ctx.draw.polygon(diamond, outline=(110, 110, 110)) + + outs = min(int(game.get("outs") or 0), 3) + r = max(1, size - 1) + for i in range(3): + x = cx + (i - 1) * (2 * r + 2 * gap) + y = ctx.height - r - 1 + dot = [x - r, y - r, x + r, y + r] + if i < outs: + ctx.draw.ellipse(dot, fill=(255, 255, 255)) + else: + ctx.draw.ellipse(dot, outline=(110, 110, 110)) + + # -- the three modes -------------------------------------------------- + + def render_live(self, ctx: SkinContext, game: dict) -> bool: + half = "▲" if game.get("inning_half") == "top" else "▼" + inning = game.get("inning") or "" + status = f"{half}{inning}" if inning else game.get("status_text", "") + score = f"{game.get('away_score', '0')}-{game.get('home_score', '0')}" + count = f"{game.get('balls', 0)}-{game.get('strikes', 0)}" + + self._draw_card(ctx, game, status, [score], "") + self._draw_bases_and_outs(ctx, game) + + # Ball-strike count in the top-left corner, over the away logo. + fit = ctx.layout.fit_text(count, (ctx.width // 4, ctx.layout.px(8, minimum=6)), LADDER_GRID) + ctx.draw_fit(fit, ctx.layout.bounds.top_band(fit.height + 1).left_col(fit.width + 2), + color=(200, 200, 200)) + return True + + def render_recent(self, ctx: SkinContext, game: dict) -> bool: + score = f"{game.get('away_score', '0')}-{game.get('home_score', '0')}" + self._draw_card(ctx, game, game.get("status_text", "Final"), + [score], game.get("series_summary", "")) + return True + + def render_upcoming(self, ctx: SkinContext, game: dict) -> bool: + matchup = f"{game.get('away_abbr', '')}@{game.get('home_abbr', '')}" + self._draw_card(ctx, game, game.get("game_date", ""), + [matchup, game.get("game_time", "")], + f"{game.get('away_record', '')} {game.get('home_record', '')}".strip()) + return True diff --git a/src/__init__.py b/src/__init__.py index 7d47ee11..f8fa0daa 100644 --- a/src/__init__.py +++ b/src/__init__.py @@ -4,5 +4,5 @@ LEDMatrix Display System Core source package for the LED Matrix Display project. """ -__version__ = "3.1.0" +__version__ = "3.2.0" diff --git a/src/background_cache_mixin.py b/src/background_cache_mixin.py deleted file mode 100644 index 623a5313..00000000 --- a/src/background_cache_mixin.py +++ /dev/null @@ -1,134 +0,0 @@ -""" -Background Cache Mixin for Sports Managers - -This mixin provides common caching functionality to eliminate code duplication -across all sports managers. It implements the background service cache pattern -where Recent/Upcoming managers consume data from the background service cache. -""" - -import time -from typing import Dict, Optional, Any, Callable - - -class BackgroundCacheMixin: - """ - Mixin class that provides background service cache functionality to sports managers. - - This mixin eliminates code duplication by providing a common implementation - for the background service cache pattern used across all sports managers. - - Note: For non-sports managers (weather, stocks, news, etc.), use - GenericCacheMixin instead. See src/generic_cache_mixin.py for details. - """ - - def _fetch_data_with_background_cache(self, - sport_key: str, - api_fetch_method: Callable, - live_manager_class: type = None) -> Optional[Dict]: - """ - Common logic for fetching data with background service cache support. - - This method implements the background service cache pattern: - 1. Live managers always fetch fresh data - 2. Recent/Upcoming managers try background cache first - 3. Fallback to direct API call if background data unavailable - - Args: - sport_key: Sport identifier (e.g., 'nba', 'nfl', 'ncaa_fb') - api_fetch_method: Method to call for direct API fetch - live_manager_class: Class to check if this is a live manager - - Returns: - Cached or fresh data from API - """ - start_time = time.time() - cache_hit = False - cache_source = None - - try: - # For Live managers, always fetch fresh data - if live_manager_class and isinstance(self, live_manager_class): - self.logger.info(f"[{sport_key.upper()}] Live manager - fetching fresh data") - result = api_fetch_method(use_cache=False) - cache_source = "live_fresh" - else: - # For Recent/Upcoming managers, try background service cache first - cache_key = self.cache_manager.generate_sport_cache_key(sport_key) - - # Check if background service has fresh data - if self.cache_manager.is_background_data_available(cache_key, sport_key): - cached_data = self.cache_manager.get_background_cached_data(cache_key, sport_key) - if cached_data: - self.logger.info(f"[{sport_key.upper()}] Using background service cache for {cache_key}") - result = cached_data - cache_hit = True - cache_source = "background_cache" - else: - self.logger.warning(f"[{sport_key.upper()}] Background cache check passed but no data returned for {cache_key}") - result = None - cache_source = "background_miss" - else: - self.logger.info(f"[{sport_key.upper()}] Background data not available for {cache_key}") - result = None - cache_source = "background_unavailable" - - # Fallback to direct API call if background data not available - if result is None: - self.logger.info(f"[{sport_key.upper()}] Fetching directly from API for {cache_key}") - result = api_fetch_method(use_cache=True) - cache_source = "api_fallback" - - # Record performance metrics - duration = time.time() - start_time - self.cache_manager.record_fetch_time(duration) - - # Log performance metrics - self._log_fetch_performance(sport_key, duration, cache_hit, cache_source) - - return result - - except Exception as e: - duration = time.time() - start_time - self.logger.error(f"[{sport_key.upper()}] Error in background cache fetch after {duration:.2f}s: {e}") - self.cache_manager.record_fetch_time(duration) - raise - - def _log_fetch_performance(self, sport_key: str, duration: float, cache_hit: bool, cache_source: str): - """ - Log detailed performance metrics for fetch operations. - - Args: - sport_key: Sport identifier - duration: Fetch operation duration in seconds - cache_hit: Whether this was a cache hit - cache_source: Source of the data (background_cache, api_fallback, etc.) - """ - # Log basic performance info - self.logger.info(f"[{sport_key.upper()}] Fetch completed in {duration:.2f}s " - f"(cache_hit={cache_hit}, source={cache_source})") - - # Log detailed metrics every 10 operations - if hasattr(self, '_fetch_count'): - self._fetch_count += 1 - else: - self._fetch_count = 1 - - if self._fetch_count % 10 == 0: - metrics = self.cache_manager.get_cache_metrics() - self.logger.info(f"[{sport_key.upper()}] Cache Performance Summary - " - f"Hit Rate: {metrics['cache_hit_rate']:.2%}, " - f"Background Hit Rate: {metrics['background_hit_rate']:.2%}, " - f"API Calls Saved: {metrics['api_calls_saved']}") - - def get_cache_performance_summary(self) -> Dict[str, Any]: - """ - Get cache performance summary for this manager. - - Returns: - Dictionary containing cache performance metrics - """ - return self.cache_manager.get_cache_metrics() - - def log_cache_performance(self): - """Log current cache performance metrics.""" - self.cache_manager.log_cache_metrics() diff --git a/src/backup_manager.py b/src/backup_manager.py index c54ce81e..e8d03b18 100644 --- a/src/backup_manager.py +++ b/src/backup_manager.py @@ -16,6 +16,7 @@ import json import logging import os import shutil +import stat import socket import tempfile import zipfile @@ -82,6 +83,10 @@ BUNDLED_FONTS: frozenset[str] = frozenset({ _CONFIG_REL = Path("config/config.json") _SECRETS_REL = Path("config/config_secrets.json") _WIFI_REL = Path("config/wifi_config.json") +# Sits in config/ next to the three above and is pure user state — a +# YouTube Music session that has to be re-authenticated by hand if lost. +# It was omitted from backups, so a restore silently signed the user out. +_YTM_REL = Path("config/ytm_auth.json") _FONTS_REL = Path("assets/fonts") _PLUGIN_UPLOADS_REL = Path("assets/plugins") _STATE_REL = Path("data/plugin_state.json") @@ -303,6 +308,9 @@ def create_backup( if (project_root / _WIFI_REL).exists(): zf.write(project_root / _WIFI_REL, _WIFI_REL.as_posix()) contents.append("wifi") + if (project_root / _YTM_REL).exists(): + zf.write(project_root / _YTM_REL, _YTM_REL.as_posix()) + contents.append("ytm_auth") # User-uploaded fonts. user_fonts = iter_user_fonts(project_root) @@ -348,6 +356,7 @@ def preview_backup_contents(project_root: Path) -> Dict[str, Any]: "has_config": (project_root / _CONFIG_REL).exists(), "has_secrets": (project_root / _SECRETS_REL).exists(), "has_wifi": (project_root / _WIFI_REL).exists(), + "has_ytm_auth": (project_root / _YTM_REL).exists(), "user_fonts": [p.name for p in iter_user_fonts(project_root)], "plugin_uploads": len(iter_plugin_uploads(project_root)), "plugins": list_installed_plugins(project_root), @@ -429,6 +438,8 @@ def validate_backup(zip_path: Path) -> Tuple[bool, str, Dict[str, Any]]: detected.append("secrets") if _WIFI_REL.as_posix() in names: detected.append("wifi") + if _YTM_REL.as_posix() in names: + detected.append("ytm_auth") if any(n.startswith(_FONTS_REL.as_posix() + "/") for n in names): detected.append("fonts") if any( @@ -481,8 +492,61 @@ def _extract_zip_safe(zip_path: Path, dest_dir: Path) -> None: def _copy_file(src: Path, dst: Path) -> None: + """Replace ``dst`` with ``src``, atomically, without needing to own ``dst``. + + ``shutil.copy2`` opens the destination for writing, so it needs write + permission on the *existing file*. Several config files are installed + root-owned and group-readable while the web interface — which is what runs + a restore — deliberately runs as a non-root user. Restoring those failed + with EACCES even though the account could create files in the same + directory perfectly well. + + Writing a temporary file alongside and renaming over the target needs only + directory permission, which the web user has. It is also atomic: a crash + mid-restore can no longer leave a half-written config behind. + + The destination's existing mode is preserved when there is one, so + restoring secrets does not silently widen them to the umask default. + """ dst.parent.mkdir(parents=True, exist_ok=True) - shutil.copy2(src, dst) + + existing_mode: Optional[int] = None + existing_owner: Optional[Tuple[int, int]] = None + if dst.exists(): + try: + info = dst.stat() + existing_mode = stat.S_IMODE(info.st_mode) + existing_owner = (info.st_uid, info.st_gid) + except OSError: + existing_mode = None + existing_owner = None + + fd, tmp_name = tempfile.mkstemp(dir=str(dst.parent), prefix=f".{dst.name}.", suffix=".tmp") + os.close(fd) + tmp_path = Path(tmp_name) + try: + shutil.copyfile(src, tmp_path) + if existing_mode is not None: + os.chmod(tmp_path, existing_mode) + else: + shutil.copymode(src, tmp_path) + if existing_owner is not None: + # Replacing a file creates a new inode owned by whoever is running, + # which would silently move a root-owned config to the web user. + # Carry the previous owner across when the OS permits it — only + # root can hand a file to another user, so this is best-effort and + # a plain restore as the web user simply keeps its own ownership. + try: + os.chown(tmp_path, existing_owner[0], existing_owner[1]) + except (OSError, PermissionError): + pass + os.replace(tmp_path, dst) + except BaseException: + try: + tmp_path.unlink() + except OSError: + pass + raise def restore_backup( @@ -513,7 +577,8 @@ def restore_backup( try: _extract_zip_safe(Path(zip_path), tmp_dir) except (ValueError, zipfile.BadZipFile, OSError) as e: - result.errors.append(f"Failed to extract backup: {e}") + logger.error("[Backup] Failed to extract backup: %s", e, exc_info=True) + result.errors.append("Failed to extract backup") return result # Main config. @@ -522,7 +587,8 @@ def restore_backup( _copy_file(tmp_dir / _CONFIG_REL, project_root / _CONFIG_REL) result.restored.append("config") except OSError as e: - result.errors.append(f"Failed to restore config.json: {e}") + logger.error("[Backup] Failed to restore config.json: %s", e, exc_info=True) + result.errors.append("Failed to restore config.json") elif (tmp_dir / _CONFIG_REL).exists(): result.skipped.append("config") @@ -532,7 +598,10 @@ def restore_backup( _copy_file(tmp_dir / _SECRETS_REL, project_root / _SECRETS_REL) result.restored.append("secrets") except OSError as e: - result.errors.append(f"Failed to restore config_secrets.json: {e}") + logger.error( + "[Backup] Failed to restore config_secrets.json: %s", e, exc_info=True + ) + result.errors.append("Failed to restore config_secrets.json") elif (tmp_dir / _SECRETS_REL).exists(): result.skipped.append("secrets") @@ -542,10 +611,26 @@ def restore_backup( _copy_file(tmp_dir / _WIFI_REL, project_root / _WIFI_REL) result.restored.append("wifi") except OSError as e: - result.errors.append(f"Failed to restore wifi_config.json: {e}") + logger.error( + "[Backup] Failed to restore wifi_config.json: %s", e, exc_info=True + ) + result.errors.append("Failed to restore wifi_config.json") elif (tmp_dir / _WIFI_REL).exists(): result.skipped.append("wifi") + # YouTube Music session. Follows restore_wifi rather than getting its + # own flag: it is device-local auth in the same sense, and a separate + # toggle for one file would be noise in the restore dialog. + if options.restore_wifi and (tmp_dir / _YTM_REL).exists(): + try: + _copy_file(tmp_dir / _YTM_REL, project_root / _YTM_REL) + result.restored.append("ytm_auth") + except OSError as e: + logger.error("[Backup] Failed to restore ytm_auth.json: %s", e, exc_info=True) + result.errors.append("Failed to restore ytm_auth.json") + elif (tmp_dir / _YTM_REL).exists(): + result.skipped.append("ytm_auth") + # User fonts — skip anything that collides with a bundled font. tmp_fonts = tmp_dir / _FONTS_REL if options.restore_fonts and tmp_fonts.exists(): @@ -560,7 +645,10 @@ def restore_backup( _copy_file(font, project_root / _FONTS_REL / font.name) restored_count += 1 except OSError as e: - result.errors.append(f"Failed to restore font {font.name}: {e}") + logger.error( + "[Backup] Failed to restore font %s: %s", font.name, e, exc_info=True + ) + result.errors.append(f"Failed to restore font {font.name}") if restored_count: result.restored.append(f"fonts ({restored_count})") elif tmp_fonts.exists(): @@ -581,7 +669,8 @@ def restore_backup( _copy_file(src, project_root / rel) count += 1 except OSError as e: - result.errors.append(f"Failed to restore {rel}: {e}") + logger.error("[Backup] Failed to restore %s: %s", rel, e, exc_info=True) + result.errors.append(f"Failed to restore {rel}") if count: result.restored.append(f"plugin_uploads ({count})") elif tmp_uploads.exists(): @@ -599,7 +688,8 @@ def restore_backup( if isinstance(p, dict) and p.get("plugin_id") ] except (OSError, json.JSONDecodeError) as e: - result.errors.append(f"Could not read plugins.json: {e}") + logger.error("[Backup] Could not read plugins.json: %s", e, exc_info=True) + result.errors.append("Could not read plugins.json") result.success = not result.errors return result diff --git a/src/base_classes/baseball.py b/src/base_classes/baseball.py index 9a46dd5a..ac84bfb7 100644 --- a/src/base_classes/baseball.py +++ b/src/base_classes/baseball.py @@ -151,7 +151,12 @@ class Baseball(SportsCore): # Only log detailed information for favorite teams if is_favorite_game: - self.logger.debug(f"Full status data: {game_event['status']}") + # Use the validated competition-level `status` here too. MiLB + # events carry no event-level one, so this debug line raised a + # KeyError and dropped the very games it was meant to help + # diagnose -- and only for favourites, which is the worst way + # for it to fail. + self.logger.debug(f"Full status data: {status}") self.logger.debug(f"Status type: {game_status}, State: {status_state}") self.logger.debug(f"Status detail: {status['type'].get('detail', '')}") self.logger.debug( @@ -164,7 +169,13 @@ class Baseball(SportsCore): # Get game state information if status_state == "in": # For live games, get detailed state - inning = game_event["status"].get( + # Use the competition-level `status` already validated by + # _extract_game_details_common. Real ESPN events duplicate + # status at the event top level, but MiLB events (synthesized + # from the MLB Stats API into an ESPN-like shape) populate + # only the competition-level one, so the top-level lookup + # raised a bare KeyError and dropped the event. + inning = status.get( "period", 1 ) # Get inning from status period @@ -187,7 +198,7 @@ class Baseball(SportsCore): if "end" in status_detail or "end" in status_short: inning_half = "top" inning = ( - game_event["status"].get("period", 1) + 1 + status.get("period", 1) + 1 ) # Use period and increment for next inning if is_favorite_game: self.logger.debug( diff --git a/src/base_classes/data_sources.py b/src/base_classes/data_sources.py index 777320ea..065ea577 100644 --- a/src/base_classes/data_sources.py +++ b/src/base_classes/data_sources.py @@ -44,9 +44,16 @@ class DataSource(ABC): """Fetch standings for a sport/league.""" def get_headers(self) -> Dict[str, str]: - """Get headers for API requests.""" + """Get headers for API requests. + + The agent carries the project URL deliberately. Around 2026-08-04 ESPN + began returning 403 for bare custom tokens like 'LEDMatrix/1.0' — and + for browser-style strings — while accepting an agent that identifies + the client and links to it. An Accept header alone does not rescue the + bare form when the request goes out through requests. + """ return { - 'User-Agent': 'LEDMatrix/1.0', + 'User-Agent': 'LEDMatrix/1.0 (+https://github.com/ChuckBuilds/LEDMatrix)', 'Accept': 'application/json' } diff --git a/src/base_classes/hockey.py b/src/base_classes/hockey.py index 9c25da35..419f09ec 100644 --- a/src/base_classes/hockey.py +++ b/src/base_classes/hockey.py @@ -38,10 +38,17 @@ class Hockey(SportsCore): status = competition["status"] powerplay = False penalties = "" + # A competitor may legitimately arrive without a "statistics" + # array (pre-game feeds, and some in-progress ones). Reading it + # unguarded raised KeyError inside the generator and dropped the + # WHOLE event, discarding valid scores and status. Default to an + # empty list so the saves/shots figures fall back to 0 instead. + home_stats = home_team.get("statistics", []) + away_stats = away_team.get("statistics", []) home_team_saves = next( ( int(c["displayValue"]) - for c in home_team["statistics"] + for c in home_stats if c.get("name") == "saves" ), 0, @@ -49,7 +56,7 @@ class Hockey(SportsCore): home_team_saves_per = next( ( float(c["displayValue"]) - for c in home_team["statistics"] + for c in home_stats if c.get("name") == "savePct" ), 0.0, @@ -57,7 +64,7 @@ class Hockey(SportsCore): away_team_saves = next( ( int(c["displayValue"]) - for c in away_team["statistics"] + for c in away_stats if c.get("name") == "saves" ), 0, @@ -65,7 +72,7 @@ class Hockey(SportsCore): away_team_saves_per = next( ( float(c["displayValue"]) - for c in away_team["statistics"] + for c in away_stats if c.get("name") == "savePct" ), 0.0, diff --git a/src/base_classes/sports/__init__.py b/src/base_classes/sports/__init__.py new file mode 100644 index 00000000..a6def39f --- /dev/null +++ b/src/base_classes/sports/__init__.py @@ -0,0 +1,17 @@ +"""Sports scoreboard base classes. + +Formerly the single module ``src/base_classes/sports.py``; now a package so +capabilities can be composed instead of accumulating in one class. See +docs/SPORTS_UNIFICATION.md for the architecture. The import path is +unchanged: ``from src.base_classes.sports import SportsCore`` still works. +""" + +from .core import SportsCore +from .modes import SportsLive, SportsRecent, SportsUpcoming + +__all__ = [ + "SportsCore", + "SportsUpcoming", + "SportsRecent", + "SportsLive", +] diff --git a/src/base_classes/sports/capabilities/__init__.py b/src/base_classes/sports/capabilities/__init__.py new file mode 100644 index 00000000..e084ad7c --- /dev/null +++ b/src/base_classes/sports/capabilities/__init__.py @@ -0,0 +1,32 @@ +"""Opt-in capabilities for the sports scoreboards. + +Each module here is a feature that only *some* sports want. They are composed +by inheritance (mixins) or selected by name (strategies) — never enabled by an +``if self._enabled:`` branch inside the base classes. + +The distinction matters: hockey has no celebrations, so ``HockeyLive`` does not +inherit :class:`~.celebrations.CelebrationMixin` and the celebration code is not +in hockey's MRO at all. A bug in it cannot reach a plugin that never opted in. + +See ``docs/SPORTS_UNIFICATION.md`` for the full rationale. +""" + +from .celebrations import CelebrationMixin +from .rotation import ( + RotationStrategy, + SimpleRotation, + SmoothWeightedRotation, + WeightedCycleRotation, + get_rotation_strategy, + register_rotation_strategy, +) + +__all__ = [ + "CelebrationMixin", + "RotationStrategy", + "SimpleRotation", + "SmoothWeightedRotation", + "WeightedCycleRotation", + "get_rotation_strategy", + "register_rotation_strategy", +] diff --git a/src/base_classes/sports/capabilities/celebrations.py b/src/base_classes/sports/capabilities/celebrations.py new file mode 100644 index 00000000..49d371cd --- /dev/null +++ b/src/base_classes/sports/capabilities/celebrations.py @@ -0,0 +1,418 @@ +"""Score / win celebration takeover — an opt-in capability. + +Four of the nine scoreboards celebrate (afl, nrl, soccer, football); the other +five do not. This is a **mixin** rather than a flag inside ``SportsLive`` so the +five that do not opt in have none of this code in their MRO: a bug here cannot +reach hockey, and hockey's config never grows keys it ignores. + +Usage — mix in *before* the mode class so its ``display`` runs first:: + + class SoccerLive(CelebrationMixin, SportsLive): + def score_phrase(self, points, team_abbr): + return secrets.choice(("GOOOOAAALLL!", f"{team_abbr} SCORES!")) + +The two lineages spelled this differently (``_check_for_goal`` / +``celebrate_opponent_goals`` in the soccer lineage, ``_check_for_score`` / +``celebrate_opponent_scores`` in football) but the bodies were identical apart +from three things, each of which is a seam here rather than a branch: + +* **wording** — :meth:`score_phrase`, the hook football uses to say "TOUCHDOWN" + from the points delta and soccer uses to say "GOOOOAAALLL"; +* **follow-up suppression** — :attr:`COALESCE_SCORING_SEQUENCE`, on for football + where a touchdown lands as +6 then +1 a few seconds later, off elsewhere where + two quick goals are two real events; +* **team identity** — matching goes through ``_favorite_key``, so nrl can match + on team id (its abbreviations are ambiguous) without core knowing why. + +The config keys are read under both spellings, so a plugin adopting the mixin +keeps working with the ``*_goals`` keys already in its published schema. +""" + +from __future__ import annotations + +import re +import time +from typing import Any, Dict, List, Optional + +from PIL import Image, ImageDraw + + +class CelebrationMixin: + """Full-screen takeover when a tracked team scores or wins.""" + + #: Collapse increments that land while a celebration is already on screen + #: into that one celebration. True for sports where a single scoring play + #: arrives as more than one score update (football: touchdown +6, then the + #: extra point +1). False where consecutive increments are distinct events — + #: suppressing there would swallow a real goal. + COALESCE_SCORING_SEQUENCE = False + + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + mode_config = getattr(self, "mode_config", {}) or {} + self.celebration_enabled = mode_config.get("celebration_enabled", True) + # Coerced and floored at init: this value is compared numerically on the + # display path, where a string from a hand-edited config would raise + # TypeError outside any try block, and a zero or negative value would + # arm a celebration that can never render. + raw_duration = mode_config.get("celebration_duration", 8) + try: + self.celebration_duration = max(1.0, float(raw_duration)) + except (TypeError, ValueError): + self.logger.warning( + "[Celebrations] Unusable celebration_duration %r; using 8s. " + "Set a positive number of seconds.", + raw_duration, + ) + self.celebration_duration = 8.0 + # Both spellings: the soccer lineage ships `celebrate_opponent_goals`, + # football ships `celebrate_opponent_scores`. Whichever the plugin's + # schema declares is the one its users have set. + self.celebrate_opponent_scores = mode_config.get( + "celebrate_opponent_scores", + mode_config.get("celebrate_opponent_goals", False), + ) + # Per-game score baselines: {game_id: {"away": int, "home": int}} + self._score_baselines: Dict[str, Dict[str, int]] = {} + # The active celebration (a game *snapshot*, so a win survives the game + # leaving live_games) or None. See _start_celebration for the shape. + self.active_celebration: Optional[Dict[str, Any]] = None + + # ------------------------------------------------------------------ + # Override points + # ------------------------------------------------------------------ + + def score_phrase(self, points: int, team_abbr: str) -> str: + """The wording for a score celebration. + + ``points`` is the score delta that triggered it, which sports with + variable-value scores use to name the play. The default is deliberately + sport-neutral; every celebrating plugin overrides it. + """ + return f"{team_abbr} SCORES!" + + def win_phrase(self, team_abbr: str) -> str: + """The wording for a win celebration.""" + return f"{team_abbr} WINS!" + + def _is_favorite(self, key: Optional[str]) -> bool: + """Whether ``key`` (whatever ``_favorite_key`` returns) is a favorite.""" + return bool(self.favorite_teams) and key in self.favorite_teams + + # ------------------------------------------------------------------ + # Detection + # ------------------------------------------------------------------ + + @staticmethod + def _score_to_int(score) -> Optional[int]: + """Coerce an ESPN score value (str / int / dict) to an int, or None.""" + try: + if score is None: + return None + if isinstance(score, str): + s = score.strip() + if not s: + return None + try: + return int(float(s)) + except ValueError: + numbers = re.findall(r"\d+", s) + return int(numbers[0]) if numbers else None + if isinstance(score, dict): + return int(float(score.get("value", score.get("displayValue", 0)))) + return int(float(score)) + except (ValueError, TypeError): + return None + + def _should_celebrate_for(self, game: Dict, side: str) -> bool: + """Whether a score by ``side`` in ``game`` should trigger a celebration.""" + if self._is_favorite(self._favorite_key(game, side)): + return True + if not self.favorite_teams: + # No favorites configured: the user opted to show this game, so + # celebrate any score in it. + return True + # Favorites exist but this team isn't one -> it's the opponent. + return self.celebrate_opponent_scores + + def prune_score_baselines(self, live_games: List[Dict]) -> None: + """Drop baselines for games no longer live. + + Only :meth:`_check_for_win` removes entries, and it only fires for games + seen to go final. A game that vanishes from the live list any other way + — postponed, dropped by the feed, or simply still live when the board + restarts — leaves its baseline behind forever, so on a board that runs + all season the dict grows without bound. + + Call this from ``update()`` with the current live set, alongside the + equivalent pruning in :meth:`SmoothWeightedRotation.next_game`. + """ + live_ids = {g.get("id") for g in live_games} + self._score_baselines = { + gid: baseline + for gid, baseline in self._score_baselines.items() + if gid in live_ids + } + + def has_active_celebration(self) -> bool: + """True while a celebration is within its display window.""" + celebration = self.active_celebration + return bool(celebration) and ( + time.time() - celebration["started_at"] < self.celebration_duration + ) + + def _check_for_score(self, game: Dict) -> None: + """Compare a live game's score against its baseline and arm a + celebration when a celebratable team's score increases.""" + if not self.celebration_enabled: + return + game_id = game.get("id") + if not game_id: + return + away = self._score_to_int(game.get("away_score")) + home = self._score_to_int(game.get("home_score")) + if away is None or home is None: + return + + baseline = self._score_baselines.get(game_id) + # Always refresh the baseline: a first sighting must never celebrate (a + # game already in progress at boot would false-fire), and a decrement + # (VAR, a correction) just re-bases silently. + self._score_baselines[game_id] = {"away": away, "home": home} + if baseline is None: + return + + away_delta = away - baseline["away"] + home_delta = home - baseline["home"] + if away_delta <= 0 and home_delta <= 0: + return + + # One takeover per scoring sequence, where the sport has such a thing. + # The baseline is already advanced above, so nothing re-fires later. + if self.COALESCE_SCORING_SEQUENCE and self.has_active_celebration(): + return + + scored_side = None + points = 0 + if away_delta > 0 and self._should_celebrate_for(game, "away"): + scored_side, points = "away", away_delta + if scored_side is None and home_delta > 0 and self._should_celebrate_for( + game, "home" + ): + scored_side, points = "home", home_delta + if scored_side is None: + return + + self._start_celebration( + game, + "score", + scored_side=scored_side, + team_abbr=game.get(f"{scored_side}_abbr", ""), + away_score=away, + home_score=home, + points=points, + ) + + def _check_for_win(self, game: Dict) -> None: + """When a game we were tracking live goes final, arm a win celebration + if a favorite won. Fires at most once per game.""" + if not self.celebration_enabled: + return + game_id = game.get("id") + if not game_id: + return + # Only celebrate wins for games we actually watched go live: one seen + # for the first time already-final (the board started after full time) + # has no baseline and must not fire. + if game_id not in self._score_baselines: + return + # Consume the baseline so this can only fire once. + self._score_baselines.pop(game_id, None) + + away = self._score_to_int(game.get("away_score")) + home = self._score_to_int(game.get("home_score")) + if away is None or home is None: + return + + if away > home: + winner_side = "away" + elif home > away: + winner_side = "home" + else: + return # draw -> no win celebration + + # Wins are gated strictly on favorites: every game ends, so the + # "no favorites -> celebrate all" score fallback would be far too noisy. + if not self._is_favorite(self._favorite_key(game, winner_side)): + return + + self._start_celebration( + game, + "win", + scored_side=winner_side, + team_abbr=game.get(f"{winner_side}_abbr", ""), + away_score=away, + home_score=home, + ) + + def _start_celebration( + self, + game: Dict, + kind: str, + scored_side: str, + team_abbr: str, + away_score: int, + home_score: int, + points: int = 0, + ) -> None: + """Arm a celebration. ``scored_side`` ('away'/'home') is the side whose + score digit gets highlighted.""" + phrase = ( + self.win_phrase(team_abbr) + if kind == "win" + else self.score_phrase(points, team_abbr) + ) + + self.active_celebration = { + "kind": kind, + "game": dict(game), # snapshot: survives the game leaving live_games + "scored_side": scored_side, + "team_abbr": team_abbr, + "away_score": away_score, + "home_score": home_score, + "started_at": time.time(), + "phrase": phrase, + } + # Pin focus to the involved game so the post-celebration scorebug + # resumes on it. + self.current_game = dict(game) + self.logger.info( + f"[Celebrations] {kind} armed: {phrase} " + f"[{game.get('away_abbr')} {away_score}-{home_score} {game.get('home_abbr')}]" + ) + + # ------------------------------------------------------------------ + # Rendering + # ------------------------------------------------------------------ + + def _fit_font(self, draw, text: str, max_width: int, fonts: List): + """The first font whose rendered ``text`` fits ``max_width``, falling + back to the last (smallest) font.""" + for font in fonts: + if draw.textlength(text, font=font) <= max_width - 2: + return font + return fonts[-1] + + def _draw_celebration_layout( + self, celebration: Dict, force_clear: bool = False + ) -> None: + """Render the full-screen score/win takeover.""" + if force_clear: + self.display_manager.clear() + + display_width = ( + self.display_manager.matrix.width + if hasattr(self.display_manager, "matrix") and self.display_manager.matrix + else self.display_width + ) + display_height = ( + self.display_manager.matrix.height + if hasattr(self.display_manager, "matrix") and self.display_manager.matrix + else self.display_height + ) + + elapsed = time.time() - celebration["started_at"] + game = celebration["game"] + + # Background: a brief color flash for the first ~1.2s, then black. + bg = (0, 0, 0, 255) + if elapsed < 1.2 and int(elapsed / 0.2) % 2 == 0: + bg = (12, 12, 48, 255) + main_img = Image.new("RGBA", (display_width, display_height), bg) + overlay = Image.new("RGBA", (display_width, display_height), (0, 0, 0, 0)) + draw = ImageDraw.Draw(overlay) + + # Logos at the edges (best-effort: a logo failure must not blank the + # celebration). + try: + center_y = display_height // 2 + home_logo = self._load_and_resize_logo( + game.get("home_id"), game.get("home_abbr"), + game.get("home_logo_path"), game.get("home_logo_url"), + ) + away_logo = self._load_and_resize_logo( + game.get("away_id"), game.get("away_abbr"), + game.get("away_logo_path"), game.get("away_logo_url"), + ) + if home_logo: + main_img.paste( + home_logo, + (display_width - home_logo.width + 2, center_y - home_logo.height // 2), + home_logo, + ) + if away_logo: + main_img.paste( + away_logo, (-2, center_y - away_logo.height // 2), away_logo + ) + except Exception as e: + self.logger.debug(f"[Celebrations] Logo load failed: {e}") + + # Phrase across the top, shrunk to fit the panel width. + phrase = celebration["phrase"] + phrase_font = self._fit_font( + draw, phrase, display_width, [self.fonts["time"], self.fonts["status"]] + ) + phrase_width = draw.textlength(phrase, font=phrase_font) + self._draw_text_with_outline( + draw, phrase, ((display_width - phrase_width) // 2, 1), phrase_font + ) + + # Score centered low, with the scoring/winning side's digit pulsing in a + # highlight color so the change reads at a glance. + away_text = str(celebration["away_score"]) + home_text = str(celebration["home_score"]) + score_font = self.fonts["score"] + segments = [ + (away_text, celebration["scored_side"] == "away"), + ("-", False), + (home_text, celebration["scored_side"] == "home"), + ] + total_width = sum(draw.textlength(seg, font=score_font) for seg, _ in segments) + highlight = (255, 255, 0) if int(elapsed * 4) % 2 == 0 else (255, 170, 0) + x = (display_width - total_width) // 2 + y = display_height - 14 + for seg, is_highlight in segments: + color = highlight if is_highlight else (255, 255, 255) + self._draw_text_with_outline(draw, seg, (int(x), y), score_font, fill=color) + x += draw.textlength(seg, font=score_font) + + main_img = Image.alpha_composite(main_img, overlay).convert("RGB") + self.display_manager.image = main_img + self.display_manager.update_display() + + def display(self, force_clear: bool = False) -> bool: + """Render an active celebration as a full-screen takeover; otherwise + defer to the normal live scorebug.""" + if not self.is_enabled: + return False + celebration = self.active_celebration + if celebration: + if self.has_active_celebration(): + try: + self._draw_celebration_layout(celebration, force_clear) + return True + except Exception as e: + self.logger.error( + f"[Celebrations] Error drawing celebration: {e}", exc_info=True + ) + # Disarm rather than retry: the same render would fail on + # every frame for the rest of the window, logging a + # traceback each time and leaving the scorebug off screen. + self.active_celebration = None + self.last_game_switch = time.time() + else: + self.active_celebration = None + # Reset the dwell so the scorebug resumes on the scoring/winning + # game for a full duration before rotation can move on. + self.last_game_switch = time.time() + return super().display(force_clear) diff --git a/src/base_classes/sports/capabilities/rotation.py b/src/base_classes/sports/capabilities/rotation.py new file mode 100644 index 00000000..4135ca9e --- /dev/null +++ b/src/base_classes/sports/capabilities/rotation.py @@ -0,0 +1,246 @@ +"""Live-rotation strategies — which live game to show next. + +The nine plugin copies grew three spellings of this, and the survey behind +``docs/SPORTS_UNIFICATION.md`` found they are all the *same* Smooth Weighted +Round-Robin algorithm in two shapes: + +* an **incremental picker** that holds weight state across calls and answers + "what next?" one game at a time (afl / nrl / soccer's ``_swrr_advance``), and +* a **precomputed cycle** that returns a full list of game ids up front + (football / baseball / basketball's ``_build_weighted_schedule`` and hockey's + ``_build_rotation_schedule``, which differ only in loop shape). + +They agree *within* a cycle — SWRR is deterministic — and differ only at cycle +boundaries, where the incremental form has no seam and the precomputed form +restarts. That is a real behavioral difference, so core ships both rather than +declaring a winner, and a plugin picks one by name: + + self.rotation = get_rotation_strategy("swrr", weight_for=self._live_weight) + +Core never learns which sport is asking. A plugin with a genuinely novel +ordering registers its own strategy instead of core growing a branch:: + + register_rotation_strategy("my-order", MyRotation) +""" + +from __future__ import annotations + +from typing import Callable, Dict, List, Optional, Type + + +def _game_id(game: Dict) -> Optional[str]: + """The rotation key for a game, or None if it has no usable id.""" + return game.get("id") + + +class RotationStrategy: + """Base class for live-rotation ordering. + + Subclasses implement :meth:`schedule`; :meth:`next_game` has a working + default derived from it. Strategies whose natural shape is incremental + override :meth:`next_game` instead and derive :meth:`schedule`. + + :param weight_for: callable mapping a game dict to a positive integer + weight — how many turns it gets per turn of a weight-1 game. Supplied by + the host so the *favorites* policy stays with the plugin and this module + stays free of any notion of what a favorite is. Defaults to equal + weights, which makes every strategy a plain round robin. + """ + + #: Name this strategy is registered under. Set by :func:`register_rotation_strategy`. + name: str = "" + + #: Ceiling on a per-game weight. A cycle is ``sum(weights)`` long and each + #: step scans every game, so an unbounded weight — a misread config field, + #: say — would spin the display thread for an unbounded time. On a Pi that + #: stalls rendering outright, so the bound is clamped like the floor is. + MAX_WEIGHT = 16 + + def __init__(self, weight_for: Optional[Callable[[Dict], int]] = None): + self._weight_for = weight_for or (lambda game: 1) + + def weights(self, games: List[Dict]) -> Dict[str, int]: + """``{game_id: weight}`` for games that have an id, in ``games`` order. + + A weight below 1 is clamped up: a zero or negative weight would starve + a game out of the rotation entirely, which no caller means to express + and which would make ``total_weight`` collapse. It is clamped down at + :attr:`MAX_WEIGHT` for the reason documented there. + """ + weights: Dict[str, int] = {} + for game in games: + gid = _game_id(game) + if gid is None: + continue + try: + weight = int(self._weight_for(game)) + except (TypeError, ValueError): + weight = 1 + weights[gid] = min(self.MAX_WEIGHT, max(1, weight)) + return weights + + def schedule(self, games: List[Dict]) -> List[str]: + """Game ids in display order for one cycle. Ids may repeat.""" + raise NotImplementedError + + def next_game(self, games: List[Dict]) -> Optional[Dict]: + """The next game to display, or None when there is nothing to show.""" + order = self.schedule(games) + if not order: + return None + by_id = {gid: g for g in games if (gid := _game_id(g)) is not None} + return by_id.get(order[0]) + + def reset(self) -> None: + """Drop any accumulated state. Stateless strategies need do nothing.""" + + +class SimpleRotation(RotationStrategy): + """Plain round robin: every live game once per cycle, weights ignored. + + The fallback for a plugin that wants strictly even rotation regardless of + favorites. + """ + + def schedule(self, games: List[Dict]) -> List[str]: + return [gid for g in games if (gid := _game_id(g)) is not None] + + +class WeightedCycleRotation(RotationStrategy): + """Precomputed SWRR cycle — the football / baseball / basketball / hockey shape. + + Returns a full cycle of ``sum(weights)`` ids with repeats spaced evenly + rather than clumped, highest weight scheduled first. When no game carries a + boost the cycle degenerates to a single pass in ``games`` order, which is + exactly the plain round robin it replaced. + """ + + def schedule(self, games: List[Dict]) -> List[str]: + weights = self.weights(games) + if not weights: + return [] + total_weight = sum(weights.values()) + if total_weight <= len(weights): + # No boost in effect — plain order, one pass. (Also the guard that + # keeps the loop below from being O(total_weight) for nothing.) + return list(weights) + + current = {gid: 0 for gid in weights} + order: List[str] = [] + for _ in range(total_weight): + for gid, weight in weights.items(): + current[gid] += weight + picked = max(current, key=lambda gid: current[gid]) + current[picked] -= total_weight + order.append(picked) + return order + + +class SmoothWeightedRotation(RotationStrategy): + """Incremental SWRR — the afl / nrl / soccer shape. + + Weight state persists across calls, so there is no fixed-length cycle and + therefore no clustering seam at a cycle boundary. A game seen for the first + time starts at weight 0 and receives its full weight on the next call, so a + favorite's game that has just gone live naturally wins the first pick after + it appears — "queued first on refresh" without a special-cased branch. + + State for games no longer live is dropped on each call, so a long-running + board does not accumulate entries for finished games. + """ + + def __init__(self, weight_for: Optional[Callable[[Dict], int]] = None): + super().__init__(weight_for) + self._current: Dict[str, int] = {} + + def reset(self) -> None: + self._current = {} + + def next_game(self, games: List[Dict]) -> Optional[Dict]: + if not games: + return None + weights = self.weights(games) + if not weights: + return None + + # Keep state only for games still live. + self._current = { + gid: value for gid, value in self._current.items() if gid in weights + } + for gid, weight in weights.items(): + self._current[gid] = self._current.get(gid, 0) + weight + + total_weight = sum(weights.values()) + # Iterate in `games` order so ties break toward the feed's ordering, + # which is what the plugin copies did and what makes the no-boost case + # identical to a plain round robin. + ids_in_order = [gid for g in games if (gid := _game_id(g)) in weights] + best = max(ids_in_order, key=lambda gid: self._current[gid]) + self._current[best] -= total_weight + return next(g for g in games if _game_id(g) == best) + + def schedule(self, games: List[Dict]) -> List[str]: + """One cycle's worth of picks, without disturbing live state. + + Derived by running the picker forward on a copy, so the returned order + is exactly what repeated :meth:`next_game` calls would produce from the + current state — callers can use it to preview or log the rotation + without perturbing it. + """ + weights = self.weights(games) + if not weights: + return [] + # type(self), not this class: a subclass that overrides next_game must + # be previewed through its own ordering, or the returned order is not + # the one repeated next_game calls would produce — which is exactly + # what this method promises. + preview = type(self)(self._weight_for) + preview._current = dict(self._current) + order: List[str] = [] + for _ in range(sum(weights.values())): + picked = preview.next_game(games) + if picked is None: + break + order.append(_game_id(picked)) + return order + + +_REGISTRY: Dict[str, Type[RotationStrategy]] = {} + + +def register_rotation_strategy(name: str, factory: Type[RotationStrategy]) -> None: + """Register a rotation strategy under ``name``. + + When a plugin needs an ordering that core does not ship, it registers its + own here instead of core growing a sport-specific branch. Re-registering a + name replaces it, so a plugin may also override a built-in for itself. + """ + if not name: + raise ValueError("rotation strategy name must be a non-empty string") + # Fail at registration, not at the first schedule() call several frames + # later, where the cause is no longer on the stack. + if not (isinstance(factory, type) and issubclass(factory, RotationStrategy)): + raise TypeError( + f"rotation strategy {name!r} must be a RotationStrategy subclass, " + f"got {factory!r}" + ) + factory.name = name + _REGISTRY[name] = factory + + +def get_rotation_strategy( + name: str, weight_for: Optional[Callable[[Dict], int]] = None +) -> RotationStrategy: + """Build the strategy registered under ``name``. + + Falls back to ``"simple"`` for an unknown name rather than raising: the name + arrives from user config, and a typo should cost the boost, not the + scoreboard. + """ + factory = _REGISTRY.get(name) or _REGISTRY["simple"] + return factory(weight_for=weight_for) + + +register_rotation_strategy("simple", SimpleRotation) +register_rotation_strategy("weighted", WeightedCycleRotation) +register_rotation_strategy("swrr", SmoothWeightedRotation) diff --git a/src/base_classes/sports/core.py b/src/base_classes/sports/core.py new file mode 100644 index 00000000..3291a1bf --- /dev/null +++ b/src/base_classes/sports/core.py @@ -0,0 +1,1127 @@ +"""SportsCore — the shared fetch/cache/config/render base for the sports +scoreboards. Split out of the former ``src/base_classes/sports.py``; see +docs/SPORTS_UNIFICATION.md for the layering. +""" + +import logging +import os +import tempfile +import time +from abc import ABC, abstractmethod +from datetime import datetime, timedelta +from pathlib import Path +from typing import Any, Dict, List, Optional, Tuple + +import pytz +import requests +from PIL import Image, ImageDraw, ImageFont +from requests.adapters import HTTPAdapter +from urllib3.util.retry import Retry + +from src.background_data_service import get_background_service + +# Import new architecture components (individual classes will import what they need) +from src.base_classes.api_extractors import APIDataExtractor +from src.base_classes.data_sources import DataSource +from src.cache_manager import CacheManager +from src.display_manager import DisplayManager +from src.dynamic_team_resolver import DynamicTeamResolver +from src.logo_downloader import LogoDownloader, download_missing_logo +try: + from src.base_odds_manager import BaseOddsManager as OddsManager +except ImportError: + OddsManager = None + +# The core install root, resolved from this module's own location so nothing +# here depends on the process working directory. +# +# The index is the number of directories between this file and the root: +# src/base_classes/sports/core.py -> sports -> base_classes -> src -> root. +# It is defined ONCE, here, because hand-counting it at each use site is +# exactly what broke when this module moved from src/base_classes/sports.py +# into the package (the old depth of 2 silently started resolving to src/). +# If this module ever moves again, this is the single line to update. +_INSTALL_ROOT = Path(__file__).resolve().parents[3] + +# Shared element-style resolver. Core always ships src.element_style, so the +# guard is never taken here — it is kept because this module's methods are +# back-copied verbatim into the plugins' bundled `sports.py`, where older +# cores genuinely lack the module (the plugins fall back to the classic +# inline config read below). +try: + from src.element_style import ElementStyleResolver, defaults_from_schema_file + STYLE_AVAILABLE = True +except ImportError: # pragma: no cover - core always ships element_style + STYLE_AVAILABLE = False + + +# --- Font catalog delegation ------------------------------------------------- +# The plugin copies each carried their own alias table mapping font *family* +# names ("press_start") to filenames. That table is a duplicate of the core +# FontManager's `common_fonts` catalog, so resolve through the catalog instead +# and let the two never drift apart. Same lazy module-level shape as +# skin_runtime._get_font_manager / base_plugin._fallback_font_manager: a +# SportsCore host has no plugin_manager to borrow a FontManager from. + +_shared_font_catalog: Optional[Any] = None + + +def _font_catalog() -> Any: + """Shared read-only FontManager used purely as a font-name catalog.""" + global _shared_font_catalog + if _shared_font_catalog is None: + from src.font_manager import FontManager + _shared_font_catalog = FontManager({}) + return _shared_font_catalog + + +def _resolve_font_family_alias(font_name: str) -> str: + """Resolve a font family alias to its filename, leaving filenames as-is. + + A config `font` value may be either a family name from the FontManager + catalog ("press_start", "four_by_six", "five_by_seven") or a literal + filename; the former resolve here, the latter pass through unchanged. + """ + try: + catalogued = _font_catalog().common_fonts.get(font_name) + except Exception: + return font_name + return os.path.basename(catalogued) if catalogued else font_name + + +def _read_bdf_native_size(bdf_path: str) -> Optional[int]: + """A BDF file's one true pixel size, delegated to FontManager. + + Deliberately NOT reimplemented here: the plugin copies grew a variant + that scans the whole file and returns a partially collected value when + parsing raises, where FontManager's stops at the first STARTCHAR and + returns None on error. + """ + try: + from src.font_manager import FontManager + return FontManager._read_bdf_native_size(bdf_path) + except Exception: + return None + + +class SportsCore(ABC): + # Which ScoreboardSkin render method this class's display path maps to. + # SportsLive inherits the default; SportsUpcoming/SportsRecent override. + SKIN_MODE = "live" + + def __init__(self, config: Dict[str, Any], display_manager: DisplayManager, cache_manager: CacheManager, logger: logging.Logger, sport_key: str): + self.logger = logger + self.config = config + self.cache_manager = cache_manager + self.config_manager = self.cache_manager.config_manager + if OddsManager: + try: + self.odds_manager = OddsManager( + self.cache_manager, self.config_manager) + except Exception as e: + self.logger.warning(f"Failed to initialize OddsManager: {e}") + self.odds_manager = None + else: + self.odds_manager = None + self.logger.warning("OddsManager not available - odds functionality disabled") + self.display_manager = display_manager + self.display_width = self.display_manager.matrix.width + self.display_height = self.display_manager.matrix.height + + self.sport_key = sport_key + self.sport = None + self.league = None + + # Initialize new architecture components (will be overridden by sport-specific classes) + self.sport_config = None + self.api_extractor: APIDataExtractor + self.data_source: DataSource + self.mode_config = config.get(f"{sport_key}_scoreboard", {}) # Changed config key + self.is_enabled: bool = self.mode_config.get("enabled", False) + self.show_odds: bool = self.mode_config.get("show_odds", False) + # Use LogoDownloader to get the correct default logo directory for this sport + default_logo_dir = Path(LogoDownloader().get_logo_directory(sport_key)) + self.logo_dir = self._initialize_logo_dir(default_logo_dir) + self.update_interval: int = self.mode_config.get( + "update_interval_seconds", 60) + self.show_records: bool = self.mode_config.get('show_records', False) + self.show_ranking: bool = self.mode_config.get('show_ranking', False) + # Number of games to show (instead of time-based windows) + self.recent_games_to_show: int = self.mode_config.get( + "recent_games_to_show", 5) # Show last 5 games + self.upcoming_games_to_show: int = self.mode_config.get( + "upcoming_games_to_show", 10) # Show next 10 games + self.show_favorite_teams_only: bool = self.mode_config.get("show_favorite_teams_only", False) + self.show_all_live: bool = self.mode_config.get("show_all_live", False) + + self.session = requests.Session() + retry_strategy = Retry( + total=5, # increased number of retries + backoff_factor=1, # increased backoff factor + # added 429 to retry list + status_forcelist=[429, 500, 502, 503, 504], + allowed_methods=["GET", "HEAD", "OPTIONS"] + ) + adapter = HTTPAdapter(max_retries=retry_strategy) + self.session.mount("https://", adapter) + self.session.mount("http://", adapter) + + self._logo_cache = {} + + # Font caches for _load_custom_font_from_element_config: per-frame + # callers (font-ladder walks) resolve the same (name, size) over and + # over, and the BDF strike size means re-reading a file header. + # Both are released in cleanup(). + self._font_cache: Dict[Tuple[str, int], Any] = {} + self._bdf_native_size_cache: Dict[str, Optional[int]] = {} + + # Set up headers + self.headers = { + 'User-Agent': 'LEDMatrix/1.0 (https://github.com/yourusername/LEDMatrix; contact@example.com)', + 'Accept': 'application/json', + 'Accept-Language': 'en-US,en;q=0.9', + 'Accept-Encoding': 'gzip, deflate, br', + 'Connection': 'keep-alive' + } + self.last_update = 0 + self.current_game = None + # Cooldown clocks for _should_log(), one per warning type. Initialized + # here rather than lazily: _should_log() reads them unguarded, so + # whichever warning fires first would otherwise raise AttributeError + # instead of logging. `_last_warning_time` is the single-clock field + # the plugin copies expose; kept for subclasses that read it. + self._warning_cooldowns: Dict[str, float] = {} + self._last_warning_time = 0 + self.fonts = self._load_fonts() + + # Optional visual skin (see docs/SKIN_SYSTEM.md). "skin" is either a + # skin id applied to all modes, or a per-mode mapping like + # {"live": "retro", "recent": "built-in"}. Loaded lazily on first + # render so a broken skin can never block startup. + self._skin_config = self.mode_config.get("skin") + self.skin_options = self.mode_config.get("skin_options", {}) or {} + self._skin = None + self._skin_load_attempted = False + self._skin_failures = 0 + self._skin_slow_renders = 0 + + # Initialize dynamic team resolver and resolve favorite teams + self.dynamic_resolver = DynamicTeamResolver() + raw_favorite_teams = self.mode_config.get("favorite_teams", []) + self.favorite_teams = self.dynamic_resolver.resolve_teams(raw_favorite_teams, sport_key) + + # Log dynamic team resolution + if raw_favorite_teams != self.favorite_teams: + self.logger.info(f"Resolved dynamic teams: {raw_favorite_teams} -> {self.favorite_teams}") + else: + self.logger.info(f"Favorite teams: {self.favorite_teams}") + + self.logger.setLevel(logging.INFO) + + # Initialize team rankings cache + self._team_rankings_cache = {} + self._rankings_cache_timestamp = 0 + self._rankings_cache_duration = 3600 # Cache rankings for 1 hour + + # Initialize background data service with optimized settings + # Hardcoded for memory optimization: 1 worker, 30s timeout, 3 retries + self.background_service = get_background_service(self.cache_manager, max_workers=1) + self.background_fetch_requests = {} # Track background fetch requests + self.background_enabled = True + self.logger.info("Background service enabled with 1 worker (memory optimized)") + + def _initialize_logo_dir(self, configured_path: Path) -> Path: + """Resolve and ensure a writable logo directory, falling back when necessary.""" + downloader = LogoDownloader() + resolved_configured = self._resolve_project_path(configured_path) + candidates = [resolved_configured] + self._get_logo_directory_fallbacks(resolved_configured) + + for candidate in candidates: + candidate_path = self._resolve_project_path(candidate) + if downloader.ensure_logo_directory(str(candidate_path)): + if candidate_path != resolved_configured: + self.logger.warning( + "Configured logo directory '%s' is not writable; using fallback '%s'", + resolved_configured, + candidate_path, + ) + return candidate_path + + self.logger.error( + "Unable to find a writable logo directory. Logos may fail to download (last attempted: %s)", + resolved_configured, + ) + return resolved_configured + + def _resolve_project_path(self, path: Path) -> Path: + """Convert relative paths to absolute ones rooted at the project directory.""" + if path.is_absolute(): + return path + return (_INSTALL_ROOT / path).resolve() + + def _get_logo_directory_fallbacks(self, configured_dir: Path) -> List[Path]: + """Return fallback directories to try when the configured directory is not writable.""" + fallbacks: List[Path] = [] + + env_override = os.environ.get("LEDMATRIX_LOGO_DIR") + if env_override: + env_path = Path(env_override) + if not env_path.is_absolute(): + env_path = self._resolve_project_path(env_path) + fallbacks.append(env_path / self.sport_key) + + cache_dir = getattr(self.cache_manager, "cache_dir", None) + if cache_dir: + fallbacks.append(Path(cache_dir) / "logos" / self.sport_key) + + try: + fallbacks.append(Path.home() / ".ledmatrix" / "logos" / self.sport_key) + except RuntimeError as e: + self.logger.debug("Could not resolve home directory (expected for service users): %s", e) + + fallbacks.append(Path(tempfile.gettempdir()) / "ledmatrix_logos" / self.sport_key) + + unique_fallbacks: List[Path] = [] + seen = set() + for candidate in fallbacks: + if candidate == configured_dir: + continue + if candidate not in seen: + unique_fallbacks.append(candidate) + seen.add(candidate) + + return unique_fallbacks + + def _get_season_schedule_dates(self) -> tuple[str, str]: + return "", "" + + def _draw_scorebug_layout(self, game: Dict, force_clear: bool = False) -> None: + """Placeholder draw method - subclasses should override.""" + # This base method will be simple, subclasses provide specifics + try: + img = Image.new('RGB', (self.display_width, self.display_height), (0, 0, 0)) + draw = ImageDraw.Draw(img) + status = game.get("status_text", "N/A") + self._draw_text_with_outline(draw, status, (2, 2), self.fonts['status']) + self.display_manager.image.paste(img, (0, 0)) + # Don't call update_display here, let subclasses handle it after drawing + except Exception as e: + self.logger.error(f"Error in base _draw_scorebug_layout: {e}", exc_info=True) + + + def _resolve_skin_id(self) -> Optional[str]: + """The skin id configured for this instance's mode, or None for the + built-in renderer. Accepts a plain id (all modes) or a per-mode + mapping ({"live": "retro-baseball", "recent": "built-in"}).""" + skin_id = self._skin_config + if isinstance(skin_id, dict): + skin_id = skin_id.get(self.SKIN_MODE) + if not skin_id or not isinstance(skin_id, str) or skin_id == "built-in": + return None + return skin_id + + def _get_skin(self): + """Lazily load the configured skin once. Returns None (built-in + renderer) when no skin is configured or loading failed.""" + if not self._skin_load_attempted: + self._skin_load_attempted = True + skin_id = self._resolve_skin_id() + if skin_id: + try: + from src.skin_system import skin_runtime + self._skin = skin_runtime.load_skin( + skin_id, sport=self.sport, sport_key=self.sport_key, + options=self.skin_options) + except Exception as e: + self.logger.error(f"Failed to load skin '{skin_id}': {e}", exc_info=True) + self._skin = None + return self._skin + + def _render_game(self, game: Dict, force_clear: bool = False) -> None: + """Render one game: try the configured skin first, fall back to the + built-in _draw_scorebug_layout. A skin that raises 3 times in a row + is disabled for the rest of the session.""" + skin = self._get_skin() + if skin is not None and self._skin_failures < 3: + try: + from src.skin_system import skin_runtime + ctx = skin_runtime.build_context(self, game) + render = getattr(skin, f"render_{self.SKIN_MODE}") + started = time.monotonic() + handled = render(ctx, dict(game)) + elapsed = time.monotonic() - started + if elapsed > 0.15 and self._skin_slow_renders < 5: + self._skin_slow_renders += 1 + self.logger.warning( + f"Skin '{self._resolve_skin_id()}' took {elapsed * 1000:.0f}ms to " + f"render {self.SKIN_MODE} — slow renders stall the whole display loop") + if handled: + self._skin_failures = 0 + self.display_manager.image.paste(ctx.canvas, (0, 0)) + self.display_manager.update_display() + return + except Exception: + self._skin_failures += 1 + outcome = ("disabling skin for this session" if self._skin_failures >= 3 + else "falling back to built-in renderer") + self.logger.error( + f"Skin '{self._resolve_skin_id()}' failed rendering {self.SKIN_MODE} " + f"({self._skin_failures}/3); {outcome}", exc_info=True) + self._draw_scorebug_layout(game, force_clear) + + def render_skin_card(self, game: Dict, size: tuple) -> Optional[Image.Image]: + """Render one game as a standalone card via the configured skin — + for vegas mode and previews. Tries render_vegas_card at the given + size, then the mode renderer on a card-sized canvas. Returns None + when no skin is active or the skin declined, so callers can use + their default rendering.""" + skin = self._get_skin() + if skin is None or self._skin_failures >= 3: + return None + try: + from src.skin_system import skin_runtime + ctx = skin_runtime.build_context(self, game, size=size) + card = skin.render_vegas_card(ctx, dict(game)) + if card is not None: + # A successful render clears accumulated strikes, mirroring + # _render_game — transient failures must not add up across + # the session and disable a working skin. + self._skin_failures = 0 + return card + ctx = skin_runtime.build_context(self, game, size=size) + render = getattr(skin, f"render_{self.SKIN_MODE}") + if render(ctx, dict(game)): + self._skin_failures = 0 + return ctx.canvas + except Exception: + # Card failures count toward the same 3-strike session disable + # as display failures — a skin broken for vegas shouldn't get + # to throw on every scroll tick forever. + self._skin_failures += 1 + self.logger.error( + f"Skin '{self._resolve_skin_id()}' card render failed " + f"({self._skin_failures}/3)", exc_info=True) + return None + + def display(self, force_clear: bool = False) -> bool: + """Common display method for all NCAA FB managers""" # Updated docstring + if not self.is_enabled: # Check if module is enabled + return False + + if not self.current_game: + # Clear display if force_clear is True, even when there's no content + # This prevents black screens when switching to modes with no content + if force_clear: + try: + self.display_manager.clear() + self.display_manager.update_display() + except Exception as e: + self.logger.debug(f"Error clearing display when no content: {e}") + + current_time = time.time() + if not hasattr(self, '_last_warning_time'): + self._last_warning_time = 0 + if current_time - getattr(self, '_last_warning_time', 0) > 300: + self.logger.warning(f"No game data available to display in {self.__class__.__name__}") + setattr(self, '_last_warning_time', current_time) + return False + + try: + self._render_game(self.current_game, force_clear) + # display_manager.update_display() should be called within subclass draw methods + # or after calling display() in the main loop. Let's keep it out of the base display. + return True + except Exception as e: + self.logger.error(f"Error during display call in {self.__class__.__name__}: {e}", exc_info=True) + return False + + + def _load_fonts(self) -> Dict[str, Any]: + """Load fonts used by the scoreboard. + + Paths go through :meth:`_resolve_font_path` so the bundled fonts are + found regardless of the process working directory — a bare + ``"assets/fonts/..."`` silently degraded every scoreboard to the PIL + default font whenever the process started elsewhere (the plugin safety + harness on CI being the case that surfaced it). + """ + fonts: Dict[str, Any] = {} + press_start = self._resolve_font_path("PressStart2P-Regular.ttf") + four_by_six = self._resolve_font_path("4x6-font.ttf") + try: + fonts['score'] = ImageFont.truetype(press_start, 10) + fonts['time'] = ImageFont.truetype(press_start, 8) + fonts['team'] = ImageFont.truetype(press_start, 8) + fonts['status'] = ImageFont.truetype(four_by_six, 6) # Using 4x6 for status + fonts['detail'] = ImageFont.truetype(four_by_six, 6) # Added detail font + fonts['rank'] = ImageFont.truetype(press_start, 10) + self.logger.info("Successfully loaded fonts") + except OSError: + # Name the directory we searched: the usual cause is an install + # whose assets/fonts is missing, and the bare message sent people + # hunting for a font-format problem instead. + self.logger.warning( + "Fonts not found under %s, using default PIL font.", + self._font_root(), + ) + fonts['score'] = ImageFont.load_default() + fonts['time'] = ImageFont.load_default() + fonts['team'] = ImageFont.load_default() + fonts['status'] = ImageFont.load_default() + fonts['detail'] = ImageFont.load_default() + fonts['rank'] = ImageFont.load_default() + return fonts + + def _draw_dynamic_odds(self, draw: ImageDraw.Draw, odds: Dict[str, Any], width: int, height: int) -> None: + """Draw odds with dynamic positioning - only show negative spread and position O/U based on favored team.""" + home_team_odds = odds.get('home_team_odds', {}) + away_team_odds = odds.get('away_team_odds', {}) + home_spread = home_team_odds.get('spread_odds') + away_spread = away_team_odds.get('spread_odds') + + # Get top-level spread as fallback + top_level_spread = odds.get('spread') + + # If we have a top-level spread and the individual spreads are None or 0, use the top-level + if top_level_spread is not None: + if home_spread is None or home_spread == 0.0: + home_spread = top_level_spread + if away_spread is None: + away_spread = -top_level_spread + + # Determine which team is favored (has negative spread) + home_favored = home_spread is not None and home_spread < 0 + away_favored = away_spread is not None and away_spread < 0 + + # Only show the negative spread (favored team) + favored_spread = None + favored_side = None + + if home_favored: + favored_spread = home_spread + favored_side = 'home' + self.logger.debug(f"Home team favored with spread: {favored_spread}") + elif away_favored: + favored_spread = away_spread + favored_side = 'away' + self.logger.debug(f"Away team favored with spread: {favored_spread}") + else: + self.logger.debug("No clear favorite - spreads: home={home_spread}, away={away_spread}") + + # Show the negative spread on the appropriate side + if favored_spread is not None: + spread_text = str(favored_spread) + font = self.fonts['detail'] # Use detail font for odds + + if favored_side == 'home': + # Home team is favored, show spread on right side + spread_width = draw.textlength(spread_text, font=font) + spread_x = width - spread_width # Top right + spread_y = 0 + self._draw_text_with_outline(draw, spread_text, (spread_x, spread_y), font, fill=(0, 255, 0)) + self.logger.debug(f"Showing home spread '{spread_text}' on right side") + else: + # Away team is favored, show spread on left side + spread_x = 0 # Top left + spread_y = 0 + self._draw_text_with_outline(draw, spread_text, (spread_x, spread_y), font, fill=(0, 255, 0)) + self.logger.debug(f"Showing away spread '{spread_text}' on left side") + + # Show over/under on the opposite side of the favored team + over_under = odds.get('over_under') + if over_under is not None: + ou_text = f"O/U: {over_under}" + font = self.fonts['detail'] # Use detail font for odds + ou_width = draw.textlength(ou_text, font=font) + + if favored_side == 'home': + # Home team is favored, show O/U on left side (opposite of spread) + ou_x = 0 # Top left + ou_y = 0 + self.logger.debug(f"Showing O/U '{ou_text}' on left side (home favored)") + elif favored_side == 'away': + # Away team is favored, show O/U on right side (opposite of spread) + ou_x = width - ou_width # Top right + ou_y = 0 + self.logger.debug(f"Showing O/U '{ou_text}' on right side (away favored)") + else: + # No clear favorite, show O/U in center + ou_x = (width - ou_width) // 2 + ou_y = 0 + self.logger.debug(f"Showing O/U '{ou_text}' in center (no clear favorite)") + + self._draw_text_with_outline(draw, ou_text, (ou_x, ou_y), font, fill=(0, 255, 0)) + + def _draw_text_with_outline(self, draw, text, position, font, fill=(255, 255, 255), outline_color=(0, 0, 0)): + """Draw text with a black outline for better readability.""" + x, y = position + for dx, dy in [(-1, -1), (-1, 0), (-1, 1), (0, -1), (0, 1), (1, -1), (1, 0), (1, 1)]: + draw.text((x + dx, y + dy), text, font=font, fill=outline_color) + draw.text((x, y), text, font=font, fill=fill) + + def _load_and_resize_logo(self, team_id: str, team_abbrev: str, logo_path: Path, logo_url: str | None ) -> Optional[Image.Image]: + """Load and resize a team logo, with caching and automatic download if missing.""" + self.logger.debug(f"Logo path: {logo_path}") + if team_abbrev in self._logo_cache: + self.logger.debug(f"Using cached logo for {team_abbrev}") + return self._logo_cache[team_abbrev] + + try: + # Try different filename variations first (for cases like TA&M vs TAANDM) + actual_logo_path = None + filename_variations = LogoDownloader.get_logo_filename_variations(team_abbrev) + + for filename in filename_variations: + test_path = logo_path.parent / filename + if test_path.exists(): + actual_logo_path = test_path + self.logger.debug(f"Found logo at alternative path: {actual_logo_path}") + break + + # If no variation found, try to download missing logo + if not actual_logo_path and not logo_path.exists(): + self.logger.info(f"Logo not found for {team_abbrev} at {logo_path}. Attempting to download.") + + # Try to download the logo from ESPN API (this will create placeholder if download fails) + download_missing_logo(self.sport_key, team_id, team_abbrev, logo_path, logo_url) + actual_logo_path = logo_path + + # Use the original path if no alternative was found + if not actual_logo_path: + actual_logo_path = logo_path + + # Only try to open the logo if the file exists + if os.path.exists(actual_logo_path): + logo = Image.open(actual_logo_path) + else: + self.logger.error(f"Logo file still doesn't exist at {actual_logo_path} after download attempt") + return None + if logo.mode != 'RGBA': + logo = logo.convert('RGBA') + + max_width = int(self.display_width * 1.5) + max_height = int(self.display_height * 1.5) + logo.thumbnail((max_width, max_height), Image.Resampling.LANCZOS) + self._logo_cache[team_abbrev] = logo + return logo + + except Exception as e: + self.logger.error(f"Error loading logo for {team_abbrev}: {e}", exc_info=True) + return None + + def _fetch_odds(self, game: Dict) -> None: + """Fetch odds for a specific game using the new architecture.""" + try: + if not self.show_odds: + return + + if not self.odds_manager: + return + + # Determine update interval based on game state + is_live = game.get('is_live', False) + update_interval = self.mode_config.get("live_odds_update_interval", 60) if is_live \ + else self.mode_config.get("odds_update_interval", 3600) + + # Fetch odds using OddsManager + odds_data = self.odds_manager.get_odds( + sport=self.sport, + league=self.league, + event_id=game['id'], + update_interval_seconds=update_interval, + ) + + if odds_data: + game['odds'] = odds_data + self.logger.debug(f"Successfully fetched and attached odds for game {game['id']}") + else: + self.logger.debug(f"No odds data returned for game {game['id']}") + + except Exception as e: + self.logger.error(f"Error fetching odds for game {game.get('id', 'N/A')}: {e}") + + def _get_timezone(self): + try: + timezone_str = self.config.get('timezone', 'UTC') + return pytz.timezone(timezone_str) + except pytz.UnknownTimeZoneError: + return pytz.utc + + def _should_log(self, warning_type: str, cooldown: int = 60) -> bool: + """Whether a warning of this kind is outside its cooldown window. + + Cooldowns are tracked **per ``warning_type``**. They previously shared + one timestamp, so the parameter was accepted and ignored: an API-error + warning would silence an unrelated cache warning for the next minute, + and whichever fired first won. Nothing in core called this, so no + behavior regressed with the fix — but every caller has always been + entitled to assume its own warning type has its own clock. + """ + current_time = time.time() + if current_time - self._warning_cooldowns.get(warning_type, 0) > cooldown: + self._warning_cooldowns[warning_type] = current_time + # Kept in step for subclasses that read it directly. + self._last_warning_time = current_time + return True + return False + + def _fetch_team_rankings(self) -> Dict[str, int]: + """Fetch team rankings using the new architecture components.""" + current_time = time.time() + + # Check if we have cached rankings that are still valid + if (self._team_rankings_cache and + current_time - self._rankings_cache_timestamp < self._rankings_cache_duration): + return self._team_rankings_cache + + try: + data = self.data_source.fetch_standings(self.sport, self.league) + + rankings = {} + rankings_data = data.get('rankings', []) + + if rankings_data: + # Use the first ranking (usually AP Top 25) + first_ranking = rankings_data[0] + teams = first_ranking.get('ranks', []) + + for team_data in teams: + team_info = team_data.get('team', {}) + team_abbr = team_info.get('abbreviation', '') + current_rank = team_data.get('current', 0) + + if team_abbr and current_rank > 0: + rankings[team_abbr] = current_rank + + # Cache the results + self._team_rankings_cache = rankings + self._rankings_cache_timestamp = current_time + + self.logger.debug(f"Fetched rankings for {len(rankings)} teams") + return rankings + + except Exception as e: + self.logger.error(f"Error fetching team rankings: {e}") + return {} + + def _extract_game_details_common(self, game_event: Dict) -> tuple[Dict | None, Dict | None, Dict | None, Dict | None, Dict | None]: + if not game_event: + return None, None, None, None, None + try: + competition = game_event["competitions"][0] + status = competition["status"] + competitors = competition["competitors"] + game_date_str = game_event["date"] + situation = competition.get("situation") + start_time_utc = None + try: + # Parse the datetime string + if game_date_str.endswith('Z'): + game_date_str = game_date_str.replace('Z', '+00:00') + dt = datetime.fromisoformat(game_date_str) + # Ensure the datetime is UTC-aware (fromisoformat may create timezone-aware but not pytz.UTC) + if dt.tzinfo is None: + # If naive, assume it's UTC + start_time_utc = dt.replace(tzinfo=pytz.UTC) + else: + # Convert to pytz.UTC for consistency + start_time_utc = dt.astimezone(pytz.UTC) + except ValueError: + logging.warning(f"Could not parse game date: {game_date_str}") + + home_team = next((c for c in competitors if c.get("homeAway") == "home"), None) + away_team = next((c for c in competitors if c.get("homeAway") == "away"), None) + + if not home_team or not away_team: + self.logger.warning(f"Could not find home or away team in event: {game_event.get('id')}") + return None, None, None, None, None + + try: + home_abbr = home_team["team"]["abbreviation"] + except KeyError: + home_abbr = home_team["team"]["name"][:3] + try: + away_abbr = away_team["team"]["abbreviation"] + except KeyError: + away_abbr = away_team["team"]["name"][:3] + + # Check if this is a favorite team game BEFORE doing expensive logging + is_favorite_game = (home_abbr in self.favorite_teams or away_abbr in self.favorite_teams) + + # Only log debug info for favorite team games + if is_favorite_game: + self.logger.debug(f"Processing favorite team game: {game_event.get('id')}") + self.logger.debug(f"Found teams: {away_abbr}@{home_abbr}, Status: {status['type']['name']}, State: {status['type']['state']}") + + game_time, game_date = "", "" + if start_time_utc: + local_time = start_time_utc.astimezone(self._get_timezone()) + game_time = local_time.strftime("%I:%M%p").lstrip('0') + + # Check date format from config + use_short_date_format = self.config.get('display', {}).get('use_short_date_format', False) + if use_short_date_format: + game_date = local_time.strftime("%-m/%-d") + else: + game_date = self.display_manager.format_date_with_ordinal(local_time) + + + home_record = home_team.get('records', [{}])[0].get('summary', '') if home_team.get('records') else '' + away_record = away_team.get('records', [{}])[0].get('summary', '') if away_team.get('records') else '' + + # Don't show "0-0" records - set to blank instead + if home_record in {"0-0", "0-0-0"}: + home_record = '' + if away_record in {"0-0", "0-0-0"}: + away_record = '' + + details = { + "id": game_event.get("id"), + "game_time": game_time, + "game_date": game_date, + "start_time_utc": start_time_utc, + "status_text": status["type"]["shortDetail"], # e.g., "Final", "7:30 PM", "Q1 12:34" + "is_live": status["type"]["state"] == "in", + "is_final": status["type"]["state"] == "post", + "is_upcoming": (status["type"]["state"] == "pre" or + status["type"]["name"].lower() in ['scheduled', 'pre-game', 'status_scheduled']), + "is_halftime": status["type"]["state"] == "halftime" or status["type"]["name"] == "STATUS_HALFTIME", # Added halftime check + "is_period_break": status["type"]["name"] == "STATUS_END_PERIOD", # Added Period Break check + "home_abbr": home_abbr, + "home_id": home_team["id"], + "home_score": home_team.get("score", "0"), + "home_logo_path": self.logo_dir / Path(f"{LogoDownloader.normalize_abbreviation(home_abbr)}.png"), + "home_logo_url": home_team["team"].get("logo"), + "home_record": home_record, + "away_record": away_record, + "away_abbr": away_abbr, + "away_id": away_team["id"], + "away_score": away_team.get("score", "0"), + "away_logo_path": self.logo_dir / Path(f"{LogoDownloader.normalize_abbreviation(away_abbr)}.png"), + "away_logo_url": away_team["team"].get("logo"), + "is_within_window": True, # Whether game is within display window + + } + return details, home_team, away_team, status, situation + except Exception as e: + # Log the problematic event structure if possible + logging.error(f"Error extracting game details: {e} from event: {game_event.get('id')}", exc_info=True) + return None, None, None, None, None + + @abstractmethod + def _extract_game_details(self, game_event: dict) -> dict | None: + details, _, _, _, _ = self._extract_game_details_common(game_event) + return details + + @abstractmethod + def _fetch_data(self) -> Optional[Dict]: + pass + + def _fetch_todays_games(self) -> Optional[Dict]: + """Fetch only today's games for live updates (not entire season).""" + try: + tz = pytz.timezone("America/New_York") # Use full name (not "EST") for DST support + now = datetime.now(tz) + yesterday = now - timedelta(days=1) + formatted_date = now.strftime("%Y%m%d") + formatted_date_yesterday = yesterday.strftime("%Y%m%d") + # Fetch todays games only + url = f"https://site.api.espn.com/apis/site/v2/sports/{self.sport}/{self.league}/scoreboard" + response = self.session.get(url, params={"dates": f"{formatted_date_yesterday}-{formatted_date}", "limit": 1000}, headers=self.headers, timeout=10) + response.raise_for_status() + data = response.json() + events = data.get('events', []) + + self.logger.info(f"Fetched {len(events)} todays games for {self.sport} - {self.league}") + return {'events': events} + except requests.exceptions.RequestException as e: + self.logger.error(f"API error fetching todays games for {self.sport} - {self.league}: {e}") + return None + + def _get_weeks_data(self) -> Optional[Dict]: + """ + Get partial data for immediate display while background fetch is in progress. + This fetches current/recent games only for quick response. + """ + try: + # Fetch current week and next few days for immediate display + now = datetime.now(pytz.utc) + immediate_events = [] + + start_date = now + timedelta(weeks=-2) + end_date = now + timedelta(weeks=1) + date_str = f"{start_date.strftime('%Y%m%d')}-{end_date.strftime('%Y%m%d')}" + url = f"https://site.api.espn.com/apis/site/v2/sports/{self.sport}/{self.league}/scoreboard" + response = self.session.get(url, params={"dates": date_str, "limit": 1000},headers=self.headers, timeout=10) + response.raise_for_status() + data = response.json() + immediate_events = data.get('events', []) + + if immediate_events: + self.logger.info(f"Fetched {len(immediate_events)} events {date_str}") + return {'events': immediate_events} + + except requests.exceptions.RequestException as e: + self.logger.warning(f"Error fetching this weeks games for {self.sport} - {self.league} - {date_str}: {e}") + return None + + def _custom_scorebug_layout(self, game: dict, draw_overlay: ImageDraw.ImageDraw): + pass + + # ------------------------------------------------------------------ + # Promoted from the plugin copies (see docs/SPORTS_UNIFICATION.md). + # Everything below is present in all nine bundled `sports.py` copies; + # the canonical form of each is documented on the method. + # ------------------------------------------------------------------ + + def _favorite_key(self, game: Dict, side: str) -> Optional[str]: + """Override point: which view-model field identifies a team when + matching against ``favorite_teams``. + + ``side`` is ``"home"`` or ``"away"``. The default is the team + abbreviation — what eight of the nine scoreboards match on, and what + users type into their favorites list. + + NRL overrides this to the team **id**, because NRL abbreviations are + not unique: "NEW" is both Newcastle Knights and New Zealand Warriors, + "CAN" both Canberra Raiders and Canterbury Bulldogs. Matching those by + abbreviation selects the wrong club. It is a seam rather than a branch + precisely so core never has to learn the string "nrl":: + + def _favorite_key(self, game, side): + return str(game.get(f"{side}_id")) + + An override that stringifies should note that a missing id becomes the + literal ``"None"``, which would spuriously match a favorites list + containing that string. The default returns ``None``, which never + matches. + """ + return game.get(f"{side}_abbr") + + def _config_schema_path(self) -> Optional[str]: + """Override point: the plugin's ``config_schema.json``, used as the + reference for style-resolver defaults. + + None (the default) means no schema is available — layout offsets are + then read with the classic inline config lookup, which is exactly what + a plugin that never shipped resolver support does today. A plugin + opting into :mod:`src.element_style` returns its own schema path:: + + def _config_schema_path(self): + return os.path.join(os.path.dirname(os.path.abspath(__file__)), + 'config_schema.json') + + It must not be derived from this module's ``__file__``: after + promotion that resolves inside ``src/base_classes/sports/``, where no + ``config_schema.json`` exists. + """ + return None + + def _font_root(self) -> str: + """Override point: the directory ``assets/fonts`` resolves against. + + Defaults to the core install root, derived from this module's own + location — the same strategy as ``FontManager._resolve_asset_path``. + A plugin bundling its own fonts overrides this to return its plugin + directory. Never a cwd-relative path: fonts must load no matter where + the process was started from (e.g. the plugin safety harness on CI). + """ + return str(_INSTALL_ROOT) + + def _resolve_font_path(self, font_name: str) -> str: + """Locate a font file by filename, independently of the process cwd. + + Tries ``assets/fonts/`` relative to the cwd first (preserving + behavior for a process started from an install root), then under + :meth:`_font_root`. Returns the cwd-relative path unchanged when the + file is nowhere to be found, so callers log the familiar path. + """ + relative = os.path.join('assets', 'fonts', font_name) + if os.path.exists(relative): + return relative + candidate = os.path.join(self._font_root(), relative) + if os.path.exists(candidate): + return candidate + return relative + + def _get_layout_offset(self, element: str, axis: str, default: int = 0) -> int: + """ + Get layout offset for a specific element and axis. + + Args: + element: Element name (e.g., 'home_logo', 'score', 'status_text') + axis: 'x_offset' or 'y_offset' (or 'away_x_offset', 'home_x_offset' for records) + default: Default value if not configured (default: 0) + + Returns: + Offset value from config or default (always returns int) + """ + schema_path = self._config_schema_path() if STYLE_AVAILABLE else None + if schema_path: + # Shared resolver (rebuilt if the config dict was swapped out, + # matching the classic path's read-config-on-every-call semantics). + # Note it is stricter than the classic read below: a boolean offset + # degrades to the default instead of counting as 1/0, which is the + # more correct reading of a pixel offset. + resolver = getattr(self, '_style_resolver_cached', None) + if resolver is None or resolver._config is not self.config: + resolver = ElementStyleResolver( + self.config, defaults_from_schema_file(schema_path)) + self._style_resolver_cached = resolver + return resolver.offset_value(element, axis, default) + try: + layout_config = self.config.get('customization', {}).get('layout', {}) + element_config = layout_config.get(element, {}) + offset_value = element_config.get(axis, default) + + # Ensure we return an integer (handle float/string from config) + if isinstance(offset_value, (int, float)): + return int(offset_value) + elif isinstance(offset_value, str): + # Try to convert string to int + try: + return int(float(offset_value)) + except (ValueError, TypeError): + self.logger.warning( + f"Invalid layout offset value for {element}.{axis}: '{offset_value}', using default {default}" + ) + return default + else: + return default + except Exception as e: + # Gracefully handle any config access errors + self.logger.debug(f"Error reading layout offset for {element}.{axis}: {e}, using default {default}") + return default + + def _load_custom_font_from_element_config( + self, + element_config: Dict[str, Any], + default_size: int = 8, + default_font: Optional[str] = None, + ) -> ImageFont.FreeTypeFont: + """ + Load a custom font from an element configuration dictionary. + + Args: + element_config: Configuration dict for a single element containing 'font' and 'font_size' keys + default_size: Default font size if not specified in config + default_font: Default font filename when not specified in config (e.g. '4x6-font.ttf' for odds) + + Returns: + PIL ImageFont object + """ + base_default = default_font or "PressStart2P-Regular.ttf" + font_name = element_config.get('font', base_default) + font_size = int(element_config.get('font_size', default_size)) # Ensure integer for PIL + + # Resolve family aliases (e.g. "press_start") to real filenames, then + # locate the file against _font_root() rather than the cwd. + resolved_name = _resolve_font_family_alias(font_name) + font_path = self._resolve_font_path(resolved_name) + + # Memoized: per-frame callers (font-ladder walks) resolve the same + # (name, size) repeatedly -- return the previously loaded face. + cache_key = (resolved_name, font_size) + cached_font = self._font_cache.get(cache_key) + if cached_font is not None: + return cached_font + + # Try to load the font + try: + if os.path.exists(font_path): + # Try loading as TTF first (works for both TTF and some BDF files with PIL) + if font_path.lower().endswith('.ttf'): + font = ImageFont.truetype(font_path, font_size) + self.logger.debug(f"Loaded font: {font_name} at size {font_size}") + self._font_cache[cache_key] = font + return font + elif font_path.lower().endswith('.bdf'): + # BDF fonts are fixed-size bitmaps, not scalable outlines -- + # FreeType only accepts the exact pixel size baked into the + # file (its "strike") and raises "invalid pixel size" for + # anything else. Try the requested size first (in case it + # happens to match), then fall back to the file's real + # native size, so a BDF font can still be selected via + # font_size-driven configs without the caller needing to + # know its exact strike size. + # + # This retry is the OLDER lineage's behavior and it is the + # correct one: the newer copies call truetype() on a BDF at + # any size (which simply fails) or refuse BDF outright. + try: + font = ImageFont.truetype(font_path, font_size) + self.logger.debug(f"Loaded BDF font: {font_name} at size {font_size}") + self._font_cache[cache_key] = font + return font + except OSError: + if font_path in self._bdf_native_size_cache: + native_size = self._bdf_native_size_cache[font_path] + else: + native_size = _read_bdf_native_size(font_path) + self._bdf_native_size_cache[font_path] = native_size + if native_size and native_size != font_size: + try: + font = ImageFont.truetype(font_path, native_size) + self.logger.debug( + f"Loaded BDF font: {font_name} at its native size {native_size} " + f"(requested {font_size} isn't a valid strike for this file)" + ) + self._font_cache[cache_key] = font + return font + except Exception as retry_exc: + self.logger.debug( + f"BDF font {font_name} also failed to load at native " + f"size {native_size}: {retry_exc}" + ) + self.logger.warning(f"Could not load BDF font {font_name} with PIL, using default") + # Fall through to default + else: + self.logger.warning(f"Unknown font file type: {font_name}, using default") + else: + self.logger.warning(f"Font file not found: {font_path}, using default") + except Exception as e: + self.logger.error(f"Error loading font {font_name}: {e}, using default") + + # Fall back to default font. Cached under the requested (name, size) + # key too, so a misconfigured or missing font pays the disk cost once + # instead of on every frame of a font-ladder walk. + default_font_path = self._resolve_font_path( + _resolve_font_family_alias(base_default)) + try: + if os.path.exists(default_font_path): + font = ImageFont.truetype(default_font_path, font_size) + else: + self.logger.warning("Default font not found, using PIL default") + font = ImageFont.load_default() + except Exception as e: + self.logger.error(f"Error loading default font: {e}") + font = ImageFont.load_default() + self._font_cache[cache_key] = font + return font + + def cleanup(self): + """Clean up resources when plugin is unloaded.""" + # Close HTTP session + if hasattr(self, 'session') and self.session: + try: + self.session.close() + except Exception as e: + self.logger.warning(f"Error closing session: {e}") + + # Clear caches + if hasattr(self, '_logo_cache'): + self._logo_cache.clear() + # Font caches hold PIL faces; without this they are an unbounded + # per-instance leak across enable/disable cycles. + if hasattr(self, '_font_cache'): + self._font_cache.clear() + if hasattr(self, '_bdf_native_size_cache'): + self._bdf_native_size_cache.clear() + + # NOTE: self.background_service is deliberately NOT shut down here. + # get_background_service() returns a PROCESS-WIDE singleton shared by + # every scoreboard; shutting it down from one unloading plugin would + # stop background fetching for all the others. Whoever owns the + # process owns its lifecycle. Do not "fix" this. + + self.logger.info(f"{self.__class__.__name__} cleanup completed") diff --git a/src/base_classes/sports.py b/src/base_classes/sports/modes.py similarity index 59% rename from src/base_classes/sports.py rename to src/base_classes/sports/modes.py index fd983446..8f6e3cce 100644 --- a/src/base_classes/sports.py +++ b/src/base_classes/sports/modes.py @@ -1,651 +1,25 @@ +"""The three display modes layered on SportsCore: SportsUpcoming, +SportsRecent and SportsLive. Split out of the former +``src/base_classes/sports.py``; see docs/SPORTS_UNIFICATION.md. +""" + import logging -import os -import tempfile import time -from abc import ABC, abstractmethod +from abc import abstractmethod from datetime import datetime, timedelta, timezone -from pathlib import Path -from typing import Any, Dict, List, Optional +from typing import Any, Dict, List -import pytz -import requests from PIL import Image, ImageDraw, ImageFont -from requests.adapters import HTTPAdapter -from urllib3.util.retry import Retry -from src.background_data_service import get_background_service - -# Import new architecture components (individual classes will import what they need) -from src.base_classes.api_extractors import APIDataExtractor -from src.base_classes.data_sources import DataSource from src.cache_manager import CacheManager from src.display_manager import DisplayManager -from src.dynamic_team_resolver import DynamicTeamResolver -from src.logo_downloader import LogoDownloader, download_missing_logo -try: - from src.base_odds_manager import BaseOddsManager as OddsManager -except ImportError: - OddsManager = None +from .core import SportsCore -class SportsCore(ABC): - def __init__(self, config: Dict[str, Any], display_manager: DisplayManager, cache_manager: CacheManager, logger: logging.Logger, sport_key: str): - self.logger = logger - self.config = config - self.cache_manager = cache_manager - self.config_manager = self.cache_manager.config_manager - if OddsManager: - try: - self.odds_manager = OddsManager( - self.cache_manager, self.config_manager) - except Exception as e: - self.logger.warning(f"Failed to initialize OddsManager: {e}") - self.odds_manager = None - else: - self.odds_manager = None - self.logger.warning("OddsManager not available - odds functionality disabled") - self.display_manager = display_manager - self.display_width = self.display_manager.matrix.width - self.display_height = self.display_manager.matrix.height - - self.sport_key = sport_key - self.sport = None - self.league = None - - # Initialize new architecture components (will be overridden by sport-specific classes) - self.sport_config = None - self.api_extractor: APIDataExtractor - self.data_source: DataSource - self.mode_config = config.get(f"{sport_key}_scoreboard", {}) # Changed config key - self.is_enabled: bool = self.mode_config.get("enabled", False) - self.show_odds: bool = self.mode_config.get("show_odds", False) - # Use LogoDownloader to get the correct default logo directory for this sport - default_logo_dir = Path(LogoDownloader().get_logo_directory(sport_key)) - self.logo_dir = self._initialize_logo_dir(default_logo_dir) - self.update_interval: int = self.mode_config.get( - "update_interval_seconds", 60) - self.show_records: bool = self.mode_config.get('show_records', False) - self.show_ranking: bool = self.mode_config.get('show_ranking', False) - # Number of games to show (instead of time-based windows) - self.recent_games_to_show: int = self.mode_config.get( - "recent_games_to_show", 5) # Show last 5 games - self.upcoming_games_to_show: int = self.mode_config.get( - "upcoming_games_to_show", 10) # Show next 10 games - self.show_favorite_teams_only: bool = self.mode_config.get("show_favorite_teams_only", False) - self.show_all_live: bool = self.mode_config.get("show_all_live", False) - - self.session = requests.Session() - retry_strategy = Retry( - total=5, # increased number of retries - backoff_factor=1, # increased backoff factor - # added 429 to retry list - status_forcelist=[429, 500, 502, 503, 504], - allowed_methods=["GET", "HEAD", "OPTIONS"] - ) - adapter = HTTPAdapter(max_retries=retry_strategy) - self.session.mount("https://", adapter) - self.session.mount("http://", adapter) - - self._logo_cache = {} - - # Set up headers - self.headers = { - 'User-Agent': 'LEDMatrix/1.0 (https://github.com/yourusername/LEDMatrix; contact@example.com)', - 'Accept': 'application/json', - 'Accept-Language': 'en-US,en;q=0.9', - 'Accept-Encoding': 'gzip, deflate, br', - 'Connection': 'keep-alive' - } - self.last_update = 0 - self.current_game = None - self.fonts = self._load_fonts() - - # Initialize dynamic team resolver and resolve favorite teams - self.dynamic_resolver = DynamicTeamResolver() - raw_favorite_teams = self.mode_config.get("favorite_teams", []) - self.favorite_teams = self.dynamic_resolver.resolve_teams(raw_favorite_teams, sport_key) - - # Log dynamic team resolution - if raw_favorite_teams != self.favorite_teams: - self.logger.info(f"Resolved dynamic teams: {raw_favorite_teams} -> {self.favorite_teams}") - else: - self.logger.info(f"Favorite teams: {self.favorite_teams}") - - self.logger.setLevel(logging.INFO) - - # Initialize team rankings cache - self._team_rankings_cache = {} - self._rankings_cache_timestamp = 0 - self._rankings_cache_duration = 3600 # Cache rankings for 1 hour - - # Initialize background data service with optimized settings - # Hardcoded for memory optimization: 1 worker, 30s timeout, 3 retries - self.background_service = get_background_service(self.cache_manager, max_workers=1) - self.background_fetch_requests = {} # Track background fetch requests - self.background_enabled = True - self.logger.info("Background service enabled with 1 worker (memory optimized)") - - def _initialize_logo_dir(self, configured_path: Path) -> Path: - """Resolve and ensure a writable logo directory, falling back when necessary.""" - downloader = LogoDownloader() - resolved_configured = self._resolve_project_path(configured_path) - candidates = [resolved_configured] + self._get_logo_directory_fallbacks(resolved_configured) - - for candidate in candidates: - candidate_path = self._resolve_project_path(candidate) - if downloader.ensure_logo_directory(str(candidate_path)): - if candidate_path != resolved_configured: - self.logger.warning( - "Configured logo directory '%s' is not writable; using fallback '%s'", - resolved_configured, - candidate_path, - ) - return candidate_path - - self.logger.error( - "Unable to find a writable logo directory. Logos may fail to download (last attempted: %s)", - resolved_configured, - ) - return resolved_configured - - def _resolve_project_path(self, path: Path) -> Path: - """Convert relative paths to absolute ones rooted at the project directory.""" - if path.is_absolute(): - return path - project_root = Path(__file__).resolve().parents[2] - return (project_root / path).resolve() - - def _get_logo_directory_fallbacks(self, configured_dir: Path) -> List[Path]: - """Return fallback directories to try when the configured directory is not writable.""" - fallbacks: List[Path] = [] - - env_override = os.environ.get("LEDMATRIX_LOGO_DIR") - if env_override: - env_path = Path(env_override) - if not env_path.is_absolute(): - env_path = self._resolve_project_path(env_path) - fallbacks.append(env_path / self.sport_key) - - cache_dir = getattr(self.cache_manager, "cache_dir", None) - if cache_dir: - fallbacks.append(Path(cache_dir) / "logos" / self.sport_key) - - try: - fallbacks.append(Path.home() / ".ledmatrix" / "logos" / self.sport_key) - except RuntimeError as e: - self.logger.debug("Could not resolve home directory (expected for service users): %s", e) - - fallbacks.append(Path(tempfile.gettempdir()) / "ledmatrix_logos" / self.sport_key) - - unique_fallbacks: List[Path] = [] - seen = set() - for candidate in fallbacks: - if candidate == configured_dir: - continue - if candidate not in seen: - unique_fallbacks.append(candidate) - seen.add(candidate) - - return unique_fallbacks - - def _get_season_schedule_dates(self) -> tuple[str, str]: - return "", "" - - def _draw_scorebug_layout(self, game: Dict, force_clear: bool = False) -> None: - """Placeholder draw method - subclasses should override.""" - # This base method will be simple, subclasses provide specifics - try: - img = Image.new('RGB', (self.display_width, self.display_height), (0, 0, 0)) - draw = ImageDraw.Draw(img) - status = game.get("status_text", "N/A") - self._draw_text_with_outline(draw, status, (2, 2), self.fonts['status']) - self.display_manager.image.paste(img, (0, 0)) - # Don't call update_display here, let subclasses handle it after drawing - except Exception as e: - self.logger.error(f"Error in base _draw_scorebug_layout: {e}", exc_info=True) - - - def display(self, force_clear: bool = False) -> bool: - """Common display method for all NCAA FB managers""" # Updated docstring - if not self.is_enabled: # Check if module is enabled - return False - - if not self.current_game: - # Clear display if force_clear is True, even when there's no content - # This prevents black screens when switching to modes with no content - if force_clear: - try: - self.display_manager.clear() - self.display_manager.update_display() - except Exception as e: - self.logger.debug(f"Error clearing display when no content: {e}") - - current_time = time.time() - if not hasattr(self, '_last_warning_time'): - self._last_warning_time = 0 - if current_time - getattr(self, '_last_warning_time', 0) > 300: - self.logger.warning(f"No game data available to display in {self.__class__.__name__}") - setattr(self, '_last_warning_time', current_time) - return False - - try: - self._draw_scorebug_layout(self.current_game, force_clear) - # display_manager.update_display() should be called within subclass draw methods - # or after calling display() in the main loop. Let's keep it out of the base display. - return True - except Exception as e: - self.logger.error(f"Error during display call in {self.__class__.__name__}: {e}", exc_info=True) - return False - - - def _load_fonts(self): - """Load fonts used by the scoreboard.""" - fonts = {} - try: - fonts['score'] = ImageFont.truetype("assets/fonts/PressStart2P-Regular.ttf", 10) - fonts['time'] = ImageFont.truetype("assets/fonts/PressStart2P-Regular.ttf", 8) - fonts['team'] = ImageFont.truetype("assets/fonts/PressStart2P-Regular.ttf", 8) - fonts['status'] = ImageFont.truetype("assets/fonts/4x6-font.ttf", 6) # Using 4x6 for status - fonts['detail'] = ImageFont.truetype("assets/fonts/4x6-font.ttf", 6) # Added detail font - fonts['rank'] = ImageFont.truetype("assets/fonts/PressStart2P-Regular.ttf", 10) - logging.info("Successfully loaded fonts") # Changed log prefix - except IOError: - logging.warning("Fonts not found, using default PIL font.") # Changed log prefix - fonts['score'] = ImageFont.load_default() - fonts['time'] = ImageFont.load_default() - fonts['team'] = ImageFont.load_default() - fonts['status'] = ImageFont.load_default() - fonts['detail'] = ImageFont.load_default() - fonts['rank'] = ImageFont.load_default() - return fonts - - def _draw_dynamic_odds(self, draw: ImageDraw.Draw, odds: Dict[str, Any], width: int, height: int) -> None: - """Draw odds with dynamic positioning - only show negative spread and position O/U based on favored team.""" - home_team_odds = odds.get('home_team_odds', {}) - away_team_odds = odds.get('away_team_odds', {}) - home_spread = home_team_odds.get('spread_odds') - away_spread = away_team_odds.get('spread_odds') - - # Get top-level spread as fallback - top_level_spread = odds.get('spread') - - # If we have a top-level spread and the individual spreads are None or 0, use the top-level - if top_level_spread is not None: - if home_spread is None or home_spread == 0.0: - home_spread = top_level_spread - if away_spread is None: - away_spread = -top_level_spread - - # Determine which team is favored (has negative spread) - home_favored = home_spread is not None and home_spread < 0 - away_favored = away_spread is not None and away_spread < 0 - - # Only show the negative spread (favored team) - favored_spread = None - favored_side = None - - if home_favored: - favored_spread = home_spread - favored_side = 'home' - self.logger.debug(f"Home team favored with spread: {favored_spread}") - elif away_favored: - favored_spread = away_spread - favored_side = 'away' - self.logger.debug(f"Away team favored with spread: {favored_spread}") - else: - self.logger.debug("No clear favorite - spreads: home={home_spread}, away={away_spread}") - - # Show the negative spread on the appropriate side - if favored_spread is not None: - spread_text = str(favored_spread) - font = self.fonts['detail'] # Use detail font for odds - - if favored_side == 'home': - # Home team is favored, show spread on right side - spread_width = draw.textlength(spread_text, font=font) - spread_x = width - spread_width # Top right - spread_y = 0 - self._draw_text_with_outline(draw, spread_text, (spread_x, spread_y), font, fill=(0, 255, 0)) - self.logger.debug(f"Showing home spread '{spread_text}' on right side") - else: - # Away team is favored, show spread on left side - spread_x = 0 # Top left - spread_y = 0 - self._draw_text_with_outline(draw, spread_text, (spread_x, spread_y), font, fill=(0, 255, 0)) - self.logger.debug(f"Showing away spread '{spread_text}' on left side") - - # Show over/under on the opposite side of the favored team - over_under = odds.get('over_under') - if over_under is not None: - ou_text = f"O/U: {over_under}" - font = self.fonts['detail'] # Use detail font for odds - ou_width = draw.textlength(ou_text, font=font) - - if favored_side == 'home': - # Home team is favored, show O/U on left side (opposite of spread) - ou_x = 0 # Top left - ou_y = 0 - self.logger.debug(f"Showing O/U '{ou_text}' on left side (home favored)") - elif favored_side == 'away': - # Away team is favored, show O/U on right side (opposite of spread) - ou_x = width - ou_width # Top right - ou_y = 0 - self.logger.debug(f"Showing O/U '{ou_text}' on right side (away favored)") - else: - # No clear favorite, show O/U in center - ou_x = (width - ou_width) // 2 - ou_y = 0 - self.logger.debug(f"Showing O/U '{ou_text}' in center (no clear favorite)") - - self._draw_text_with_outline(draw, ou_text, (ou_x, ou_y), font, fill=(0, 255, 0)) - - def _draw_text_with_outline(self, draw, text, position, font, fill=(255, 255, 255), outline_color=(0, 0, 0)): - """Draw text with a black outline for better readability.""" - x, y = position - for dx, dy in [(-1, -1), (-1, 0), (-1, 1), (0, -1), (0, 1), (1, -1), (1, 0), (1, 1)]: - draw.text((x + dx, y + dy), text, font=font, fill=outline_color) - draw.text((x, y), text, font=font, fill=fill) - - def _load_and_resize_logo(self, team_id: str, team_abbrev: str, logo_path: Path, logo_url: str | None ) -> Optional[Image.Image]: - """Load and resize a team logo, with caching and automatic download if missing.""" - self.logger.debug(f"Logo path: {logo_path}") - if team_abbrev in self._logo_cache: - self.logger.debug(f"Using cached logo for {team_abbrev}") - return self._logo_cache[team_abbrev] - - try: - # Try different filename variations first (for cases like TA&M vs TAANDM) - actual_logo_path = None - filename_variations = LogoDownloader.get_logo_filename_variations(team_abbrev) - - for filename in filename_variations: - test_path = logo_path.parent / filename - if test_path.exists(): - actual_logo_path = test_path - self.logger.debug(f"Found logo at alternative path: {actual_logo_path}") - break - - # If no variation found, try to download missing logo - if not actual_logo_path and not logo_path.exists(): - self.logger.info(f"Logo not found for {team_abbrev} at {logo_path}. Attempting to download.") - - # Try to download the logo from ESPN API (this will create placeholder if download fails) - download_missing_logo(self.sport_key, team_id, team_abbrev, logo_path, logo_url) - actual_logo_path = logo_path - - # Use the original path if no alternative was found - if not actual_logo_path: - actual_logo_path = logo_path - - # Only try to open the logo if the file exists - if os.path.exists(actual_logo_path): - logo = Image.open(actual_logo_path) - else: - self.logger.error(f"Logo file still doesn't exist at {actual_logo_path} after download attempt") - return None - if logo.mode != 'RGBA': - logo = logo.convert('RGBA') - - max_width = int(self.display_width * 1.5) - max_height = int(self.display_height * 1.5) - logo.thumbnail((max_width, max_height), Image.Resampling.LANCZOS) - self._logo_cache[team_abbrev] = logo - return logo - - except Exception as e: - self.logger.error(f"Error loading logo for {team_abbrev}: {e}", exc_info=True) - return None - - def _fetch_odds(self, game: Dict) -> None: - """Fetch odds for a specific game using the new architecture.""" - try: - if not self.show_odds: - return - - if not self.odds_manager: - return - - # Determine update interval based on game state - is_live = game.get('is_live', False) - update_interval = self.mode_config.get("live_odds_update_interval", 60) if is_live \ - else self.mode_config.get("odds_update_interval", 3600) - - # Fetch odds using OddsManager - odds_data = self.odds_manager.get_odds( - sport=self.sport, - league=self.league, - event_id=game['id'], - update_interval_seconds=update_interval, - ) - - if odds_data: - game['odds'] = odds_data - self.logger.debug(f"Successfully fetched and attached odds for game {game['id']}") - else: - self.logger.debug(f"No odds data returned for game {game['id']}") - - except Exception as e: - self.logger.error(f"Error fetching odds for game {game.get('id', 'N/A')}: {e}") - - def _get_timezone(self): - try: - timezone_str = self.config.get('timezone', 'UTC') - return pytz.timezone(timezone_str) - except pytz.UnknownTimeZoneError: - return pytz.utc - - def _should_log(self, warning_type: str, cooldown: int = 60) -> bool: - """Check if we should log a warning based on cooldown period.""" - current_time = time.time() - if current_time - self._last_warning_time > cooldown: - self._last_warning_time = current_time - return True - return False - - def _fetch_team_rankings(self) -> Dict[str, int]: - """Fetch team rankings using the new architecture components.""" - current_time = time.time() - - # Check if we have cached rankings that are still valid - if (self._team_rankings_cache and - current_time - self._rankings_cache_timestamp < self._rankings_cache_duration): - return self._team_rankings_cache - - try: - data = self.data_source.fetch_standings(self.sport, self.league) - - rankings = {} - rankings_data = data.get('rankings', []) - - if rankings_data: - # Use the first ranking (usually AP Top 25) - first_ranking = rankings_data[0] - teams = first_ranking.get('ranks', []) - - for team_data in teams: - team_info = team_data.get('team', {}) - team_abbr = team_info.get('abbreviation', '') - current_rank = team_data.get('current', 0) - - if team_abbr and current_rank > 0: - rankings[team_abbr] = current_rank - - # Cache the results - self._team_rankings_cache = rankings - self._rankings_cache_timestamp = current_time - - self.logger.debug(f"Fetched rankings for {len(rankings)} teams") - return rankings - - except Exception as e: - self.logger.error(f"Error fetching team rankings: {e}") - return {} - - def _extract_game_details_common(self, game_event: Dict) -> tuple[Dict | None, Dict | None, Dict | None, Dict | None, Dict | None]: - if not game_event: - return None, None, None, None, None - try: - competition = game_event["competitions"][0] - status = competition["status"] - competitors = competition["competitors"] - game_date_str = game_event["date"] - situation = competition.get("situation") - start_time_utc = None - try: - # Parse the datetime string - if game_date_str.endswith('Z'): - game_date_str = game_date_str.replace('Z', '+00:00') - dt = datetime.fromisoformat(game_date_str) - # Ensure the datetime is UTC-aware (fromisoformat may create timezone-aware but not pytz.UTC) - if dt.tzinfo is None: - # If naive, assume it's UTC - start_time_utc = dt.replace(tzinfo=pytz.UTC) - else: - # Convert to pytz.UTC for consistency - start_time_utc = dt.astimezone(pytz.UTC) - except ValueError: - logging.warning(f"Could not parse game date: {game_date_str}") - - home_team = next((c for c in competitors if c.get("homeAway") == "home"), None) - away_team = next((c for c in competitors if c.get("homeAway") == "away"), None) - - if not home_team or not away_team: - self.logger.warning(f"Could not find home or away team in event: {game_event.get('id')}") - return None, None, None, None, None - - try: - home_abbr = home_team["team"]["abbreviation"] - except KeyError: - home_abbr = home_team["team"]["name"][:3] - try: - away_abbr = away_team["team"]["abbreviation"] - except KeyError: - away_abbr = away_team["team"]["name"][:3] - - # Check if this is a favorite team game BEFORE doing expensive logging - is_favorite_game = (home_abbr in self.favorite_teams or away_abbr in self.favorite_teams) - - # Only log debug info for favorite team games - if is_favorite_game: - self.logger.debug(f"Processing favorite team game: {game_event.get('id')}") - self.logger.debug(f"Found teams: {away_abbr}@{home_abbr}, Status: {status['type']['name']}, State: {status['type']['state']}") - - game_time, game_date = "", "" - if start_time_utc: - local_time = start_time_utc.astimezone(self._get_timezone()) - game_time = local_time.strftime("%I:%M%p").lstrip('0') - - # Check date format from config - use_short_date_format = self.config.get('display', {}).get('use_short_date_format', False) - if use_short_date_format: - game_date = local_time.strftime("%-m/%-d") - else: - game_date = self.display_manager.format_date_with_ordinal(local_time) - - - home_record = home_team.get('records', [{}])[0].get('summary', '') if home_team.get('records') else '' - away_record = away_team.get('records', [{}])[0].get('summary', '') if away_team.get('records') else '' - - # Don't show "0-0" records - set to blank instead - if home_record in {"0-0", "0-0-0"}: - home_record = '' - if away_record in {"0-0", "0-0-0"}: - away_record = '' - - details = { - "id": game_event.get("id"), - "game_time": game_time, - "game_date": game_date, - "start_time_utc": start_time_utc, - "status_text": status["type"]["shortDetail"], # e.g., "Final", "7:30 PM", "Q1 12:34" - "is_live": status["type"]["state"] == "in", - "is_final": status["type"]["state"] == "post", - "is_upcoming": (status["type"]["state"] == "pre" or - status["type"]["name"].lower() in ['scheduled', 'pre-game', 'status_scheduled']), - "is_halftime": status["type"]["state"] == "halftime" or status["type"]["name"] == "STATUS_HALFTIME", # Added halftime check - "is_period_break": status["type"]["name"] == "STATUS_END_PERIOD", # Added Period Break check - "home_abbr": home_abbr, - "home_id": home_team["id"], - "home_score": home_team.get("score", "0"), - "home_logo_path": self.logo_dir / Path(f"{LogoDownloader.normalize_abbreviation(home_abbr)}.png"), - "home_logo_url": home_team["team"].get("logo"), - "home_record": home_record, - "away_record": away_record, - "away_abbr": away_abbr, - "away_id": away_team["id"], - "away_score": away_team.get("score", "0"), - "away_logo_path": self.logo_dir / Path(f"{LogoDownloader.normalize_abbreviation(away_abbr)}.png"), - "away_logo_url": away_team["team"].get("logo"), - "is_within_window": True, # Whether game is within display window - - } - return details, home_team, away_team, status, situation - except Exception as e: - # Log the problematic event structure if possible - logging.error(f"Error extracting game details: {e} from event: {game_event.get('id')}", exc_info=True) - return None, None, None, None, None - - @abstractmethod - def _extract_game_details(self, game_event: dict) -> dict | None: - details, _, _, _, _ = self._extract_game_details_common(game_event) - return details - - @abstractmethod - def _fetch_data(self) -> Optional[Dict]: - pass - - def _fetch_todays_games(self) -> Optional[Dict]: - """Fetch only today's games for live updates (not entire season).""" - try: - tz = pytz.timezone("America/New_York") # Use full name (not "EST") for DST support - now = datetime.now(tz) - yesterday = now - timedelta(days=1) - formatted_date = now.strftime("%Y%m%d") - formatted_date_yesterday = yesterday.strftime("%Y%m%d") - # Fetch todays games only - url = f"https://site.api.espn.com/apis/site/v2/sports/{self.sport}/{self.league}/scoreboard" - response = self.session.get(url, params={"dates": f"{formatted_date_yesterday}-{formatted_date}", "limit": 1000}, headers=self.headers, timeout=10) - response.raise_for_status() - data = response.json() - events = data.get('events', []) - - self.logger.info(f"Fetched {len(events)} todays games for {self.sport} - {self.league}") - return {'events': events} - except requests.exceptions.RequestException as e: - self.logger.error(f"API error fetching todays games for {self.sport} - {self.league}: {e}") - return None - - def _get_weeks_data(self) -> Optional[Dict]: - """ - Get partial data for immediate display while background fetch is in progress. - This fetches current/recent games only for quick response. - """ - try: - # Fetch current week and next few days for immediate display - now = datetime.now(pytz.utc) - immediate_events = [] - - start_date = now + timedelta(weeks=-2) - end_date = now + timedelta(weeks=1) - date_str = f"{start_date.strftime('%Y%m%d')}-{end_date.strftime('%Y%m%d')}" - url = f"https://site.api.espn.com/apis/site/v2/sports/{self.sport}/{self.league}/scoreboard" - response = self.session.get(url, params={"dates": date_str, "limit": 1000},headers=self.headers, timeout=10) - response.raise_for_status() - data = response.json() - immediate_events = data.get('events', []) - - if immediate_events: - self.logger.info(f"Fetched {len(immediate_events)} events {date_str}") - return {'events': immediate_events} - - except requests.exceptions.RequestException as e: - self.logger.warning(f"Error fetching this weeks games for {self.sport} - {self.league} - {date_str}: {e}") - return None - - def _custom_scorebug_layout(self, game: dict, draw_overlay: ImageDraw.ImageDraw): - pass class SportsUpcoming(SportsCore): + SKIN_MODE = "upcoming" + def __init__(self, config: Dict[str, Any], display_manager: DisplayManager, cache_manager: CacheManager, logger: logging.Logger, sport_key: str): super().__init__(config, display_manager, cache_manager, logger, sport_key) self.upcoming_games = [] # Store all fetched upcoming games initially @@ -660,6 +34,71 @@ class SportsUpcoming(SportsCore): self.last_game_switch = 0 self.game_display_duration = 15 # Display each upcoming game for 15 seconds + def _select_games_for_display( + self, processed_games: List[Dict], favorite_teams: List[str] + ) -> List[Dict]: + """ + Single-pass game selection with proper deduplication and counting. + + When a game involves two favorite teams, it counts toward BOTH teams' limits. + This prevents unexpected game counts from the multi-pass algorithm. + + Team identity goes through the ``_favorite_key`` override point rather + than reading ``home_abbr``/``away_abbr`` directly, because abbreviations + are not unique in every league (NRL matches on team ID instead). + """ + sorted_games = sorted( + processed_games, + key=lambda g: g.get("start_time_utc") + or datetime.max.replace(tzinfo=timezone.utc), + ) + + if not favorite_teams: + return sorted_games + + selected_games = [] + selected_ids = set() + team_counts = {team: 0 for team in favorite_teams} + + for game in sorted_games: + game_id = game.get("id") + if game_id in selected_ids: + continue + + home = self._favorite_key(game, "home") + away = self._favorite_key(game, "away") + + home_fav = home in favorite_teams + away_fav = away in favorite_teams + + if not home_fav and not away_fav: + continue + + home_needs = home_fav and team_counts[home] < self.upcoming_games_to_show + away_needs = away_fav and team_counts[away] < self.upcoming_games_to_show + + if home_needs or away_needs: + selected_games.append(game) + selected_ids.add(game_id) + if home_fav: + team_counts[home] += 1 + if away_fav: + team_counts[away] += 1 + + self.logger.debug( + f"Selected game {away}@{home}: team_counts={team_counts}" + ) + + if all(c >= self.upcoming_games_to_show for c in team_counts.values()): + self.logger.debug("All favorite teams satisfied, stopping selection") + break + + self.logger.info( + f"Selected {len(selected_games)} games for {len(favorite_teams)} " + f"favorite teams: {team_counts}" + ) + return selected_games + def update(self): """Update upcoming games data.""" if not self.is_enabled: return @@ -706,8 +145,7 @@ class SportsUpcoming(SportsCore): if (game['home_abbr'] in self.favorite_teams or game['away_abbr'] in self.favorite_teams): favorite_games_found += 1 - if self.show_odds: - self._fetch_odds(game) + # Odds are NOT fetched here -- see after selection below. # Enhanced logging for debugging self.logger.info(f"Found {all_upcoming_games} total upcoming games in data") @@ -751,6 +189,20 @@ class SportsUpcoming(SportsCore): # Limit to the specified number of upcoming games team_games = team_games[:self.upcoming_games_to_show] + # Odds are fetched here, for the games that survived selection, + # rather than in the loop that collects them. That loop walks every + # upcoming game in the schedule window, and for a college league + # the window is enormous -- a live rig logged 946 upcoming games in + # one cycle and displayed 1 of them. The comment up there claimed + # odds were fetched "only for games that will be displayed", but + # the only narrowing it applied was show_favorite_teams_only, which + # is not the default; in the usual case nothing narrowed it at all + # and every game cost a separate ESPN request on a Pi that is also + # driving the panel. + if self.show_odds: + for game in team_games: + self._fetch_odds(game) + # Log changes or periodically should_log = ( current_time - self.last_log_time >= self.log_interval or @@ -973,7 +425,7 @@ class SportsUpcoming(SportsCore): self.logger.debug(f"Switched to game index {self.current_game_index}") if self.current_game: - self._draw_scorebug_layout(self.current_game, force_clear) + self._render_game(self.current_game, force_clear) return True # update_display() is called within _draw_scorebug_layout for upcoming return False @@ -984,6 +436,7 @@ class SportsUpcoming(SportsCore): class SportsRecent(SportsCore): + SKIN_MODE = "recent" def __init__(self, config: Dict[str, Any], display_manager: DisplayManager, cache_manager: CacheManager, logger: logging.Logger, sport_key: str): super().__init__(config, display_manager, cache_manager, logger, sport_key) @@ -994,6 +447,96 @@ class SportsRecent(SportsCore): self.update_interval = self.mode_config.get("recent_update_interval", 3600) # Check for recent games every hour self.last_game_switch = 0 self.game_display_duration = 15 # Display each recent game for 15 seconds + # Tracks when each game was first seen with an expired clock, keyed by + # game id. Promoted alongside the zero-clock helpers below; without it + # the first _get_zero_clock_duration() call raises AttributeError. + self._zero_clock_timestamps: Dict[str, float] = {} # Track games at 0:00 + + # -- Zero-clock tracking ------------------------------------------------ + # Byte-identical in all nine plugin copies. Note that afl/nrl/soccer define + # these but never call them — their clocks count up, so 0:00 means kickoff + # rather than expiry (see CLOCK_COUNTS_DOWN on SportsLive). That makes the + # pair a future `CountdownClockMixin` candidate so it stops appearing in the + # MRO of sports that cannot use it — B2 work, not now. + + def _get_zero_clock_duration(self, game_id: str) -> float: + """Track how long a game has been at 0:00 clock.""" + current_time = time.time() + if game_id not in self._zero_clock_timestamps: + self._zero_clock_timestamps[game_id] = current_time + return 0.0 + return current_time - self._zero_clock_timestamps[game_id] + + def _clear_zero_clock_tracking(self, game_id: str) -> None: + """Clear tracking when game clock moves away from 0:00 or game ends.""" + if game_id in self._zero_clock_timestamps: + del self._zero_clock_timestamps[game_id] + + def _select_recent_games_for_display( + self, processed_games: List[Dict], favorite_teams: List[str] + ) -> List[Dict]: + """ + Single-pass game selection for recent games with proper deduplication. + + When a game involves two favorite teams, it counts toward BOTH teams' limits. + Games are sorted by most recent first. + + Team identity goes through the ``_favorite_key`` override point rather + than reading ``home_abbr``/``away_abbr`` directly, because abbreviations + are not unique in every league (NRL matches on team ID instead). + """ + sorted_games = sorted( + processed_games, + key=lambda g: g.get("start_time_utc") + or datetime.min.replace(tzinfo=timezone.utc), + reverse=True, + ) + + if not favorite_teams: + return sorted_games + + selected_games = [] + selected_ids = set() + team_counts = {team: 0 for team in favorite_teams} + + for game in sorted_games: + game_id = game.get("id") + if game_id in selected_ids: + continue + + home = self._favorite_key(game, "home") + away = self._favorite_key(game, "away") + + home_fav = home in favorite_teams + away_fav = away in favorite_teams + + if not home_fav and not away_fav: + continue + + home_needs = home_fav and team_counts[home] < self.recent_games_to_show + away_needs = away_fav and team_counts[away] < self.recent_games_to_show + + if home_needs or away_needs: + selected_games.append(game) + selected_ids.add(game_id) + if home_fav: + team_counts[home] += 1 + if away_fav: + team_counts[away] += 1 + + self.logger.debug( + f"Selected recent game {away}@{home}: team_counts={team_counts}" + ) + + if all(c >= self.recent_games_to_show for c in team_counts.values()): + self.logger.debug("All favorite teams satisfied, stopping selection") + break + + self.logger.info( + f"Selected {len(selected_games)} recent games for {len(favorite_teams)} " + f"favorite teams: {team_counts}" + ) + return selected_games def update(self): """Update recent games data.""" @@ -1274,7 +817,7 @@ class SportsRecent(SportsCore): self.logger.debug(f"Switched to game index {self.current_game_index}") if self.current_game: - self._draw_scorebug_layout(self.current_game, force_clear) + self._render_game(self.current_game, force_clear) return True # update_display() is called within _draw_scorebug_layout for recent return False @@ -1284,6 +827,17 @@ class SportsRecent(SportsCore): return False class SportsLive(SportsCore): + # Per-sport constants for the "is this live game actually over?" check. + # These are values, not behavior, so they are class attributes rather than + # override points (see docs/SPORTS_UNIFICATION.md "Override points"). + # + # FINAL_PERIOD: the period at/after which an expired clock can mean "over". + # 4 for four-quarter sports; hockey overrides to 3. + # CLOCK_COUNTS_DOWN: whether "0:00" means the clock expired. False for + # sports whose clock counts up (soccer/afl/nrl), where 0:00 is kickoff — + # running the expiry branch there would evict games that just started. + FINAL_PERIOD = 4 + CLOCK_COUNTS_DOWN = True def __init__(self, config: Dict[str, Any], display_manager: DisplayManager, cache_manager: CacheManager, logger: logging.Logger, sport_key: str): super().__init__(config, display_manager, cache_manager, logger, sport_key) @@ -1301,11 +855,117 @@ class SportsLive(SportsCore): self.count_log_interval = 5 # Only log count data every 5 seconds # Initialize test_mode - defaults to False (live mode) self.test_mode = self.mode_config.get("test_mode", False) + # Freshness bookkeeping for _detect_stale_games(). The base class only + # *reads* this map; a subclass's update() stamps entries as it ingests a + # feed: {game_id: {"clock": ts, "score": ts, "last_seen": ts}}. + # Until a subclass writes "last_seen", the staleness branch of + # _detect_stale_games is inert and only the game-over check applies. + self.game_update_timestamps: Dict[str, Dict[str, float]] = {} + self.stale_game_timeout = self.mode_config.get("stale_game_timeout", 300) # 5 minutes default @abstractmethod def _test_mode_update(self) -> None: return + def _is_game_really_over(self, game: Dict) -> bool: + """Check if a game appears to be over even if API says it's live. + + Two independent signals: + 1. ``period_text`` says "final" — universal across every sport. + 2. The clock has expired at/after :attr:`FINAL_PERIOD` — only meaningful + where :attr:`CLOCK_COUNTS_DOWN` is true. + + Fails *safe*: anything ambiguous returns False and the game keeps being + displayed. The only caller, :meth:`_detect_stale_games`, removes games + on a True, so a false positive silently drops a live game. + """ + game_str = f"{game.get('away_abbr')}@{game.get('home_abbr')}" + + # `period_text` may be present-but-None; `or ""` keeps that from raising + # AttributeError — the caller has no try/except around this call. + period_text = (game.get("period_text") or "").lower() + if "final" in period_text: + self.logger.debug( + f"_is_game_really_over({game_str}): " + f"returning True - 'final' in period_text='{period_text}'" + ) + return True + + if not self.CLOCK_COUNTS_DOWN: + # Count-up clock: 0:00 means the match has not started. + self.logger.debug( + f"_is_game_really_over({game_str}): returning False " + f"(count-up clock, period_text='{period_text}')" + ) + return False + + raw_clock = game.get("clock") + # `or 0` rather than a get() default: feeds routinely send an explicit + # null period, and `None >= FINAL_PERIOD` raises TypeError — which would + # take down the whole live-update pass, since the only caller + # (_detect_stale_games) has no try/except around it. + period = game.get("period") or 0 + + # Only check clock-based finish if we have a valid clock string. A + # missing or non-string clock is NOT coerced to "0:00": sports without a + # game clock (e.g. baseball, where `period` is the inning) would + # otherwise be declared over from the FINAL_PERIOD-th period onward. + if isinstance(raw_clock, str) and raw_clock.strip() and period >= self.FINAL_PERIOD: + clock = raw_clock + # Compare numerically rather than against a literal set: feeds spell + # an expired clock "0:00", ":00" and "00:00" depending on sport, and + # a membership test silently misses every spelling not listed. + clock_normalized = clock.replace(":", "").strip() + if clock_normalized.isdigit() and int(clock_normalized) == 0: + self.logger.debug( + f"_is_game_really_over({game_str}): " + f"returning True - clock at 0:00 (clock='{clock}', period={period})" + ) + return True + + self.logger.debug( + f"_is_game_really_over({game_str}): returning False" + ) + return False + + def _detect_stale_games(self, games: List[Dict]) -> None: + """Remove games that appear stale or haven't updated. + + Mutates ``games`` **in place** and returns None. Removal is by value + (``list.remove`` uses ``dict.__eq__``), so two structurally-equal game + dicts in the same list would drop the first occurrence. + """ + current_time = time.time() + + for game in games[:]: # Copy list to iterate safely + game_id = game.get("id") + if not game_id: + continue + + # Check if game data is stale + timestamps = self.game_update_timestamps.get(game_id, {}) + last_seen = timestamps.get("last_seen", 0) + + if last_seen > 0 and current_time - last_seen > self.stale_game_timeout: + self.logger.warning( + f"Removing stale game {game.get('away_abbr')}@{game.get('home_abbr')} " + f"(last seen {int(current_time - last_seen)}s ago)" + ) + games.remove(game) + if game_id in self.game_update_timestamps: + del self.game_update_timestamps[game_id] + continue + + # Also check if game appears to be over + if self._is_game_really_over(game): + self.logger.debug( + f"Removing game that appears over: {game.get('away_abbr')}@{game.get('home_abbr')} " + f"(clock={game.get('clock')}, period={game.get('period')}, period_text={game.get('period_text')})" + ) + games.remove(game) + if game_id in self.game_update_timestamps: + del self.game_update_timestamps[game_id] + def update(self): """Update live game data and handle game switching.""" if not self.is_enabled: diff --git a/src/base_odds_manager.py b/src/base_odds_manager.py index 3520ce67..4b455069 100644 --- a/src/base_odds_manager.py +++ b/src/base_odds_manager.py @@ -12,6 +12,8 @@ Follows LEDMatrix configuration management patterns: """ import logging +import time + import requests import json from typing import Dict, Any, Optional, List @@ -42,10 +44,35 @@ class BaseOddsManager: self.config_manager = config_manager self.logger = logging.getLogger(__name__) self.base_url = "https://sports.core.api.espn.com/v2/sports" + + # This path used a bare requests.get, so it identified itself as + # python-requests/x.y -- the one thing ESPN is known to reject. Around + # 2026-08-04 it began 403ing browser strings and bare custom tokens + # alike; what it accepts is a token with a URL that says who is + # calling. Every other ESPN caller in the tree already sends this + # (src/common/api_helper.py, src/base_classes/data_sources.py); the + # odds path was simply missed, and it is the one whose failures cost + # the caller its whole update budget. + # + # Deliberately no retry adapter, unlike api_helper: retries multiply + # request_timeout, which is set to 5s precisely to stay inside that + # budget. One try, then the cooldown below. + self.session = requests.Session() + self.session.headers.update({ + 'User-Agent': 'LEDMatrix/1.0 (+https://github.com/ChuckBuilds/LEDMatrix)', + 'Accept': 'application/json', + }) # Configuration with defaults self.update_interval = 3600 # 1 hour default - self.request_timeout = 30 # 30 seconds default + # Well under the plugin executor's 30s operation budget. At 30s a + # single stalled ESPN request consumed the entire budget and the whole + # update() was killed -- and odds are fetched per live game, inside the + # live update loop, with show_odds defaulting on. Losing one game's + # odds beats losing the update that carries every game's score. + self.request_timeout = 5 + # Set when a request fails; until then, skip the network entirely. + self._skip_network_until = 0.0 self.cache_ttl = 1800 # 30 minutes default # Load configuration if available @@ -73,6 +100,14 @@ class BaseOddsManager: except Exception as e: self.logger.warning(f"Failed to load BaseOddsManager configuration: {e}") + # After a network failure, stop trying for this long and serve cache only. + # A short per-request timeout bounds one stall, but a full Sunday slate is + # ~16 games fetched in a loop, so 16 consecutive timeouts still blow the + # budget. When ESPN is unreachable it is unreachable for all of them, so + # the first failure is enough to know: skip the rest of this pass and try + # again shortly. + _FAILURE_COOLDOWN = 60.0 + def get_odds(self, sport: str | None, league: str | None, event_id: str, update_interval_seconds: int = None) -> Optional[Dict[str, Any]]: """ @@ -101,8 +136,18 @@ class BaseOddsManager: self.logger.info(f"Using cached odds from ESPN for {cache_key}") return cached_data + if time.monotonic() < self._skip_network_until: + # A recent request failed, so ESPN is very likely still unreachable. + # Returning now keeps the caller's update inside its time budget + # instead of paying the timeout again for every remaining game. + self.logger.debug( + "Skipping odds fetch for %s: a recent request failed, holding off " + "for another %.0fs", cache_key, + self._skip_network_until - time.monotonic()) + return None + self.logger.info(f"Cache miss - fetching fresh odds from ESPN for {cache_key}") - + try: # Map league names to ESPN API format league_mapping = { @@ -117,10 +162,12 @@ class BaseOddsManager: url = f"{self.base_url}/{sport}/leagues/{espn_league}/events/{event_id}/competitions/{event_id}/odds" self.logger.info(f"Requesting odds from URL: {url}") - response = requests.get(url, timeout=self.request_timeout) + response = self.session.get(url, timeout=self.request_timeout) response.raise_for_status() raw_data = response.json() + self._skip_network_until = 0.0 # reachable again + self.logger.debug(f"Received raw odds data from ESPN: {json.dumps(raw_data, indent=2)}") odds_data = self._extract_espn_data(raw_data) @@ -140,7 +187,11 @@ class BaseOddsManager: return odds_data except requests.exceptions.RequestException as e: - self.logger.error(f"Error fetching odds from ESPN API for {cache_key}: {e}") + self._skip_network_until = time.monotonic() + self._FAILURE_COOLDOWN + self.logger.error( + "Error fetching odds from ESPN API for %s: %s. Holding off on odds " + "for %.0fs so a slate of games does not pay this timeout each.", + cache_key, e, self._FAILURE_COOLDOWN) except json.JSONDecodeError: self.logger.error(f"Error decoding JSON response from ESPN API for {cache_key}.") @@ -163,19 +214,25 @@ class BaseOddsManager: item = data["items"][0] self.logger.debug(f"First item keys: {list(item.keys())}") - # The ESPN API returns odds data directly in the item, not in a providers array - # Extract the odds data directly from the item + # The ESPN API returns odds data directly in the item, not in a + # providers array. ESPN sends explicit JSON nulls for absent + # sides ("homeTeamOdds": null), so every level uses `or {}` — + # .get's default only applies when the key is missing entirely. + home = item.get("homeTeamOdds") or {} + away = item.get("awayTeamOdds") or {} extracted_data = { "details": item.get("details"), "over_under": item.get("overUnder"), "spread": item.get("spread"), "home_team_odds": { - "money_line": item.get("homeTeamOdds", {}).get("moneyLine"), - "spread_odds": item.get("homeTeamOdds", {}).get("current", {}).get("pointSpread", {}).get("value") + "money_line": home.get("moneyLine"), + "spread_odds": ((home.get("current") or {}) + .get("pointSpread") or {}).get("value") }, "away_team_odds": { - "money_line": item.get("awayTeamOdds", {}).get("moneyLine"), - "spread_odds": item.get("awayTeamOdds", {}).get("current", {}).get("pointSpread", {}).get("value") + "money_line": away.get("moneyLine"), + "spread_odds": ((away.get("current") or {}) + .get("pointSpread") or {}).get("value") } } self.logger.debug(f"Returning extracted odds data: {json.dumps(extracted_data, indent=2)}") @@ -260,9 +317,13 @@ class BaseOddsManager: Returns: Formatted odds summary string """ - if not self.is_odds_available(odds_data): + # Gate only on truly-empty / negative-cached data. is_odds_available + # deliberately ignores money lines (its callers decide whether to + # RENDER an odds widget), but a summary of money-line-only odds is + # still meaningful — the parts loop below handles them. + if not odds_data or odds_data.get('no_odds'): return "No odds available" - + parts = [] # Add spread information diff --git a/src/cache/disk_cache.py b/src/cache/disk_cache.py index 64b72f07..03cfb2f1 100644 --- a/src/cache/disk_cache.py +++ b/src/cache/disk_cache.py @@ -14,6 +14,13 @@ import zlib from typing import Dict, Any, Optional, Protocol from datetime import datetime +# How old an abandoned write's temp file must be before the sweep removes it. +# A real write holds its temp file for milliseconds, so an hour is far beyond +# any in-flight write while still clearing the same day's debris. Deliberately +# not tied to the retention policies: those describe how long data stays +# useful, and a half-written file was never useful. +_ORPHAN_TEMP_MAX_AGE_SECONDS = 3600 + class CacheStrategyProtocol(Protocol): @@ -112,6 +119,22 @@ class DiskCache: record_ts = None now = time.time() + + # An explicit per-entry ttl wins over the caller's max_age. The + # caller that wrote the record knows what its data is; max_age is + # inferred from substrings in the key ("live", "odds", "stock") and + # is only a fallback for records that never said. Until now the ttl + # was stored and ignored, so `set(key, data, ttl=...)` did nothing + # at all -- 48 plugin call sites and 4 in the core were writing a + # number no read path consulted. + effective_max_age = max_age + if isinstance(record, dict): + stored_ttl = record.get('ttl') + if isinstance(stored_ttl, (int, float)) and not isinstance(stored_ttl, bool) \ + and stored_ttl >= 0: + effective_max_age = stored_ttl + max_age = effective_max_age + # max_age=None means "never expires" (mirrors MemoryCache and the # cache_manager docstring). Guard it explicitly — otherwise the # comparison below raises TypeError and the record is treated as a @@ -331,6 +354,23 @@ class DiskCache: """Get the cache directory path.""" return self.cache_dir + @staticmethod + def _is_orphaned_temp(filename: str) -> bool: + """Whether a name is one of set()'s temp files rather than real data. + + Matches only what this class creates: mkstemp with a prefix of + ".." , so ".weather.json.a1b2c3d4". The shape is + checked rather than just the leading dot, because this predicate + deletes things -- a stray dotfile someone left in the cache directory + is not ours to remove, and a completed ".json" never is either. + """ + if not filename.startswith('.') or filename.endswith('.json'): + return False + head, sep, suffix = filename.rpartition('.json.') + # head is the key (non-empty after the leading dot), suffix is + # mkstemp's random component. + return bool(sep) and len(head) > 1 and bool(suffix) + def cleanup_expired_files(self, cache_strategy: CacheStrategyProtocol, retention_policies: Dict[str, int]) -> Dict[str, Any]: """ Clean up expired cache files based on retention policies. @@ -365,11 +405,50 @@ class DiskCache: try: with self._lock: # Get snapshot of files while holding lock briefly - filenames = [f for f in os.listdir(self.cache_dir) if f.endswith('.json')] + entries = os.listdir(self.cache_dir) except OSError as list_error: self.logger.error("Error listing cache directory %s: %s", self.cache_dir, list_error, exc_info=True) stats['errors'] += 1 return stats + + filenames = [f for f in entries if f.endswith('.json')] + + # Sweep temp files abandoned by a write that never finished. set() + # removes its own in a finally, so these are the ones where the + # process died between mkstemp and os.replace -- a SIGKILL, a lost + # restart race, a power cut. Nothing ever collected them: they are + # named "..json.", and the scan above only matches + # names ending in .json, so they accumulated indefinitely. Measured + # on a live rig: 76 files, 1,050 MB, 81% of the whole cache + # directory, the oldest six months old. + stats['orphan_temp_files_deleted'] = 0 + for filename in (f for f in entries if self._is_orphaned_temp(f)): + # Counted as scanned like any other candidate, so files_deleted + # can never exceed files_scanned and the summary line reads + # honestly ("77/8864", not "77/0"). + stats['files_scanned'] += 1 + path = os.path.join(self.cache_dir, filename) + try: + # An in-flight write lives for milliseconds, so anything + # this old is certainly abandoned rather than in progress. + if (current_time - os.path.getmtime(path)) <= _ORPHAN_TEMP_MAX_AGE_SECONDS: + continue + with self._lock: + size = os.path.getsize(path) + os.remove(path) + stats['files_deleted'] += 1 + stats['orphan_temp_files_deleted'] += 1 + stats['space_freed_bytes'] += size + except FileNotFoundError: + continue # another sweep got there first + except OSError as e: + stats['errors'] += 1 + self.logger.warning("Error deleting orphaned temp file %s: %s", filename, e) + + if stats['orphan_temp_files_deleted']: + self.logger.info( + "Removed %d abandoned cache temp file(s)", + stats['orphan_temp_files_deleted']) # Process files outside the lock to avoid blocking get/set operations for filename in filenames: diff --git a/src/cache/memory_cache.py b/src/cache/memory_cache.py index 33c9963b..55ca010d 100644 --- a/src/cache/memory_cache.py +++ b/src/cache/memory_cache.py @@ -4,11 +4,58 @@ Memory Cache Handles in-memory caching with TTL support, size limits, and automatic cleanup. """ +import os import time import threading import logging from typing import Dict, Any, Optional +# Historical fixed ceiling, kept as the fallback when RAM cannot be read. +DEFAULT_MAX_SIZE = 1000 + + +def _total_memory_mb() -> Optional[float]: + """Physical RAM in MB, or None where /proc/meminfo is unavailable.""" + try: + with open('/proc/meminfo', 'r', encoding='utf-8') as fh: + for line in fh: + if line.startswith('MemTotal:'): + return int(line.split()[1]) / 1024 + except (OSError, ValueError, IndexError): + return None + return None + + +def default_max_size() -> int: + """Entry ceiling scaled to this machine's RAM. + + One fixed ceiling cannot serve both a 512 MB Pi Zero 2 W and an 8 GB Pi 5. + Entries here are parsed API payloads that routinely run tens of kilobytes + each, so a thousand of them is a comfortable cache on a large board and a + substantial fraction of total RAM on a small one — where the process + competing for that RAM is also driving the panel. Set + LEDMATRIX_CACHE_MAX_ENTRIES to override. + """ + override = os.environ.get('LEDMATRIX_CACHE_MAX_ENTRIES') + if override: + try: + value = int(override) + if value > 0: + return value + except ValueError: + pass + + total_mb = _total_memory_mb() + if total_mb is None: + return DEFAULT_MAX_SIZE + if total_mb < 1536: # 512 MB and 1 GB boards + return 150 + if total_mb < 3072: # 2 GB + return 400 + if total_mb < 6144: # 4 GB + return 800 + return 1500 # 8 GB and up + class MemoryCache: """Manages in-memory cache with TTL and size limits.""" @@ -57,6 +104,16 @@ class MemoryCache: if timestamp is None: return None + # An explicit per-entry ttl wins over the caller's max_age, matching + # DiskCache. max_age is inferred from substrings in the key and is + # only a fallback for records that did not say what they wanted. + record = self._cache[key] + if isinstance(record, dict): + stored_ttl = record.get('ttl') + if isinstance(stored_ttl, (int, float)) and not isinstance(stored_ttl, bool) \ + and stored_ttl >= 0: + max_age = stored_ttl + # Check expiration if max_age is not None and (now - timestamp) > max_age: # Expired - remove it @@ -77,6 +134,32 @@ class MemoryCache: with self._lock: self._cache[key] = value self._timestamps[key] = time.time() + # Enforce the ceiling here rather than leaving it to the periodic + # cleanup, which only runs every cleanup_interval seconds (300 by + # default). A burst of inserts between two sweeps could otherwise + # take the cache far past _max_size, which is the memory growth this + # limit exists to prevent -- and on a 1GB board that is the + # difference between a bounded cache and an unreachable Pi. + self._evict_over_limit_locked() + + def _evict_over_limit_locked(self) -> int: + """Drop oldest entries until the cache is within _max_size. + + Caller must hold self._lock. Returns the number of entries removed. + """ + excess = len(self._cache) - self._max_size + if excess <= 0: + return 0 + oldest = sorted( + self._timestamps.items(), + key=lambda item: float(item[1]) if isinstance(item[1], (int, float)) else 0.0 + ) + removed = 0 + for key, _ in oldest[:excess]: + self._cache.pop(key, None) + self._timestamps.pop(key, None) + removed += 1 + return removed def clear(self, key: Optional[str] = None) -> None: """ @@ -133,22 +216,8 @@ class MemoryCache: self._timestamps.pop(key, None) removed_count += 1 - # Enforce size limit by removing oldest entries if cache is too large - if len(self._cache) > self._max_size: - # Sort by timestamp (oldest first) - sorted_entries = sorted( - self._timestamps.items(), - key=lambda x: float(x[1]) if isinstance(x[1], (int, float)) else 0 - ) - - # Remove oldest entries until we're under the limit - excess_count = len(self._cache) - self._max_size - for i in range(excess_count): - if i < len(sorted_entries): - key = sorted_entries[i][0] - self._cache.pop(key, None) - self._timestamps.pop(key, None) - removed_count += 1 + # Same ceiling enforcement set() uses, so the two cannot drift. + removed_count += self._evict_over_limit_locked() self._last_cleanup = current_time diff --git a/src/cache_manager.py b/src/cache_manager.py index 5065dcdb..67624de1 100644 --- a/src/cache_manager.py +++ b/src/cache_manager.py @@ -33,24 +33,34 @@ import logging import threading import tempfile from src.exceptions import CacheError -from src.cache.memory_cache import MemoryCache +from src.cache.memory_cache import MemoryCache, default_max_size from src.cache.disk_cache import DiskCache from src.cache.cache_strategy import CacheStrategy from src.cache.cache_metrics import CacheMetrics from src.logging_config import get_logger -class DateTimeEncoder(json.JSONEncoder): - """JSON encoder that serialises ``datetime`` objects as ISO-8601 strings.""" - - def default(self, obj): - """Return ISO-8601 string for datetime; delegate all other types to the base encoder.""" - if isinstance(obj, datetime): - return obj.isoformat() - return super().default(obj) +# Canonical implementation lives in src.cache.disk_cache; re-exported here +# because this module's docstring documents it and external code may import +# it from either path. +from src.cache.disk_cache import DateTimeEncoder # noqa: F401 - deliberate re-export class CacheManager: """Manages caching of API responses to reduce API calls.""" - + + # Which cache directories already have a cleanup thread in this process. + # + # The sweep is directory-scoped work -- it lists a directory and deletes + # from it -- so one per directory is the right number no matter how many + # managers exist. Nothing enforced that before: every instance started its + # own, and because the loop closes over `self`, a discarded manager could + # never be collected and its thread woke to re-scan the same directory + # every 24 hours for the life of the process. Startup validation runs + # twice and built a throwaway manager each time, so a display process + # carried three threads for one cache. + _cleanup_owners: Dict[str, 'CacheManager'] = {} + _cleanup_owners_lock = threading.Lock() + + def __init__(self) -> None: # Initialize logger first self.logger: logging.Logger = get_logger(__name__) @@ -74,7 +84,9 @@ class CacheManager: self.logger.warning("ConfigManager not available, using default cache intervals") # Initialize cache components using composition - self._memory_cache_component = MemoryCache(max_size=1000, cleanup_interval=300.0) + self._memory_cache_component = MemoryCache( + max_size=default_max_size(), cleanup_interval=300.0 + ) self._disk_cache_component = DiskCache(cache_dir=self.cache_dir, logger=self.logger) self._strategy_component = CacheStrategy(config_manager=self.config_manager, logger=self.logger) self._metrics_component = CacheMetrics(logger=self.logger) @@ -598,8 +610,10 @@ class CacheManager: Args: key: Cache key data: Data to cache - ttl: Optional time-to-live in seconds (stored for compatibility but - expiration is still controlled via max_age when reading) + ttl: Time-to-live in seconds for this entry. Takes precedence over + the max_age a reader would otherwise apply, which is inferred + from the key and is only a fallback for entries that did not + say. Omit it to keep that inferred behaviour. """ cache_data = { 'data': data, @@ -720,11 +734,29 @@ class CacheManager: } def start_cleanup_thread(self) -> None: - """Start background thread for periodic disk cache cleanup.""" + """Start background thread for periodic disk cache cleanup. + + At most one thread per cache directory per process: the sweep is + directory-scoped, so a second one only duplicates the scan. + """ if self._cleanup_thread and self._cleanup_thread.is_alive(): self.logger.debug("Cleanup thread already running") return - + + with CacheManager._cleanup_owners_lock: + owner = CacheManager._cleanup_owners.get(self.cache_dir) + if owner is not None and owner is not self: + thread = owner._cleanup_thread + if thread is not None and thread.is_alive(): + self.logger.debug( + "Cleanup thread for %s already owned by another cache " + "manager in this process; not starting a second", + self.cache_dir) + return + # The owner's thread died or was stopped -- take over. + CacheManager._cleanup_owners[self.cache_dir] = self + + def cleanup_loop(): """Background loop that runs cleanup periodically.""" self.logger.info("Disk cache cleanup thread started (interval: %d hours)", @@ -772,10 +804,17 @@ class CacheManager: Signals the thread to stop and waits for it to finish (with timeout). This allows for clean shutdown during testing or application termination. """ + # Release ownership first and unconditionally, so a manager that never + # started a thread (or whose thread already exited) cannot keep the + # directory claimed and block a live manager from sweeping it. + with CacheManager._cleanup_owners_lock: + if CacheManager._cleanup_owners.get(self.cache_dir) is self: + del CacheManager._cleanup_owners[self.cache_dir] + if not self._cleanup_thread or not self._cleanup_thread.is_alive(): self.logger.debug("Cleanup thread not running") return - + self.logger.info("Stopping disk cache cleanup thread...") self._cleanup_stop_event.set() # Signal thread to stop diff --git a/src/common/README.md b/src/common/README.md index cccaa40b..4246ccff 100644 --- a/src/common/README.md +++ b/src/common/README.md @@ -99,11 +99,6 @@ Helpers for ensuring directory permissions and ownership are correct when running as a service (used by `CacheManager` to set up its persistent cache directory). -## CLI Helpers (`cli.py`) - -Shared CLI argument parsing helpers used by `scripts/dev/*` and other -command-line entry points. - ## Best Practices 1. **Use centralized logging**: Import from `src.logging_config` instead of creating loggers directly diff --git a/src/common/api_helper.py b/src/common/api_helper.py index 9d9b076f..d6974b22 100644 --- a/src/common/api_helper.py +++ b/src/common/api_helper.py @@ -56,7 +56,9 @@ class APIHelper: # Default headers self.session.headers.update({ - 'User-Agent': 'LEDMatrix-Common/1.0', + # Identifies the client and links to it: ESPN began 403ing bare + # custom tokens (and browser strings) around 2026-08-04. + 'User-Agent': 'LEDMatrix/1.0 (+https://github.com/ChuckBuilds/LEDMatrix)', 'Accept': 'application/json', 'Accept-Language': 'en-US,en;q=0.9', 'Accept-Encoding': 'gzip, deflate, br', @@ -270,20 +272,35 @@ class APIHelper: def clear_cache(self, pattern: Optional[str] = None) -> None: """ Clear cache data. - + + Uses CacheManager's real surface (clear_cache / delete / + list_cache_files); safely no-ops on managers without it. The old + implementation guarded on a nonexistent ``clear`` method, so it + silently never cleared anything. + Args: - pattern: Optional pattern to match cache keys + pattern: Optional substring to match cache keys; only matching + entries are deleted. """ - if self.cache_manager: - if hasattr(self.cache_manager, 'clear'): - if pattern: - # Clear only keys matching pattern - keys = self.cache_manager.keys() - for key in keys: - if pattern in key: - self.cache_manager.delete(key) - else: - self.cache_manager.clear() + if not self.cache_manager: + return + if pattern: + if (hasattr(self.cache_manager, 'list_cache_files') + and hasattr(self.cache_manager, 'delete')): + for entry in self.cache_manager.list_cache_files(): + key = entry.get('key') if isinstance(entry, dict) else None + if key and pattern in key: + self.cache_manager.delete(key) + else: + self.logger.debug( + "Cache manager lacks list_cache_files/delete; " + "cannot clear by pattern") + elif hasattr(self.cache_manager, 'clear_cache'): + self.cache_manager.clear_cache() + elif hasattr(self.cache_manager, 'clear'): + self.cache_manager.clear() + else: + self.logger.debug("Cache manager exposes no clear method; no-op") def _get_from_cache(self, key: str) -> Optional[Any]: """Get data from cache.""" diff --git a/src/common/basketball_plugin_example.py b/src/common/basketball_plugin_example.py deleted file mode 100644 index 9921a2fe..00000000 --- a/src/common/basketball_plugin_example.py +++ /dev/null @@ -1,328 +0,0 @@ -""" -Example: Basketball Plugin using LEDMatrix Common Helpers - -This example shows how to refactor the basketball plugin to use the -ledmatrix-common package for cleaner, more maintainable code. -""" - -from pathlib import Path -from typing import Any, Dict, List, Optional - - -# Import common helpers -from src.common import ( - LogoHelper, TextHelper, APIHelper, DisplayHelper, - GameHelper, ConfigHelper -) -from src.plugin_system.base_plugin import BasePlugin - - -class BasketballPluginManager(BasePlugin): - """ - Basketball scoreboard plugin using LEDMatrix Common helpers. - - This version is much cleaner and more maintainable than the original - because it delegates common functionality to the shared helpers. - """ - - def __init__( - self, - plugin_id: str, - config: Dict[str, Any], - display_manager, - cache_manager, - plugin_manager - ): - """Initialize the basketball plugin with common helpers.""" - super().__init__(plugin_id, config, display_manager, cache_manager, plugin_manager) - - # Get display dimensions - self.display_width = display_manager.matrix.width - self.display_height = display_manager.matrix.height - - # Initialize common helpers - self._init_helpers() - - # Load configuration - self._load_config() - - # State tracking - self.current_games = [] - self.current_game = None - - # Log initialization - enabled_leagues = [k for k, v in self.league_configs.items() if v['enabled']] - self.logger.info(f"Basketball plugin initialized with leagues: {enabled_leagues}") - - def _init_helpers(self): - """Initialize all common helpers.""" - # Logo helper for team logos - self.logo_helper = LogoHelper( - display_width=self.display_width, - display_height=self.display_height, - logger=self.logger - ) - - # Text helper for rendering - self.text_helper = TextHelper(logger=self.logger) - self.fonts = self.text_helper.load_fonts() - - # API helper for ESPN data - self.api_helper = APIHelper( - cache_manager=self.cache_manager, - logger=self.logger - ) - - # Display helper for layouts - self.display_helper = DisplayHelper( - display_width=self.display_width, - display_height=self.display_height, - logger=self.logger - ) - - # Game helper for data processing - self.game_helper = GameHelper( - timezone_str=self.config.get('timezone', 'UTC'), - logger=self.logger - ) - - # Config helper for configuration management - self.config_helper = ConfigHelper(logger=self.logger) - - def _load_config(self): - """Load and validate configuration.""" - # Get basketball-specific config - basketball_config = self.config_helper.get_sports_config(self.config, 'basketball') - - # Build league configurations - self.league_configs = { - 'nba': { - 'enabled': basketball_config.get('nba_enabled', True), - 'url': 'https://site.api.espn.com/apis/site/v2/sports/basketball/nba/scoreboard', - 'logo_dir': Path('assets/sports/nba_logos'), - 'favorite_teams': basketball_config.get('nba_favorite_teams', []), - 'display_modes': { - 'nba_live': basketball_config.get('nba_display_modes_live', True), - 'nba_recent': basketball_config.get('nba_display_modes_recent', True), - 'nba_upcoming': basketball_config.get('nba_display_modes_upcoming', True), - }, - }, - 'wnba': { - 'enabled': basketball_config.get('wnba_enabled', False), - 'url': 'https://site.api.espn.com/apis/site/v2/sports/basketball/wnba/scoreboard', - 'logo_dir': Path('assets/sports/wnba_logos'), - 'favorite_teams': basketball_config.get('wnba_favorite_teams', []), - 'display_modes': { - 'wnba_live': basketball_config.get('wnba_display_modes_live', True), - 'wnba_recent': basketball_config.get('wnba_display_modes_recent', True), - 'wnba_upcoming': basketball_config.get('wnba_display_modes_upcoming', True), - }, - }, - 'ncaam': { - 'enabled': basketball_config.get('ncaam_basketball_enabled', False), - 'url': 'https://site.api.espn.com/apis/site/v2/sports/basketball/mens-college-basketball/scoreboard', - 'logo_dir': Path('assets/sports/ncaa_logos'), - 'favorite_teams': basketball_config.get('ncaam_basketball_favorite_teams', []), - 'display_modes': { - 'ncaam_basketball_live': basketball_config.get('ncaam_basketball_display_modes_live', True), - 'ncaam_basketball_recent': basketball_config.get('ncaam_basketball_display_modes_recent', True), - 'ncaam_basketball_upcoming': basketball_config.get('ncaam_basketball_display_modes_upcoming', True), - }, - }, - 'ncaaw': { - 'enabled': basketball_config.get('ncaaw_basketball_enabled', False), - 'url': 'https://site.api.espn.com/apis/site/v2/sports/basketball/womens-college-basketball/scoreboard', - 'logo_dir': Path('assets/sports/ncaa_logos'), - 'favorite_teams': basketball_config.get('ncaaw_basketball_favorite_teams', []), - 'display_modes': { - 'ncaaw_basketball_live': basketball_config.get('ncaaw_basketball_display_modes_live', True), - 'ncaaw_basketball_recent': basketball_config.get('ncaaw_basketball_display_modes_recent', True), - 'ncaaw_basketball_upcoming': basketball_config.get('ncaaw_basketball_display_modes_upcoming', True), - }, - }, - } - - def update(self) -> None: - """Update game data for all enabled leagues.""" - try: - all_games = [] - - for league_key, league_config in self.league_configs.items(): - if not league_config['enabled']: - continue - - games = self._fetch_league_games(league_key, league_config) - for game in games: - game['league_key'] = league_key - game['league_config'] = league_config - all_games.extend(games) - - self.current_games = all_games - self.logger.debug(f"Updated basketball data: {len(all_games)} total games") - - except Exception as e: - self.logger.error(f"Error updating basketball data: {e}", exc_info=True) - - def _fetch_league_games(self, league_key: str, league_config: Dict) -> List[Dict]: - """Fetch games for a specific league using API helper.""" - try: - # Use API helper to fetch ESPN data with caching - data = self.api_helper.fetch_espn_scoreboard( - sport='basketball', - league=league_key, - cache_key=f"basketball_{league_key}", - cache_ttl=300 # 5 minutes cache - ) - - if not data: - return [] - - # Use game helper to process events - events = data.get('events', []) - games = self.game_helper.process_games(events, sport='basketball') - - # Add logo paths to games - for game in games: - logo_dir = league_config['logo_dir'] - game['home_logo_path'] = logo_dir / f"{game['home_abbr']}.png" - game['away_logo_path'] = logo_dir / f"{game['away_abbr']}.png" - - return games - - except Exception as e: - self.logger.error(f"Error fetching {league_key} games: {e}", exc_info=True) - return [] - - def display(self, force_clear: bool = False, display_mode: str = None) -> None: - """Display basketball games using display helper.""" - try: - mode = display_mode or self._determine_display_mode() - - if not mode: - self._display_no_games() - return - - # Filter games for mode - filtered_games = self._filter_games_for_mode(mode) - - if not filtered_games: - self._display_no_games() - return - - # Display first game - self.current_game = filtered_games[0] - self._draw_scorebug_layout(self.current_game, force_clear) - - except Exception as e: - self.logger.error(f"Error displaying game: {e}", exc_info=True) - - def _determine_display_mode(self) -> Optional[str]: - """Determine display mode based on available games.""" - # Priority: live > recent > upcoming - for game in self.current_games: - if game.get('is_live'): - return f"{game['league_key']}_live" - for game in self.current_games: - if game.get('is_final'): - return f"{game['league_key']}_recent" - for game in self.current_games: - if game.get('is_upcoming'): - return f"{game['league_key']}_upcoming" - return None - - def _filter_games_for_mode(self, mode: str) -> List[Dict]: - """Filter games based on display mode.""" - filtered = [] - - for game in self.current_games: - league_config = game.get('league_config', {}) - display_modes = league_config.get('display_modes', {}) - - if mode in display_modes and display_modes[mode]: - if 'live' in mode and game.get('is_live'): - filtered.append(game) - elif 'recent' in mode and game.get('is_final'): - filtered.append(game) - elif 'upcoming' in mode and game.get('is_upcoming'): - filtered.append(game) - - return filtered[:5] - - def _draw_scorebug_layout(self, game: Dict, force_clear: bool = False) -> None: - """Draw the basketball scorebug layout using display helper.""" - try: - # Load logos using logo helper - home_logo = self.logo_helper.load_logo( - game['home_abbr'], - game['home_logo_path'] - ) - away_logo = self.logo_helper.load_logo( - game['away_abbr'], - game['away_logo_path'] - ) - - if not home_logo or not away_logo: - self.logger.error("Failed to load logos") - self._display_error("Logo Error") - return - - # Use display helper to create scorebug layout - final_img = self.display_helper.draw_scorebug_layout( - game_data=game, - fonts=self.fonts, - home_logo=home_logo, - away_logo=away_logo - ) - - # Display the image - self.display_manager.image.paste(final_img, (0, 0)) - self.display_manager.update_display() - - except Exception as e: - self.logger.error(f"Error drawing scorebug: {e}", exc_info=True) - - def _display_no_games(self) -> None: - """Display 'no games' message using display helper.""" - try: - img = self.display_helper.draw_no_data_message("No Games") - self.display_manager.image = img.copy() - self.display_manager.update_display() - except Exception as e: - self.logger.error(f"Error displaying no games: {e}", exc_info=True) - - def _display_error(self, message: str) -> None: - """Display error message using display helper.""" - try: - img = self.display_helper.draw_error_message(message) - self.display_manager.image = img.copy() - self.display_manager.update_display() - except Exception as e: - self.logger.error(f"Error displaying error message: {e}", exc_info=True) - - def get_display_duration(self) -> float: - """Get display duration.""" - return self.config.get('display_duration', 15) - - def cleanup(self) -> None: - """Cleanup resources.""" - self.current_games = [] - self.logger.info("Basketball plugin cleaned up") - - -# Example usage and benefits: -""" -Benefits of using LEDMatrix Common helpers: - -1. **Cleaner Code**: The plugin is much shorter and more readable -2. **Reusable Components**: Common functionality is shared across plugins -3. **Better Testing**: Each helper can be tested independently -4. **Easier Maintenance**: Bug fixes in helpers benefit all plugins -5. **Consistent Behavior**: All plugins use the same underlying logic -6. **Reduced Dependencies**: Plugins don't need to import LEDMatrix core -7. **Better Error Handling**: Centralized error handling in helpers -8. **Configuration Management**: Consistent config handling across plugins - -The original basketball plugin was 326 lines. This version is much cleaner -and delegates most functionality to the common helpers, making it easier to -maintain and extend. -""" diff --git a/src/common/cli.py b/src/common/cli.py deleted file mode 100644 index ec33caaa..00000000 --- a/src/common/cli.py +++ /dev/null @@ -1,111 +0,0 @@ -""" -LEDMatrix Common CLI - -Command-line interface for LEDMatrix Common utilities. -""" - -import argparse -import sys -from pathlib import Path - - -def main(): - """Main CLI entry point.""" - parser = argparse.ArgumentParser( - description="LEDMatrix Common Utilities", - prog="ledmatrix-common" - ) - - subparsers = parser.add_subparsers(dest='command', help='Available commands') - - # Test command - test_parser = subparsers.add_parser('test', help='Test common utilities') - test_parser.add_argument('--display-width', type=int, default=128, help='Display width') - test_parser.add_argument('--display-height', type=int, default=64, help='Display height') - - # Validate command - validate_parser = subparsers.add_parser('validate', help='Validate configuration') - validate_parser.add_argument('config_file', help='Configuration file to validate') - - args = parser.parse_args() - - if args.command == 'test': - test_utilities(args.display_width, args.display_height) - elif args.command == 'validate': - validate_config(args.config_file) - else: - parser.print_help() - - -def test_utilities(display_width: int, display_height: int): - """Test common utilities.""" - print(f"Testing LEDMatrix Common utilities with {display_width}x{display_height} display") - - try: - from ledmatrix_common import LogoHelper, TextHelper, DisplayHelper, GameHelper, ConfigHelper - - # Test LogoHelper - print("Testing LogoHelper...") - logo_helper = LogoHelper(display_width, display_height) - print(f"Logo cache stats: {logo_helper.get_cache_stats()}") - - # Test TextHelper - print("Testing TextHelper...") - text_helper = TextHelper() - fonts = text_helper.load_fonts() - print(f"Loaded {len(fonts)} fonts") - - # Test DisplayHelper - print("Testing DisplayHelper...") - display_helper = DisplayHelper(display_width, display_height) - img = display_helper.create_base_image() - print(f"Created {img.size} base image") - - # Test GameHelper - print("Testing GameHelper...") - GameHelper() - print("GameHelper initialized") - - # Test ConfigHelper - print("Testing ConfigHelper...") - ConfigHelper() - print("ConfigHelper initialized") - - print("All tests passed!") - - except ImportError as e: - print(f"Import error: {e}") - sys.exit(1) - except Exception as e: - print(f"Test error: {e}") - sys.exit(1) - - -def validate_config(config_file: str): - """Validate configuration file.""" - config_path = Path(config_file) - - if not config_path.exists(): - print(f"Configuration file not found: {config_file}") - sys.exit(1) - - try: - from ledmatrix_common import ConfigHelper - - config_helper = ConfigHelper() - config = config_helper.load_config(config_path) - - if config: - print(f"Configuration loaded successfully from {config_file}") - print(f"Found {len(config)} top-level keys") - else: - print(f"Failed to load configuration from {config_file}") - sys.exit(1) - - except Exception as e: - print(f"Validation error: {e}") - sys.exit(1) - - -if __name__ == '__main__': - main() diff --git a/src/common/config_helper.py b/src/common/config_helper.py index 79840188..b3e5aeac 100644 --- a/src/common/config_helper.py +++ b/src/common/config_helper.py @@ -5,6 +5,7 @@ Handles configuration management and validation for LED matrix plugins. Extracted from LEDMatrix core to provide reusable functionality for plugins. """ +import copy import json import logging from pathlib import Path @@ -160,17 +161,20 @@ class ConfigHelper: override_config: Configuration to merge in (takes precedence) Returns: - Merged configuration dictionary + Merged configuration dictionary (fully independent of both + inputs — a shallow copy would alias un-overridden nested dicts, + so mutating the result would mutate the caller's base config). """ - merged = base_config.copy() - + merged = copy.deepcopy(base_config) + for key, value in override_config.items(): if key in merged and isinstance(merged[key], dict) and isinstance(value, dict): # Recursively merge nested dictionaries merged[key] = self.merge_configs(merged[key], value) else: - # Override with new value - merged[key] = value + # Override with new value — deep-copied so mutating the + # merged result can't reach back into override_config. + merged[key] = copy.deepcopy(value) return merged diff --git a/src/common/display_helper.py b/src/common/display_helper.py index 13dc3030..c88c9e63 100644 --- a/src/common/display_helper.py +++ b/src/common/display_helper.py @@ -115,17 +115,13 @@ class DisplayHelper: if home_logo and away_logo: self._draw_logos(main_img, home_logo, away_logo) - # Draw status/period text (top center) - if status_text or period_text: - status_display = f"{period_text} {status_text}".strip() - if status_display: - self._draw_centered_text(draw, status_display, - fonts.get('time', fonts.get('status')), - y_position=1) - - # Draw clock if available - if clock: - self._draw_centered_text(draw, clock, fonts.get('time'), y_position=1) + # Draw one combined top line (period/status/clock all share y=1 — + # drawing them separately overprinted each other). + top_line = " ".join(p for p in [period_text, status_text, clock] if p) + if top_line: + self._draw_centered_text(draw, top_line, + fonts.get('time', fonts.get('status')), + y_position=1) # Draw scores (center) score_text = f"{away_score}-{home_score}" @@ -153,26 +149,28 @@ class DisplayHelper: """ Draw a ticker/scrolling text layout. + Renders a single static frame with the text at the left edge; the + caller advances the scroll by re-rendering or shifting. The + scroll_speed parameter is accepted for API compatibility but does + not affect this frame. (Previously the text was drawn starting at + x=display_width — entirely off-canvas — so every frame was blank.) + Args: text: Text to display font: Font to use background_color: Background color text_color: Text color - scroll_speed: Pixels to scroll per frame - + scroll_speed: Accepted for compatibility; unused per-frame + Returns: PIL Image with ticker layout """ img = self.create_base_image(background_color) draw = ImageDraw.Draw(img) - - # Start text off-screen to the right - x_position = self.display_width - - # Draw text - self._draw_text_with_outline(draw, text, (x_position, self.display_height // 2 - 6), + + self._draw_text_with_outline(draw, text, (0, self.display_height // 2 - 6), font, fill=text_color) - + return img def draw_centered_text(self, text: str, font: ImageFont.ImageFont, @@ -214,15 +212,9 @@ class DisplayHelper: Returns: PIL Image with error message """ - img = self.create_base_image((50, 0, 0)) # Dark red background - - # Use default font + # Dark red background, white text font = ImageFont.load_default() - - # Draw centered error message - self._draw_centered_text(message, font, (50, 0, 0), (255, 255, 255)) - - return img + return self.draw_centered_text(message, font, (50, 0, 0), (255, 255, 255)) def draw_no_data_message(self, message: str = "No Data") -> Image.Image: """ @@ -234,11 +226,8 @@ class DisplayHelper: Returns: PIL Image with no data message """ - img = self.create_base_image((0, 0, 0)) font = ImageFont.load_default() - self._draw_centered_text(message, font, (0, 0, 0), (150, 150, 150)) - - return img + return self.draw_centered_text(message, font, (0, 0, 0), (150, 150, 150)) def get_display_dimensions(self) -> Tuple[int, int]: """ diff --git a/src/common/logo_helper.py b/src/common/logo_helper.py index 7d0dc4df..743e7b7b 100644 --- a/src/common/logo_helper.py +++ b/src/common/logo_helper.py @@ -6,6 +6,8 @@ Extracted from LEDMatrix core to provide reusable functionality for plugins. """ import logging +import os +import tempfile from pathlib import Path from typing import Dict, List, Optional, Union @@ -19,6 +21,10 @@ from src.common.permission_utils import ( ) +# Well above any real team logo; bounds what a remote URL can write to disk. +MAX_LOGO_BYTES = 10 * 1024 * 1024 + + class LogoHelper: """ Helper class for logo loading, caching, and resizing. @@ -187,10 +193,17 @@ class LogoHelper: def normalize_abbreviation(self, team_abbr: str) -> str: """ Normalize team abbreviation for consistent filename usage. - + + NOTE: this deliberately differs from + LogoDownloader.normalize_abbreviation (src/logo_downloader.py), + which replaces filesystem-unsafe characters (/ \\ : * ? " < > |) + but does not strip spaces. Plugins call the LogoDownloader + version; changing either implementation changes which logo + filenames resolve on existing installs. + Args: team_abbr: Raw team abbreviation - + Returns: Normalized abbreviation """ @@ -219,7 +232,10 @@ class LogoHelper: return { 'cached_logos': len(self._logo_cache), 'cache_size_limit': self.cache_size, - 'cache_usage_percent': (len(self._logo_cache) / self.cache_size) * 100 + 'cache_usage_percent': ( + (len(self._logo_cache) / self.cache_size) * 100 + if self.cache_size else 0 + ), } def _resize_logo(self, logo: Image.Image, max_width: Optional[int] = None, @@ -251,21 +267,64 @@ class LogoHelper: self._cache_order.append(cache_key) def _download_logo(self, url: str, file_path: Path) -> None: - """Download logo from URL.""" + """Download logo from URL. + + The response size is capped and the saved file is verified as a + decodable image before it is left on disk: a logo URL is remote + input, and without this an oversized or malformed response would + be cached for every later load_logo() call to trip over. + + The body is streamed and counted as it arrives rather than read + through response.content, which buffers the whole thing first — + a server that omits Content-Length and never stops sending would + exhaust memory before any size check could run. Nothing lands at + file_path until the download completes and decodes, so a failed + download cannot leave a truncated logo behind either. + """ # Ensure directory exists with proper permissions ensure_directory_permissions(file_path.parent, get_assets_dir_mode()) - - # Download with timeout - response = self.session.get(url, timeout=30) - response.raise_for_status() - - # Save to file - with open(file_path, 'wb') as f: - f.write(response.content) - + + # A unique temp name, not a fixed ".part": two plugins can + # ask for the same logo at once, and a shared name would let them + # interleave writes into one file, publish the mixture, or delete + # each other's partial. Same directory, so os.replace stays atomic. + fd, tmp_name = tempfile.mkstemp( + dir=str(file_path.parent), prefix=file_path.name + '.', suffix='.part') + tmp_path = Path(tmp_name) + try: + # fdopen outermost so the descriptor mkstemp handed back is + # always adopted and closed, including when the request itself + # raises — load_logo_with_download swallows that, so a leak + # here would accumulate quietly on a URL that keeps failing. + with os.fdopen(fd, 'wb') as f: + with self.session.get(url, timeout=30, stream=True) as response: + response.raise_for_status() + downloaded = 0 + for chunk in response.iter_content(chunk_size=64 * 1024): + if not chunk: + continue + downloaded += len(chunk) + if downloaded > MAX_LOGO_BYTES: + raise ValueError( + f"Logo at {url} exceeds the " + f"{MAX_LOGO_BYTES}-byte limit; not saved") + f.write(chunk) + + # Verify it decodes before it becomes the cached logo. PIL + # raises DecompressionBombError past its own pixel limit; a + # partial or non-image response raises UnidentifiedImageError + # (an OSError subclass). + with Image.open(tmp_path) as probe: + probe.load() + + os.replace(tmp_path, file_path) + except BaseException: + tmp_path.unlink(missing_ok=True) + raise + # Set proper file permissions after saving ensure_file_permissions(file_path, get_assets_file_mode()) - + self.logger.debug(f"Downloaded logo to {file_path}") def _create_placeholder_logo(self, team_abbr: str, diff --git a/src/common/permission_utils.py b/src/common/permission_utils.py index 3679c253..40ed98a9 100644 --- a/src/common/permission_utils.py +++ b/src/common/permission_utils.py @@ -146,6 +146,60 @@ def ensure_file_permissions(path: Path, mode: int = 0o644) -> None: raise +_shared_group_gid_cache: Optional[int] = None + + +def get_shared_group_gid() -> Optional[int]: + """ + Return the gid that should own config/secrets files shared between the + root-run ``ledmatrix.service`` (main display) and the non-root user that + ``ledmatrix-web.service`` runs as (see install_web_service.sh, which sets + ``User=$SUDO_USER``). + + Resolved once from the project root directory's current group (normally + the login user's group from the initial ``git clone``), since that user + is stable across reinstalls unlike any single file's ownership. + + Returns: + The gid, or None if it cannot be determined. + """ + global _shared_group_gid_cache + if _shared_group_gid_cache is not None: + return _shared_group_gid_cache + try: + project_root = Path(__file__).resolve().parent.parent.parent + _shared_group_gid_cache = project_root.stat().st_gid + return _shared_group_gid_cache + except OSError: + return None + + +def ensure_shared_group_ownership(path: Path) -> None: + """ + Best-effort chgrp of ``path`` to the shared group (see + :func:`get_shared_group_gid`) when running as root. + + Only root can change a file's group to one the calling process isn't a + member of, which is exactly the case that causes the web interface + (running as a non-root user) to get ``PermissionError`` reading files + the root-run display service just wrote with a 0o640/2775 mode: the mode + is group-readable, but without this the group is root's, not the web + user's. Silently does nothing if not running as root or on any error — + this is a hardening step, not a required one. + """ + if os.geteuid() != 0: + return + gid = get_shared_group_gid() + if gid is None: + return + try: + if path.exists() and path.stat().st_gid != gid: + os.chown(path, -1, gid) + logger.debug(f"Set shared group ownership (gid {gid}) on {path}") + except OSError as e: + logger.debug(f"Could not set shared group ownership on {path}: {e}") + + def get_config_file_mode(file_path: Path) -> int: """ Return appropriate permission mode for config files. diff --git a/src/common/scroll_helper.py b/src/common/scroll_helper.py index fd4c5552..88c6d498 100644 --- a/src/common/scroll_helper.py +++ b/src/common/scroll_helper.py @@ -110,20 +110,30 @@ class ScrollHelper: self.is_scrolling = False self.scroll_complete = False - def create_scrolling_image(self, content_items: list, + def create_scrolling_image(self, content_items: list, item_gap: int = 32, - element_gap: int = 16) -> Image.Image: + element_gap: int = 16, + lead_gap: Optional[int] = None) -> Image.Image: """ Create a wide image containing all content items for scrolling. - + Args: content_items: List of PIL Images to include in scroll item_gap: Gap between different items element_gap: Gap between elements within an item - + lead_gap: Blank columns before the first item. Defaults to a full + display width, which makes a standalone ticker scroll in from + off-screen. Callers that loop many plugins back-to-back (Vegas + mode) pass a smaller value, since a full display width of black + reads as the panel being switched off at the start of every + cycle. + Returns: PIL Image containing all content arranged horizontally """ + if lead_gap is None: + lead_gap = self.display_width + lead_gap = max(0, int(lead_gap)) if not content_items: # Create empty image if no content # Still set total_scroll_width to 0 to indicate no scrollable content @@ -144,13 +154,13 @@ class ScrollHelper: total_width += element_gap * len(content_items) # Add initial gap before first item - total_width += self.display_width - + total_width += lead_gap + # Create the full scrolling image full_image = Image.new('RGB', (total_width, self.display_height), (0, 0, 0)) - + # Position items - current_x = self.display_width # Start with initial gap + current_x = lead_gap # Start with initial gap for i, img in enumerate(content_items): # Paste the item image @@ -318,7 +328,7 @@ class ScrollHelper: elapsed_time = current_time - (self.scroll_start_time or current_time) # The image already includes display_width padding, so we only need total_scroll_width required_total_distance = self.total_scroll_width - self.logger.info( + self.logger.debug( "Scroll progress: elapsed=%.2fs, target=%.2fs, total_scrolled=%.0f/%d px (%.1f%%)", elapsed_time, self.calculated_duration, @@ -338,13 +348,72 @@ class ScrollHelper: """ if not self.cached_image or self.cached_array is None: return None - - # Use integer pixel positioning for high FPS scrolling (like stock ticker) + start_x_int = int(self.scroll_position) end_x_int = start_x_int + self.display_width - - # Fast integer pixel path (no interpolation - high frame rate provides smoothness) + + # Integer positioning quantises motion to whole pixels, so the number of + # distinct frames per second equals the scroll speed in px/s, no matter + # how fast the loop renders. At 50px/s and 78fps that made 36% of frames + # identical: the extra frames cost work and bought nothing. Blending + # between the two neighbouring positions gives motion at the frame rate + # instead of the step rate. + if self.sub_pixel_scrolling: + fractional = self.scroll_position - start_x_int + if fractional > 0.0: + return self._blend_visible_portion(start_x_int, fractional) + return self._get_visible_portion_integer(start_x_int, end_x_int) + + def _blend_visible_portion(self, start_x: int, fractional: float) -> Image.Image: + """ + Linear blend between the frames at ``start_x`` and ``start_x + 1``. + + Implemented with numpy rather than scipy.ndimage.shift: scipy is not + installed on the target devices (HAS_SCIPY is False there), which is why + the pre-existing sub-pixel path was dead code — get_visible_portion never + consulted the flag, and the scipy fallback would not have interpolated + anyway. + + Args: + start_x: Left column of the earlier of the two frames + fractional: How far between the two, in [0, 1) + + Returns: + The blended frame + """ + width = self.display_width + strip_width = self.cached_array.shape[1] + + if start_x + width + 1 <= strip_width: + # Slice the backing array directly. Going via + # _get_visible_portion_integer would build two PIL images only for + # them to be converted straight back to arrays, which measured 15x + # the cost of the integer path. + near = self.cached_array[:, start_x:start_x + width] + far = self.cached_array[:, start_x + 1:start_x + 1 + width] + else: + # Close enough to the end that one of the slices wraps; let the + # integer path handle that and pay the conversion. Continuous mode + # extends the strip before reaching here, so this is the rare case. + near = np.asarray( + self._get_visible_portion_integer(start_x, start_x + width)) + far = np.asarray( + self._get_visible_portion_integer(start_x + 1, start_x + 1 + width)) + + # Fixed-point rather than float32: integer multiply-add on uint16 is + # markedly faster than float maths on the Pi's ARM cores, and 8 bits of + # weight is finer than the panel can show. + weight = int(fractional * 256.0) + blended = ( + (near.astype(np.uint16) * (256 - weight) + + far.astype(np.uint16) * weight) >> 8 + ).astype(np.uint8) + + return Image.frombytes( + 'RGB', (width, self.display_height), + np.ascontiguousarray(blended).tobytes() + ) def _get_visible_portion_integer(self, start_x: int, end_x: int) -> Image.Image: """Fast integer pixel extraction (no interpolation). @@ -638,6 +707,128 @@ class ScrollHelper: """ return self.scroll_complete + def append_content(self, content_items: list, + item_gap: int = 32, + element_gap: int = 0) -> bool: + """ + Append items to the right of the existing strip, preserving scroll state. + + Lets a caller keep one continuous strip instead of replacing it. Vegas + mode uses this so the next group of plugins scrolls in from the right + rather than the strip being swapped out underneath the viewer — a swap + shows as a flash and a hard cut to already-full-screen content. + + ``scroll_position`` and ``total_distance_scrolled`` are untouched, so + motion continues uninterrupted; only the strip gets longer. Because + completion is measured against ``total_scroll_width``, extending the + strip also defers completion, which is the intent. + + Args: + content_items: Images to append, in order + item_gap: Gap between appended items, and between the existing + content and the first appended item + element_gap: Extra gap after each item, mirroring + create_scrolling_image + + Returns: + True if content was appended + """ + if not content_items: + return False + + if self.cached_image is None or self.cached_array is None: + # Nothing to extend yet — this is just the first build. + self.create_scrolling_image( + content_items, item_gap=item_gap, element_gap=element_gap, lead_gap=0) + return True + + gap = max(0, item_gap) + addition_width = ( + sum(img.width for img in content_items) + + gap * len(content_items) # one leading gap per item + + element_gap * len(content_items) + ) + + addition = Image.new('RGB', (addition_width, self.display_height), (0, 0, 0)) + x = 0 + for img in content_items: + x += gap # separate from whatever precedes + addition.paste(img, (x, 0)) + x += img.width + element_gap + + # numpy concatenate then one conversion back, rather than allocating a + # full-width PIL image and pasting twice: the strip can be tens of + # thousands of columns wide and this runs on the render path. + self.cached_array = np.concatenate( + (self.cached_array, np.array(addition)), axis=1) + self.cached_image = Image.fromarray(self.cached_array) + self.total_scroll_width = self.cached_image.width + self.scroll_complete = False + + self.logger.info( + "Appended %d item(s) (%dpx) to scroll strip: now %dpx, position %.0f", + len(content_items), addition_width, self.total_scroll_width, + self.scroll_position + ) + return True + + def drop_scrolled_prefix(self, keep_before: int = 0) -> int: + """ + Discard columns that have already scrolled past, to bound memory. + + A continuously extended strip would otherwise grow without limit. All + the positional state is shifted by the amount removed so the visible + frame and the completion arithmetic are unchanged: + ``total_distance_scrolled`` and ``total_scroll_width`` both shrink by the + same amount, preserving their difference. + + Args: + keep_before: Columns to retain behind the current position, as a + safety margin against a caller reading slightly behind it + + Returns: + Number of columns actually removed + """ + if self.cached_image is None or self.cached_array is None: + return 0 + + # While the viewport wraps, get_visible_portion fills its right-hand side + # from the *head* of the strip, so trimming the head would change what + # is on screen. Continuous mode extends before ever reaching that state; + # refusing here keeps "trimming is invisible" true unconditionally. + if self.scroll_position + self.display_width > self.cached_image.width: + return 0 + + cut = int(self.scroll_position) - max(0, keep_before) + if cut <= 0: + return 0 + # Never trim so far that the remaining strip is narrower than the + # viewport, or get_visible_portion has nothing to slice. + cut = min(cut, max(0, self.cached_image.width - self.display_width)) + if cut <= 0: + return 0 + + # .copy() so the original buffer is released rather than kept alive by + # a numpy view. + self.cached_array = self.cached_array[:, cut:].copy() + self.cached_image = Image.fromarray(self.cached_array) + self.total_scroll_width = self.cached_image.width + self.scroll_position -= cut + self.total_distance_scrolled = max(0.0, self.total_distance_scrolled - cut) + + self.logger.debug( + "Dropped %dpx of scrolled strip: now %dpx, position %.0f", + cut, self.total_scroll_width, self.scroll_position + ) + return cut + + def remaining_unscrolled(self) -> int: + """Columns of strip still to the right of the viewport.""" + if self.cached_image is None: + return 0 + return max(0, self.total_scroll_width - int(self.scroll_position) + - self.display_width) + def reset_scroll(self) -> None: """ Reset scroll position to beginning. diff --git a/src/common/sports_scroll.py b/src/common/sports_scroll.py new file mode 100644 index 00000000..f9a32e37 --- /dev/null +++ b/src/common/sports_scroll.py @@ -0,0 +1,485 @@ +"""Shared scroll-display scaffolding for the sports scoreboards. + +Ten plugins ship a `scroll_display.py`. A method-level comparison of the eight +that share a shape (f1 and ufc are genuine forks) found a sharp split, and this +module is drawn along it rather than around all of it: + +* The **orchestration layer is converged** — ``get_all_vegas_content_items`` is + byte-identical in all eight, and ``clear_all``, ``get_scroll_info``, + ``get_dynamic_duration``, ``is_complete`` and ``display_frame`` are 96-100% + similar. That is what lives here. +* The **content layer has genuinely diverged** — ``prepare_scroll_content`` has + eight distinct bodies across eight plugins (145 lines, 53% similarity at + worst) and ``_load_separator_icons`` seven (6% at worst). Those build each + sport's game cards and icon strip; they are *not* drift to be merged but + per-sport rendering. They stay override points here, permanently. + +Promoting the content layer would be exactly the mistake +``docs/SPORTS_UNIFICATION.md`` warns against — merging on the intuition that +same-named methods are the same method. Same name, different job. + +The one behavior this module adds over the plugin copies is native support for +``global_config['target_fps']``: the bundled copies hardcode ~100 FPS via +``scroll_delay=0.01`` and never consult the global smooth-scrolling target. A +plugin inheriting from here gets it for free. + +Usage:: + + class HockeyScrollDisplay(SportsScrollDisplay): + SCROLL_LEAGUE_KEYS = ("nhl", "ncaa_mens", "ncaam_hockey") + + def prepare_scroll_content(self, games, game_type, leagues, rankings=None): + ... # build this sport's cards + + class HockeyScrollDisplayManager(SportsScrollDisplayManager): + display_class = HockeyScrollDisplay +""" + +from __future__ import annotations + +import logging +import time +from typing import Any, Dict, List, Optional + +from PIL import Image + +from src.common.scroll_helper import ScrollHelper + +logger = logging.getLogger(__name__) + +#: Defaults every copy agreed on. A subclass overrides +#: :meth:`SportsScrollDisplay.scroll_settings_defaults` to change them — +#: the soccer lineage uses a 24px gap and min/max duration keys instead. +DEFAULT_SCROLL_SETTINGS: Dict[str, Any] = { + "scroll_speed": 50.0, + "scroll_delay": 0.01, + "gap_between_games": 48, + "show_league_separators": True, + "dynamic_duration": True, +} + +#: Bounds on the px/second -> px/frame conversion, applied before the helper +#: sees the value. FPS is *not* clamped here — ScrollHelper.set_target_fps +#: already does that, and a second copy of the range would drift from it. +MIN_PIXELS_PER_FRAME = 0.1 +MAX_PIXELS_PER_FRAME = 5.0 + +#: Pacing to assume when scroll_delay is 0, i.e. the plugin has not set one. +ASSUMED_FPS_WHEN_UNPACED = 100.0 + + +class SportsScrollDisplay: + """One scrolling strip of game cards. + + Subclasses supply the content (:meth:`prepare_scroll_content`) and, + optionally, the per-sport league ladder and separator icons. Everything + else — helper configuration, frame pumping, completion, state — is here. + """ + + #: Config keys to walk when looking for per-league ``scroll_settings``, + #: most-preferred first. A sport's own league names, which is the *only* + #: reason the eight copies of ``_get_scroll_settings`` differ. Empty means + #: the plugin has no per-league scroll settings. + SCROLL_LEAGUE_KEYS: tuple = () + + #: Config block holding scroll settings when the plugin keeps them in one + #: place rather than per league (the afl/nrl/soccer shape). + SCROLL_CONFIG_KEY: Optional[str] = None + + def __init__( + self, + display_manager, + config: Dict[str, Any], + custom_logger: Optional[logging.Logger] = None, + global_config: Optional[Dict[str, Any]] = None, + ): + """ + :param display_manager: the core display manager + :param config: the plugin's configuration + :param custom_logger: the plugin's logger, so scroll lines are attributed + :param global_config: the LEDMatrix global config — the source of + ``target_fps``. Optional so an older caller that does not pass it + keeps working at the config-derived pacing. + """ + self.display_manager = display_manager + self.config = config + self.logger = custom_logger or logger + self.global_config = global_config or {} + + if getattr(display_manager, "matrix", None) is not None: + self.display_width = display_manager.matrix.width + self.display_height = display_manager.matrix.height + else: + self.display_width = getattr(display_manager, "width", 128) + self.display_height = getattr(display_manager, "height", 32) + + self.scroll_helper = ScrollHelper( + self.display_width, self.display_height, self.logger + ) + self._configure_scroll_helper() + + self._logo_cache: Dict[str, Image.Image] = {} + self._separator_icons: Dict[str, Image.Image] = {} + self._load_separator_icons() + + self._current_games: List[Dict] = [] + self._current_game_type: str = "" + self._current_leagues: List[str] = [] + self._vegas_content_items: List[Image.Image] = [] + self._is_scrolling = False + self._scroll_start_time: Optional[float] = None + self._last_log_time: float = 0 + self._log_interval: float = 5.0 + self._frame_count: int = 0 + self._fps_sample_start: float = time.time() + + # ------------------------------------------------------------------ + # Override points + # ------------------------------------------------------------------ + + def prepare_scroll_content( + self, + games: List[Dict], + game_type: str, + leagues: List[str], + rankings_cache: Optional[Dict[str, int]] = None, + ) -> bool: + """Render ``games`` into one wide image and hand it to the scroll helper. + + **Per-sport by nature, not by drift** — the eight plugin copies have + eight different bodies because each draws its own card. Implementations + build the strip, hand it over with + ``self.scroll_helper.set_scrolling_image(...)`` (or + ``create_scrolling_image(...)`` from a list of cards), and record + ``self._current_games`` / ``_current_game_type`` / ``_current_leagues``. + + :returns: True when there is content to scroll. + """ + raise NotImplementedError( + f"{type(self).__name__} must implement prepare_scroll_content(); " + "it builds this sport's game cards and is not shared code." + ) + + def _load_separator_icons(self) -> None: + """Populate ``self._separator_icons``. Per-sport; no-op by default.""" + + def scroll_settings_defaults(self) -> Dict[str, Any]: + """The baseline scroll settings before any config is applied.""" + defaults = dict(DEFAULT_SCROLL_SETTINGS) + defaults["game_card_width"] = self.display_width + return defaults + + # ------------------------------------------------------------------ + # Settings + # ------------------------------------------------------------------ + + def _get_scroll_settings(self, league: Optional[str] = None) -> Dict[str, Any]: + """Resolve scroll settings: defaults, then the most specific override. + + Precedence: the named ``league``, then each entry of + :attr:`SCROLL_LEAGUE_KEYS` in order, then :attr:`SCROLL_CONFIG_KEY`. + The eight plugin copies implement exactly this and differ only in which + league names they walk — which is why the ladder is data here rather + than a body per sport. + """ + settings = self.scroll_settings_defaults() + + candidates: List[str] = [] + if league: + candidates.append(league) + candidates.extend(self.SCROLL_LEAGUE_KEYS) + for key in candidates: + override = (self.config.get(key) or {}).get("scroll_settings") + if override: + return {**settings, **override} + + if self.SCROLL_CONFIG_KEY: + override = self.config.get(self.SCROLL_CONFIG_KEY) or {} + if override: + return {**settings, **override} + return settings + + def _resolve_target_fps(self) -> Optional[float]: + """The global smooth-scrolling FPS target, or None to keep config pacing. + + Coerced before use: a malformed value in the global config must degrade + to the existing ``scroll_delay`` pacing, never raise on a display path. + """ + raw = self.global_config.get("target_fps") or self.global_config.get( + "scroll_target_fps" + ) + try: + return float(raw) if raw is not None else None + except (TypeError, ValueError): + self.logger.debug("Ignoring unusable target_fps: %r", raw) + return None + + def _coerce_float(self, value: Any, default: float) -> float: + """A usable float from config, or ``default``. + + ``dict.get(key, default)`` only helps when the key is *absent*; a key + present with ``null`` or a string returns that value verbatim and blows + up in the arithmetic below — inside ``__init__``, so the whole display + fails to construct. + """ + if value is None: + return default + try: + return float(value) + except (TypeError, ValueError): + self.logger.warning( + "Ignoring unusable scroll setting %r; using %s", value, default + ) + return default + + def _configure_scroll_helper(self) -> None: + """Apply config to the scroll helper. Safe to call again after a change.""" + settings = self._get_scroll_settings() + + scroll_speed = self._coerce_float(settings.get("scroll_speed"), 50.0) + scroll_delay = self._coerce_float(settings.get("scroll_delay"), 0.01) + dynamic_duration = bool(settings.get("dynamic_duration", True)) + + self.scroll_helper.set_scroll_delay(scroll_delay) + self.scroll_helper.set_dynamic_duration_settings( + enabled=dynamic_duration, + min_duration=settings.get("min_duration", 30), + max_duration=settings.get("max_duration", 600), + buffer=0.2, # ensure the strip clears the panel completely + ) + # Frame-based scrolling: motion advances per rendered frame rather than + # per wall-clock second, which is what makes the pacing stable. + self.scroll_helper.set_frame_based_scrolling(True) + + # Config states speed in px/second; frame-based mode wants px/frame. + if scroll_delay > 0: + pixels_per_frame = scroll_speed * scroll_delay + else: + pixels_per_frame = scroll_speed / ASSUMED_FPS_WHEN_UNPACED + pixels_per_frame = max( + MIN_PIXELS_PER_FRAME, min(MAX_PIXELS_PER_FRAME, pixels_per_frame) + ) + self.scroll_helper.set_scroll_speed(pixels_per_frame) + + effective_pps = ( + pixels_per_frame / scroll_delay + if scroll_delay > 0 + else pixels_per_frame * ASSUMED_FPS_WHEN_UNPACED + ) + self.logger.info( + f"ScrollHelper configured: {pixels_per_frame:.2f} px/frame, " + f"delay={scroll_delay}s (effective {effective_pps:.1f} px/s from " + f"{scroll_speed} px/s config), dynamic_duration={dynamic_duration}" + ) + + # The reason this module exists upstream: the bundled copies hardcode + # ~100 FPS via scroll_delay and never consult the global target. + # No hasattr guard here, unlike the plugin copies: they probe because + # they may run against an older core, whereas this module ships in the + # same release as the ScrollHelper it calls. The helper clamps. + target_fps = self._resolve_target_fps() + if target_fps: + self.scroll_helper.set_target_fps(target_fps) + self.logger.info(f"Target FPS set to {target_fps}") + + # ------------------------------------------------------------------ + # Frame pumping + # ------------------------------------------------------------------ + + def display_scroll_frame(self) -> bool: + """Advance and render one frame. + + :returns: True if a frame was drawn; False when there is no content or + the frame could not be rendered. + """ + if not self.scroll_helper.cached_image: + return False + + try: + # Inside the try, not before it: advancing the position and cropping + # the visible slice are as capable of raising as the display push, + # and the promise below is that no frame failure reaches the + # plugin's loop. + self.scroll_helper.update_scroll_position() + visible = self.scroll_helper.get_visible_portion() + if not visible: + return False + + self.display_manager.image = visible + self.display_manager.update_display() + self._frame_count += 1 + self.scroll_helper.log_frame_rate() + self._log_scroll_progress() + except Exception: + # A display failure must not propagate into the plugin's loop. + self.logger.exception("Error displaying scroll frame") + return False + return True + + def _log_scroll_progress(self) -> None: + """Emit a throttled progress line.""" + now = time.time() + if now - self._last_log_time < self._log_interval: + return + self._last_log_time = now + elapsed = now - self._fps_sample_start + fps = self._frame_count / elapsed if elapsed > 0 else 0.0 + self.logger.debug( + f"Scrolling {len(self._current_games)} {self._current_game_type} " + f"game(s) at {fps:.1f} FPS" + ) + + def is_scroll_complete(self) -> bool: + """True when the strip has scrolled fully past the panel.""" + return self.scroll_helper.is_scroll_complete() + + def reset_scroll(self) -> None: + """Return the strip to its starting position, keeping the content.""" + self.scroll_helper.reset_scroll() + self._frame_count = 0 + self._fps_sample_start = time.time() + self.logger.debug("Scroll position reset") + + def clear(self) -> None: + """Drop cached content and reset tracking state.""" + self.scroll_helper.clear_cache() + self._current_games = [] + self._current_game_type = "" + self._current_leagues = [] + self._vegas_content_items = [] + self._is_scrolling = False + self._scroll_start_time = None + self.logger.debug("Scroll display cleared") + + # ------------------------------------------------------------------ + # Introspection + # ------------------------------------------------------------------ + + def get_dynamic_duration(self) -> int: + """How long this content needs to scroll fully, in seconds.""" + return self.scroll_helper.get_dynamic_duration() + + def has_cached_content(self) -> bool: + """Whether content is prepared and ready to scroll.""" + return bool(self.scroll_helper.cached_image) + + def get_current_game_count(self) -> int: + return len(self._current_games) + + def get_current_leagues(self) -> List[str]: + return list(self._current_leagues) + + def get_scroll_info(self) -> Dict[str, Any]: + """Helper state plus this display's tracking state, for logging/debug.""" + info = self.scroll_helper.get_scroll_info() + info.update( + { + "game_count": len(self._current_games), + "game_type": self._current_game_type, + "leagues": self._current_leagues, + "is_scrolling": self._is_scrolling, + } + ) + return info + + +class SportsScrollDisplayManager: + """One :class:`SportsScrollDisplay` per game type ('live'/'recent'/'upcoming'). + + Subclasses set :attr:`display_class`; everything else was near-identical + across the eight plugin copies. + """ + + #: The SportsScrollDisplay subclass to instantiate per game type. + display_class = SportsScrollDisplay + + def __init__( + self, + display_manager, + config: Dict[str, Any], + custom_logger: Optional[logging.Logger] = None, + global_config: Optional[Dict[str, Any]] = None, + ): + self.display_manager = display_manager + self.config = config + self.logger = custom_logger or logger + self.global_config = global_config or {} + self._scroll_displays: Dict[str, SportsScrollDisplay] = {} + # "" rather than None, matching SportsScrollDisplay's own empty value — + # both are falsy, so `game_type or self._current_game_type` behaved + # either way, but two spellings of "nothing active" across two classes + # is a trap for anyone comparing state between them. + self._current_game_type: str = "" + + def get_scroll_display(self, game_type: str) -> SportsScrollDisplay: + """The display for ``game_type``, created on first use.""" + if game_type not in self._scroll_displays: + self._scroll_displays[game_type] = self.display_class( + self.display_manager, + self.config, + self.logger, + global_config=self.global_config, + ) + return self._scroll_displays[game_type] + + def prepare_and_display( + self, + games: List[Dict], + game_type: str, + leagues: List[str], + rankings_cache: Optional[Dict[str, int]] = None, + ) -> bool: + """Build content for ``game_type`` and make it the active strip.""" + scroll_display = self.get_scroll_display(game_type) + try: + success = scroll_display.prepare_scroll_content( + games, game_type, leagues, rankings_cache + ) + except Exception: + # prepare_scroll_content is subclass-implemented and builds cards + # straight from feed data, which is exactly where this PR's other + # crashes came from. One sport's bad payload must not take down the + # shared orchestration for the others. + self.logger.exception( + "Error preparing scroll content for game_type=%s", game_type + ) + return False + if success: + self._current_game_type = game_type + return success + + def display_frame(self, game_type: Optional[str] = None) -> bool: + """Advance the active strip (or a named one) by one frame.""" + game_type = game_type or self._current_game_type + if not game_type: + return False + scroll_display = self._scroll_displays.get(game_type) + if scroll_display is None: + return False + return scroll_display.display_scroll_frame() + + def is_complete(self, game_type: Optional[str] = None) -> bool: + """True when the strip has finished — including when there isn't one, + so a caller waiting on completion is never wedged.""" + game_type = game_type or self._current_game_type + if not game_type: + return True + scroll_display = self._scroll_displays.get(game_type) + if scroll_display is None: + return True + return scroll_display.is_scroll_complete() + + def clear_all(self) -> None: + """Clear every display and forget which one was active.""" + for scroll_display in self._scroll_displays.values(): + scroll_display.clear() + self._current_game_type = "" + + def get_all_vegas_content_items(self) -> List[Image.Image]: + """Every display's Vegas items, for splicing into the marquee.""" + items: List[Image.Image] = [] + for scroll_display in self._scroll_displays.values(): + vegas_items = getattr(scroll_display, "_vegas_content_items", None) + if vegas_items: + items.extend(vegas_items) + return items diff --git a/src/common/sync_manager.py b/src/common/sync_manager.py index d51bbb27..12fddc8f 100644 --- a/src/common/sync_manager.py +++ b/src/common/sync_manager.py @@ -19,6 +19,7 @@ Port default: 5765 (UDP). Open this port on both Pis if ufw is active: import io import json +import math import os import socket import struct @@ -37,6 +38,13 @@ _RAW_MAGIC = b'SYNC_RAW' _RAW_HEADER = struct.Struct(' None: hw = self._hw_config @@ -273,11 +286,10 @@ class DisplaySyncManager: break data.extend(chunk) img = Image.open(io.BytesIO(data)) - _MAX_W, _MAX_H = 100_000, 256 # generous for any real scroll image - if img.width > _MAX_W or img.height > _MAX_H: + if img.width > _MAX_FRAME_W or img.height > _MAX_FRAME_H: self.logger.warning( "Sync: rejected oversized scroll image %dx%d (max %dx%d) from %s", - img.width, img.height, _MAX_W, _MAX_H, addr, + img.width, img.height, _MAX_FRAME_W, _MAX_FRAME_H, addr, ) continue try: @@ -396,7 +408,7 @@ class DisplaySyncManager: data = header + arr.tobytes() if len(data) <= 65000: self._send_sock.sendto(data, (self._peer_ip, self.port)) - elif not getattr(self, '_oversized_frame_warned', False): + elif not self._oversized_frame_warned: self._oversized_frame_warned = True self.logger.warning( "Sync: frame too large for UDP (%d bytes, max 65000) — " @@ -451,43 +463,76 @@ class DisplaySyncManager: ) self.write_status_file() + def _handle_received_frame(self, img: Image.Image, sender_ip: str) -> None: + """Record a decoded leader frame and enter follower mode if needed.""" + with self._frame_lock: + self._latest_frame = img + self._last_leader_frame_time = time.time() + self._leader_ip = sender_ip + + if self._follower_state == FollowerState.STANDALONE: + self._follower_state = FollowerState.FOLLOWER + self.logger.info( + "Sync: leader active at %s — switching to follower mode", + sender_ip, + ) + self.write_status_file() + def _follower_recv_loop(self) -> None: while self._running: try: data, addr = self._recv_sock.recvfrom(65535) sender_ip = addr[0] - if data[:8] == _RAW_MAGIC or len(data) > 512: - # Frame data: prefer magic-tagged raw RGB; fall back to legacy PNG + if data[:8] == _RAW_MAGIC: + # Magic-tagged raw RGB frame — self-describing, no guessing. try: - if data[:8] == _RAW_MAGIC: - w, h = _RAW_HEADER.unpack(data[8:12]) - raw = data[12:] - img = Image.frombuffer( - "RGB", (w, h), raw, "raw", "RGB", 0, 1 - ) - else: - # Fallback: try legacy PNG - img = Image.open(io.BytesIO(data)) - img.load() - with self._frame_lock: - self._latest_frame = img - self._last_leader_frame_time = time.time() - self._leader_ip = sender_ip - - if self._follower_state == FollowerState.STANDALONE: - self._follower_state = FollowerState.FOLLOWER - self.logger.info( - "Sync: leader active at %s — switching to follower mode", - sender_ip, - ) - self.write_status_file() + w, h = _RAW_HEADER.unpack(data[8:12]) + raw = data[12:] + img = Image.frombuffer( + "RGB", (w, h), raw, "raw", "RGB", 0, 1 + ) + self._handle_received_frame(img, sender_ip) except Exception as exc: self.logger.debug("Sync: frame decode error: %s", exc) else: - # Control message + # No magic prefix. Whether the payload parses as JSON + # decides between a control message and a legacy + # (pre-magic) PNG frame — both wire formats are + # self-describing, so no size heuristic is needed. A + # >512-byte control message used to be misrouted into + # image decode and silently dropped. try: msg = json.loads(data.decode("utf-8")) + except (json.JSONDecodeError, UnicodeDecodeError): + # Not JSON — try a legacy PNG frame. + try: + img = Image.open(io.BytesIO(data)) + if img.width > _MAX_FRAME_W or img.height > _MAX_FRAME_H: + # Same cap the TCP image path applies: decode + # is deferred until load(), so check first. + self.logger.debug( + "Sync: rejected oversized legacy frame %dx%d from %s", + img.width, img.height, sender_ip, + ) + continue + img.load() + self._handle_received_frame(img, sender_ip) + except Exception as exc: + self.logger.debug("Sync: frame decode error: %s", exc) + continue + + # It parsed, so it is a control message and never a + # frame. Read and validate its fields under a guard — + # a UDP payload is attacker-shaped, so a non-object + # body makes .get() raise AttributeError and an "sx" + # carrying a non-numeric x raises ValueError/TypeError + # — but dispatch the callback *outside* it. Running + # the callback in here would let a fault in someone + # else's code read as a malformed packet and be + # logged as one. + fire_new_cycle = False + try: t = msg.get("t") if t == "hello_ack": self._leader_ip = sender_ip @@ -501,7 +546,17 @@ class DisplaySyncManager: self.write_status_file() elif t == "sx": # Vegas scroll-position sync — tiny message, renders locally - self._latest_scroll_x = float(msg["x"]) + scroll_x = float(msg["x"]) + if not math.isfinite(scroll_x): + # json.loads accepts the NaN/Infinity literals, + # and float("nan") accepts the strings, so a + # non-finite x reaches here intact. Left alone + # it poisons every offset computed from it — + # NaN comparisons are all false, so the + # follower renders a frame it can never scroll + # back from. Treat it as malformed. + raise ValueError(f"non-finite scroll x: {msg['x']!r}") + self._latest_scroll_x = scroll_x self._last_leader_frame_time = time.time() self._leader_ip = sender_ip if self._follower_state == FollowerState.STANDALONE: @@ -511,19 +566,22 @@ class DisplaySyncManager: sender_ip, ) self.write_status_file() - if self._on_new_cycle: - self._on_new_cycle() # build initial scroll image + fire_new_cycle = True # build initial scroll image elif t == "nc": # Leader started a new scroll cycle — rebuild local image - if self._on_new_cycle: - self._on_new_cycle() - except (json.JSONDecodeError, UnicodeDecodeError, KeyError): - pass + fire_new_cycle = True + except (KeyError, AttributeError, TypeError, ValueError) as exc: + self.logger.debug("Sync: malformed control message: %s", exc) + continue + + if fire_new_cycle and self._on_new_cycle: + self._on_new_cycle() except socket.timeout: continue except Exception as exc: self.logger.debug("Sync follower recv error: %s", exc) + time.sleep(0.1) def _follower_announce_loop(self) -> None: hw = self._hw_config diff --git a/src/config_manager.py b/src/config_manager.py index e4c5cc4b..453e73b1 100644 --- a/src/config_manager.py +++ b/src/config_manager.py @@ -38,6 +38,7 @@ from src.config_manager_atomic import ( from src.common.permission_utils import ( ensure_directory_permissions, ensure_file_permissions, + ensure_shared_group_ownership, get_config_file_mode, get_config_dir_mode ) @@ -105,18 +106,13 @@ class ConfigManager: Returns: SaveResult with status and details """ - # Load current secrets to preserve them - secrets_content = {} - if os.path.exists(self.secrets_path): - try: - with open(self.secrets_path, 'r') as f_secrets: - secrets_content = json.load(f_secrets) - except Exception as e: - self.logger.warning(f"Could not load secrets file {self.secrets_path} during save: {e}") - + # Load current secrets to preserve them (raises if unreadable — see + # _load_secrets_for_save) + secrets_content = self._load_secrets_for_save() + # Strip secrets from main config before saving config_to_write = self._strip_secrets_recursive(new_config_data, secrets_content) - + # Use atomic manager to save atomic_mgr = self._get_atomic_manager() result = atomic_mgr.save_config_atomic( @@ -234,6 +230,11 @@ class ConfigManager: # Load and merge secrets if they exist (be permissive on errors) if os.path.exists(self.secrets_path): + # Self-heal stale group ownership (e.g. the root-run display + # service wrote this file before the web user was granted + # group access) before every load attempt; no-op unless + # running as root and the group is already wrong. + ensure_shared_group_ownership(Path(self.secrets_path)) try: with open(self.secrets_path, 'r') as f: secrets = json.load(f) @@ -268,35 +269,86 @@ class ConfigManager: self.logger.error(error_msg, exc_info=True) raise ConfigError(error_msg, config_path=self.config_path) from e + @staticmethod + def _is_parallel_secrets_list(value: Any) -> bool: + """True for the parallel-placeholder list shape emitted by + ``secret_helpers.separate_secrets`` for array-item secrets: a + non-empty list whose elements are ALL dicts (``{}`` marks an item + with no secrets). Any other list-shaped secrets value is a + whole-key secret (e.g. a list of secret scalars).""" + return (isinstance(value, list) and bool(value) + and all(isinstance(item, dict) for item in value)) + def _strip_secrets_recursive(self, data_to_filter: Dict[str, Any], secrets: Dict[str, Any]) -> Dict[str, Any]: """Recursively remove secret keys from a dictionary.""" result = {} for key, value in data_to_filter.items(): - if key in secrets: - if isinstance(value, dict) and isinstance(secrets[key], dict): - # This key is a shared group, recurse - stripped_sub_dict = self._strip_secrets_recursive(value, secrets[key]) - if stripped_sub_dict: # Only add if there's non-secret data left - result[key] = stripped_sub_dict - # Else, it's a secret key at this level, so we skip it - else: + if key not in secrets: # This key is not in secrets, so we keep it result[key] = value + continue + sec = secrets[key] + if isinstance(value, dict) and isinstance(sec, dict): + # This key is a shared group, recurse + stripped_sub_dict = self._strip_secrets_recursive(value, sec) + if stripped_sub_dict: # Only add if there's non-secret data left + result[key] = stripped_sub_dict + elif isinstance(value, list) and self._is_parallel_secrets_list(sec): + # Parallel-list shape from separate_secrets: sec[i] holds the + # secret fields of value[i] ({} = item i has none). Strip each + # item and ALWAYS keep the list — indices must survive so the + # merge-on-load can realign secrets with their items. The + # regular list's length is authoritative: extra secrets + # entries are ignored. + stripped_items = [] + for i, item in enumerate(value): + s_item = sec[i] if i < len(sec) else {} + if isinstance(item, dict) and s_item: + stripped_items.append(self._strip_secrets_recursive(item, s_item)) + else: + stripped_items.append(item) + result[key] = stripped_items + # Else: whole-key secret (scalar, list of secret scalars, or a + # shape mismatch) -> drop the key entirely. Never leak. return result + def _load_secrets_for_save(self) -> Dict[str, Any]: + """Load config_secrets.json for stripping before a save. + + A missing secrets file is fine (nothing to strip). But a file that + EXISTS and cannot be read or parsed means stripping is impossible — + and the in-memory config being saved has secrets deep-merged into it, + so proceeding would write them into config.json in plaintext. That + was the historical behavior; it is now a hard refusal. The save + raises so the caller (and user) fixes the secrets file instead of + silently leaking its contents into the world-readable main config. + """ + if not os.path.exists(self.secrets_path): + return {} + try: + with open(self.secrets_path, 'r') as f_secrets: + return json.load(f_secrets) + # Only the expected read/parse failures — an unexpected implementation + # error should propagate as itself, not masquerade as a secrets-file + # problem. (JSONDecodeError and UnicodeDecodeError are ValueErrors.) + except (OSError, ValueError, RecursionError) as e: + error_msg = ( + f"Refusing to save config: secrets file {self.secrets_path} exists " + f"but could not be loaded ({e}). Saving without it would write " + f"merged secret values into config.json in plaintext. Fix or " + f"remove the secrets file, then retry." + ) + self.logger.error("[Config] %s", error_msg, exc_info=True) + raise ConfigError(error_msg, config_path=self.secrets_path) from e + def save_config(self, new_config_data: Dict[str, Any]) -> None: - """Save configuration to the main JSON file, stripping out secrets.""" - secrets_content = {} - if os.path.exists(self.secrets_path): - try: - with open(self.secrets_path, 'r') as f_secrets: - secrets_content = json.load(f_secrets) - except Exception as e: - self.logger.warning(f"Could not load secrets file {self.secrets_path} during save: {e}") - # Continue without stripping if secrets can't be loaded, or handle as critical error - # For now, we'll proceed cautiously and save the full new_config_data if secrets are unreadable - # to prevent accidental data loss if the secrets file is temporarily corrupt. - # A more robust approach might be to fail the save or use a cached version of secrets. + """Save configuration to the main JSON file, stripping out secrets. + + Raises ConfigError when the secrets file exists but cannot be loaded, + because stripping would be impossible and secrets would leak into + config.json. + """ + secrets_content = self._load_secrets_for_save() config_to_write = self._strip_secrets_recursive(new_config_data, secrets_content) @@ -333,11 +385,39 @@ class ConfigManager: return None def _deep_merge(self, target: Dict[str, Any], source: Dict[str, Any]) -> None: - """Deep merge source dict into target dict.""" + """Deep merge source dict into target dict. + + Sole call site: merging config_secrets.json into the loaded config. + Understands the parallel-list shape separate_secrets emits for + array-item secrets (see _is_parallel_secrets_list): each secrets + list item is merged into the config list item at the same index + ({} placeholders skipped). The config list's length is + authoritative — a user deleting an array item from config.json + must not have it resurrected from a stale secrets entry.""" for key, value in source.items(): if key in target and isinstance(target[key], dict) and isinstance(value, dict): self._deep_merge(target[key], value) + elif (key in target and isinstance(target[key], list) + and self._is_parallel_secrets_list(value)): + tlist = target[key] + for i, s_item in enumerate(value): + if i >= len(tlist): + # Interpolate only config-side data here — nothing + # iterated out of the secrets dict (not even the key + # name) may reach the log. + self.logger.warning( + "A secrets list is longer than the config list it " + "parallels (config has %d item(s)); ignoring the " + "extra entries", len(tlist)) + break + if not s_item: + continue # {} placeholder: item i has no secrets + if isinstance(tlist[i], dict): + self._deep_merge(tlist[i], s_item) + else: + tlist[i] = s_item # shape drift; the secret wins else: + # Scalars AND whole-secret scalar arrays: replace (legacy). target[key] = value def _create_config_from_template(self) -> None: @@ -363,6 +443,7 @@ class ConfigManager: # Set proper file permissions after creation config_path_obj = Path(self.config_path) ensure_file_permissions(config_path_obj, get_config_file_mode(config_path_obj)) + ensure_shared_group_ownership(config_path_obj) self.logger.info(f"Created config.json from template at {os.path.abspath(self.config_path)}") @@ -442,10 +523,6 @@ class ConfigManager: """Get display configuration.""" return self.config.get('display', {}) - def get_clock_config(self) -> Dict[str, Any]: - """Get clock configuration.""" - return self.config.get('clock', {}) - def get_config(self) -> Dict[str, Any]: """Get the full configuration dictionary. @@ -475,6 +552,11 @@ class ConfigManager: self.logger.error(error_msg) raise ConfigError(error_msg, config_path=path_to_load) + if file_type == "secrets": + # Best-effort self-heal: no-op unless running as root and the + # group is stale (see load_config for why this can happen). + ensure_shared_group_ownership(Path(path_to_load)) + try: with open(path_to_load, 'r') as f: return json.load(f) @@ -482,7 +564,18 @@ class ConfigManager: error_msg = f"Error parsing {file_type} configuration file: {path_to_load}" self.logger.error(error_msg, exc_info=True) raise ConfigError(error_msg, config_path=path_to_load) from e - except (IOError, OSError, PermissionError) as e: + except PermissionError as e: + if file_type == "secrets": + # Match load_config()'s tolerance: a secrets file the web + # process can't read (e.g. written 0640 by the root-run + # display service before the group was fixed up) shouldn't + # 500 the settings page — degrade to "no secrets" instead. + self.logger.warning(f"Secrets file not readable ({path_to_load}): {e}. Returning empty secrets.") + return {} + error_msg = f"Error loading {file_type} configuration file {path_to_load}: {str(e)}" + self.logger.error(error_msg, exc_info=True) + raise ConfigError(error_msg, config_path=path_to_load) from e + except (IOError, OSError) as e: error_msg = f"Error loading {file_type} configuration file {path_to_load}: {str(e)}" self.logger.error(error_msg, exc_info=True) raise ConfigError(error_msg, config_path=path_to_load) from e @@ -539,6 +632,7 @@ class ConfigManager: # Ensure final file has correct permissions try: ensure_file_permissions(path_obj, file_mode) + ensure_shared_group_ownership(path_obj) except OSError as perm_error: # If we can't set permissions but file was written, log warning but don't fail self.logger.warning( diff --git a/src/config_manager_atomic.py b/src/config_manager_atomic.py index 3e56d4d3..7b2610a1 100644 --- a/src/config_manager_atomic.py +++ b/src/config_manager_atomic.py @@ -17,6 +17,7 @@ from enum import Enum from src.exceptions import ConfigError from src.logging_config import get_logger +from src.common.permission_utils import ensure_shared_group_ownership class SaveResultStatus(Enum): @@ -410,6 +411,13 @@ class AtomicConfigManager: # This is important because temp files may have different permissions # and we need root service to be able to read config.json os.chmod(destination, target_mode) + + # Also fix group ownership when this save is running as root + # (the display service): 0o640 alone only helps the non-root web + # user read a root-written secrets file if its group already + # matches the web user's group, which isn't guaranteed. See + # permission_utils.ensure_shared_group_ownership for why. + ensure_shared_group_ownership(destination) except Exception as e: raise ConfigError(f"Error during atomic move: {e}") from e diff --git a/src/display_controller.py b/src/display_controller.py index ace55b58..e739e08e 100644 --- a/src/display_controller.py +++ b/src/display_controller.py @@ -44,6 +44,20 @@ from src.common.sync_manager import DisplaySyncManager, SyncRole # Get logger with consistent configuration logger = get_logger(__name__) +# How long startup will wait for plugins to fetch their first data before +# showing anything. Each plugin's update blocks for up to the executor's 30s +# timeout and they run one after another, so the uncapped total is the sum of +# every slow plugin: 82 seconds on the worst boot measured, with a blank panel +# throughout. Whatever does not finish in time is picked up by the scheduled +# update tick moments later, with the display already running. +_INITIAL_UPDATE_BUDGET_SECONDS = 20.0 + +# The least budget worth starting a plugin with. Below this the plugin is +# deferred instead: granting it a floor would let the pass run past its +# deadline, and granting it the true remainder would record a timeout for a +# slot it never had a chance to use. +_MIN_INITIAL_UPDATE_TIMEOUT_SECONDS = 2.0 + # Vegas mode import (lazy loaded to avoid circular imports) _vegas_mode_imported = False VegasModeCoordinator = None @@ -90,7 +104,8 @@ class DisplayController: # Validate startup configuration try: from src.startup_validator import StartupValidator - validator = StartupValidator(self.config_manager) + validator = StartupValidator(self.config_manager, + cache_manager=self.cache_manager) is_valid, errors, warnings = validator.validate_all() if warnings: @@ -258,7 +273,8 @@ class DisplayController: # Validate plugins after plugin manager is created try: from src.startup_validator import StartupValidator - validator = StartupValidator(self.config_manager, self.plugin_manager) + validator = StartupValidator(self.config_manager, self.plugin_manager, + cache_manager=self.cache_manager) is_valid, errors, warnings = validator.validate_all() if warnings: @@ -381,6 +397,10 @@ class DisplayController: logger.debug("%d plugin(s) disabled in config", disabled_count) logger.info("Plugin system initialized in %.3f seconds", time.time() - plugin_time) + # Parallel loading appends modes in load-completion order, which + # varies between restarts; apply the user's configured rotation + # order (no-op when not configured). + self._apply_plugin_rotation_order() logger.info("Total available modes: %d", len(self.available_modes)) logger.info("Available modes: %s", self.available_modes) @@ -457,7 +477,7 @@ class DisplayController: # Initial data update for plugins (ensures data available on first display) logger.info("Performing initial plugin data update...") update_start = time.time() - self._update_modules() + self._update_modules(deadline=update_start + _INITIAL_UPDATE_BUDGET_SECONDS) logger.info("Initial plugin update completed in %.3f seconds", time.time() - update_start) # Initialize Vegas mode coordinator @@ -813,14 +833,42 @@ class DisplayController: self._cached_target_brightness = normal_brightness # persist for minute-gate return normal_brightness - def _update_modules(self): - """Update all plugin modules.""" + def _update_modules(self, deadline: Optional[float] = None): + """Update all plugin modules. + + Args: + deadline: Wall-clock time after which remaining plugins are left + for the scheduled update tick instead of being waited on. Each + update blocks this thread for up to the executor's timeout, and + they run one after another, so without a bound the total is the + sum of every slow plugin on the system. Measured at startup on + a live rig: 82 seconds, 55 and 26 on the two boots before -- all + of it with nothing on the panel. + """ if not self.plugin_manager: return - + # Update all loaded plugins plugins_dict = getattr(self.plugin_manager, 'loaded_plugins', None) or getattr(self.plugin_manager, 'plugins', {}) + deferred = [] for plugin_id, plugin_instance in plugins_dict.items(): + update_timeout = None + if deadline is not None: + update_timeout = deadline - time.time() + if update_timeout < _MIN_INITIAL_UPDATE_TIMEOUT_SECONDS: + # Too little left to be worth starting. Deferring rather + # than granting a floor keeps the budget a real ceiling -- + # clamping up to a minimum let a plugin that began with a + # sliver left run on past the deadline -- and a plugin + # handed a slot it cannot use would just be recorded as + # having timed out. + # + # Nothing is lost either way: a plugin that has never + # updated is immediately due, so run_scheduled_updates() + # picks it up within seconds, with the display already + # running. + deferred.append(plugin_id) + continue # Check circuit breaker before attempting update if hasattr(self.plugin_manager, 'health_tracker') and self.plugin_manager.health_tracker: if self.plugin_manager.health_tracker.should_skip_plugin(plugin_id): @@ -829,7 +877,13 @@ class DisplayController: # Use PluginExecutor if available for safe execution if hasattr(self.plugin_manager, 'plugin_executor'): - success = self.plugin_manager.plugin_executor.execute_update(plugin_instance, plugin_id) + # The remaining budget is the timeout, so the pass cannot + # run past its deadline. Bounding the loop alone did not do + # it: the last plugin to start could still block for the + # executor's full 30s, which turned a 20s budget into a 31.8s + # pass on the rig. + success = self.plugin_manager.plugin_executor.execute_update( + plugin_instance, plugin_id, timeout=update_timeout) if success and hasattr(self.plugin_manager, 'plugin_last_update'): self.plugin_manager.plugin_last_update[plugin_id] = time.time() else: @@ -848,6 +902,12 @@ class DisplayController: if hasattr(self.plugin_manager, 'health_tracker') and self.plugin_manager.health_tracker: self.plugin_manager.health_tracker.record_failure(plugin_id, exc) + if deferred: + logger.info( + "Initial update budget spent; %d plugin(s) left to the update " + "tick so the display can start: %s", + len(deferred), ", ".join(deferred)) + def _tick_plugin_updates_for_vegas(self) -> None: """Run scheduled plugin updates and tell Vegas mode which plugins actually got fresh data, so it can hot-swap them into the scroll @@ -1133,6 +1193,29 @@ class DisplayController: remaining = self.on_demand_expires_at - time.time() return max(0.0, remaining) + def _publish_current_mode_state(self) -> None: + """Publish the currently active display mode/plugin to cache for the web UI.""" + try: + state = { + 'mode': self.current_display_mode, + 'plugin_id': self.mode_to_plugin_id.get(self.current_display_mode), + 'mode_index': self.current_mode_index, + 'total_modes': len(self.available_modes), + 'on_demand_active': self.on_demand_active, + 'is_display_active': self.is_display_active, + 'last_updated': time.time(), + } + self.cache_manager.set('display_current_state', state) + self._last_published_mode = self.current_display_mode + except (OSError, RuntimeError, ValueError, TypeError) as err: + logger.error("Failed to publish current display state: %s", err, exc_info=True) + + def _publish_current_mode_state_if_changed(self) -> None: + """Publish current mode state only when it actually changed, to avoid + writing to the shared cache on every render tick.""" + if self.current_display_mode != getattr(self, '_last_published_mode', None): + self._publish_current_mode_state() + def _publish_on_demand_state(self) -> None: """Publish current on-demand state to cache for external consumers.""" try: @@ -1611,6 +1694,12 @@ class DisplayController: logger.warning("Error checking live priority for %s: %s", mode_name, e) return live + def _vegas_keeps_live_in_ticker(self) -> bool: + """Whether live content should stay in the ticker instead of preempting it.""" + coordinator = getattr(self, 'vegas_coordinator', None) + config = getattr(coordinator, 'vegas_config', None) + return bool(getattr(config, 'live_in_ticker', False)) + def _check_live_priority(self, advance=False): """Return the live-priority mode to display, or None if nothing is live. @@ -1652,6 +1741,7 @@ class DisplayController: logger.info("Starting display with cached data (fast startup mode)") self.current_display_mode = self.available_modes[self.current_mode_index] if self.available_modes else 'none' logger.info(f"Initial mode set to: {self.current_display_mode} (index: {self.current_mode_index}, total modes: {len(self.available_modes)})") + self._publish_current_mode_state() while True: # Apply plugin enable/disable edits saved via the web UI. The @@ -1712,9 +1802,11 @@ class DisplayController: logger.debug(f"Error clearing display when inactive: {e}") logger.info(f"Display not active (is_display_active={self.is_display_active}), sleeping...") + self._publish_current_mode_state() self._sleep_with_plugin_updates(60) continue + self._publish_current_mode_state_if_changed() logger.debug("Display active, processing mode: %s", self.current_display_mode) # Plugins update on their own schedules - no forced sync updates needed @@ -1821,14 +1913,24 @@ class DisplayController: # Check for live priority content and switch to it immediately. # advance=True so multiple simultaneously-live games take turns # (round-robin) instead of pinning to the first plugin. - if not self.on_demand_active and not wifi_status_data: + # Skipped when the ticker is keeping live content: switching + # the rotation underneath Vegas would move current_mode_index + # and stash a resume point for a takeover that never happens. + if (not self.on_demand_active and not wifi_status_data + and not (self._is_vegas_mode_active() + and self._vegas_keeps_live_in_ticker())): live_priority_mode = self._check_live_priority(advance=True) self._apply_live_priority(live_priority_mode) # Vegas scroll mode - continuous ticker across all plugins # Priority: on-demand > wifi-status > live-priority > vegas > normal rotation if self._is_vegas_mode_active() and not wifi_status_data: - live_mode = self._check_live_priority() + # Live content normally preempts the ticker entirely. With + # vegas_scroll.live_in_ticker the marquee keeps running and + # the live plugin takes extra turns inside it instead -- + # see StreamManager._apply_priority_weights. + live_mode = (None if self._vegas_keeps_live_in_ticker() + else self._check_live_priority()) if not live_mode: try: # Run Vegas mode iteration @@ -2843,11 +2945,52 @@ class DisplayController: except Exception as e: logger.error("Plugin reconcile: error enabling %s: %s", plugin_id, e, exc_info=True) + # Newly enabled plugins were appended at the end; put them in the + # configured rotation slot before resyncing the index. + self._apply_plugin_rotation_order() self._resync_mode_index_after_change(previous_mode) - logger.info("Plugin reconcile complete: +%s -%s (%d modes)", + logger.info("[DisplayController] Plugin reconcile complete: +%s -%s (%d modes)", sorted(to_add), sorted(to_remove), len(self.available_modes)) return True + def _apply_plugin_rotation_order(self) -> None: + """Reorder available_modes to follow display.plugin_rotation_order. + + The configured value is a list of plugin ids; their modes rotate in + that order (each plugin's own modes keep their declared order), with + any enabled-but-unlisted plugins appended afterwards in their current + relative order. An empty/missing list leaves available_modes exactly + as built (today's behavior). Mirrors vegas_mode/config.py's + get_ordered_plugins() semantics for the primary rotation. + """ + configured = (self.config.get("display", {}) or {}).get("plugin_rotation_order", []) or [] + # Defensive: hand-edited or migrated configs may hold a non-list or + # non-string entries; keep the existing rotation rather than applying + # a garbage order. + if not isinstance(configured, list): + logger.warning("[DisplayController] Ignoring invalid plugin_rotation_order (not a list): %r", + type(configured).__name__) + return + configured = [p for p in configured if isinstance(p, str)] + if not configured or not self.available_modes: + return + + ordered_ids = [p for p in configured if p in self.plugin_display_modes] + new_modes: List[str] = [] + for plugin_id in ordered_ids: + for mode in self.plugin_display_modes[plugin_id]: + if mode in self.available_modes and mode not in new_modes: + new_modes.append(mode) + # Unlisted plugins' modes (and any mode not attributable to a plugin) + # follow in their existing relative order. + for mode in self.available_modes: + if mode not in new_modes: + new_modes.append(mode) + if new_modes != self.available_modes: + self.available_modes = new_modes + logger.info("[DisplayController] Applied plugin rotation order %s -> modes: %s", + configured, self.available_modes) + def _resync_mode_index_after_change(self, previous_mode: Optional[str]) -> None: """Clamp rotation state after available_modes changed. Stays on the previous mode if it survived, otherwise restarts cleanly within range.""" diff --git a/src/display_manager.py b/src/display_manager.py index 9de558a1..9cc7f622 100644 --- a/src/display_manager.py +++ b/src/display_manager.py @@ -25,6 +25,7 @@ the same object. import json import os +import socket import tempfile if os.getenv("EMULATOR", "false") == "true": from RGBMatrixEmulator import RGBMatrix, RGBMatrixOptions @@ -186,8 +187,14 @@ class DisplayManager: self.config = config or {} self._force_fallback = force_fallback self._suppress_test_pattern = suppress_test_pattern - # When True, update_display() and clear() skip hardware writes (used during off-screen content capture) - self._capture_mode_active = False + # Per-thread capture state. update_display() and clear() skip hardware + # writes while the *calling* thread is capturing content off-screen. + # + # Thread-local rather than a plain flag because Vegas mode prepares + # upcoming content on a background thread: a shared flag set there would + # suppress the render loop's own frame pushes for the duration, freezing + # the panel exactly when the point was to avoid a freeze. + self._capture_state = threading.local() # Double-sided mode state (resolved in _setup_matrix). When disabled, # the logical image is blitted to the matrix unchanged. self._double_sided = None # dict {copies, axis, logical_width, logical_height} or None @@ -252,6 +259,26 @@ class DisplayManager: # Initialize managers # Calendar manager is now initialized by DisplayController + # Orientation setting -> rpi-rgb-led-matrix "Rotate:" pixel-mapper suffix. + # "normal" needs no suffix since 0 degrees is the identity transform. + _ORIENTATION_ROTATE_DEGREES = {'normal': None, '90': 90, '180': 180, '270': 270} + + def _build_pixel_mapper_config(self, hardware_config: dict) -> str: + """Compose the raw pixel_mapper_config string with the orientation setting. + + `pixel_mapper_config` stays available as a free-form advanced field (e.g. + for "U-mapper" chain layouts); `orientation` is the user-facing dropdown + for physical mounting (e.g. panels mounted upside down) and is appended as + a "Rotate:" mapper rather than overwriting any existing config. + """ + base_mapper = (hardware_config.get('pixel_mapper_config') or '').strip() + orientation = hardware_config.get('orientation', 'normal') + degrees = self._ORIENTATION_ROTATE_DEGREES.get(orientation) + if degrees is None: + return base_mapper + rotate_mapper = f'Rotate:{degrees}' + return f'{base_mapper};{rotate_mapper}' if base_mapper else rotate_mapper + def _setup_matrix(self): """Initialize the RGB matrix with configuration settings.""" _init_error_str = None @@ -277,7 +304,7 @@ class DisplayManager: options.pwm_bits = hardware_config.get('pwm_bits', 10) options.pwm_lsb_nanoseconds = hardware_config.get('pwm_lsb_nanoseconds', 150) options.led_rgb_sequence = hardware_config.get('led_rgb_sequence', 'RGB') - options.pixel_mapper_config = hardware_config.get('pixel_mapper_config', '') + options.pixel_mapper_config = self._build_pixel_mapper_config(hardware_config) options.row_address_type = hardware_config.get('row_address_type', 0) options.multiplexing = hardware_config.get('multiplexing', 0) options.panel_type = hardware_config.get('panel_type', '') @@ -491,6 +518,91 @@ class DisplayManager: logger.warning(f"[BRIGHTNESS] Matrix does not support brightness property: {e}", exc_info=True) return -1 + @staticmethod + def _local_ip() -> Optional[str]: + """This device's address on the network it routes through, or None. + + Deliberately not `hostname -I` or a systemctl probe for AP mode, which + is how the web launcher does it: both spawn processes with multi-second + timeouts, and this runs on the startup path the rest of this change + exists to shorten. Connecting a UDP socket sends no packets -- it only + asks the kernel which source address it would use -- so it costs + microseconds and works with the network down, as long as a route + exists. + """ + sock = None + try: + sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) + sock.settimeout(0.2) + sock.connect(("8.8.8.8", 80)) # nosec B104 - no traffic; selects a route + ip = sock.getsockname()[0] + return ip if ip and not ip.startswith("127.") else None + except OSError: + return None + finally: + if sock is not None: + try: + sock.close() + except OSError: + pass + + def _fitting_font(self, lines, width): + """The largest font from the usual ladder that fits every line.""" + candidates = [self.font, + ("assets/fonts/4x6-font.ttf", 6)] + for candidate in candidates: + try: + font = candidate + if isinstance(candidate, tuple): + font = ImageFont.truetype(candidate[0], candidate[1]) + if all(self.draw.textlength(t, font=font) <= width for t in lines): + return font + except (OSError, ValueError, AttributeError): + continue + return self.font + + def _draw_startup_banner(self, lines, width: int, height: int) -> None: + """Centre `lines` over whatever the test pattern already drew. + + This screen stays on the panel for the whole initial plugin update, and + on a headless Pi it is the only place the device's address appears + without going looking for it -- so it has to be readable off a wall, + not merely present. + + The font is chosen to fit rather than fixed at 8px: "Initializing" is + 96px in PressStart2P, which ran off the side of a 64px panel even + before an address was added. And the pattern is punched out behind the + text, because the diagonal runs through the middle of the panel, which + is exactly where this sits. + + The text stays blue. It is not decoration: the pattern draws one pure + channel per element -- red border, green diagonal, blue text -- so that + a glance at the panel says whether led_rgb_sequence is right. Swap the + wiring to BGR and the border comes up blue and this text red. Drawing + it white would light all three channels and destroy the only blue + reference on the screen, which is why it is worth a comment rather + than a quiet preference. + """ + if not lines: + return + font = self._fitting_font(lines, width - 2) + line_height = self.draw.textbbox((0, 0), "Ag", font=font)[3] + 1 + block_height = line_height * len(lines) + block_top = max(1, (height - block_height) // 2) + block_width = max(self.draw.textlength(t, font=font) for t in lines) + block_left = max(0, (width - block_width) // 2) + + self.draw.rectangle( + [block_left - 2, block_top - 1, + block_left + block_width + 1, block_top + block_height], + fill=(0, 0, 0)) + + for row, line in enumerate(lines): + line_width = self.draw.textlength(line, font=font) + self.draw.text( + (max(0, (width - line_width) // 2), block_top + row * line_height), + line, font=font, fill=(0, 0, 255)) + def _draw_test_pattern(self): """Draw a test pattern to verify the display is working.""" try: @@ -510,8 +622,11 @@ class DisplayManager: # Draw a diagonal line self.draw.line([0, 0, self.matrix.width-1, self.matrix.height-1], fill=(0, 255, 0)) - # Draw some text - changed from "TEST" to "Initializing" with smaller font - self.draw.text((10, 10), "Initializing", font=self.font, fill=(0, 0, 255)) + lines = ["Initializing"] + ip = self._local_ip() + if ip: + lines.append(ip) + self._draw_startup_banner(lines, self.matrix.width, self.matrix.height) # Update the display once after everything is drawn self.update_display() @@ -520,6 +635,15 @@ class DisplayManager: except Exception as e: logger.error(f"Error drawing test pattern: {e}", exc_info=True) + @property + def _capture_mode_active(self) -> bool: + """True while the calling thread is capturing content off-screen.""" + return getattr(self._capture_state, 'active', False) + + @_capture_mode_active.setter + def _capture_mode_active(self, value: bool) -> None: + self._capture_state.active = bool(value) + @contextmanager def capture_mode(self): """Suppress hardware output during off-screen content capture. @@ -536,6 +660,59 @@ class DisplayManager: finally: self._capture_mode_active = False + @contextmanager + def render_size(self, width: int, height: Optional[int] = None): + """Temporarily present a smaller logical canvas to plugins. + + Plugins lay out against ``display_manager.matrix.width`` (and the + ``width``/``height`` properties, which defer to it), so the only way to + get a *narrower layout* rather than a cropped one is to tell the plugin + the screen is narrower while it renders. Trimming after the fact cannot + fix a forecast spread across five columns or a progress bar drawn at + 100% width — those need the plugin to make different layout decisions. + + Vegas mode uses this so a plugin can occupy a fraction of a wide panel + and still look deliberately composed. Reuses the same _LogicalMatrix + indirection that double-sided mode relies on, so plugins see a + consistent size from every accessor. + + Only meaningful inside :meth:`capture_mode` — this swaps the shared + image buffer, so the render loop must not be writing to it concurrently. + + Args: + width: Logical width to report, clamped to at least 1 and to the + real panel width (a larger canvas would overflow the hardware). + height: Logical height, defaulting to the current height. + """ + real_matrix = self.matrix + prev_image = getattr(self, 'image', None) + prev_draw = getattr(self, 'draw', None) + + current_w = self.width + current_h = self.height + target_w = max(1, min(int(width), current_w)) + target_h = max(1, min(int(height) if height else current_h, current_h)) + + if target_w == current_w and target_h == current_h: + # Nothing to do; avoid pointless wrapping and buffer churn. + yield + return + + try: + if real_matrix is not None: + self.matrix = _LogicalMatrix(real_matrix, target_w, target_h) + # With no hardware, the width/height properties fall through to + # self.image, so swapping the buffer below is enough on its own. + self.image = Image.new('RGB', (target_w, target_h)) + self.draw = ImageDraw.Draw(self.image) + yield + finally: + self.matrix = real_matrix + if prev_image is not None: + self.image = prev_image + if prev_draw is not None: + self.draw = prev_draw + def _composite_double_sided(self): """Tile the logical screen across the full physical chain. diff --git a/src/dynamic_team_resolver.py b/src/dynamic_team_resolver.py index ab969e17..7cbe09cd 100644 --- a/src/dynamic_team_resolver.py +++ b/src/dynamic_team_resolver.py @@ -167,10 +167,14 @@ class DynamicTeamResolver: # Sort by ranking (1, 2, 3, etc.) sorted_rankings = dict(sorted(rankings.items(), key=lambda x: x[1])) - - # Cache the results - self._rankings_cache = sorted_rankings - self._cache_timestamp = current_time + + # Cache the results ON THE CLASS. Assigning through self + # would create instance attributes that shadow the shared + # class-level cache, making it per-instance — and every + # scoreboard constructs its own resolver, so the cache + # would never actually be shared. + DynamicTeamResolver._rankings_cache = sorted_rankings + DynamicTeamResolver._cache_timestamp = current_time self.logger.info(f"Fetched rankings for {len(sorted_rankings)} teams") return sorted_rankings @@ -216,9 +220,11 @@ class DynamicTeamResolver: return any(pattern in team_name.upper() for pattern in dynamic_patterns) def clear_cache(self): - """Clear the rankings cache to force fresh data on next request.""" - self._rankings_cache = {} - self._cache_timestamp = 0 + """Clear the SHARED rankings cache to force fresh data on next + request. Writes through the class — assigning via self would only + shadow the shared cache for this instance.""" + DynamicTeamResolver._rankings_cache = {} + DynamicTeamResolver._cache_timestamp = 0 self.logger.info("Cleared dynamic team rankings cache") diff --git a/src/element_style.py b/src/element_style.py new file mode 100644 index 00000000..1d9a08ea --- /dev/null +++ b/src/element_style.py @@ -0,0 +1,628 @@ +""" +Shared per-element style resolution for plugins (the x-style-elements system). + +Plugins expose user-customizable text styling — font, size, color, and x/y +pixel offsets per named element — through their ``config_schema.json``. Two +declaration forms exist in the plugin ecosystem: + +- The compact ``x-style-elements`` map on the ``customization`` object + (of-the-day is the reference). ``expand_style_elements()`` turns it into + the full per-element property blocks the web-UI config form renders. +- The manual ``customization`` block: hand-written per-element objects with + ``font`` / ``font_size`` / ``text_color`` defaults (the scoreboards, + ledmatrix-music). No expansion needed — the defaults are read as-is. + +At render time a plugin builds an ``ElementStyleResolver`` from its config +and the schema-file defaults, then asks for each element's resolved style:: + + from src.element_style import ElementStyleResolver, defaults_from_schema_file + + resolver = ElementStyleResolver(config, defaults_from_schema_file(schema_path)) + title = resolver.style('title_text', classic_font='PressStart2P-Regular.ttf', + classic_size=8, classic_color=(255, 255, 255)) + # title.font (PIL font / freetype.Face), title.color (RGB tuple), + # title.offset ((dx, dy)), title.user_forced, title.user_forced_color + +The central subtlety is what "the user set it" means. The web UI's save flow +(``schema_manager.merge_with_defaults``) writes the FULL schema-default +object into ``config.json`` on every save, whether or not the user touched +the styling section — so a value merely being *present* in config is not an +override. A value only counts as user-forced when it genuinely differs from +the schema default for that element. When nothing is forced, ``style()`` +returns exactly the ``classic_*`` values the caller passes (the plugin's +pre-customization styling), so an untouched config renders byte-identically +to the classic code path. Note the classic values and the schema defaults +may legitimately differ (e.g. football's status_text: schema declares 4x6, +the classic loader fell back to PressStart) — the schema default is the +override *reference*, the classic values are the *fallback*. + +``style()`` never raises: any malformed config value degrades to the classic +style with a logged warning. Font faces are cached module-wide by +(resolved path, size), and font files resolve independently of the caller's +cwd (cwd ``assets/fonts/`` first for compatibility, then the core install +root derived from this module's own location). +""" + +import copy +import json +import logging +import os +from dataclasses import dataclass +from typing import Any, Dict, Optional, Tuple, Union + +from PIL import ImageFont + +try: + import freetype +except ImportError: # pragma: no cover - freetype ships with the core + freetype = None + +logger = logging.getLogger(__name__) + +# Core install root (the directory that contains src/ and assets/fonts/), +# derived from this file so fonts resolve regardless of the caller's cwd. +_CORE_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) + +_FONTS_SUBDIR = os.path.join('assets', 'fonts') + +# Last-resort font when a requested file can't be found or loaded. +_FALLBACK_FONT_NAME = 'PressStart2P-Regular.ttf' + +# (resolved absolute path, size) -> loaded font face. BDF faces are stateful +# in principle, but the core's own FontManager shares faces the same way. +_font_cache: Dict[Tuple[str, int], Any] = {} + +# Config keys a style element block carries, in schema/UI order. +_STYLE_KEYS = ('font', 'font_size', 'text_color') + + +@dataclass(frozen=True) +class ElementStyle: + """A fully resolved style for one named element.""" + + font: Any # PIL ImageFont or freetype.Face + color: Tuple[int, int, int] # resolved RGB + offset: Tuple[int, int] # user layout (x, y) offset, default (0, 0) + font_name: str # resolved font filename + font_size: int # resolved pixel size + user_forced: bool # font or size genuinely overridden + user_forced_color: bool # color genuinely overridden + + +# --------------------------------------------------------------------------- +# Font loading (cwd-independent, cached) +# --------------------------------------------------------------------------- + +def resolve_font_path(font_name: str) -> Optional[str]: + """Locate a font file by name, independent of the caller's cwd. + + Tries, in order: an absolute path as given; ``assets/fonts/`` + relative to the cwd (the classic loaders' behavior, kept first so a + process running from a different checkout keeps its own fonts); then + ``assets/fonts/`` under the core install root. Returns an + absolute path, or None when the file doesn't exist anywhere. + """ + if not font_name or not isinstance(font_name, str): + return None + if os.path.isabs(font_name): + return font_name if os.path.isfile(font_name) else None + # A relative name must be a bare filename. font_name comes from plugin + # config, which the web UI writes; a value like "../../config/config.json" + # would otherwise escape assets/fonts/ once joined and let a config probe + # arbitrary paths for existence. os.path.basename collapses any such value + # to its last component, so a name that isn't already bare is rejected. + if os.path.basename(font_name) != font_name: + return None + candidates = ( + os.path.join(os.getcwd(), _FONTS_SUBDIR, font_name), + os.path.join(_CORE_ROOT, _FONTS_SUBDIR, font_name), + ) + for candidate in candidates: + if os.path.isfile(candidate): + return os.path.abspath(candidate) + return None + + +def load_font(font_name: str, size: int) -> Any: + """Load a font by filename at a pixel size, with caching and fallback. + + ``.bdf`` files load as ``freetype.Face`` (matching FontManager), other + files through ``PIL.ImageFont.truetype``. A missing or unloadable font + degrades to ``PressStart2P-Regular.ttf`` at the requested size, then to + PIL's built-in default — this function never raises. + """ + try: + size = max(1, int(size)) + except (TypeError, ValueError): + size = 8 + + path = resolve_font_path(font_name) + if path is None: + logger.warning("Font file not found: %s, using fallback", font_name) + return _load_fallback_font(size) + + cache_key = (path, size) + cached = _font_cache.get(cache_key) + if cached is not None: + return cached + + try: + if path.lower().endswith('.bdf'): + if freetype is None: + raise RuntimeError("freetype not available for BDF fonts") + face = freetype.Face(path) + # Character size in 1/64th points at 72dpi == pixel size. + face.set_char_size(size * 64, size * 64, 72, 72) + font: Any = face + else: + font = ImageFont.truetype(path, size) + except Exception as e: + logger.warning("Error loading font %s at %spx: %s, using fallback", + path, size, e) + return _load_fallback_font(size) + + _font_cache[cache_key] = font + return font + + +def _load_fallback_font(size: int) -> Any: + """PressStart2P at the requested size, else PIL's built-in default.""" + path = resolve_font_path(_FALLBACK_FONT_NAME) + if path is not None: + cache_key = (path, size) + cached = _font_cache.get(cache_key) + if cached is not None: + return cached + try: + font = ImageFont.truetype(path, size) + _font_cache[cache_key] = font + return font + except Exception as e: + logger.error("Error loading fallback font: %s", e) + return ImageFont.load_default() + + +# --------------------------------------------------------------------------- +# Schema parsing +# --------------------------------------------------------------------------- + +def expand_style_elements(schema: Dict[str, Any]) -> Dict[str, Any]: + """Expand a ``customization.x-style-elements`` declaration into the full + per-element property blocks the web-UI config form renders. + + Each declared element becomes an object with ``font`` / ``font_size`` / + ``text_color`` properties (only the sub-fields the declaration carries), + tagged ``x-style-managed: true``; elements declaring ``offsets: true`` + additionally get an entry under ``customization.layout`` with + ``x_offset`` / ``y_offset`` integers defaulting to 0. Hand-written + element blocks with the same key are left untouched. + + Returns the schema unchanged (same object) when there is nothing to + expand; otherwise returns an expanded deep copy. Never raises. + """ + try: + customization = schema.get('properties', {}).get('customization') + if not isinstance(customization, dict): + return schema + declaration = customization.get('x-style-elements') + if not isinstance(declaration, dict) or not declaration: + return schema + + expanded = copy.deepcopy(schema) + customization = expanded['properties']['customization'] + customization.setdefault('type', 'object') + props = customization.setdefault('properties', {}) + layout_props: Dict[str, Any] = {} + + for element_key, spec in declaration.items(): + if not isinstance(spec, dict): + continue + if element_key not in props: + props[element_key] = _element_block_from_spec(element_key, spec) + if spec.get('offsets'): + layout_props[element_key] = _offset_block_from_spec( + element_key, spec) + + if layout_props: + layout = props.setdefault('layout', { + 'type': 'object', + 'title': 'Layout Offsets', + 'description': 'Pixel offsets applied to each element ' + '(positive x moves right, positive y moves down)', + 'x-advanced': True, + 'properties': {}, + 'additionalProperties': False, + }) + layout.setdefault('properties', {}) + for element_key, block in layout_props.items(): + layout['properties'].setdefault(element_key, block) + + return expanded + except Exception as e: + logger.warning("Error expanding x-style-elements: %s", e) + return schema + + +def _element_block_from_spec(element_key: str, + spec: Dict[str, Any]) -> Dict[str, Any]: + """Build one expanded per-element schema block from its declaration.""" + properties: Dict[str, Any] = {} + order = [] + + font_spec = spec.get('font') + if isinstance(font_spec, dict): + font_prop: Dict[str, Any] = { + 'type': 'string', + 'title': 'Font Family', + 'x-advanced': True, + } + if 'default' in font_spec: + font_prop['default'] = font_spec['default'] + if isinstance(font_spec.get('enum'), list): + font_prop['enum'] = list(font_spec['enum']) + properties['font'] = font_prop + order.append('font') + + size_spec = spec.get('size') + if isinstance(size_spec, dict): + size_prop: Dict[str, Any] = { + 'type': 'integer', + 'title': 'Font Size', + 'description': 'Font size in pixels', + 'x-advanced': True, + } + if 'default' in size_spec: + size_prop['default'] = size_spec['default'] + if 'min' in size_spec: + size_prop['minimum'] = size_spec['min'] + if 'max' in size_spec: + size_prop['maximum'] = size_spec['max'] + properties['font_size'] = size_prop + order.append('font_size') + + color_spec = spec.get('color') + if isinstance(color_spec, dict): + color_prop: Dict[str, Any] = { + 'type': 'array', + 'title': 'Text Color', + 'items': {'type': 'integer', 'minimum': 0, 'maximum': 255}, + 'minItems': 3, + 'maxItems': 3, + 'x-widget': 'color-picker', + } + if 'default' in color_spec: + color_prop['default'] = list(color_spec['default']) + properties['text_color'] = color_prop + order.append('text_color') + + return { + 'type': 'object', + 'title': spec.get('title', element_key), + 'x-style-managed': True, + 'x-propertyOrder': order, + 'additionalProperties': False, + 'properties': properties, + } + + +def _offset_block_from_spec(element_key: str, + spec: Dict[str, Any]) -> Dict[str, Any]: + """Build one layout. offset block (x/y, default 0).""" + axis = { + 'type': 'integer', + 'default': 0, + 'x-advanced': True, + } + return { + 'type': 'object', + 'title': spec.get('title', element_key), + 'x-style-managed': True, + 'additionalProperties': False, + 'properties': { + 'x_offset': dict(axis, title='X Offset'), + 'y_offset': dict(axis, title='Y Offset'), + }, + } + + +def defaults_from_schema(schema: Dict[str, Any]) -> Dict[str, Any]: + """Extract per-element style defaults from a config schema dict. + + Understands both declaration forms: the compact ``x-style-elements`` + map, and hand-written per-element blocks under + ``customization.properties`` (their ``font`` / ``font_size`` / + ``text_color`` property defaults). Returns a config-shaped dict:: + + {"customization": {"": {"font": ..., "font_size": ..., + "text_color": [...]}, ...}} + + Elements with no declared defaults are omitted. Never raises. + """ + elements: Dict[str, Dict[str, Any]] = {} + try: + customization = schema.get('properties', {}).get('customization') + if not isinstance(customization, dict): + return {'customization': elements} + + declaration = customization.get('x-style-elements') + if isinstance(declaration, dict): + for element_key, spec in declaration.items(): + if not isinstance(spec, dict): + continue + defaults: Dict[str, Any] = {} + font_spec = spec.get('font') + if isinstance(font_spec, dict) and 'default' in font_spec: + defaults['font'] = font_spec['default'] + size_spec = spec.get('size') + if isinstance(size_spec, dict) and 'default' in size_spec: + defaults['font_size'] = size_spec['default'] + color_spec = spec.get('color') + if isinstance(color_spec, dict) and 'default' in color_spec: + defaults['text_color'] = list(color_spec['default']) + if defaults: + elements[element_key] = defaults + + properties = customization.get('properties') + if isinstance(properties, dict): + for element_key, block in properties.items(): + if element_key == 'layout' or element_key in elements: + continue + if not isinstance(block, dict): + continue + block_props = block.get('properties') + if not isinstance(block_props, dict): + continue + defaults = {} + for style_key in _STYLE_KEYS: + prop = block_props.get(style_key) + if isinstance(prop, dict) and 'default' in prop: + defaults[style_key] = prop['default'] + if defaults: + elements[element_key] = defaults + except Exception as e: + logger.warning("Error extracting style defaults from schema: %s", e) + return {'customization': elements} + + +def defaults_from_schema_file(schema_path: Union[str, os.PathLike]) -> Dict[str, Any]: + """``defaults_from_schema`` for a schema file on disk. A missing or + malformed file yields empty defaults (with a logged warning) — every + configured value then counts as a user override, which is the safe + degradation. Never raises.""" + try: + with open(schema_path, 'r', encoding='utf-8') as f: + schema = json.load(f) + if not isinstance(schema, dict): + raise ValueError("schema is not a JSON object") + except Exception as e: + logger.warning("Could not read style defaults from %s: %s", + schema_path, e) + return {'customization': {}} + return defaults_from_schema(schema) + + +# --------------------------------------------------------------------------- +# Resolver +# --------------------------------------------------------------------------- + +def _normalize_color(value: Any) -> Optional[Tuple[int, int, int]]: + """An (r, g, b) tuple of ints in 0..255, or None for anything else.""" + if isinstance(value, (list, tuple)) and len(value) == 3: + try: + rgb = tuple(int(c) for c in value) + except (TypeError, ValueError): + return None + if all(0 <= c <= 255 for c in rgb): + return rgb # type: ignore[return-value] + return None + + +class ElementStyleResolver: + """Resolves per-element user styling against schema defaults. + + Built from a plugin's live config dict and the defaults extracted from + its own ``config_schema.json`` (``defaults_from_schema_file``). The + config dict is held by reference as ``_config`` — consumers compare + identity (``resolver._config is not self.config``) to decide when a + resolver must be rebuilt after ``on_config_change`` swaps the dict. + + A configured font/size/color counts as user-forced only when it differs + from the schema default (see module docstring); otherwise ``style()`` + returns the caller's classic values verbatim, keeping untouched configs + byte-identical to pre-customization rendering. + """ + + def __init__(self, config: Optional[Dict[str, Any]], + defaults: Optional[Dict[str, Any]] = None): + # Keep the exact object for identity-based invalidation, even if the + # caller hands us something odd; reads are guarded. + self._config = config + if isinstance(defaults, dict): + element_defaults = defaults.get('customization', {}) + else: + element_defaults = {} + self._defaults: Dict[str, Any] = ( + element_defaults if isinstance(element_defaults, dict) else {}) + self._memo: Dict[Any, ElementStyle] = {} + + # -- internal accessors ------------------------------------------------- + + def _customization(self) -> Dict[str, Any]: + config = self._config if isinstance(self._config, dict) else {} + customization = config.get('customization', {}) + return customization if isinstance(customization, dict) else {} + + def _element_config(self, element_key: str) -> Dict[str, Any]: + element = self._customization().get(element_key, {}) + return element if isinstance(element, dict) else {} + + def _element_defaults(self, element_key: str) -> Dict[str, Any]: + defaults = self._defaults.get(element_key, {}) + return defaults if isinstance(defaults, dict) else {} + + # -- public API --------------------------------------------------------- + + def style(self, element_key: str, + classic_font: str = _FALLBACK_FONT_NAME, + classic_size: int = 8, + classic_color: Optional[Tuple[int, int, int]] = None) -> ElementStyle: + """Resolve one element's style. Never raises. + + Args: + element_key: Key under ``config['customization']`` (e.g. + ``'title_text'``). + classic_font: Font filename the plugin's classic (pre- + customization) code used for this element. + classic_size: Classic pixel size. + classic_color: Classic RGB color, or None when the caller only + cares about the font (``.color`` then falls back to the + schema default color, else white). + + Returns: + ElementStyle with the loaded font face, RGB color, (x, y) + offset, and the ``user_forced`` / ``user_forced_color`` flags. + """ + try: + memo_key = (element_key, classic_font, classic_size, + _normalize_color(classic_color) or classic_color) + memoized = self._memo.get(memo_key) + if memoized is not None: + return memoized + except Exception: + memo_key = None + + try: + resolved = self._resolve(element_key, classic_font, + classic_size, classic_color) + except Exception as e: + logger.warning("Error resolving style for element '%s': %s — " + "using classic style", element_key, e) + resolved = self._classic_style(classic_font, classic_size, + classic_color) + if memo_key is not None: + self._memo[memo_key] = resolved + return resolved + + def offset(self, element_key: str) -> Tuple[int, int]: + """The user's ``customization.layout.`` (x, y) pixel + offset, defaulting to (0, 0). Never raises.""" + return (self.offset_value(element_key, 'x_offset', 0), + self.offset_value(element_key, 'y_offset', 0)) + + def offset_value(self, element_key: str, axis: str, default: int = 0) -> int: + """One ``customization.layout..`` value as an int. + + ``axis`` is usually ``'x_offset'`` / ``'y_offset'`` but any key is + honored (e.g. the scoreboards' ``'away_x_offset'``). Numeric + strings are coerced; anything else degrades to ``default``. Never + raises. + """ + try: + layout = self._customization().get('layout', {}) + if not isinstance(layout, dict): + return int(default) + element = layout.get(element_key, {}) + if not isinstance(element, dict): + return int(default) + value = element.get(axis, default) + if isinstance(value, bool): + return int(default) + if isinstance(value, (int, float)): + return int(value) + if isinstance(value, str): + try: + return int(float(value)) + except (TypeError, ValueError): + logger.warning( + "Invalid layout offset for %s.%s: %r, using %s", + element_key, axis, value, default) + return int(default) + return int(default) + except Exception as e: + logger.warning("Error reading layout offset %s.%s: %s", + element_key, axis, e) + try: + return int(default) + except (TypeError, ValueError): + return 0 + + # -- resolution internals ----------------------------------------------- + + def _resolve(self, element_key: str, classic_font: str, + classic_size: int, + classic_color: Optional[Tuple[int, int, int]]) -> ElementStyle: + element_config = self._element_config(element_key) + element_defaults = self._element_defaults(element_key) + + # Font family: forced only when it differs from the schema default + # (falling back to the classic font as the reference when the + # schema declares none). + default_font = element_defaults.get('font', classic_font) + configured_font = element_config.get('font') + font_forced = (isinstance(configured_font, str) and configured_font + and configured_font != default_font) + + # Font size: same rule, with defensive int coercion. + default_size = self._coerce_size( + element_defaults.get('font_size'), None) + if default_size is None: + default_size = self._coerce_size(classic_size, 8) + configured_size = self._coerce_size(element_config.get('font_size'), + None) + size_forced = (configured_size is not None + and configured_size != default_size) + + font_name = configured_font if font_forced else classic_font + font_size = configured_size if size_forced else self._coerce_size( + classic_size, 8) + user_forced = bool(font_forced or size_forced) + + # Color: forced only when it differs from the schema default (or, + # absent one, from the classic color). + default_color = _normalize_color(element_defaults.get('text_color')) + configured_color = _normalize_color(element_config.get('text_color')) + reference_color = (default_color if default_color is not None + else _normalize_color(classic_color)) + color_forced = (configured_color is not None + and configured_color != reference_color) + if color_forced: + color = configured_color + else: + color = (_normalize_color(classic_color) or default_color + or (255, 255, 255)) + + return ElementStyle( + font=load_font(font_name, font_size), + color=color, + offset=self.offset(element_key), + font_name=font_name, + font_size=font_size, + user_forced=user_forced, + user_forced_color=bool(color_forced), + ) + + def _classic_style(self, classic_font: str, classic_size: int, + classic_color: Optional[Tuple[int, int, int]]) -> ElementStyle: + """The untouched fallback style — used when resolution itself + fails, so ``style()`` can keep its never-raises promise.""" + size = self._coerce_size(classic_size, 8) + return ElementStyle( + font=load_font(classic_font, size), + color=_normalize_color(classic_color) or (255, 255, 255), + offset=(0, 0), + font_name=classic_font, + font_size=size, + user_forced=False, + user_forced_color=False, + ) + + @staticmethod + def _coerce_size(value: Any, default: Optional[int]) -> Optional[int]: + """An int pixel size, or ``default`` for None/garbage.""" + if value is None or isinstance(value, bool): + return default + try: + size = int(value) + except (TypeError, ValueError): + return default + return size if size > 0 else default diff --git a/src/font_manager.py b/src/font_manager.py index 08a3f31e..8be283be 100644 --- a/src/font_manager.py +++ b/src/font_manager.py @@ -659,6 +659,25 @@ class FontManager: # ==================== Font Discovery ==================== + @staticmethod + def _resolve_asset_path(relative_path: str) -> str: + """Resolve a repo-relative asset path independently of the process cwd. + + Prefers the working directory (preserving behavior when the process + runs from the install root), then falls back to the install root + derived from this module's own location. Without the fallback, any + process started outside the install root (e.g. the plugin safety + harness on CI) silently loses every font and degrades to PIL's + default face. + """ + if os.path.exists(relative_path): + return relative_path + install_root = Path(__file__).resolve().parent.parent + candidate = install_root / relative_path + if candidate.exists(): + return str(candidate) + return relative_path + def _initialize_fonts(self): """Initialize font catalog and validate configuration.""" self._scan_fonts_directory() @@ -667,7 +686,7 @@ class FontManager: def _scan_fonts_directory(self): """Scan assets/fonts directory for available fonts.""" - fonts_dir = "assets/fonts" + fonts_dir = self._resolve_asset_path("assets/fonts") if not os.path.exists(fonts_dir): logger.warning(f"Fonts directory not found: {fonts_dir}") return @@ -683,6 +702,7 @@ class FontManager: def _register_common_fonts(self): """Register common font aliases from common_fonts dictionary.""" for family_name, font_path in self.common_fonts.items(): + font_path = self._resolve_asset_path(font_path) # Check if font file exists if os.path.exists(font_path): # Register the common font name (overrides auto-generated name if exists) diff --git a/src/font_test_manager.py b/src/font_test_manager.py deleted file mode 100644 index 8869ebbd..00000000 --- a/src/font_test_manager.py +++ /dev/null @@ -1,135 +0,0 @@ -import os -import freetype -from PIL import ImageDraw, ImageFont -import logging -from typing import Dict, Any -from src.display_manager import DisplayManager - -# Configure logging -logging.basicConfig(level=logging.INFO) -logger = logging.getLogger(__name__) - -class FontTestManager: - """Manager for testing fonts with easy BDF/TTF switching.""" - - def __init__(self, config: Dict[str, Any], display_manager: DisplayManager): - self.display_manager = display_manager - self.config = config - self.logger = logging.getLogger('FontTest') - - # FONT CONFIGURATION - EASY SWITCHING - # Set to 'bdf' or 'ttf' to switch font types - self.font_type = 'bdf' # Change this to 'ttf' to use TTF font - - # Font configurations - self.font_configs = { - 'bdf': { - 'path': "assets/fonts/cozette.bdf", - 'display_name': "Cozette BTF", - 'description': "BTF font Test" - }, - 'ttf': { - 'path': "assets/fonts/5by7.regular.ttf", - 'display_name': "5by7 TTF", - 'description': "TTF font test" - } - } - - # Get current font configuration - self.current_config = self.font_configs[self.font_type] - self.font_path = self.current_config['path'] - - # Verify font exists - if not os.path.exists(self.font_path): - self.logger.error(f"Font file not found: {self.font_path}") - raise FileNotFoundError(f"Font file not found: {self.font_path}") - - # Load the font based on type - if self.font_type == 'bdf': - self._load_bdf_font() - else: - self._load_ttf_font() - - self.logger.info(f"Initialized FontTestManager with {self.current_config['description']}") - - def _load_bdf_font(self): - """Load BDF font using freetype.""" - try: - self.face = freetype.Face(self.font_path) - self.logger.info(f"Successfully loaded BDF font from {self.font_path}") - except Exception as e: - self.logger.error(f"Failed to load BDF font: {e}") - raise - - def _load_ttf_font(self): - """Load TTF font using PIL.""" - try: - self.font = ImageFont.truetype(self.font_path, 8) # Size 8 for 5x7 font - self.logger.info(f"Successfully loaded TTF font from {self.font_path}") - except Exception as e: - self.logger.error(f"Failed to load TTF font: {e}") - raise - - def update(self): - """No update needed for static display.""" - - def display(self, force_clear: bool = False): - """Display the font with sample text.""" - try: - # Clear the display - self.display_manager.clear() - - # Draw font name at the top - self.display_manager.draw_text(self.current_config['display_name'], y=2, color=(255, 255, 255)) - - # Draw sample text - draw = ImageDraw.Draw(self.display_manager.image) - sample_text = "ABCDEFGHIJKLMNOPQRSTUVWXYZ" - - # Calculate starting position - x = 10 # Start 10 pixels from the left - y = 10 # Start 10 pixels from the top - - # Draw text based on font type - if self.font_type == 'bdf': - self._draw_bdf_text(draw, sample_text, x, y) - else: - self._draw_ttf_text(draw, sample_text, x, y) - - # Update the display once - self.display_manager.update_display() - - # Log that display is complete - self.logger.info("Font test display complete.") - - except Exception as e: - self.logger.error(f"Error displaying font test: {e}", exc_info=True) - - def _draw_bdf_text(self, draw, text, x, y): - """Draw text using BDF font.""" - for char in text: - # Load the glyph - self.face.load_char(char) - bitmap = self.face.glyph.bitmap - - # Draw the glyph - for i in range(bitmap.rows): - for j in range(bitmap.width): - try: - # Get the byte containing the pixel - byte_index = i * bitmap.pitch + (j // 8) - if byte_index < len(bitmap.buffer): - byte = bitmap.buffer[byte_index] - # Check if the specific bit is set - if byte & (1 << (7 - (j % 8))): - draw.point((x + j, y + i), fill=(255, 255, 255)) - except IndexError: - self.logger.warning(f"Index out of range for char '{char}' at position ({i}, {j})") - continue - - # Move to next character position - x += self.face.glyph.advance.x >> 6 - - def _draw_ttf_text(self, draw, text, x, y): - """Draw text using TTF font.""" - draw.text((x, y), text, font=self.font, fill=(255, 255, 255)) \ No newline at end of file diff --git a/src/generic_cache_mixin.py b/src/generic_cache_mixin.py deleted file mode 100644 index 3c727eb0..00000000 --- a/src/generic_cache_mixin.py +++ /dev/null @@ -1,150 +0,0 @@ -""" -Generic Cache Mixin for Any Manager - -This mixin provides caching functionality that can be used by any manager -that needs to cache data, not just sports managers. It's a more general -version of BackgroundCacheMixin that works for weather, stocks, news, etc. -""" - -import time -from typing import Dict, Optional, Any, Callable - - -class GenericCacheMixin: - """ - Generic mixin class that provides caching functionality to any manager. - - This mixin can be used by weather, stock, news, or any other manager - that needs to cache data with performance monitoring. - - Note: For sports managers that need background service cache integration, - use BackgroundCacheMixin instead. See src/background_cache_mixin.py for details. - """ - - def _fetch_data_with_cache(self, - cache_key: str, - api_fetch_method: Callable, - cache_ttl: int = 300, - force_refresh: bool = False) -> Optional[Dict]: - """ - Generic caching pattern for any manager. - - Args: - cache_key: Unique cache key for this data - api_fetch_method: Method to call for fresh data - cache_ttl: Time-to-live in seconds (default: 5 minutes) - force_refresh: Skip cache and fetch fresh data - - Returns: - Cached or fresh data from API - """ - start_time = time.time() - cache_hit = False - cache_source = None - - try: - # Check cache first (unless forcing refresh) - if not force_refresh: - cached_data = self.cache_manager.get_cached_data(cache_key, cache_ttl) - if cached_data: - self.logger.info(f"Using cached data for {cache_key}") - cache_hit = True - cache_source = "cache" - self.cache_manager.record_cache_hit('regular') - - # Record performance metrics - duration = time.time() - start_time - self.cache_manager.record_fetch_time(duration) - self._log_fetch_performance(cache_key, duration, cache_hit, cache_source) - - return cached_data - - # Fetch fresh data - self.logger.info(f"Fetching fresh data for {cache_key}") - result = api_fetch_method() - cache_source = "api_fresh" - - # Store in cache if we got data - if result: - self.cache_manager.save_cache(cache_key, result) - self.cache_manager.record_cache_miss('regular') - else: - self.logger.warning(f"No data returned for {cache_key}") - - # Record performance metrics - duration = time.time() - start_time - self.cache_manager.record_fetch_time(duration) - - # Log performance - self._log_fetch_performance(cache_key, duration, cache_hit, cache_source) - - return result - - except Exception as e: - duration = time.time() - start_time - self.logger.error(f"Error fetching data for {cache_key} after {duration:.2f}s: {e}") - self.cache_manager.record_fetch_time(duration) - raise - - def _log_fetch_performance(self, cache_key: str, duration: float, cache_hit: bool, cache_source: str): - """ - Log detailed performance metrics for fetch operations. - - Args: - cache_key: Cache key that was accessed - duration: Fetch operation duration in seconds - cache_hit: Whether this was a cache hit - cache_source: Source of the data (cache, api_fresh, etc.) - """ - # Log basic performance info - self.logger.info(f"Fetch completed for {cache_key} in {duration:.2f}s " - f"(cache_hit={cache_hit}, source={cache_source})") - - # Log detailed metrics every 10 operations - if hasattr(self, '_fetch_count'): - self._fetch_count += 1 - else: - self._fetch_count = 1 - - if self._fetch_count % 10 == 0: - metrics = self.cache_manager.get_cache_metrics() - self.logger.info(f"Cache Performance Summary - " - f"Hit Rate: {metrics['cache_hit_rate']:.2%}, " - f"API Calls Saved: {metrics['api_calls_saved']}, " - f"Avg Fetch Time: {metrics['average_fetch_time']:.2f}s") - - def get_cache_performance_summary(self) -> Dict[str, Any]: - """ - Get cache performance summary for this manager. - - Returns: - Dictionary containing cache performance metrics - """ - return self.cache_manager.get_cache_metrics() - - def log_cache_performance(self): - """Log current cache performance metrics.""" - self.cache_manager.log_cache_metrics() - - def clear_cache_for_key(self, cache_key: str): - """Clear cache for a specific key.""" - self.cache_manager.clear_cache(cache_key) - self.logger.info(f"Cleared cache for {cache_key}") - - def get_cache_info(self, cache_key: str) -> Dict[str, Any]: - """ - Get information about a cached item. - - Args: - cache_key: Cache key to check - - Returns: - Dictionary with cache information - """ - # This would need to be implemented in CacheManager - # For now, just return basic info - return { - 'key': cache_key, - 'exists': self.cache_manager.get_cached_data(cache_key, 0) is not None, - 'ttl': 'unknown' # Would need to be implemented - } diff --git a/src/image_utils.py b/src/image_utils.py deleted file mode 100644 index 6179f9bf..00000000 --- a/src/image_utils.py +++ /dev/null @@ -1,22 +0,0 @@ -"""Deprecated: use src/adaptive_images.py (fit_image) instead. - -This module predates the adaptive image system and has no known callers. -It is kept only so any out-of-tree code importing it keeps working. -""" - -import logging -from PIL import Image - -logger = logging.getLogger(__name__) - -def scale_to_max_dimensions(img, max_width, max_height): - h_to_w_ratio = img.height / img.width - w_to_h_ratio = img.width / img.height - - if img.height > max_height: - img = img.resize((int(max_height * w_to_h_ratio), max_height), Image.Resampling.LANCZOS) - - if img.width > max_width: - img = img.resize((max_width, int(max_width * h_to_w_ratio)), Image.Resampling.LANCZOS) - - return img diff --git a/src/layout_manager.py b/src/layout_manager.py deleted file mode 100644 index 1d852fc2..00000000 --- a/src/layout_manager.py +++ /dev/null @@ -1,409 +0,0 @@ -""" -Layout Manager for LED Matrix Display -Handles custom layouts, element positioning, and display composition. -""" - -import json -import os -import logging -from typing import Dict, List, Any -from datetime import datetime - -logger = logging.getLogger(__name__) - -class LayoutManager: - def __init__(self, display_manager=None, config_path="config/custom_layouts.json"): - self.display_manager = display_manager - self.config_path = config_path - self.layouts = self.load_layouts() - self.current_layout = None - - def load_layouts(self) -> Dict[str, Any]: - """Load saved layouts from file.""" - try: - if os.path.exists(self.config_path): - with open(self.config_path, 'r') as f: - return json.load(f) - return {} - except Exception as e: - logger.error(f"Error loading layouts: {e}") - return {} - - def save_layouts(self) -> bool: - """Save layouts to file.""" - try: - from pathlib import Path - from src.common.permission_utils import ( - ensure_directory_permissions, - get_config_dir_mode - ) - config_path_obj = Path(self.config_path) - ensure_directory_permissions(config_path_obj.parent, get_config_dir_mode()) - with open(self.config_path, 'w') as f: - json.dump(self.layouts, f, indent=2) - return True - except Exception as e: - logger.error(f"Error saving layouts: {e}") - return False - - def create_layout(self, name: str, elements: List[Dict], description: str = "") -> bool: - """Create a new layout.""" - try: - self.layouts[name] = { - 'elements': elements, - 'description': description, - 'created': datetime.now().isoformat(), - 'modified': datetime.now().isoformat() - } - return self.save_layouts() - except Exception as e: - logger.error(f"Error creating layout '{name}': {e}") - return False - - def update_layout(self, name: str, elements: List[Dict], description: str = None) -> bool: - """Update an existing layout.""" - try: - if name not in self.layouts: - return False - - self.layouts[name]['elements'] = elements - self.layouts[name]['modified'] = datetime.now().isoformat() - - if description is not None: - self.layouts[name]['description'] = description - - return self.save_layouts() - except Exception as e: - logger.error(f"Error updating layout '{name}': {e}") - return False - - def delete_layout(self, name: str) -> bool: - """Delete a layout.""" - try: - if name in self.layouts: - del self.layouts[name] - return self.save_layouts() - return False - except Exception as e: - logger.error(f"Error deleting layout '{name}': {e}") - return False - - def get_layout(self, name: str) -> Dict[str, Any]: - """Get a specific layout.""" - return self.layouts.get(name, {}) - - def list_layouts(self) -> List[str]: - """Get list of all layout names.""" - return list(self.layouts.keys()) - - def set_current_layout(self, name: str) -> bool: - """Set the current active layout.""" - if name in self.layouts: - self.current_layout = name - return True - return False - - def render_layout(self, layout_name: str = None, data_context: Dict = None) -> bool: - """Render a layout to the display.""" - if not self.display_manager: - logger.error("No display manager available") - return False - - layout_name = layout_name or self.current_layout - if not layout_name or layout_name not in self.layouts: - logger.error(f"Layout '{layout_name}' not found") - return False - - try: - # Clear the display - self.display_manager.clear() - - # Get layout elements - elements = self.layouts[layout_name]['elements'] - - # Render each element - for element in elements: - self.render_element(element, data_context or {}) - - # Update the display - self.display_manager.update_display() - return True - - except Exception as e: - logger.error(f"Error rendering layout '{layout_name}': {e}") - return False - - def render_element(self, element: Dict, data_context: Dict) -> None: - """Render a single element.""" - element_type = element.get('type') - x = element.get('x', 0) - y = element.get('y', 0) - properties = element.get('properties', {}) - - try: - if element_type == 'text': - self._render_text_element(x, y, properties, data_context) - elif element_type == 'weather_icon': - self._render_weather_icon_element(x, y, properties, data_context) - elif element_type == 'rectangle': - self._render_rectangle_element(x, y, properties) - elif element_type == 'line': - self._render_line_element(x, y, properties) - elif element_type == 'clock': - self._render_clock_element(x, y, properties) - elif element_type == 'data_text': - self._render_data_text_element(x, y, properties, data_context) - else: - logger.warning(f"Unknown element type: {element_type}") - - except Exception as e: - logger.error(f"Error rendering element {element_type}: {e}") - - def _render_text_element(self, x: int, y: int, properties: Dict, data_context: Dict) -> None: - """Render a text element.""" - text = properties.get('text', 'Sample Text') - color = tuple(properties.get('color', [255, 255, 255])) - font_size = properties.get('font_size', 'normal') - - # Support template variables in text - text = self._process_template_text(text, data_context) - - # Select font - if font_size == 'small': - font = self.display_manager.small_font - elif font_size == 'large': - font = self.display_manager.regular_font - else: - font = self.display_manager.regular_font - - self.display_manager.draw_text(text, x, y, color, font=font) - - def _render_weather_icon_element(self, x: int, y: int, properties: Dict, data_context: Dict) -> None: - """Render a weather icon element.""" - condition = properties.get('condition', 'sunny') - size = properties.get('size', 16) - - # Use weather data from context if available - if 'weather' in data_context and 'condition' in data_context['weather']: - condition = data_context['weather']['condition'].lower() - - self.display_manager.draw_weather_icon(condition, x, y, size) - - def _render_rectangle_element(self, x: int, y: int, properties: Dict) -> None: - """Render a rectangle element.""" - width = properties.get('width', 10) - height = properties.get('height', 10) - color = tuple(properties.get('color', [255, 255, 255])) - filled = properties.get('filled', False) - - if filled: - self.display_manager.draw.rectangle( - [x, y, x + width, y + height], - fill=color - ) - else: - self.display_manager.draw.rectangle( - [x, y, x + width, y + height], - outline=color - ) - - def _render_line_element(self, x: int, y: int, properties: Dict) -> None: - """Render a line element.""" - x2 = properties.get('x2', x + 10) - y2 = properties.get('y2', y) - color = tuple(properties.get('color', [255, 255, 255])) - width = properties.get('width', 1) - - self.display_manager.draw.line([x, y, x2, y2], fill=color, width=width) - - def _render_clock_element(self, x: int, y: int, properties: Dict) -> None: - """Render a clock element.""" - format_str = properties.get('format', '%H:%M') - color = tuple(properties.get('color', [255, 255, 255])) - - current_time = datetime.now().strftime(format_str) - self.display_manager.draw_text(current_time, x, y, color) - - def _render_data_text_element(self, x: int, y: int, properties: Dict, data_context: Dict) -> None: - """Render a data-driven text element.""" - data_key = properties.get('data_key', '') - format_str = properties.get('format', '{value}') - color = tuple(properties.get('color', [255, 255, 255])) - default_value = properties.get('default', 'N/A') - - # Extract data from context - value = self._get_nested_value(data_context, data_key, default_value) - - # Format the text - try: - text = format_str.format(value=value) - except (ValueError, TypeError, KeyError, IndexError): - text = str(value) - - self.display_manager.draw_text(text, x, y, color) - - def _process_template_text(self, text: str, data_context: Dict) -> str: - """Process template variables in text.""" - try: - # Simple template processing - replace {key} with values from context - for key, value in data_context.items(): - placeholder = f"{{{key}}}" - if placeholder in text: - text = text.replace(placeholder, str(value)) - return text - except Exception as e: - logger.error(f"Error processing template text: {e}") - return text - - def _get_nested_value(self, data: Dict, key: str, default=None): - """Get a nested value from a dictionary using dot notation.""" - try: - keys = key.split('.') - value = data - for k in keys: - value = value[k] - return value - except (KeyError, TypeError): - return default - - def create_preset_layouts(self) -> None: - """Create some preset layouts for common use cases.""" - # Basic clock layout - clock_layout = [ - { - 'type': 'clock', - 'x': 10, - 'y': 10, - 'properties': { - 'format': '%H:%M', - 'color': [255, 255, 255] - } - }, - { - 'type': 'clock', - 'x': 10, - 'y': 20, - 'properties': { - 'format': '%m/%d', - 'color': [100, 100, 255] - } - } - ] - self.create_layout('basic_clock', clock_layout, 'Simple clock with date') - - # Weather layout - weather_layout = [ - { - 'type': 'weather_icon', - 'x': 5, - 'y': 5, - 'properties': { - 'condition': 'sunny', - 'size': 20 - } - }, - { - 'type': 'data_text', - 'x': 30, - 'y': 8, - 'properties': { - 'data_key': 'weather.temperature', - 'format': '{value}°', - 'color': [255, 200, 0], - 'default': '--°' - } - }, - { - 'type': 'data_text', - 'x': 30, - 'y': 18, - 'properties': { - 'data_key': 'weather.condition', - 'format': '{value}', - 'color': [200, 200, 200], - 'default': 'Unknown' - } - } - ] - self.create_layout('weather_display', weather_layout, 'Weather icon with temperature and condition') - - # Mixed dashboard layout - dashboard_layout = [ - { - 'type': 'clock', - 'x': 2, - 'y': 2, - 'properties': { - 'format': '%H:%M', - 'color': [255, 255, 255] - } - }, - { - 'type': 'weather_icon', - 'x': 50, - 'y': 2, - 'properties': { - 'size': 16 - } - }, - { - 'type': 'data_text', - 'x': 70, - 'y': 5, - 'properties': { - 'data_key': 'weather.temperature', - 'format': '{value}°', - 'color': [255, 200, 0], - 'default': '--°' - } - }, - { - 'type': 'line', - 'x': 0, - 'y': 15, - 'properties': { - 'x2': 128, - 'y2': 15, - 'color': [100, 100, 100] - } - }, - { - 'type': 'data_text', - 'x': 2, - 'y': 18, - 'properties': { - 'data_key': 'stocks.AAPL.price', - 'format': 'AAPL: ${value}', - 'color': [0, 255, 0], - 'default': 'AAPL: N/A' - } - } - ] - self.create_layout('dashboard', dashboard_layout, 'Mixed dashboard with clock, weather, and stocks') - - logger.info("Created preset layouts") - - def get_layout_preview(self, layout_name: str) -> Dict[str, Any]: - """Get a preview representation of a layout.""" - if layout_name not in self.layouts: - return {} - - layout = self.layouts[layout_name] - elements = layout['elements'] - - # Create a simple preview representation - preview = { - 'name': layout_name, - 'description': layout.get('description', ''), - 'element_count': len(elements), - 'elements': [] - } - - for element in elements: - preview['elements'].append({ - 'type': element.get('type'), - 'position': f"({element.get('x', 0)}, {element.get('y', 0)})", - 'properties': list(element.get('properties', {}).keys()) - }) - - return preview \ No newline at end of file diff --git a/src/logging_config.py b/src/logging_config.py index 4211e861..843257eb 100644 --- a/src/logging_config.py +++ b/src/logging_config.py @@ -5,6 +5,7 @@ Provides consistent logging configuration across the LEDMatrix application. Supports structured logging with context information and appropriate log levels. """ +import copy import logging import sys import os @@ -65,24 +66,29 @@ class ContextualFormatter(logging.Formatter): self.include_context = include_context def format(self, record: logging.LogRecord) -> str: - """Format log record with context.""" - # Add context to message if present + """Format log record with context. + + Works on a shallow copy of the record: a record is formatted once + PER HANDLER, so mutating record.msg in place (the old behavior) + prepended the context prefix again for every additional handler. + """ if self.include_context: context_parts = [] - + if hasattr(record, 'plugin_id'): context_parts.append(f"[Plugin: {record.plugin_id}]") - + if hasattr(record, 'operation_id'): context_parts.append(f"[Op: {record.operation_id}]") - + if hasattr(record, 'context') and isinstance(record.context, dict): for key, value in record.context.items(): context_parts.append(f"[{key}: {value}]") - + if context_parts: + record = copy.copy(record) record.msg = ' '.join(context_parts) + ' ' + str(record.msg) - + return super().format(record) @@ -124,7 +130,12 @@ def setup_logging( # Console handler (always add) console_handler = logging.StreamHandler(sys.stdout) console_handler.setLevel(level) - console_handler.setFormatter(formatter) + # Under systemd, tag each line so the journal records the real severity + # rather than filing everything as informational. The file handler below + # keeps the plain formatter: the prefix is meaningful to journald and noise + # anywhere else. + console_handler.setFormatter( + JournalPriorityFormatter(formatter) if _under_systemd() else formatter) root_logger.addHandler(console_handler) # File handler (if specified) @@ -139,23 +150,119 @@ def setup_logging( sys.stderr.write(f"Warning: Could not set up file logging to {log_file}: {e}\n") -def get_logger(name: str, plugin_id: Optional[str] = None) -> logging.Logger: +#: syslog priorities, which is what systemd parses from a "" prefix on +#: stdout. Mapped from Python's levels. +_SYSLOG_PRIORITY = { + logging.CRITICAL: 2, # LOG_CRIT + logging.ERROR: 3, # LOG_ERR + logging.WARNING: 4, # LOG_WARNING + logging.INFO: 6, # LOG_INFO + logging.DEBUG: 7, # LOG_DEBUG +} + + +class JournalPriorityFormatter(logging.Formatter): + """Wraps a formatter, prefixing each line with its syslog priority. + + Under systemd everything this process writes to stdout lands in the journal + as PRIORITY=6, whatever the Python level was. Measured on a live rig: 55 + ERROR lines and 13 WARNING lines in a day, every one of them recorded as + informational, so `journalctl -p err -u ledmatrix` returned nothing at all + while errors were being logged. Anyone triaging has to grep the message + text instead, which is both slower and wrong -- a search for "oom" matches + the radar logging "zoom=9". + + systemd reads a leading "" on each line and uses it as the priority + (sd-daemon(3)), so this needs no extra dependency. Multi-line records get + the prefix on every line, since the journal splits them and an unprefixed + continuation would fall back to the default. + """ + + def __init__(self, inner: logging.Formatter): + super().__init__() + self._inner = inner + + @property + def inner(self) -> logging.Formatter: + """The formatter doing the actual work. + + Whether journald tagging is applied depends on JOURNAL_STREAM, so it is + on under systemd and off in a terminal -- and anything asserting which + formatter setup_logging() selected would otherwise get a different + answer in CI than on a developer's machine. Exposing the inner one lets + those checks stay about format_type, which is what they mean. + """ + return self._inner + + def format(self, record: logging.LogRecord) -> str: + text = self._inner.format(record) + prefix = f"<{_SYSLOG_PRIORITY.get(record.levelno, 6)}>" + return "\n".join(prefix + line for line in text.split("\n")) + + +def _under_systemd() -> bool: + """True when stdout really is the journal. + + systemd sets JOURNAL_STREAM to "dev:ino" for services whose output it + captures. Presence alone is not enough to act on: the variable is + inherited by child processes and survives redirection, so a subprocess + whose stdout is a pipe or a file still sees it and would emit the "" + priority prefixes as literal noise into that output. systemd's own + guidance is to fstat the descriptor and compare st_dev/st_ino, which is + what distinguishes "the journal is somewhere in my ancestry" from "my + stdout is the journal". + """ + declared = os.environ.get("JOURNAL_STREAM") + if not declared: + return False + try: + dev_text, ino_text = declared.split(":", 1) + declared_ids = (int(dev_text), int(ino_text)) + except (ValueError, AttributeError): + return False + try: + stat_result = os.fstat(sys.stdout.fileno()) + except (OSError, ValueError, AttributeError): + # No usable stdout: captured by pytest, detached, or already closed. + return False + return (stat_result.st_dev, stat_result.st_ino) == declared_ids + + +class PluginLoggerAdapter(logging.LoggerAdapter): + """LoggerAdapter that stamps every record with its plugin_id. + + A plain `logging.Logger` attribute (the old approach) is never copied + onto individual `LogRecord`s, so `ContextualFormatter`/`StructuredFormatter` + only ever saw `plugin_id` on calls that explicitly passed + `extra={'plugin_id': ...}` (i.e. `log_with_context`). This adapter injects + it into `extra` on every call, so `self.logger.info(...)` in plugin code + is tagged automatically. + """ + + def process(self, msg, kwargs): + extra = dict(kwargs.get('extra') or {}) + extra.setdefault('plugin_id', self.extra.get('plugin_id')) + kwargs['extra'] = extra + return msg, kwargs + + +def get_logger(name: str, plugin_id: Optional[str] = None): """ Get a logger with consistent configuration. - + Args: name: Logger name (typically __name__) plugin_id: Optional plugin ID for automatic context - + Returns: - Configured logger instance + Configured logger instance (or a PluginLoggerAdapter when plugin_id + is given, which supports the same .debug/.info/.warning/.error API) """ logger = logging.getLogger(name) - - # Add plugin_id as attribute for formatters + if plugin_id: - logger.plugin_id = plugin_id - + return PluginLoggerAdapter(logger, {'plugin_id': plugin_id}) + return logger @@ -206,8 +313,11 @@ def log_warning(logger: logging.Logger, message: str, **kwargs) -> None: def log_error(logger: logging.Logger, message: str, **kwargs) -> None: - """Log error message with context.""" - log_with_context(logger, logging.ERROR, message, **kwargs, exc_info=True) + """Log error message with context. Defaults exc_info=True; a caller + passing exc_info explicitly wins (the old hardcoded keyword raised + TypeError on that duplicate).""" + kwargs.setdefault('exc_info', True) + log_with_context(logger, logging.ERROR, message, **kwargs) def log_debug(logger: logging.Logger, message: str, **kwargs) -> None: diff --git a/src/logo_downloader.py b/src/logo_downloader.py index e4dad335..b799b7c1 100644 --- a/src/logo_downloader.py +++ b/src/logo_downloader.py @@ -118,7 +118,14 @@ class LogoDownloader: @staticmethod def normalize_abbreviation(abbreviation: str) -> str: - """Normalize team abbreviation for consistent filename usage.""" + """Normalize team abbreviation for consistent filename usage. + + Public API: sports scoreboard plugins call this directly. + NOTE: LogoHelper.normalize_abbreviation (src/common/logo_helper.py) + is a deliberately different variant (strips spaces, fewer character + replacements) — keep both behaviors stable; logo filenames on + existing installs depend on them. + """ # Handle special characters that can cause filesystem issues normalized = abbreviation.upper() diff --git a/src/plugin_system/base_plugin.py b/src/plugin_system/base_plugin.py index a990ecd9..dbe14a49 100644 --- a/src/plugin_system/base_plugin.py +++ b/src/plugin_system/base_plugin.py @@ -86,7 +86,9 @@ class BasePlugin(ABC): self.display_manager: Any = display_manager self.cache_manager: Any = cache_manager self.plugin_manager: Any = plugin_manager - self.logger: logging.Logger = get_logger(f"plugin.{plugin_id}", plugin_id=plugin_id) + # get_logger returns a PluginLoggerAdapter here (plugin_id given), which + # stamps every record with plugin_id so it survives into formatted output. + self.logger = get_logger(f"plugin.{plugin_id}", plugin_id=plugin_id) self.enabled: bool = config.get("enabled", True) self.logger.info("Initialized plugin: %s", plugin_id) @@ -143,6 +145,77 @@ class BasePlugin(ABC): """ raise NotImplementedError("Plugins must implement display()") + # ------------------------------------------------------------------------- + # Global (whole-device) configuration + # ------------------------------------------------------------------------- + @property + def global_config(self) -> Dict[str, Any]: + """ + The full LEDMatrix configuration, for reading device-wide settings. + + ``self.config`` is only this plugin's own slice, so cross-cutting + settings — ``target_fps``, ``timezone``, ``location`` — were previously + unreachable from a plugin without reaching into a manager by hand. + + Resolution order mirrors the timezone helpers the sports plugins + already ship: ``plugin_manager.config_manager`` first (the cores that + hang it there), then ``cache_manager.config_manager``. Returns ``{}`` + when neither is available, so callers can use plain ``.get()`` without + guarding, and a plugin on a core that predates this property still + loads — ``getattr(self, 'global_config', {})`` simply yields the + default. + + Treat as read-only: the returned dict is the live config the core is + using, so mutating it edits every other consumer's view and can be + persisted back to disk. + + Assignment is still allowed and wins over the resolved value. Several + shipped plugins (news, stock-news, ledmatrix-stocks, ledmatrix- + elections, ledmatrix-leaderboard, nfl-draft) set + ``self.global_config`` to their own ``config['global']`` sub-dict; a + property without a setter would raise AttributeError and stop those + plugins loading. + + Example: + fps = self.global_config.get('target_fps') + """ + override = getattr(self, '_global_config_override', None) + if override is not None: + return override + for owner in (self.plugin_manager, self.cache_manager): + config_manager = getattr(owner, 'config_manager', None) + if config_manager is None: + continue + try: + config = config_manager.get_config() + except Exception: + # A broken or unreadable config must never stop a plugin from + # loading; fall through to the next source, then to {}. + self.logger.debug( + "Could not read global config from %s", + type(owner).__name__, exc_info=True, + ) + continue + # Only a real mapping is usable: callers do .get() on this and feed + # the result to numeric code, so handing back whatever a stub or a + # half-built manager returned would fail later and further away. + # + # An empty dict is treated as "nothing here yet" rather than a + # valid answer, so resolution continues to the next source. Both + # managers default to the same config/config.json, so falling + # through cannot pick up a different file's settings -- but it does + # rescue the case where the first manager simply hasn't loaded yet, + # which would otherwise return {} and silently disable every + # setting read through this property. + if isinstance(config, dict) and config: + return config + return {} + + @global_config.setter + def global_config(self, value: Dict[str, Any]) -> None: + """Let a plugin substitute its own view (see the getter's docstring).""" + self._global_config_override = value + # ------------------------------------------------------------------------- # Adaptive layout support (opt-in) # ------------------------------------------------------------------------- @@ -291,8 +364,10 @@ class BasePlugin(ABC): # Handle None case if duration is None: pass # Fall through to config - # Try to convert to float if it's a number or numeric string - elif isinstance(duration, (int, float)): + # Try to convert to float if it's a number or numeric string. + # bool is excluded: it's an int subclass, and True would + # otherwise read as a 1-second duration. + elif isinstance(duration, (int, float)) and not isinstance(duration, bool): if duration > 0: return float(duration) else: @@ -330,8 +405,9 @@ class BasePlugin(ABC): # Fall back to config config_duration = self.config.get("display_duration", 15.0) try: - # Ensure config value is also a valid float - if isinstance(config_duration, (int, float)): + # Ensure config value is also a valid float (bool excluded — an + # int subclass that would otherwise read True as 1 second) + if isinstance(config_duration, (int, float)) and not isinstance(config_duration, bool): if config_duration > 0: return float(config_duration) else: @@ -479,6 +555,48 @@ class BasePlugin(ABC): """ return False + def get_vegas_priority_weight(self) -> Optional[int]: + """How many slots per Vegas cycle this plugin should get, or None. + + The Vegas ticker is otherwise a strict round robin: every plugin + appears exactly once per cycle. With a dozen plugins enabled that puts + minutes between a live score and its next appearance. A weight of N + gives the plugin N slots per cycle, spread evenly through it rather + than clumped together. + + Return ``None`` (the default) to let the core decide. It gives a + plugin ``vegas_scroll.live_weight`` when ``has_live_priority()`` and + ``has_live_content()`` are both true, and 1 otherwise -- so live sports + already get extra turns without implementing this at all. + + Implement it only when the plugin knows something the core cannot. The + motivating case is favorite teams: the core can see *that* a game is + live but not *whose*, so a scoreboard that wants its favorite's game + shown more often than other live games has to say so:: + + def get_vegas_priority_weight(self): + if not (self.has_live_priority() and self.has_live_content()): + return None # let the core decide + cfg = self.global_config.get('display', {}).get('vegas_scroll', {}) + if self._favorite_is_live(): + return cfg.get('favorite_live_weight', 5) + return cfg.get('live_weight', 3) + + The weight is per *plugin*, not per game. A scoreboard showing four + live games still occupies one slot at a time and rotates its own games + within that slot; this controls how often the plugin itself comes + round. + + Raising is safe: the core logs it and falls back to its own + live-content check, so a broken weight calculation costs the plugin + the favorite distinction but not the live boost. + + Returns: + Slots per cycle (clamped to 1..10 by the caller), or None to + defer to the core's own live-content weighting. + """ + return None + def get_live_modes(self) -> List[str]: """ Get list of display modes that should be used during live priority takeover. @@ -503,6 +621,40 @@ class BasePlugin(ABC): # ------------------------------------------------------------------------- # Vegas scroll mode support # ------------------------------------------------------------------------- + def get_vegas_render_width(self) -> int: + """ + Width the Vegas ticker wants this plugin's content to occupy. + + On a wide panel a layout built to fill the screen reads as sparse in a + ticker — a forecast spread over five columns, a progress bar drawn at + 100% width, a stat block with the panel's whole width between its + elements. Vegas asks for a narrower render so the plugin can choose a + tighter arrangement instead of being cropped afterwards. + + Vegas also narrows ``display_manager`` for the duration of the call, so + a plugin that already sizes itself from ``matrix.width`` needs no + changes. Read this only when you size content some other way. + + Controlled by the plugin's own ``vegas_width_pct`` config value, else + the global ``display.vegas_scroll.render_width_pct``. + + Returns: + Target width in pixels. Outside a Vegas content request, the full + display width. + """ + requested = getattr(self, '_vegas_render_width', None) + if isinstance(requested, int) and requested > 0: + return requested + + display_manager = getattr(self, 'display_manager', None) + matrix = getattr(display_manager, 'matrix', None) + if matrix is not None and getattr(matrix, 'width', None): + return int(matrix.width) + width = getattr(display_manager, 'width', None) + if callable(width): + width = width() + return int(width) if width else 128 + def get_vegas_content(self) -> Optional[Any]: """ Get content for Vegas-style continuous scroll mode. @@ -687,10 +839,12 @@ class BasePlugin(ABC): self.logger.error("'enabled' must be a boolean") return False - # Check display_duration if present + # Check display_duration if present. bool is excluded explicitly: + # it's an int subclass, and get_display_duration rejects it too. if "display_duration" in self.config: duration = self.config["display_duration"] - if not isinstance(duration, (int, float)) or duration <= 0: + if (not isinstance(duration, (int, float)) + or isinstance(duration, bool) or duration <= 0): self.logger.error("'display_duration' must be a positive number") return False diff --git a/src/plugin_system/compatibility.py b/src/plugin_system/compatibility.py new file mode 100644 index 00000000..9b676cd2 --- /dev/null +++ b/src/plugin_system/compatibility.py @@ -0,0 +1,292 @@ +"""One place that answers "can this plugin run on this core?". + +Two callers ask that question and they must not drift apart: + +- `PluginLoader._warn_if_incompatible` — at load time, **advisory**. A plugin + already on disk keeps loading regardless, because the guarded-import pattern + means most incompatibilities degrade rather than break. +- `PluginStoreManager.install_plugin` — at install/update time, **blocking**. + This is the point where refusing costs the user nothing (they keep the + version they already had) and allowing can cost them a plugin that fails to + load with only a log line to explain it. + +## The trustworthiness problem + +The core's own `__version__` has not always been right. `v3.1.0` was tagged +2026-05-31 while `src/__init__.py` still said `"1.0.0"`; the bump landed +2026-07-12. Devices installed from that release report `1.0.0` — below the +floor that essentially every published plugin declares. + +So a core reporting a version below `TRUSTWORTHY_FLOOR` is treated as +**unknown, not old**: it neither warns nor blocks. Blocking on it would be far +worse than the problem being solved — nearly every manifest in the ecosystem +floors at `2.0.0`, so a strict gate would stop those users installing *any* +plugin. They are unprotected until they update the core, which is also what +fixes their version string. See `docs/SPORTS_UNIFICATION.md`, phase B4. +""" + +from __future__ import annotations + +import re +from typing import Any, Dict, Optional, Tuple + +# Below this, the core's self-reported version is not evidence of anything. +# See the module docstring. +TRUSTWORTHY_FLOOR: Tuple[int, int, int] = (2, 0, 0) + + +def parse_semver(value: Any) -> Optional[Tuple[int, int, int]]: + """Parse ``X.Y.Z`` (extra parts and suffixes ignored) into a comparable + 3-tuple, or ``None`` when unparseable. A leading ``v`` is tolerated.""" + if not isinstance(value, str): + return None + text = value.strip().lstrip('v') + # Drop the prerelease/build suffix before scraping digits. Without this the + # scrape pulls them into the numbers: "3.2.0+build42" parsed as (3, 2, 42) + # and "3.2.0-rc1" as (3, 2, 1) -- a release candidate ranking *above* its + # own release, and a build of 3.2.0 failing an exact "3.2.0" match. + # + # Prereleases compare equal to their release here rather than below it. + # Full prerelease ordering is more than any caller needs, and equal is far + # closer to right than the old behaviour. + for sep in ('+', '-'): + head, found, _tail = text.partition(sep) + if found: + text = head + parts = text.split('.') + try: + nums = [int(''.join(ch for ch in p if ch.isdigit()) or 0) for p in parts[:3]] + except ValueError: + return None + while len(nums) < 3: + nums.append(0) + return tuple(nums) # type: ignore[return-value] + + +# `parse_semver` is deliberately lenient — it strips non-digits and yields +# (0, 0, 0) for a string with no numbers at all, which is fine for a floor +# (a floor of 0.0.0 never blocks anything) but wrong for a range, where the +# same leniency would turn an unreadable spec into a *refusal*. Range specs +# are therefore validated against this first, so garbage reads as "no +# evidence" rather than "incompatible". +_VERSION_TOKEN = re.compile(r"^v?\d+(\.\d+){0,2}(-[\w.-]+)?(\+[\w.-]+)?$") + + +def _parse_strict(value: str) -> Optional[Tuple[int, int, int]]: + """`parse_semver`, but ``None`` unless the string really looks like one.""" + if not isinstance(value, str) or not _VERSION_TOKEN.match(value.strip()): + return None + return parse_semver(value) + + +def _satisfies_range(core: Tuple[int, int, int], spec: str) -> Optional[bool]: + """Does ``core`` satisfy one `compatible_versions` entry? + + Returns ``None`` when the spec cannot be parsed — the caller treats that as + "no evidence" rather than as a refusal, so an unrecognised spelling never + costs a user a working install. + + Supports the forms `schema/manifest_schema.json` permits: `>=`, `<=`, `>`, + `<`, `~`, `^`, a bare exact version, and an inclusive `A - B` range. + Prerelease/build suffixes are tolerated and ignored, matching `parse_semver`. + """ + spec = spec.strip() + if not spec: + return None + + if " - " in spec: # inclusive range, e.g. "2.0.0 - 3.1.0" + low_raw, _, high_raw = spec.partition(" - ") + low, high = _parse_strict(low_raw), _parse_strict(high_raw) + if low is None or high is None: + return None + return low <= core <= high + + for op in (">=", "<=", ">", "<", "~", "^"): + if spec.startswith(op): + target = _parse_strict(spec[len(op):]) + if target is None: + return None + if op == ">=": + return core >= target + if op == "<=": + return core <= target + if op == ">": + return core > target + if op == "<": + return core < target + if op == "~": + # Patch-level changes only: >=X.Y.Z, =X.Y.Z, <(X+1).0.0 + return target <= core < (target[0] + 1, 0, 0) + + exact = _parse_strict(spec) + return None if exact is None else core == exact + + +def satisfies_compatible_versions( + manifest: Dict[str, Any], core: Tuple[int, int, int] +) -> Optional[bool]: + """Evaluate the manifest's `compatible_versions` array against ``core``. + + The array is a set of *alternatives*: satisfying any one entry means the + plugin declares itself compatible. Returns ``None`` when the field is + absent or no entry could be parsed, so callers can distinguish "declared + incompatible" from "did not say". + + This is the field `schema/manifest_schema.json` marks **required**, and it + is the only one that can express an upper bound — `ledmatrix_min_version` + is a floor and cannot say "not compatible with 4.x". + """ + specs = manifest.get('compatible_versions') + if not isinstance(specs, list) or not specs: + return None + + verdicts = [_satisfies_range(core, s) for s in specs if isinstance(s, str)] + parsed = [v for v in verdicts if v is not None] + if not parsed: + return None + return any(parsed) + + +def declared_min_version(manifest: Dict[str, Any]) -> Optional[str]: + """The core version this plugin says it needs, or ``None`` if it doesn't say. + + Checked in order of specificity. `ledmatrix_min` is the deprecated spelling + of `ledmatrix_min_version` (`store_manager._validate_manifest_fields` flags + it); both are read because a large share of published manifests still carry + the old one. + + Container types are validated rather than assumed. A hand-edited or + third-party manifest can carry `requires` as a list or `versions` as a + mapping, and both used to raise out of here (`AttributeError` and + `KeyError` respectively). That now matters far more than it did: the + untrustworthy-core branch of :func:`check` calls this for *every* manifest, + so one malformed file would take down the install path rather than just + itself. A shape we do not recognise means "no declared floor". + """ + declared = manifest.get('min_ledmatrix_version') + if not declared: + requires = manifest.get('requires') + if isinstance(requires, dict): + declared = requires.get('min_ledmatrix_version') + if declared: + return declared + + versions = manifest.get('versions') + if isinstance(versions, list) and versions and isinstance(versions[0], dict): + return (versions[0].get('ledmatrix_min_version') + or versions[0].get('ledmatrix_min')) + return None + + +def is_update_available(installed_version: str, latest_version: str) -> bool: + """Return True when the registry's ``latest_version`` is strictly newer + than the installed version. + + THE shared comparator for "should this plugin be updated?" — used by both + the web UI's update badge (`api_v3._is_plugin_update_available`) and the + store's `update_plugin` reinstall decision, so the two can never disagree. + + Uses PEP 440-aware comparison (``packaging``), which also normalizes + equivalent spellings: ``v1.2.0`` == ``1.2.0`` and ``1.2`` == ``1.2.0``, so + cosmetic differences never trigger a reinstall — and a locally modified + plugin whose version is *ahead* of the registry is never "updated" + (downgraded). If either version string can't be parsed the mismatch is + surfaced (True) so the user can reconcile, rather than silently hiding a + potential update. + """ + if not installed_version or not latest_version: + return False + if not isinstance(installed_version, str) or not isinstance(latest_version, str): + # A malformed manifest/registry can carry a number (1.2) or worse; + # packaging would raise TypeError. Surface the mismatch instead. + return True + if installed_version == latest_version: + return False + try: + from packaging.version import parse as _parse_version, InvalidVersion + except ImportError: + # packaging is a core dependency, but if it's somehow unavailable we + # can't compare semantically — surface the mismatch we already know + # exists (the two strings differ). + return True + try: + return _parse_version(latest_version) > _parse_version(installed_version) + except InvalidVersion: + # Unparseable version string: we can't tell direction, so surface the + # mismatch rather than silently hiding a potential update. + return True + + +def check(manifest: Dict[str, Any], core_version: str) -> Tuple[bool, Optional[str]]: + """Return ``(compatible, reason)``. + + Two fields can say a plugin is incompatible and **the more restrictive + wins**: + + - `compatible_versions` — the schema-required array of semver ranges, and + the only one that can express an upper bound. + - `ledmatrix_min_version` (or the deprecated `ledmatrix_min`) — the + per-release floor inside `versions[]`. + + They agree across every published manifest today except `7-segment-clock`, + but they *can* disagree, and a plugin that says `["2.0.0 - 2.9.9"]` means + "not compatible with 3.x" no matter what its floor says. + + ``compatible`` is False **only** on evidence: the core reports a parseable, + trustworthy version and a field genuinely excludes it. Every uncertain case + resolves to compatible — nothing declared, an unparseable version on either + side, or a core below `TRUSTWORTHY_FLOOR`. Refusing on a guess breaks a + working install, which is the more expensive mistake here. + + ``reason`` is user-facing text, present only when incompatible. + """ + current = parse_semver(core_version) + name = manifest.get('name') or manifest.get('id') or 'This plugin' + + if current is None or current < TRUSTWORTHY_FLOOR: + # The version is not evidence of what this core HAS. But a floor above + # the ecosystem baseline says the plugin needs modules that arrived + # *after* 2.0.0 — and a core reporting below that either is the v3.1.0 + # release (which ships __version__ = "1.0.0" and has none of the 3.2.0 + # modules) or is genuinely ancient. Either way it will not have them. + # + # This is the only protection available to that population: they cannot + # be told apart from a real 1.0.0 install, so the gate cannot reason + # about them, and the *plugin's* guarded-import fallback disappears at + # the B6 sunset. Refusing the install leaves them on the version they + # already run instead of handing them one that fails to load. + # + # Floors at or below 2.0.0 are still allowed, which is every manifest + # published today — so this does not lock anyone out of the store. + declared = declared_min_version(manifest) + needed = parse_semver(declared) + if needed is not None and needed > TRUSTWORTHY_FLOOR: + return False, ( + f"{name} requires LEDMatrix {declared} or newer. This system " + f"reports {core_version}, which is too old to identify " + f"reliably — update LEDMatrix, then install it." + ) + return True, None + + # Ranges first: they are the canonical field and can rule out a core that + # clears the floor. + if satisfies_compatible_versions(manifest, current) is False: + specs = ", ".join( + s for s in manifest.get('compatible_versions', []) if isinstance(s, str)) + return False, ( + f"{name} supports LEDMatrix {specs}, but this system is running " + f"{core_version}. Install a build in that range, or a plugin " + f"version that supports {core_version}." + ) + + declared = declared_min_version(manifest) + needed = parse_semver(declared) + if needed is not None and needed > current: + return False, ( + f"{name} requires LEDMatrix {declared} or newer, but this system is " + f"running {core_version}. Update LEDMatrix first, then install it." + ) + + return True, None diff --git a/src/plugin_system/plugin_health.py b/src/plugin_system/plugin_health.py index ecbec076..3908d9ca 100644 --- a/src/plugin_system/plugin_health.py +++ b/src/plugin_system/plugin_health.py @@ -7,7 +7,7 @@ and circuit breaker state. Provides automatic recovery mechanisms. import time import logging -from typing import Dict, Optional, Any +from typing import Dict, Optional, Any, Tuple from enum import Enum @@ -64,10 +64,48 @@ class PluginHealthTracker: cache_key, max_age=None, memory_ttl=0 if force_reload else None ) - if cached: - return cached - - # Default state + if isinstance(cached, dict) and cached: + # Complete it rather than trusting it: a persisted record can be + # missing fields the callers index directly (a partial write, a + # restored backup, an older schema), and returning it verbatim makes + # record_success / record_failure raise KeyError, which takes the + # display down in a restart loop that survives reboots because the + # bad entry is on disk. + state, repaired = self._repair_health_state(cached) + if repaired: + self.logger.warning( + f"Repaired health state for {plugin_id}: " + f"{sorted(repaired)} missing or invalid, using defaults for those." + ) + return state + + # Not a dict at all: written by something other than + # _save_health_state (a key collision, a corrupted entry). Nothing to + # salvage. + if cached is not None and not isinstance(cached, dict): + self.logger.warning( + f"Discarding malformed health state for {plugin_id}: expected " + f"dict, got {type(cached).__name__}. Falling back to defaults." + ) + + return self._default_health_state() + + def _save_health_state(self, plugin_id: str, state: Dict[str, Any]) -> None: + """Save health state to cache.""" + cache_key = self._get_health_key(plugin_id) + self.cache_manager.set(cache_key, state) # Persist indefinitely + self._health_state[plugin_id] = state + + # The fields callers index directly (state['circuit_state'] and friends). + # A cached dict missing any of them raises KeyError deep in record_success / + # record_failure, so the value is completed before it is handed out. + _COUNTER_FIELDS = ('consecutive_failures', 'total_failures', 'total_successes') + _TIMESTAMP_FIELDS = ('last_success_time', 'last_failure_time', + 'circuit_opened_time', 'half_open_start_time') + + @staticmethod + def _default_health_state() -> Dict[str, Any]: + """A fresh state with every field the callers expect.""" return { 'consecutive_failures': 0, 'total_failures': 0, @@ -77,15 +115,56 @@ class PluginHealthTracker: 'circuit_state': CircuitState.CLOSED.value, 'circuit_opened_time': None, 'half_open_start_time': None, - 'last_error': None + 'last_error': None, } - - def _save_health_state(self, plugin_id: str, state: Dict[str, Any]) -> None: - """Save health state to cache.""" - cache_key = self._get_health_key(plugin_id) - self.cache_manager.set(cache_key, state) # Persist indefinitely - self._health_state[plugin_id] = state - + + @classmethod + def _repair_health_state(cls, cached: Dict[str, Any]) -> Tuple[Dict[str, Any], list]: + """Return `cached` completed against the defaults, plus what was repaired. + + Per-field rather than all-or-nothing: a record that has real failure + counts but is missing `last_error` should keep the counts, not be reset + to healthy. Only values that are absent or the wrong type fall back to + the default, so a partial or older-schema record survives with whatever + it does carry, while every field the callers index is guaranteed present + and of a usable type. + """ + state = cls._default_health_state() + repaired = [] + for field, default in state.items(): + if field not in cached: + repaired.append(field) + continue + value = cached[field] + if field in cls._COUNTER_FIELDS: + ok = isinstance(value, int) and not isinstance(value, bool) and value >= 0 + elif field in cls._TIMESTAMP_FIELDS: + # bool is a subclass of int, so True would pass as a timestamp + # and then compare as 1.0 -- expiring a cooldown the instant it + # opens, or (False) making the elapsed check never fire. + ok = value is None or ( + isinstance(value, (int, float)) and not isinstance(value, bool) + ) + elif field == 'circuit_state': + # Membership first requires the value to be hashable: a list or + # dict here would raise TypeError out of the repair itself, + # which is the crash this whole path exists to prevent. + ok = isinstance(value, str) and value in { + member.value for member in CircuitState + } + else: # last_error + ok = value is None or isinstance(value, str) + if ok: + state[field] = value + else: + repaired.append(field) + # Anything the schema has since grown (degraded, degraded_reason) is + # read with .get() by its callers, so carry it through untouched. + for field, value in cached.items(): + if field not in state: + state[field] = value + return state, repaired + def get_health_state(self, plugin_id: str, force_reload: bool = False) -> Dict[str, Any]: """Get current health state for a plugin. @@ -99,11 +178,21 @@ class PluginHealthTracker: ) return self._health_state[plugin_id] + # Fields the circuit breaker is rebuilt from after a restart. Everything + # else in a health record is reporting, read only for display. + _DURABLE_FIELDS = ('consecutive_failures', 'circuit_state', + 'circuit_opened_time', 'half_open_start_time') + + def _durable(self, state: Dict[str, Any]) -> tuple: + """The part of a health record whose loss would change behaviour.""" + return tuple(state.get(field) for field in self._DURABLE_FIELDS) + def record_success(self, plugin_id: str) -> None: """Record a successful plugin execution.""" state = self.get_health_state(plugin_id) current_time = time.time() - + durable_before = self._durable(state) + # Reset consecutive failures state['consecutive_failures'] = 0 state['total_successes'] = state.get('total_successes', 0) + 1 @@ -119,9 +208,20 @@ class PluginHealthTracker: # Shouldn't happen, but handle it state['circuit_state'] = CircuitState.CLOSED.value state['circuit_opened_time'] = None - - self._save_health_state(plugin_id, state) - + + # A healthy plugin reports success every cycle, and in that steady state + # the only fields changed above are a counter and a timestamp that + # nothing reads back after a restart. Persisting them anyway rewrites a + # small file per plugin per cycle: on a rig running 24 plugins, a + # five-minute sample measured 22 rewrites, about 4.4 a minute or 6,300 a + # day. Those land on an SD card, where the cost is an erase-block cycle + # rather than the 400 bytes involved, and where wear is what eventually + # kills the card. + # In-memory state is still updated every time, so the health API and web + # UI show exactly what they did before; only the write is skipped. + if self._durable(state) != durable_before: + self._save_health_state(plugin_id, state) + def record_failure(self, plugin_id: str, error: Optional[Exception] = None) -> None: """Record a failed plugin execution.""" state = self.get_health_state(plugin_id) diff --git a/src/plugin_system/plugin_loader.py b/src/plugin_system/plugin_loader.py index 733bfde1..d8ae0e3d 100644 --- a/src/plugin_system/plugin_loader.py +++ b/src/plugin_system/plugin_loader.py @@ -14,7 +14,7 @@ import sys import subprocess import threading from pathlib import Path -from typing import Dict, Any, Optional, Tuple, Type +from typing import Dict, Any, List, Optional, Tuple, Type import logging from packaging.requirements import InvalidRequirement, Requirement @@ -45,6 +45,76 @@ def requirements_has_real_deps(requirements_file: str) -> bool: return False +def _extra_dependencies(dist_name: str, extras) -> Optional[List[Requirement]]: + """Dependencies a distribution declares *only* behind the given extras. + + Returns None when the installed metadata cannot be read or parsed, so the + caller can fall back to running pip rather than assuming anything. + """ + try: + meta = importlib.metadata.metadata(dist_name) + except importlib.metadata.PackageNotFoundError: + return None + + gated: List[Requirement] = [] + for raw in meta.get_all('Requires-Dist') or []: + try: + dep = Requirement(raw) + except InvalidRequirement: + return None + if dep.marker is None: + continue + # Keep only what the distribution gates behind an extra we asked for: + # satisfied when `extra` is that name, but not when no extra is + # requested. A marker that holds either way (python_version, sys_platform) + # belongs to the base install and is already covered by the version check. + if dep.marker.evaluate({'extra': ''}): + continue + if any(dep.marker.evaluate({'extra': extra}) for extra in extras): + gated.append(dep) + return gated + + +def _extras_are_satisfied(req: Requirement, _visited: Optional[set] = None) -> bool: + """Check the dependencies pulled in by req's extras are installed. + + Follows extras through nested extras. A gated dependency can itself request + one (`requests[socks]`), and checking only that `requests` is installed at + an acceptable version says nothing about whether the socks extra's own + dependency is there -- so the caller would skip pip and the plugin would + fail at import instead. Plain dependencies are still checked one level + deep, which is all that is needed to tell "the extra was installed" from + "the extra was never installed". + + `_visited` carries the (distribution, extras) pairs already seen, so a + dependency cycle between extras terminates instead of recursing forever. + Anything unreadable returns False, so the caller still falls through to pip. + """ + if _visited is None: + _visited = set() + marker = (req.name.lower(), frozenset(e.lower() for e in req.extras)) + if marker in _visited: + # Already accounted for higher up the chain; treating a cycle as + # satisfied here is safe because the outer frame still has to pass. + return True + _visited.add(marker) + + gated = _extra_dependencies(req.name, req.extras) + if gated is None: + return False + + for dep in gated: + try: + dep_version = importlib.metadata.version(dep.name) + except importlib.metadata.PackageNotFoundError: + return False + if dep.specifier and not dep.specifier.contains(dep_version, prereleases=True): + return False + if dep.extras and not _extras_are_satisfied(dep, _visited): + return False + return True + + def requirements_are_satisfied(requirements_file: str) -> bool: """ Check whether every real requirement line in requirements.txt is already @@ -76,9 +146,6 @@ def requirements_are_satisfied(requirements_file: str) -> bool: except InvalidRequirement: return False - if req.extras: - return False # verifying extras' sub-dependencies isn't worth it here - if req.marker is not None and not req.marker.evaluate(): continue # not applicable on this platform/interpreter @@ -90,6 +157,9 @@ def requirements_are_satisfied(requirements_file: str) -> bool: if req.specifier and not req.specifier.contains(installed_version, prereleases=True): return False + if req.extras and not _extras_are_satisfied(req): + return False + return True @@ -702,34 +772,25 @@ class PluginLoader: newer than the running core. Advisory only — never raises — so a plugin that guards optional features with try/except keeps working. """ - declared = ( - manifest.get('min_ledmatrix_version') - or manifest.get('requires', {}).get('min_ledmatrix_version') - ) - if not declared: - versions = manifest.get('versions') or [] - if versions and isinstance(versions[0], dict): - declared = (versions[0].get('ledmatrix_min_version') - or versions[0].get('ledmatrix_min')) - needed = self._parse_semver(declared) - if needed is None: + from src import __version__ as core_version + from src.plugin_system import compatibility + + compatible, _reason = compatibility.check(manifest, core_version) + if compatible: + # Distinguish "fine" from "couldn't tell" for anyone reading logs: + # a core below the trustworthy floor is skipped, not cleared. + current = compatibility.parse_semver(core_version) + if current is None or current < compatibility.TRUSTWORTHY_FLOOR: + self.logger.debug( + "Skipping version compatibility check for %s: core __version__ " + "(%s) is below the ecosystem floor", plugin_id, core_version) return - from src import __version__ as core_version - current = self._parse_semver(core_version) - # Anti-spam guard: if the core's own version number is stale (below - # the ecosystem floor every shipped plugin declares), comparing would - # warn on nearly everything — skip with a debug note instead. - if current is None or current < (2, 0, 0): - self.logger.debug( - "Skipping version compatibility check for %s: core __version__ " - "(%s) is below the ecosystem floor", plugin_id, core_version) - return - if needed > current: - self.logger.warning( - "Plugin %s declares min LEDMatrix version %s but this core is %s — " - "features it relies on may be missing; update the core or expect " - "degraded fallbacks", plugin_id, declared, core_version) + declared = compatibility.declared_min_version(manifest) + self.logger.warning( + "Plugin %s declares min LEDMatrix version %s but this core is %s — " + "features it relies on may be missing; update the core or expect " + "degraded fallbacks", plugin_id, declared, core_version) def load_plugin( self, diff --git a/src/plugin_system/plugin_manager.py b/src/plugin_system/plugin_manager.py index 4a571658..543017c9 100644 --- a/src/plugin_system/plugin_manager.py +++ b/src/plugin_system/plugin_manager.py @@ -71,11 +71,15 @@ class PluginManager: self.plugin_loader = PluginLoader(logger=self.logger) self.plugin_executor = PluginExecutor(default_timeout=30.0, logger=self.logger) self.state_manager = PluginStateManager(logger=self.logger) - self.schema_manager = SchemaManager(plugins_dir=self.plugins_dir, logger=self.logger) + self.schema_manager = SchemaManager(plugins_dir=self.plugins_dir, logger=self.logger, + config_manager=self.config_manager) # Lock protecting plugin_manifests and plugin_directories from # concurrent mutation (background reconciliation) and reads (requests). self._discovery_lock = threading.RLock() + #: Directories already reported as unloadable, so the warning is + #: emitted once rather than on every discovery scan. + self._skip_reported: set = set() # Lock protecting plugin_last_update from concurrent mutation/iteration. # It's written from run_scheduled_updates()/update_all_plugins() (main @@ -113,9 +117,26 @@ class PluginManager: self._update_queue: "queue.Queue[Optional[Tuple[str, float]]]" = queue.Queue() self._pending_updates: set = set() self._pending_lock = threading.Lock() + # Serializes the "is this plugin eligible?" -> "claim it (RUNNING)" + # transition. Two schedulers run concurrently in practice — the render + # loop's _tick_plugin_updates() and Vegas mode's vegas-plugin-tick + # daemon thread, which is never joined — so without this both can + # observe ENABLED and both call update() on the same plugin. Held only + # across the check and the state transition, never across update() + # itself: that would serialize slow plugins behind each other and + # reintroduce the stall the async worker exists to avoid. + self._reservation_lock = threading.Lock() self._plugin_locks: Dict[str, threading.Lock] = {} self._plugin_locks_guard = threading.Lock() self._update_worker: Optional[threading.Thread] = None + # Plugin ids whose update() has finished since the last time anyone + # asked. Updates are dispatched to a worker thread, so a caller that + # wants to know "whose data just changed" cannot learn it by diffing + # plugin_last_update around run_scheduled_updates() -- that call only + # enqueues, and the timestamp is stamped later, on the worker. See + # run_scheduled_updates_with_changes(). + self._completed_updates: set = set() + self._completed_updates_lock = threading.Lock() self._synchronous_updates = False if self.config_manager is not None: try: @@ -178,18 +199,59 @@ class PluginManager: continue manifest_path = item / "manifest.json" - if manifest_path.exists(): - try: - with open(manifest_path, 'r', encoding='utf-8') as f: - manifest = json.load(f) - plugin_id = manifest.get('id') - if plugin_id: - plugin_ids.append(plugin_id) - new_manifests[plugin_id] = manifest - new_directories[plugin_id] = item - except (json.JSONDecodeError, PermissionError, OSError) as e: - self.logger.warning("Error reading manifest from %s: %s", manifest_path, e, exc_info=True) - continue + if not manifest_path.exists(): + # Once per directory per process. Discovery runs on every + # web UI page load and every config reconcile, so warning + # unconditionally would put a line in the journal each + # time someone opened a page -- the same log-volume + # problem this is meant to help diagnose. + # A directory here that carries no manifest is not a + # plugin. Said once, because the alternative is a plugin + # that is enabled in config, enabled in plugin state, + # present on disk, and simply absent from the running + # process with nothing anywhere to say why. Working that + # out afterwards means reading cache-file mtimes. + if item.name not in self._skip_reported: + self._skip_reported.add(item.name) + self.logger.warning( + "Skipping %s: no manifest.json, so it cannot be " + "loaded as a plugin", item.name) + continue + try: + with open(manifest_path, 'r', encoding='utf-8') as f: + manifest = json.load(f) + except (json.JSONDecodeError, PermissionError, OSError) as e: + self.logger.warning("Error reading manifest from %s: %s", manifest_path, e, exc_info=True) + continue + + # json.load accepts any JSON value, so a manifest holding + # null, [] or "text" parses and then raises AttributeError on + # .get(). Nothing here catches that -- the outer handler takes + # OSError/PermissionError only -- so a single malformed + # manifest aborted the whole scan and every other plugin on + # disk, however healthy, silently failed to register. + if not isinstance(manifest, dict): + if item.name not in self._skip_reported: + self._skip_reported.add(item.name) + self.logger.warning( + "Skipping %s: its manifest.json is %s, not a JSON " + "object", item.name, type(manifest).__name__) + continue + + plugin_id = manifest.get('id') + if not plugin_id: + # Parsed but unusable. This was the quietest path of all: + # the manifest is read successfully and then dropped. + if item.name not in self._skip_reported: + self._skip_reported.add(item.name) + self.logger.warning( + "Skipping %s: its manifest.json has no \"id\", so " + "there is nothing to register it under", item.name) + continue + + plugin_ids.append(plugin_id) + new_manifests[plugin_id] = manifest + new_directories[plugin_id] = item except (OSError, PermissionError) as e: self.logger.error("Error scanning directory %s: %s", directory, e, exc_info=True) @@ -395,6 +457,37 @@ class PluginManager: self.state_manager.set_state(plugin_id, PluginState.ERROR, error=e) return False + #: Config keys the **core** reads out of a plugin's own config block. The + #: plugin never declares them, so a schema with + #: ``"additionalProperties": false`` — 37 of the 42 published ones — reports + #: them as violations and the plugin gets flagged degraded in the web UI for + #: using a documented core feature. + #: + #: Listed explicitly rather than matched on a ``vegas_`` prefix, because + #: ``vegas_mode`` is the opposite case: plugins *do* declare that one, and a + #: prefix rule would silently stop validating it. + #: + #: Read by: ``vegas_mode/plugin_adapter.py`` (``vegas_width_pct``, + #: ``vegas_overflow``) and ``base_plugin.py`` (``vegas_max_width_screens``). + CORE_OWNED_CONFIG_KEYS = frozenset({ + 'vegas_width_pct', + 'vegas_overflow', + 'vegas_max_width_screens', + }) + + def _strip_core_owned_keys(self, config: Dict[str, Any]) -> Dict[str, Any]: + """A shallow copy of ``config`` without the core's own tuning keys. + + Only the top level is touched, and only when such a key is present, so + the common case allocates nothing extra. + """ + if not isinstance(config, dict): + return config + if not self.CORE_OWNED_CONFIG_KEYS.intersection(config): + return config + return {k: v for k, v in config.items() + if k not in self.CORE_OWNED_CONFIG_KEYS} + def _validate_config_schema_soft(self, plugin_id: str, config: Dict[str, Any]) -> None: """Validate a plugin's config against its JSON schema — warn/degrade only. @@ -419,7 +512,7 @@ class PluginManager: try: is_valid, errors = self.schema_manager.validate_config_against_schema( - config, schema, plugin_id + self._strip_core_owned_keys(config), schema, plugin_id ) except Exception as e: # pragma: no cover - defensive # Validation machinery itself failed — do not penalise the plugin. @@ -777,25 +870,69 @@ class PluginManager: if self.health_tracker and self.health_tracker.should_skip_plugin(plugin_id): continue - # Check if plugin can execute - if not self.state_manager.can_execute(plugin_id): - continue - interval = self._get_plugin_update_interval(plugin_id, plugin_instance) if interval is None: continue - with self._plugin_last_update_lock: - last_update = self.plugin_last_update.get(plugin_id, 0.0) + # Eligibility check, due check and the RUNNING transition happen + # together, so a concurrent scheduler cannot claim the same plugin. + if not self._reserve_for_update(plugin_id, current_time, interval): + continue - if last_update == 0.0 or (current_time - last_update) >= interval: - if self._synchronous_updates: - # Kill-switch path: the original inline execution - # (blocks the caller until update() completes/times out) - self.state_manager.set_state(plugin_id, PluginState.RUNNING) - self._execute_update_now(plugin_id, plugin_instance, current_time) - else: - self._enqueue_update(plugin_id, current_time) + if self._synchronous_updates: + # Kill-switch path: the original inline execution + # (blocks the caller until update() completes/times out) + self._execute_update_now(plugin_id, plugin_instance, current_time) + else: + self._enqueue_update(plugin_id, current_time) + + def _reserve_for_update( + self, + plugin_id: str, + current_time: Optional[float] = None, + interval: Optional[float] = None, + ) -> bool: + """Atomically claim a plugin for update, returning True if we won it. + + can_execute() and the RUNNING transition have to happen under one lock. + As two separate calls, two scheduler threads can both see ENABLED and + both go on to run the same plugin's update() concurrently — unsafe for + any plugin that isn't reentrant (shared mutable state, a non-thread-safe + HTTP session or cache). + + The due-time check is inside the lock too. Leaving it outside would let + a second thread that had already decided "due" claim the plugin the + instant the first finished, running update() twice in one interval. + + Args: + plugin_id: Plugin to claim. + current_time: Now, for the due check. Omit to skip that check. + interval: Seconds between updates. Omit to skip the due check. + + Returns: + True if this caller reserved the plugin and must dispatch it, + False if it is ineligible, not yet due, or already claimed. + """ + with self._reservation_lock: + if not self.state_manager.can_execute(plugin_id): + return False + + if current_time is not None and interval is not None: + with self._plugin_last_update_lock: + last_update = self.plugin_last_update.get(plugin_id, 0.0) + if last_update != 0.0 and (current_time - last_update) < interval: + return False + + self.state_manager.set_state(plugin_id, PluginState.RUNNING) + return True + + def _release_reservation(self, plugin_id: str) -> None: + """Hand a claimed plugin back when it never got dispatched. + + Without this a plugin reserved but not queued would sit in RUNNING + forever, and can_execute() would refuse it on every later tick. + """ + self.state_manager.set_state(plugin_id, PluginState.ENABLED) def get_plugin_lock(self, plugin_id: str) -> threading.Lock: """Per-plugin lock keeping update() and display() mutually exclusive. @@ -812,16 +949,40 @@ class PluginManager: return lock def _enqueue_update(self, plugin_id: str, scheduled_time: float) -> None: - """Queue a due update for the background worker (dedup while pending).""" + """Queue an already-reserved update for the background worker. + + The caller has reserved the plugin (RUNNING), which is what blocks + re-entry and shows the truthful state in the web UI while the item + waits its turn. The pending set stays as a second line of defence; if + it ever fires the reservation has to be handed back, or the plugin + would sit in RUNNING with nothing queued to release it. + """ with self._pending_lock: if plugin_id in self._pending_updates: + self.logger.warning( + "Plugin %s reserved for update but already queued; " + "releasing the reservation", plugin_id) + self._release_reservation(plugin_id) return self._pending_updates.add(plugin_id) - # RUNNING is set at enqueue time so can_execute() blocks re-entry and - # the web UI shows the truthful state while the item waits its turn. - self.state_manager.set_state(plugin_id, PluginState.RUNNING) - self._ensure_update_worker() - self._update_queue.put((plugin_id, scheduled_time)) + try: + self._ensure_update_worker() + self._update_queue.put((plugin_id, scheduled_time)) + except Exception as exc: # pylint: disable=broad-except + # Thread.start() raises RuntimeError when the OS refuses a new + # thread — a real condition on a Pi under memory pressure. Nothing + # is queued to release the plugin at that point, so the claim has to + # be undone here, or it sits in RUNNING with nothing to clear it and + # can_execute() refuses it for the rest of the process. Swallowed + # rather than raised so the remaining plugins in this tick still get + # their turn. + self.logger.error( + "Could not queue update for plugin %s (%s: %s); releasing the " + "reservation so the next tick can retry", + plugin_id, type(exc).__name__, exc, exc_info=True) + with self._pending_lock: + self._pending_updates.discard(plugin_id) + self._release_reservation(plugin_id) def _ensure_update_worker(self) -> None: if self._update_worker is not None and self._update_worker.is_alive(): @@ -906,9 +1067,18 @@ class PluginManager: return finished['done'] = True try: + # Drop the queue reservation *before* the state goes back to + # ENABLED. The other order leaves a window where a scheduler + # sees ENABLED, reserves the plugin, then finds it still in + # _pending_updates -- the enqueue is dropped and the plugin + # would sit in RUNNING with nothing left to release it. + if lock is not None: + with self._pending_lock: + self._pending_updates.discard(plugin_id) if success: with self._plugin_last_update_lock: self.plugin_last_update[plugin_id] = scheduled_time + self._note_update_completed(plugin_id) self.state_manager.record_update(plugin_id) self.state_manager.set_state(plugin_id, PluginState.ENABLED) if self.health_tracker: @@ -918,8 +1088,6 @@ class PluginManager: finally: if lock is not None: lock.release() - with self._pending_lock: - self._pending_updates.discard(plugin_id) if lock is None: # Synchronous / no-lock path: unchanged behavior. @@ -975,28 +1143,41 @@ class PluginManager: def run_scheduled_updates_with_changes(self, current_time: Optional[float] = None) -> List[str]: """ - Like run_scheduled_updates(), but also returns the plugin_ids whose - plugin_last_update timestamp actually advanced during this call. + Like run_scheduled_updates(), but also reports which plugins have + fresh data -- the ids whose update() has finished since the last + call, not necessarily the ones enqueued by this one. - The before/after snapshots and the update pass itself are each - individually lock-protected against concurrent plugin_last_update - mutation (Vegas mode calls this from its own background - update-tick thread, racing the main render loop's plugin updates), - so callers get an atomic "who got fresh data" answer without - reaching into plugin_last_update themselves. The lock is not held - across the update pass so slow/blocking plugin update() calls don't - serialize against other plugin_last_update readers. + That distinction is the whole point. This used to snapshot + plugin_last_update, call run_scheduled_updates(), and diff. But + run_scheduled_updates() only *enqueues*: the work runs on the + update worker and the timestamp is stamped there, after this method + has already returned. The two snapshots were therefore always + identical and the result was always empty, so Vegas never learned + that any plugin's data had changed and kept scrolling whatever a + segment was first built from -- last night's live game still drawn + as live the next morning. The only path that ever worked was the + synchronous kill-switch, where update() runs inline. + + Reporting completions instead of enqueues costs a poll's worth of + latency (the Vegas tick runs every ~4s) and is correct regardless of + which side of the queue the work lands on. """ - with self._plugin_last_update_lock: - old_times = dict(self.plugin_last_update) - self.run_scheduled_updates(current_time) + return self.drain_completed_updates() - with self._plugin_last_update_lock: - return [ - plugin_id for plugin_id, new_time in self.plugin_last_update.items() - if new_time > old_times.get(plugin_id, 0.0) - ] + def _note_update_completed(self, plugin_id: str) -> None: + """Record that a plugin's update() finished, for the next poll.""" + with self._completed_updates_lock: + self._completed_updates.add(plugin_id) + + def drain_completed_updates(self) -> List[str]: + """Return and clear the plugin ids whose update() has since finished.""" + with self._completed_updates_lock: + if not self._completed_updates: + return [] + done = sorted(self._completed_updates) + self._completed_updates.clear() + return done def update_all_plugins(self) -> None: """ @@ -1010,18 +1191,18 @@ class PluginManager: if not hasattr(plugin_instance, "update"): continue - # Check if plugin can execute - if not self.state_manager.can_execute(plugin_id): + # Eligibility check and the RUNNING transition together, so a + # concurrent scheduler cannot claim the same plugin (see + # _reserve_for_update). + if not self._reserve_for_update(plugin_id): continue - - # Update state to RUNNING - self.state_manager.set_state(plugin_id, PluginState.RUNNING) - + try: success = self.plugin_executor.execute_update(plugin_instance, plugin_id) if success: with self._plugin_last_update_lock: self.plugin_last_update[plugin_id] = time.time() + self._note_update_completed(plugin_id) self.state_manager.record_update(plugin_id) self.state_manager.set_state(plugin_id, PluginState.ENABLED) else: diff --git a/src/plugin_system/resource_monitor.py b/src/plugin_system/resource_monitor.py index 42ded4ec..1b63d74e 100644 --- a/src/plugin_system/resource_monitor.py +++ b/src/plugin_system/resource_monitor.py @@ -9,7 +9,7 @@ import time import logging import threading from typing import Dict, Optional, Any, Callable -from dataclasses import dataclass, field +from dataclasses import dataclass, field, fields try: import psutil @@ -49,6 +49,20 @@ class ResourceMetrics: self.total_execution_time = self.total_execution_time / self.call_count +#: How often a plugin's metrics are written to the cache, in seconds. +#: +#: Persisting on every call meant a small file rewritten roughly nine times a +#: minute per plugin. On a rig with fourteen active plugins that was ~126 +#: writes a minute for metrics alone, and since each ~350-byte file costs a +#: 4KB block plus an ext4 journal entry, it dominated the device's write +#: volume -- on an SD card, which wears out. +#: +#: The in-memory copy stays authoritative and exact; only the cross-process +#: snapshot the web UI reads is delayed, and telemetry up to half a minute old +#: is still a fair description of a long-running plugin. +_METRICS_PERSIST_INTERVAL = 30.0 + + class PluginResourceMonitor: """ Monitors resource usage for plugins. @@ -75,6 +89,10 @@ class PluginResourceMonitor: # Resource metrics per plugin self._metrics: Dict[str, ResourceMetrics] = {} self._limits: Dict[str, ResourceLimits] = {} + # When each plugin's metrics last reached the cache. Metrics change on + # every call, so they cannot be de-duplicated the way health state can; + # they are rate-limited instead. See _METRICS_PERSIST_INTERVAL. + self._metrics_persisted_at: Dict[str, float] = {} # Thread-local storage for execution tracking self._local = threading.local() @@ -102,6 +120,66 @@ class PluginResourceMonitor: "psutil not available - resource monitoring will be limited to execution time only" ) + def _metrics_from_cache(self, plugin_id: str, cached: Any) -> "ResourceMetrics": + """Build metrics from a cached record, ignoring anything unrecognised. + + ResourceMetrics(**cached) raises TypeError on a single unexpected key, + and that exception escapes into plugin_manager, which reports it as + "plugin operation failed". Every plugin fails, and the plugin + system never finishes initialising. + + Seen on a live rig: every plugin failing with + + ResourceMetrics.__init__() got an unexpected keyword argument + 'consecutive_failures' + + which is a plugin_health field, not a metrics one. How a health-shaped + record came to sit under a plugin_metrics key on that machine is not + established -- a restored backup that mixed two machines' caches is the + likeliest explanation -- but the loader should not be brittle enough for + it to matter. plugin_health already repairs its records field by field + rather than trusting whatever is on disk; this does the same. + + Unknown keys are dropped and named once, so a genuine schema change is + visible in the log instead of silently discarded. + """ + if not isinstance(cached, dict): + self.logger.warning( + "Ignoring cached metrics for %s: expected a mapping, got %s", + plugin_id, type(cached).__name__) + return ResourceMetrics() + + known = {f.name for f in fields(ResourceMetrics)} + unknown = sorted(set(cached) - known) + if unknown: + self.logger.warning( + "Dropping unrecognised field(s) from cached metrics for %s: %s", + plugin_id, ", ".join(unknown)) + # A dataclass does not enforce its annotations, so + # ResourceMetrics(call_count="not a number") builds happily and only + # blows up later, deep inside monitor_call ("can only concatenate str + # (not \"int\") to str"). Coerce here, where there is still a cache + # key to name in the warning. + declared = {f.name: f.type for f in fields(ResourceMetrics)} + usable = {} + for key, value in cached.items(): + if key not in known: + continue + try: + usable[key] = int(value) if declared[key] in ('int', int) else float(value) + except (TypeError, ValueError): + self.logger.warning( + "Cached metrics for %s have a bad %s (%r); starting fresh", + plugin_id, key, value) + return ResourceMetrics() + try: + return ResourceMetrics(**usable) + except (TypeError, ValueError) as e: + self.logger.warning( + "Cached metrics for %s unusable (%s); starting fresh", + plugin_id, e) + return ResourceMetrics() + def _get_metrics_key(self, plugin_id: str) -> str: """Get cache key for plugin metrics.""" return f"plugin_metrics:{plugin_id}" @@ -126,7 +204,7 @@ class PluginResourceMonitor: cache_key, max_age=None, memory_ttl=0 if force_reload else None ) if cached: - metrics = ResourceMetrics(**cached) + metrics = self._metrics_from_cache(plugin_id, cached) else: metrics = ResourceMetrics() self._metrics[plugin_id] = metrics @@ -232,18 +310,8 @@ class PluginResourceMonitor: # CPU is harder to measure per-call, so we track it separately metrics.cpu_percent = self._get_process_cpu_percent() - # Persist metrics - cache_key = self._get_metrics_key(plugin_id) - self.cache_manager.set(cache_key, { - 'memory_mb': metrics.memory_mb, - 'cpu_percent': metrics.cpu_percent, - 'execution_time': metrics.execution_time, - 'call_count': metrics.call_count, - 'total_execution_time': metrics.total_execution_time, - 'max_execution_time': metrics.max_execution_time, - 'min_execution_time': metrics.min_execution_time if metrics.min_execution_time != float('inf') else 0.0, - 'last_update_time': metrics.last_update_time - }) + # Persist metrics, at most once per interval per plugin. + self._persist_metrics(plugin_id, metrics) # Check limits if limits: @@ -363,6 +431,44 @@ class PluginResourceMonitor: summaries[plugin_id] = self.get_metrics_summary(plugin_id) return summaries + def _persist_metrics(self, plugin_id: str, metrics: ResourceMetrics, + force: bool = False) -> None: + """Write a plugin's metrics to the cache, at most once per interval. + + Caller must hold ``self._lock``. + """ + # Monotonic, not wall clock: these devices have no RTC, so the clock + # jumps by however far off boot-time was the moment NTP first syncs. + # A forward jump would allow an early write, a backward one would + # stall the snapshot well past the interval. + # + # The sentinel for "never written" is None, not 0.0. monotonic() is + # time since boot on Linux, and systemd starts this service *at* boot, + # so `now - 0.0 < 30` was true for the first half-minute of every + # single run -- the throttle swallowed the very first snapshot, which + # is the one that matters most after a restart. + now = time.monotonic() + last_written = self._metrics_persisted_at.get(plugin_id) + if (not force and last_written is not None + and now - last_written < _METRICS_PERSIST_INTERVAL): + return + cache_key = self._get_metrics_key(plugin_id) + self.cache_manager.set(cache_key, { + 'memory_mb': metrics.memory_mb, + 'cpu_percent': metrics.cpu_percent, + 'execution_time': metrics.execution_time, + 'call_count': metrics.call_count, + 'total_execution_time': metrics.total_execution_time, + 'max_execution_time': metrics.max_execution_time, + 'min_execution_time': (metrics.min_execution_time + if metrics.min_execution_time != float('inf') + else 0.0), + 'last_update_time': metrics.last_update_time, + }) + # Only after the write lands. Marking it first would mean a failed + # set() bought the next interval's silence without leaving a snapshot. + self._metrics_persisted_at[plugin_id] = now + def reset_metrics(self, plugin_id: str) -> None: """Reset metrics for a plugin.""" with self._lock: @@ -370,4 +476,7 @@ class PluginResourceMonitor: self._metrics[plugin_id] = ResourceMetrics() cache_key = self._get_metrics_key(plugin_id) self.cache_manager.delete(cache_key) + # Let the next call persist immediately rather than leaving the + # deleted key absent for the rest of the interval. + self._metrics_persisted_at.pop(plugin_id, None) diff --git a/src/plugin_system/saved_repositories.py b/src/plugin_system/saved_repositories.py index bd0e1872..c8da3c5b 100644 --- a/src/plugin_system/saved_repositories.py +++ b/src/plugin_system/saved_repositories.py @@ -6,6 +6,7 @@ Manages saved GitHub repository URLs for easy plugin discovery and installation. import json import logging +import os from pathlib import Path from typing import List, Dict, Optional @@ -43,20 +44,45 @@ class SavedRepositoriesManager: return [] def _save_repositories(self) -> bool: - """Save repositories to file.""" + """Save repositories to file atomically. + + Writes to a temp file in the same directory and os.replace()s it + over the target, so a failed write can never truncate or + half-overwrite an existing saved_repositories.json. + """ + tmp_path = self.config_path.with_suffix(self.config_path.suffix + '.tmp') try: # Ensure directory exists self.config_path.parent.mkdir(parents=True, exist_ok=True) - - with open(self.config_path, 'w') as f: + + with open(tmp_path, 'w') as f: json.dump(self.repositories, f, indent=2) - + f.flush() + os.fsync(f.fileno()) + os.replace(tmp_path, self.config_path) + self.logger.info(f"Saved {len(self.repositories)} repositories to {self.config_path}") return True except Exception as e: self.logger.error(f"Error saving repositories: {e}") + try: + tmp_path.unlink(missing_ok=True) + except OSError: + pass return False + @staticmethod + def _clean_url(repo_url: str) -> str: + """Normalize a repo URL: strip whitespace, trailing slashes, and a + trailing ``.git`` suffix ONLY. (The old ``.replace('.git', '')`` + was an unanchored substring replace that mangled URLs merely + containing ``.git``, e.g. ``https://github.com/user/my.github.io``.) + """ + repo_url = repo_url.strip().rstrip('/') + if repo_url.endswith('.git'): + repo_url = repo_url[:-4] + return repo_url + def get_all(self) -> List[Dict[str, str]]: """Get all saved repositories.""" return self.repositories.copy() @@ -72,15 +98,14 @@ class SavedRepositoriesManager: Returns: True if added successfully """ - # Clean URL - repo_url = repo_url.strip().rstrip('/').replace('.git', '') - + repo_url = self._clean_url(repo_url) + # Check if already exists for repo in self.repositories: if repo.get('url') == repo_url: self.logger.warning(f"Repository already exists: {repo_url}") return False - + # Extract name from URL if not provided if not name: parts = repo_url.split('/') @@ -88,15 +113,20 @@ class SavedRepositoriesManager: name = parts[-1] else: name = repo_url - + # Add repository self.repositories.append({ 'url': repo_url, 'name': name, 'type': 'registry' if 'plugins.json' in repo_url or 'ledmatrix-plugins' in repo_url.lower() else 'single' }) - - return self._save_repositories() + + if not self._save_repositories(): + # Keep memory consistent with disk: a failed save must not leave + # a phantom entry that only this process can see. + self.repositories.pop() + return False + return True def remove(self, repo_url: str) -> bool: """ @@ -108,21 +138,25 @@ class SavedRepositoriesManager: Returns: True if removed successfully """ - # Clean URL - repo_url = repo_url.strip().rstrip('/').replace('.git', '') - - original_count = len(self.repositories) - self.repositories = [r for r in self.repositories if r.get('url') != repo_url] - - if len(self.repositories) < original_count: - return self._save_repositories() + repo_url = self._clean_url(repo_url) + + previous = self.repositories + remaining = [r for r in previous if r.get('url') != repo_url] + + if len(remaining) < len(previous): + self.repositories = remaining + if not self._save_repositories(): + # Failed save: restore so memory matches disk. + self.repositories = previous + return False + return True else: self.logger.warning(f"Repository not found: {repo_url}") return False def has(self, repo_url: str) -> bool: """Check if a repository is already saved.""" - repo_url = repo_url.strip().rstrip('/').replace('.git', '') + repo_url = self._clean_url(repo_url) return any(r.get('url') == repo_url for r in self.repositories) def get_registry_repositories(self) -> List[Dict[str, str]]: diff --git a/src/plugin_system/schema_manager.py b/src/plugin_system/schema_manager.py index 385b53fd..3db4a5a2 100644 --- a/src/plugin_system/schema_manager.py +++ b/src/plugin_system/schema_manager.py @@ -26,7 +26,25 @@ class SchemaManager: - Cache invalidation on plugin changes """ - def __init__(self, plugins_dir: Optional[Path] = None, project_root: Optional[Path] = None, logger: Optional[logging.Logger] = None): + # Plugin config keys that mean "where this device is". A plugin declaring + # any of these in its schema gets the device-wide ``location`` block from + # config.json as the *default* for that field, instead of whatever city the + # plugin author happened to ship. A value the user set on the plugin itself + # always wins -- this only ever replaces the schema default, so an explicit + # per-plugin location is still honoured. + # + # Only these fully-namespaced keys are substituted. A bare ``state`` or + # ``city`` key is deliberately left alone: plugins use those for unrelated + # things (ledmatrix-elections' ``state`` is a two-letter code, not a place + # name), and silently rewriting them would break those plugins. + DEVICE_LOCATION_KEYS: Dict[str, str] = { + 'location_city': 'city', + 'location_state': 'state', + 'location_country': 'country', + } + + def __init__(self, plugins_dir: Optional[Path] = None, project_root: Optional[Path] = None, + logger: Optional[logging.Logger] = None, config_manager: Optional[Any] = None): """ Initialize the Schema Manager. @@ -34,10 +52,14 @@ class SchemaManager: plugins_dir: Base plugins directory path project_root: Project root directory path logger: Optional logger instance + config_manager: Optional config manager, used to resolve the + device-wide ``location`` that seeds plugin location defaults. + Omitting it simply leaves schema defaults untouched. """ self.logger = logger or logging.getLogger(__name__) self.plugins_dir = plugins_dir self.project_root = project_root or Path.cwd() + self.config_manager = config_manager # Schema cache: plugin_id -> schema dict self._schema_cache: Dict[str, Dict[str, Any]] = {} @@ -115,7 +137,17 @@ class SchemaManager: if not isinstance(schema, dict): self.logger.error(f"Invalid schema format for {plugin_id}: not a dictionary") return None - + + # Expand any customization.x-style-elements declaration into the + # full per-element style blocks (font/size/color + layout + # offsets) the web-UI config form renders. No-op for schemas + # without the declaration; never raises. + try: + from src.element_style import expand_style_elements + schema = expand_style_elements(schema) + except ImportError: + pass + # Cache the schema self._schema_cache[plugin_id] = schema @@ -202,10 +234,70 @@ class SchemaManager: return defaults + def get_device_location(self) -> Optional[Dict[str, Any]]: + """ + Return the device-wide ``location`` block from config.json, or None. + + This is the City/State/Country the user sets once under General + settings. Returns None when there is no config manager wired, the + config can't be read, or no location has been configured. + """ + if self.config_manager is None: + return None + try: + config = self.config_manager.load_config() + except Exception as e: + # A config that can't be read must never stop defaults being + # generated -- the plugin's own schema defaults still apply. + self.logger.debug(f"Could not read device location from config: {e}") + return None + if not isinstance(config, dict): + return None + location = config.get('location') + return location if isinstance(location, dict) else None + + def apply_device_location(self, defaults: Dict[str, Any]) -> Dict[str, Any]: + """ + Replace location-shaped schema defaults with the device's own location. + + Without this, a plugin that ships ``"location_city": "Dallas"`` as its + schema default silently reports Dallas weather (and centres its radar + there) for every user who never opened that plugin's config form -- + even though they set their real city under General settings. The + substituted value is still only a *default*: ``merge_with_defaults`` + lets any per-plugin value the user saved win over it. + + Mutates and returns ``defaults`` for convenience. + """ + if not defaults: + return defaults + if not any(key in defaults for key in self.DEVICE_LOCATION_KEYS): + return defaults + + location = self.get_device_location() + if not location: + return defaults + + for key, field in self.DEVICE_LOCATION_KEYS.items(): + if key not in defaults: + continue + value = location.get(field) + # Only a non-empty string is a real answer; a blank or missing + # field means "not configured", which leaves the schema default. + if isinstance(value, str) and value.strip(): + defaults[key] = value.strip() + + return defaults + def generate_default_config(self, plugin_id: str, use_cache: bool = True) -> Dict[str, Any]: """ Generate default configuration for a plugin from its schema. + Location fields (see ``DEVICE_LOCATION_KEYS``) default to the device's + configured location rather than the plugin author's. That substitution + is applied on the way out rather than being cached, so changing the + device location takes effect without invalidating the defaults cache. + Args: plugin_id: Plugin identifier use_cache: If True, return cached defaults if available @@ -215,7 +307,7 @@ class SchemaManager: """ # Check cache first if use_cache and plugin_id in self._defaults_cache: - return self._defaults_cache[plugin_id].copy() + return self.apply_device_location(self._defaults_cache[plugin_id].copy()) schema = self.load_schema(plugin_id, use_cache=use_cache) if not schema: @@ -239,10 +331,11 @@ class SchemaManager: if 'live_priority' not in defaults: defaults['live_priority'] = schema.get('properties', {}).get('live_priority', {}).get('default', False) - # Cache the defaults + # Cache the defaults *before* the device location is layered on, so a + # later change to the device location is picked up by the next call. self._defaults_cache[plugin_id] = defaults.copy() - return defaults + return self.apply_device_location(defaults) def validate_config_against_schema(self, config: Dict[str, Any], schema: Dict[str, Any], plugin_id: Optional[str] = None) -> Tuple[bool, List[str]]: @@ -284,6 +377,19 @@ class SchemaManager: "type": "boolean", "default": False, "description": "Enable live priority takeover when plugin has live content" + }, + # Skin selection (docs/SKIN_SYSTEM.md). Deliberately NOT an + # enum here: validation must keep passing when a configured + # skin gets uninstalled (rendering falls back to built-in). + # The install-dependent enum is injected only at serve time + # (inject_skin_selector) for the web UI dropdown. + "skin": { + "type": ["string", "object", "null"], + "description": "Visual skin id, or a per-mode mapping like {\"live\": \"my-skin\"}" + }, + "skin_options": { + "type": "object", + "description": "Options passed through to the selected skin" } } @@ -354,6 +460,53 @@ class SchemaManager: self.logger.error(error_msg) return False, [error_msg] + def inject_skin_selector(self, schema: Dict[str, Any], plugin_id: str, + current_value: Any = None) -> Dict[str, Any]: + """Return a copy of a plugin's schema with a "skin" dropdown added + when installed skins target this plugin (docs/SKIN_SYSTEM.md). + + Serve-time only — validation never sees this enum, so a config + referencing an uninstalled skin stays valid (rendering falls back + to the built-in layout). The currently-configured value is always + included in the enum for the same reason: the dropdown must be able + to display a selection whose skin was removed. + """ + # A per-mode mapping ({"live": ..., "recent": ...}) can't be edited + # through a string dropdown — injecting one would let the form save + # a string over the mapping. Leave the schema alone; per-mode users + # edit via the raw JSON config editor. + if isinstance(current_value, dict): + return schema + + try: + from src.skin_system import skin_runtime + matching = skin_runtime.skins_for_plugin(plugin_id) + except Exception as e: + self.logger.debug(f"Skin discovery failed for {plugin_id}: {e}") + return schema + + choices = sorted(matching.keys()) + if isinstance(current_value, str) and current_value and \ + current_value != "built-in" and current_value not in choices: + choices.append(current_value) + if not choices: + return schema + + enhanced = copy.deepcopy(schema) + enhanced.setdefault("properties", {}) + if "skin" not in enhanced["properties"]: + names = {sid: (matching.get(sid, {}).get("name") or sid) for sid in choices} + enhanced["properties"]["skin"] = { + "type": "string", + "title": "Visual Skin", + "description": "Replace this scoreboard's look with an installed skin " + "(data, scheduling, and vegas mode are unaffected)", + "enum": ["built-in", *choices], + "enumNames": ["Built-in", *(names[sid] for sid in choices)], + "default": "built-in" + } + return enhanced + def _format_validation_error(self, error: ValidationError, plugin_id: Optional[str] = None) -> str: """ Format a validation error into a readable message. diff --git a/src/plugin_system/store_manager.py b/src/plugin_system/store_manager.py index 095eb206..5cf31cf8 100644 --- a/src/plugin_system/store_manager.py +++ b/src/plugin_system/store_manager.py @@ -149,18 +149,27 @@ class PluginStoreManager: # loser can end up renaming the winner's in-progress install aside # mid-download, stealing its own rollback safety net. Keyed by # plugin_id so unrelated plugins still update concurrently. - self._reinstall_locks: Dict[str, threading.Lock] = {} + # Reentrant: install_plugin takes this lock, and _reinstall_with_rollback + # holds it across its call to install_plugin. A plain Lock would + # self-deadlock on that nesting. + self._reinstall_locks: Dict[str, "threading.RLock"] = {} self._reinstall_locks_guard = threading.Lock() # Ensure plugins directory exists self.plugins_dir.mkdir(exist_ok=True) - def _get_reinstall_lock(self, plugin_id: str) -> threading.Lock: - """Lazily create (or fetch) the per-plugin reinstall lock.""" + def _get_reinstall_lock(self, plugin_id: str): + """Lazily create (or fetch) the per-plugin reinstall lock. + + Reentrant by necessity: `install_plugin` acquires it to protect its + set-aside/restore, and `_reinstall_with_rollback` holds it across its + own call to `install_plugin`. With a plain `Lock` that nesting + deadlocks the request thread. + """ with self._reinstall_locks_guard: lock = self._reinstall_locks.get(plugin_id) if lock is None: - lock = threading.Lock() + lock = threading.RLock() self._reinstall_locks[plugin_id] = lock return lock @@ -1134,7 +1143,7 @@ class PluginStoreManager: """ registry = self.fetch_registry() plugins = registry.get('plugins', []) or [] - plugin_info = next((p for p in plugins if p['id'] == plugin_id), None) + plugin_info = self._match_registry_entry(plugins, plugin_id) if not plugin_info: return None @@ -1174,6 +1183,37 @@ class PluginStoreManager: return plugin_info + @staticmethod + def _match_registry_entry(plugins: List[Dict], plugin_id: str) -> Optional[Dict]: + """Find a registry entry by its id, or by the directory it installs to. + + Four shipped plugins have a registry ``id`` that differs from the ``id`` + in their own manifest: ``weather`` installs to ``plugins/ledmatrix-weather``, + and likewise stocks, music and leaderboard. Installation already prefers + the manifest id for the directory name, so on disk, in ``config.json`` + and in a backup manifest those plugins are called ``ledmatrix-weather``. + + Only the registry calls them ``weather``, and nothing resolved that in + reverse: restoring a backup asked the store for ``ledmatrix-weather`` + and got "Plugin not found in registry", silently dropping four enabled + plugins from a restored device. + + Matching ``plugin_path`` fixes it without renaming any published id, + which would orphan ``plugin_state.json`` entries keyed on the old ones. + Exact id always wins, so an entry whose *path* happens to collide with + another entry's id cannot shadow it. + """ + if not plugin_id: + return None + exact = next((p for p in plugins if p.get('id') == plugin_id), None) + if exact is not None: + return exact + for entry in plugins: + path = (entry.get('plugin_path') or '').rstrip('/') + if path and path.rsplit('/', 1)[-1] == plugin_id: + return entry + return None + def get_registry_info(self, plugin_id: str) -> Optional[Dict]: """ Get plugin information from the registry cache only (no GitHub API calls). @@ -1189,9 +1229,93 @@ class PluginStoreManager: """ registry = self.fetch_registry() plugins = registry.get('plugins', []) or [] - return next((p for p in plugins if p.get('id') == plugin_id), None) + return self._match_registry_entry(plugins, plugin_id) def install_plugin(self, plugin_id: str, branch: Optional[str] = None) -> bool: + """Install a plugin, keeping any existing install until the new one is + known good. + + `_install_plugin_impl` deletes the existing directory *before* + downloading, so every failure after that point — a dropped connection, a + malformed manifest, or the compatibility gate refusing the new version — + left the user with no plugin at all. `_reinstall_with_rollback` gives the + *update* path exactly this protection; a direct install had none, and the + compatibility gate added a new way to reach it. + + Pass-through when nothing is installed, and when called from + `_reinstall_with_rollback`, which has already moved the old copy aside. + + The aside name embeds '.standalone-backup-' so plugin discovery + (`plugin_manager._scan_directory_for_plugins`) skips it even though it + still holds a manifest.json. + + Held under the per-plugin reinstall lock for the same reason + `_reinstall_with_rollback` is: the web UI runs Flask with + threaded=True, so a double-clicked Install button gives two threads the + same plugin_id. Interleaved, one thread's restore would delete the + other's freshly installed copy. The lock is reentrant because the + rollback path already holds it when it calls in here. + """ + with self._get_reinstall_lock(plugin_id): + plugin_path = self.plugins_dir / plugin_id + if not plugin_path.exists(): + return self._install_plugin_impl(plugin_id, branch) + + backup_path = plugin_path.with_name( + f"{plugin_path.name}.standalone-backup-preinstall") + if backup_path.exists() and not self._safe_remove_directory(backup_path): + # Can't stage a safety net. Better to attempt the install than + # to refuse outright, which is what callers got before this + # existed. + self.logger.warning( + "Could not clear stale pre-install backup for %s at %s; " + "installing without a rollback net", plugin_id, backup_path) + return self._install_plugin_impl(plugin_id, branch) + + try: + plugin_path.rename(backup_path) + except OSError as e: + self.logger.warning( + "Could not set aside existing install of %s (%s); " + "installing without a rollback net", plugin_id, e) + return self._install_plugin_impl(plugin_id, branch) + + try: + installed = self._install_plugin_impl(plugin_id, branch) + except Exception: + self._restore_preinstall_backup(plugin_id, plugin_path, backup_path) + raise + + if installed: + if not self._safe_remove_directory(backup_path): + self.logger.warning( + "Install of %s succeeded but the previous copy at %s " + "could not be removed; it will be cleared on the next " + "install", plugin_id, backup_path) + return True + + self._restore_preinstall_backup(plugin_id, plugin_path, backup_path) + return False + + def _restore_preinstall_backup( + self, plugin_id: str, plugin_path: Path, backup_path: Path + ) -> None: + """Put the previous install back after a failed (re)install.""" + self.logger.error( + "Install of %s failed; restoring the previous version", plugin_id) + try: + if plugin_path.exists(): + # Partial download debris from the failed install. + self._safe_remove_directory(plugin_path) + backup_path.rename(plugin_path) + self.logger.info("Restored previous install of %s", plugin_id) + except OSError as e: + self.logger.error( + "CRITICAL: could not restore %s from %s: %s. The previous " + "install is preserved there — rename it back manually.", + plugin_id, backup_path, e) + + def _install_plugin_impl(self, plugin_id: str, branch: Optional[str] = None) -> bool: """ Install a plugin from the official registry. Always installs the latest commit from the repository's default branch (or specified branch). @@ -1214,6 +1338,11 @@ class PluginStoreManager: self.logger.error(f"Plugin not found in registry: {plugin_id}") return False + # Visual skins share the registry but install to skins/, not to a + # plugin directory (docs/SKIN_SYSTEM.md) + if (plugin_info.get('type') or 'plugin') == 'skin': + return self._install_skin_from_info(plugin_id, plugin_info, branch) + repo_url = plugin_info.get('repo') if not repo_url: self.logger.error(f"Plugin {plugin_id} missing repository URL") @@ -1328,6 +1457,26 @@ class PluginStoreManager: self._safe_remove_directory(plugin_path) return False + # Refuse a plugin that needs a newer core than this one. The + # registry carries no compatibility field, so the floor is only + # knowable once the files are down — checking here, before + # dependency installation, is the earliest possible point. + # + # Refusing costs the user nothing: on an update this returns + # False and _reinstall_with_rollback restores the version they + # already had. Allowing it costs them a plugin that raises + # ModuleNotFoundError at load and is reported only as one line + # in the journal. See docs/SPORTS_UNIFICATION.md (phase B4/B6). + from src import __version__ as core_version + from src.plugin_system import compatibility + + compatible, reason = compatibility.check(manifest, core_version) + if not compatible: + self.logger.error( + "Refusing to install %s: %s", plugin_id, reason) + self._safe_remove_directory(plugin_path) + return False + if 'entry_point' not in manifest: manifest['entry_point'] = 'manager.py' manifest_modified = True @@ -2254,19 +2403,171 @@ class PluginStoreManager: return None + _SKIN_ID_PATTERN = re.compile(r'^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$') + + def _resolve_skin_target(self, skin_id: str) -> Optional[Path]: + """Validate an externally-supplied skin id and resolve it to a path + strictly inside the skins directory. Returns None (after logging) + for ids that are malformed or would escape the directory — registry + entries and manifests are external input and must not be able to + write or delete outside skins/.""" + from src.skin_system import skin_runtime + + if not isinstance(skin_id, str) or not self._SKIN_ID_PATTERN.match(skin_id) \ + or '..' in skin_id: + self.logger.error(f"Rejecting unsafe skin id: {skin_id!r}") + return None + skins_dir = skin_runtime.get_skins_directory().resolve() + target = (skins_dir / skin_id).resolve() + if target.parent != skins_dir: + self.logger.error(f"Skin id {skin_id!r} escapes the skins directory; rejecting") + return None + return target + + def _install_skin_from_info(self, skin_id: str, skin_info: Dict, + branch: Optional[str] = None) -> bool: + """Install a registry entry of type "skin" into skins//. + + Reuses the plugin download machinery (git / monorepo zip / archive) + but validates skin.json instead of manifest.json and never installs + dependencies — skins are render-only (stdlib + PIL + the provided + SkinContext), which is also what keeps them safe to iterate on. + + Downloads into a staging directory and validates there; the + existing installation is only replaced after the new one passes, + so a failed download or bad manifest can't destroy a working skin. + """ + from src.skin_system import skin_runtime + from src.skin_system.skin_base import SKIN_API_VERSION + + repo_url = skin_info.get('repo') + if not repo_url: + self.logger.error(f"Skin {skin_id} missing repository URL") + return False + + target = self._resolve_skin_target(skin_id) + if target is None: + return False + skins_dir = target.parent + skins_dir.mkdir(parents=True, exist_ok=True) + # Leading "_" keeps staging invisible to skin discovery + staging = skins_dir / f"_staging-{skin_id}" + if staging.exists() and not self._safe_remove_directory(staging): + return False + + subpath = skin_info.get('plugin_path') + branch_candidates = self._distinct_sequence([ + branch, + skin_info.get('branch'), + skin_info.get('default_branch'), + skin_info.get('last_commit_branch'), + 'main', + 'master' + ]) + + try: + branch_used = None + if subpath: + for candidate in branch_candidates: + download_url = f"{repo_url}/archive/refs/heads/{candidate}.zip" + if self._install_from_monorepo(download_url, subpath, staging): + branch_used = candidate + break + else: + branch_used = self._install_via_git(repo_url, staging, branch_candidates) + if branch_used is None and not staging.exists(): + for candidate in branch_candidates: + download_url = f"{repo_url}/archive/refs/heads/{candidate}.zip" + if self._install_via_download(download_url, staging): + branch_used = candidate + break + + if branch_used is None and not staging.exists(): + self.logger.error(f"Failed to install skin {skin_id} via git or archive download") + return False + + try: + with open(staging / 'skin.json', 'r', encoding='utf-8') as f: + manifest = json.load(f) + except (OSError, json.JSONDecodeError) as e: + self.logger.error(f"Skin {skin_id} has no valid skin.json: {e}") + return False + + missing = [k for k in ('id', 'name', 'version', 'skin_api_version', 'class_name') + if not manifest.get(k)] + if missing: + self.logger.error(f"Skin {skin_id} manifest missing fields: {missing}") + return False + + # Unlike plugins, a mismatched id is rejected rather than + # renamed: the manifest id is external input, and the registry + # id is what the user asked to install. + if manifest['id'] != skin_id: + self.logger.error( + f"Skin manifest id {manifest['id']!r} doesn't match registry id " + f"{skin_id!r}; not installing") + return False + + def _api_major(v): + try: + return int(str(v).split('.')[0]) + except (ValueError, IndexError): + return None + + if _api_major(manifest['skin_api_version']) != _api_major(SKIN_API_VERSION): + self.logger.error( + f"Skin {skin_id} targets skin API {manifest['skin_api_version']} but this " + f"LEDMatrix provides {SKIN_API_VERSION}; not installing") + return False + + # Validated — swap into place + if target.exists() and not self._safe_remove_directory(target): + self.logger.error(f"Could not replace existing skin directory: {target}") + return False + shutil.move(str(staging), str(target)) + skin_runtime.discover_skins(force_refresh=True) + self.logger.info(f"Successfully installed skin: {skin_id} (branch: {branch_used})") + return True + finally: + if staging.exists(): + self._safe_remove_directory(staging) + + def uninstall_skin(self, skin_id: str) -> bool: + """Remove an installed skin. Plugin configs referencing it keep + validating; rendering falls back to the built-in layout.""" + from src.skin_system import skin_runtime + + target = self._resolve_skin_target(skin_id) + if target is None: + return False + if not target.exists(): + self.logger.info(f"Skin {skin_id} not found (already uninstalled)") + return True + if self._safe_remove_directory(target): + skin_runtime.discover_skins(force_refresh=True) + self.logger.info(f"Successfully uninstalled skin: {skin_id}") + return True + return False + def uninstall_plugin(self, plugin_id: str) -> bool: """ Uninstall a plugin by removing its directory. - + Args: plugin_id: Plugin identifier - + Returns: True if uninstalled successfully (or already not installed) """ plugin_path = self._find_plugin_path(plugin_id) - + if plugin_path is None or not plugin_path.exists(): + # A skin id passed to the plugin uninstall path (the store UI + # uses one uninstall flow) removes the skin instead + skin_target = self._resolve_skin_target(plugin_id) \ + if self._SKIN_ID_PATTERN.match(str(plugin_id)) else None + if skin_target is not None and skin_target.exists(): + return self.uninstall_skin(plugin_id) self.logger.info(f"Plugin {plugin_id} not found (already uninstalled)") return True # Already uninstalled, consider this success @@ -2699,7 +3000,10 @@ class PluginStoreManager: remote_branch = plugin_info_remote.get('branch') or plugin_info_remote.get('default_branch') # Compare local manifest version against registry latest_version - # to avoid unnecessary reinstalls for monorepo plugins + # to avoid unnecessary reinstalls for monorepo plugins. Uses the + # same semantic comparator as the web UI's update badge, so + # equivalent spellings ("v1.2.0" vs "1.2.0") never trigger a + # reinstall and a locally-ahead version is never downgraded. try: local_manifest_path = plugin_path / "manifest.json" if local_manifest_path.exists(): @@ -2707,8 +3011,16 @@ class PluginStoreManager: local_manifest = json.load(f) local_version = local_manifest.get('version', '') remote_version = plugin_info_remote.get('latest_version', '') - if local_version and remote_version and local_version == remote_version: - self.logger.info(f"Plugin {plugin_id} already at latest version {local_version}") + from src.plugin_system.compatibility import is_update_available + # No truthiness gate: the shared comparator already treats + # a missing version on either side as "no update", and the + # store must agree with the UI badge in that case too. A + # missing manifest (not just a missing version field) + # still falls through to the reinstall recovery path. + if not is_update_available(local_version, remote_version): + self.logger.info( + f"Plugin {plugin_id} already at latest version " + f"(installed {local_version}, registry {remote_version})") return True except Exception as e: self.logger.debug(f"Could not compare versions for {plugin_id}: {e}") diff --git a/src/plugin_system/testing/harness.py b/src/plugin_system/testing/harness.py index ee4d6613..51c688a2 100644 --- a/src/plugin_system/testing/harness.py +++ b/src/plugin_system/testing/harness.py @@ -73,6 +73,11 @@ class RenderResult: golden_ok: Optional[bool] = None golden_diff_pixels: int = 0 golden_max_delta: int = 0 + # what display() handed back; the controller skips a mode only on False + display_returned: Any = None + # empty-frame check: rendered nothing while not reporting "no content" + empty_claimed: Optional[bool] = None # True when that happened + empty_ok: Optional[bool] = None # False only in strict mode # fill / scale-up check (populated only for sizes >= 2x the design size) fill_checked: bool = False fill_ok: Optional[bool] = None # False only in strict mode @@ -92,6 +97,8 @@ class RenderResult: return False if self.fill_ok is False: return False + if self.empty_ok is False: + return False return True @@ -132,21 +139,25 @@ def _instantiate(plugin_id: str, manifest: Dict[str, Any], plugin_dir: Path, return plugin_instance -def _render_mode(plugin_instance: Any, mode: str) -> None: +def _render_mode(plugin_instance: Any, mode: str) -> Any: """Render a specific screen. Prefer an explicit display_mode kwarg; otherwise drive the plugin's internal mode state machine (first display() call renders - modes[current_mode_index] when current_display_mode is None).""" + modes[current_mode_index] when current_display_mode is None). + + Returns whatever display() returned. The display controller skips a mode + whose display() returns False, so that value decides whether an empty mode + is rotated past or sat on -- which makes it worth reporting rather than + discarding.""" sig = inspect.signature(plugin_instance.display) if "display_mode" in sig.parameters: - plugin_instance.display(force_clear=True, display_mode=mode) - return + return plugin_instance.display(force_clear=True, display_mode=mode) modes = getattr(plugin_instance, "modes", None) if modes and mode in modes: plugin_instance.current_mode_index = list(modes).index(mode) if hasattr(plugin_instance, "current_display_mode"): plugin_instance.current_display_mode = None - plugin_instance.display(force_clear=False) + return plugin_instance.display(force_clear=False) def _freeze(freeze_time: Optional[str]): @@ -234,7 +245,7 @@ def _render_size(plugin_id, manifest, plugin_dir, config, mock_data, logger.warning("update() raised a non-connectivity error for %s [%s]: %s", plugin_id, mode, e) if result.error is None: - _render_mode(inst, mode) + result.display_returned = _render_mode(inst, mode) result.image = dm.get_image() result.overflow = dm.check_overflow() except Exception as e: # noqa: BLE001 — a display crash is a real failure @@ -341,6 +352,44 @@ def fill_metrics(image: Image.Image) -> Tuple[float, float, float]: return (extent_x, extent_y, ink) +def check_empty_claimed(results: List[RenderResult], + strict: bool = False) -> List[RenderResult]: + """Flag a mode that rendered nothing without reporting "no content". + + The display controller skips a mode whose ``display()`` returns False, and + treats anything else -- including None -- as "content was shown". A mode + that draws nothing and does not return False therefore holds whatever is on + the panel for its whole display duration. Since a mode switch clears first, + that is a blank screen. Two sports plugins shipped exactly this: their + ``display()`` returned None on every path, so an out-of-season league sat + blank for its full duration rather than being rotated past. + + Warn-only by default, because a blank frame is not automatically wrong: a + scroll mode whose first frame is its blank scroll-in buffer renders empty + and is behaving correctly. ``strict=True`` sets ``empty_claimed`` such that + ``RenderResult.ok`` fails -- opt in per plugin via harness.json + ``{"empty_check": "strict"}`` once its modes are known to draw on the + fixture data. + + Note this can only catch what the fixtures actually render. A plugin whose + harness fixture seeds content never exercises its empty path here; the + source-level gate in the plugins repo covers that case. + """ + for r in results: + if r.image is None or r.error is not None: + continue + # An explicit False is the plugin correctly saying "nothing to show". + if r.display_returned is False: + continue + if r.image.convert("L").point( + lambda p: 255 if p > _LIT_THRESHOLD else 0).getbbox() is not None: + continue + r.empty_claimed = True + if strict: + r.empty_ok = False + return results + + def check_scale_up(results: List[RenderResult], design_size: Tuple[int, int] = (128, 32), min_extent: float = _MIN_FILL_EXTENT, diff --git a/src/plugin_system/testing/plugin_test_base.py b/src/plugin_system/testing/plugin_test_base.py index f873ea9e..806f3d3f 100644 --- a/src/plugin_system/testing/plugin_test_base.py +++ b/src/plugin_system/testing/plugin_test_base.py @@ -2,6 +2,11 @@ Base test class for LEDMatrix plugins. Provides common fixtures and helper methods for plugin testing. + +Note: this is the plugin-author-facing base class shipped with the +core (importable as src.plugin_system.testing.plugin_test_base). The +repo's own plugin tests use a separate, richer harness in +test/plugins/test_plugin_base.py — the two are intentionally distinct. """ import unittest diff --git a/src/plugin_system/testing/visual_display_manager.py b/src/plugin_system/testing/visual_display_manager.py index 7f84a94e..5211a226 100644 --- a/src/plugin_system/testing/visual_display_manager.py +++ b/src/plugin_system/testing/visual_display_manager.py @@ -10,11 +10,23 @@ without requiring hardware or the RGBMatrixEmulator. Used for: Unlike MockDisplayManager (which logs calls but doesn't render) or MagicMock (which tracks nothing visual), this class creates a real PIL Image canvas and draws text using the actual project fonts. + +MAINTENANCE WARNING: this class is a deliberate fork of +src/display_manager.py so it can run without hardware. It mirrors +these DisplayManager methods by name and behavior: _load_fonts, +_draw_bdf_text, get_font_height, get_text_width, draw_text, +draw_text_with_icons, draw_weather_icon (and the _draw_sun/_draw_cloud/ +_draw_rain/_draw_snow/_draw_storm family), format_date_with_ordinal, +capture_mode, set_scrolling_state, is_currently_scrolling, +process_deferred_updates, update_display, render_size. A behavior +change to any of those in DisplayManager must be mirrored here, or +plugin visual tests will pass against stale behavior. """ import math import os import time +from contextlib import contextmanager from pathlib import Path from typing import Any, List, Optional, Tuple @@ -62,6 +74,9 @@ class VisualTestDisplayManager: # Matrix proxy (plugins access display_manager.matrix.width/height) self.matrix = _MatrixProxy(width, height) + # Set while inside capture_mode(); mirrors DisplayManager's flag. + self._capture_mode_active = False + # Scrolling state (interface compat, no-op) self._scrolling_state = { 'is_scrolling': False, @@ -174,6 +189,50 @@ class VisualTestDisplayManager: """No-op for hardware; marks that display was updated.""" self.update_called = True + @contextmanager + def render_size(self, width: int, height: Optional[int] = None): + """ + Interface parity with DisplayManager.render_size(). + + Vegas mode narrows the canvas so plugins lay out compactly instead of + being cropped. The harness must offer the same context or that path + cannot be exercised offline — and because the adapter catches broadly, + a missing method shows up as "no content" rather than an error. + """ + prev_image = self.image + prev_draw = self.draw + prev_w, prev_h = self._width, self._height + + target_w = max(1, min(int(width), prev_w)) + target_h = max(1, min(int(height) if height else prev_h, prev_h)) + + try: + self._width, self._height = target_w, target_h + self.matrix = _MatrixProxy(target_w, target_h) + self.image = Image.new('RGB', (target_w, target_h), (0, 0, 0)) + self.draw = ImageDraw.Draw(self.image) + yield + finally: + self._width, self._height = prev_w, prev_h + self.matrix = _MatrixProxy(prev_w, prev_h) + self.image = prev_image + self.draw = prev_draw + + @contextmanager + def capture_mode(self): + """ + Interface parity with DisplayManager.capture_mode(). + + There is no hardware to suppress here, but Vegas mode's PluginAdapter + wraps every off-screen content fetch in this context, so the harness + must provide it for that code path to be exercisable in tests. + """ + self._capture_mode_active = True + try: + yield + finally: + self._capture_mode_active = False + def draw_text(self, text: str, x: Optional[int] = None, y: Optional[int] = None, color: Tuple[int, int, int] = (255, 255, 255), small_font: bool = False, font: Optional[Any] = None, centered: bool = False) -> None: diff --git a/src/skin_system/__init__.py b/src/skin_system/__init__.py new file mode 100644 index 00000000..c33f7bf8 --- /dev/null +++ b/src/skin_system/__init__.py @@ -0,0 +1,31 @@ +""" +Skin system: user-installable visual overlays for sports scoreboards. + +A skin replaces only the rendering of a scoreboard (live / recent / +upcoming) while the host plugin keeps doing data fetching, scheduling, +caching, live priority, and vegas mode. See docs/SKIN_SYSTEM.md. +""" + +from src.skin_system.skin_base import ( + SKIN_API_VERSION, + VIEW_MODEL_VERSION, + ScoreboardSkin, + SkinContext, +) +from src.skin_system.skin_runtime import ( + build_context, + discover_skins, + get_skins_directory, + load_skin, +) + +__all__ = [ + "SKIN_API_VERSION", + "VIEW_MODEL_VERSION", + "ScoreboardSkin", + "SkinContext", + "build_context", + "discover_skins", + "get_skins_directory", + "load_skin", +] diff --git a/src/skin_system/fixtures/baseball_live.json b/src/skin_system/fixtures/baseball_live.json new file mode 100644 index 00000000..c5bcec34 --- /dev/null +++ b/src/skin_system/fixtures/baseball_live.json @@ -0,0 +1,39 @@ +{ + "id": "401570001", + "game_time": "7:05PM", + "game_date": "Jul 16th", + "start_time_utc": "2026-07-16T23:05:00+00:00", + "status_text": "Bot 7th", + "is_live": true, + "is_final": false, + "is_upcoming": false, + "is_halftime": false, + "is_period_break": false, + "home_abbr": "LAD", + "home_id": "19", + "home_score": "5", + "home_logo_path": "src/skin_system/fixtures/placeholder_home.png", + "home_logo_url": null, + "home_record": "58-33", + "away_abbr": "SF", + "away_id": "26", + "away_score": "3", + "away_logo_path": "src/skin_system/fixtures/placeholder_away.png", + "away_logo_url": null, + "away_record": "49-42", + "is_within_window": true, + "status": "STATUS_IN_PROGRESS", + "status_state": "in", + "inning": 7, + "inning_half": "bottom", + "balls": 3, + "strikes": 2, + "outs": 2, + "bases_occupied": [ + true, + true, + true + ], + "start_time": "2026-07-16T23:05:00Z", + "series_summary": "LAD leads 2-1" +} \ No newline at end of file diff --git a/src/skin_system/fixtures/baseball_recent.json b/src/skin_system/fixtures/baseball_recent.json new file mode 100644 index 00000000..15ed238f --- /dev/null +++ b/src/skin_system/fixtures/baseball_recent.json @@ -0,0 +1,39 @@ +{ + "id": "401570001", + "game_time": "7:05PM", + "game_date": "Jul 16th", + "start_time_utc": "2026-07-16T23:05:00+00:00", + "status_text": "Final", + "is_live": false, + "is_final": true, + "is_upcoming": false, + "is_halftime": false, + "is_period_break": false, + "home_abbr": "LAD", + "home_id": "19", + "home_score": "5", + "home_logo_path": "src/skin_system/fixtures/placeholder_home.png", + "home_logo_url": null, + "home_record": "58-33", + "away_abbr": "SF", + "away_id": "26", + "away_score": "3", + "away_logo_path": "src/skin_system/fixtures/placeholder_away.png", + "away_logo_url": null, + "away_record": "49-42", + "is_within_window": true, + "status": "STATUS_FINAL", + "status_state": "post", + "inning": 9, + "inning_half": "top", + "balls": 0, + "strikes": 0, + "outs": 3, + "bases_occupied": [ + false, + false, + false + ], + "start_time": "2026-07-16T23:05:00Z", + "series_summary": "Series tied 2-2" +} \ No newline at end of file diff --git a/src/skin_system/fixtures/baseball_upcoming.json b/src/skin_system/fixtures/baseball_upcoming.json new file mode 100644 index 00000000..48ccb857 --- /dev/null +++ b/src/skin_system/fixtures/baseball_upcoming.json @@ -0,0 +1,39 @@ +{ + "id": "401570001", + "game_time": "7:05PM", + "game_date": "Jul 16th", + "start_time_utc": "2026-07-16T23:05:00+00:00", + "status_text": "7:05 PM", + "is_live": false, + "is_final": false, + "is_upcoming": true, + "is_halftime": false, + "is_period_break": false, + "home_abbr": "LAD", + "home_id": "19", + "home_score": "0", + "home_logo_path": "src/skin_system/fixtures/placeholder_home.png", + "home_logo_url": null, + "home_record": "58-33", + "away_abbr": "SF", + "away_id": "26", + "away_score": "0", + "away_logo_path": "src/skin_system/fixtures/placeholder_away.png", + "away_logo_url": null, + "away_record": "49-42", + "is_within_window": true, + "status": "STATUS_SCHEDULED", + "status_state": "pre", + "inning": 0, + "inning_half": "top", + "balls": 0, + "strikes": 0, + "outs": 0, + "bases_occupied": [ + false, + false, + false + ], + "start_time": "2026-07-16T23:05:00Z", + "series_summary": "" +} \ No newline at end of file diff --git a/src/skin_system/fixtures/basketball_live.json b/src/skin_system/fixtures/basketball_live.json new file mode 100644 index 00000000..06e86880 --- /dev/null +++ b/src/skin_system/fixtures/basketball_live.json @@ -0,0 +1,28 @@ +{ + "id": "401570001", + "game_time": "7:05PM", + "game_date": "Jul 16th", + "start_time_utc": "2026-07-16T23:05:00+00:00", + "status_text": "Q4 2:34", + "is_live": true, + "is_final": false, + "is_upcoming": false, + "is_halftime": false, + "is_period_break": false, + "home_abbr": "OKC", + "home_id": "19", + "home_score": "5", + "home_logo_path": "src/skin_system/fixtures/placeholder_home.png", + "home_logo_url": null, + "home_record": "", + "away_abbr": "MIN", + "away_id": "26", + "away_score": "3", + "away_logo_path": "src/skin_system/fixtures/placeholder_away.png", + "away_logo_url": null, + "away_record": "", + "is_within_window": true, + "period": 4, + "period_text": "Q4", + "clock": "2:34" +} \ No newline at end of file diff --git a/src/skin_system/fixtures/basketball_recent.json b/src/skin_system/fixtures/basketball_recent.json new file mode 100644 index 00000000..96e15d06 --- /dev/null +++ b/src/skin_system/fixtures/basketball_recent.json @@ -0,0 +1,28 @@ +{ + "id": "401570001", + "game_time": "7:05PM", + "game_date": "Jul 16th", + "start_time_utc": "2026-07-16T23:05:00+00:00", + "status_text": "Final", + "is_live": false, + "is_final": true, + "is_upcoming": false, + "is_halftime": false, + "is_period_break": false, + "home_abbr": "OKC", + "home_id": "19", + "home_score": "5", + "home_logo_path": "src/skin_system/fixtures/placeholder_home.png", + "home_logo_url": null, + "home_record": "", + "away_abbr": "MIN", + "away_id": "26", + "away_score": "3", + "away_logo_path": "src/skin_system/fixtures/placeholder_away.png", + "away_logo_url": null, + "away_record": "", + "is_within_window": true, + "period": 4, + "period_text": "Final", + "clock": "0:00" +} \ No newline at end of file diff --git a/src/skin_system/fixtures/basketball_upcoming.json b/src/skin_system/fixtures/basketball_upcoming.json new file mode 100644 index 00000000..32cd7df6 --- /dev/null +++ b/src/skin_system/fixtures/basketball_upcoming.json @@ -0,0 +1,28 @@ +{ + "id": "401570001", + "game_time": "7:05PM", + "game_date": "Jul 16th", + "start_time_utc": "2026-07-16T23:05:00+00:00", + "status_text": "7:05 PM", + "is_live": false, + "is_final": false, + "is_upcoming": true, + "is_halftime": false, + "is_period_break": false, + "home_abbr": "OKC", + "home_id": "19", + "home_score": "0", + "home_logo_path": "src/skin_system/fixtures/placeholder_home.png", + "home_logo_url": null, + "home_record": "", + "away_abbr": "MIN", + "away_id": "26", + "away_score": "0", + "away_logo_path": "src/skin_system/fixtures/placeholder_away.png", + "away_logo_url": null, + "away_record": "", + "is_within_window": true, + "period": 0, + "period_text": "", + "clock": "0:00" +} \ No newline at end of file diff --git a/src/skin_system/fixtures/football_live.json b/src/skin_system/fixtures/football_live.json new file mode 100644 index 00000000..b680e640 --- /dev/null +++ b/src/skin_system/fixtures/football_live.json @@ -0,0 +1,36 @@ +{ + "id": "401570001", + "game_time": "7:05PM", + "game_date": "Jul 16th", + "start_time_utc": "2026-07-16T23:05:00+00:00", + "status_text": "Q3 8:12", + "is_live": true, + "is_final": false, + "is_upcoming": false, + "is_halftime": false, + "is_period_break": false, + "home_abbr": "KC", + "home_id": "19", + "home_score": "21", + "home_logo_path": "src/skin_system/fixtures/placeholder_home.png", + "home_logo_url": null, + "home_record": "58-33", + "away_abbr": "BUF", + "away_id": "26", + "away_score": "17", + "away_logo_path": "src/skin_system/fixtures/placeholder_away.png", + "away_logo_url": null, + "away_record": "49-42", + "is_within_window": true, + "period": 3, + "period_text": "Q3", + "clock": "8:12", + "home_timeouts": 2, + "away_timeouts": 3, + "down_distance_text": "3rd & 4", + "down_distance_text_long": "3rd & 4 at KC 22", + "is_redzone": true, + "possession": "12", + "possession_indicator": "away", + "scoring_event": null +} \ No newline at end of file diff --git a/src/skin_system/fixtures/football_recent.json b/src/skin_system/fixtures/football_recent.json new file mode 100644 index 00000000..1f2efb5c --- /dev/null +++ b/src/skin_system/fixtures/football_recent.json @@ -0,0 +1,36 @@ +{ + "id": "401570001", + "game_time": "7:05PM", + "game_date": "Jul 16th", + "start_time_utc": "2026-07-16T23:05:00+00:00", + "status_text": "Final", + "is_live": false, + "is_final": true, + "is_upcoming": false, + "is_halftime": false, + "is_period_break": false, + "home_abbr": "KC", + "home_id": "19", + "home_score": "21", + "home_logo_path": "src/skin_system/fixtures/placeholder_home.png", + "home_logo_url": null, + "home_record": "58-33", + "away_abbr": "BUF", + "away_id": "26", + "away_score": "17", + "away_logo_path": "src/skin_system/fixtures/placeholder_away.png", + "away_logo_url": null, + "away_record": "49-42", + "is_within_window": true, + "period": 4, + "period_text": "Final", + "clock": "0:00", + "home_timeouts": 0, + "away_timeouts": 0, + "down_distance_text": "", + "down_distance_text_long": "", + "is_redzone": false, + "possession": null, + "possession_indicator": null, + "scoring_event": null +} \ No newline at end of file diff --git a/src/skin_system/fixtures/football_upcoming.json b/src/skin_system/fixtures/football_upcoming.json new file mode 100644 index 00000000..7b28c0a9 --- /dev/null +++ b/src/skin_system/fixtures/football_upcoming.json @@ -0,0 +1,36 @@ +{ + "id": "401570001", + "game_time": "7:05PM", + "game_date": "Jul 16th", + "start_time_utc": "2026-07-16T23:05:00+00:00", + "status_text": "7:05 PM", + "is_live": false, + "is_final": false, + "is_upcoming": true, + "is_halftime": false, + "is_period_break": false, + "home_abbr": "KC", + "home_id": "19", + "home_score": "0", + "home_logo_path": "src/skin_system/fixtures/placeholder_home.png", + "home_logo_url": null, + "home_record": "58-33", + "away_abbr": "BUF", + "away_id": "26", + "away_score": "0", + "away_logo_path": "src/skin_system/fixtures/placeholder_away.png", + "away_logo_url": null, + "away_record": "49-42", + "is_within_window": true, + "period": 0, + "period_text": "", + "clock": "0:00", + "home_timeouts": 3, + "away_timeouts": 3, + "down_distance_text": "", + "down_distance_text_long": "", + "is_redzone": false, + "possession": null, + "possession_indicator": null, + "scoring_event": null +} \ No newline at end of file diff --git a/src/skin_system/fixtures/hockey_live.json b/src/skin_system/fixtures/hockey_live.json new file mode 100644 index 00000000..2d53425d --- /dev/null +++ b/src/skin_system/fixtures/hockey_live.json @@ -0,0 +1,32 @@ +{ + "id": "401570001", + "game_time": "7:05PM", + "game_date": "Jul 16th", + "start_time_utc": "2026-07-16T23:05:00+00:00", + "status_text": "P3 14:55", + "is_live": true, + "is_final": false, + "is_upcoming": false, + "is_halftime": false, + "is_period_break": false, + "home_abbr": "COL", + "home_id": "19", + "home_score": "2", + "home_logo_path": "src/skin_system/fixtures/placeholder_home.png", + "home_logo_url": null, + "home_record": "58-33", + "away_abbr": "VGK", + "away_id": "26", + "away_score": "2", + "away_logo_path": "src/skin_system/fixtures/placeholder_away.png", + "away_logo_url": null, + "away_record": "49-42", + "is_within_window": true, + "period": 3, + "period_text": "P3", + "clock": "14:55", + "power_play": true, + "penalties": [], + "home_shots": 27, + "away_shots": 31 +} \ No newline at end of file diff --git a/src/skin_system/fixtures/hockey_recent.json b/src/skin_system/fixtures/hockey_recent.json new file mode 100644 index 00000000..0127162a --- /dev/null +++ b/src/skin_system/fixtures/hockey_recent.json @@ -0,0 +1,32 @@ +{ + "id": "401570001", + "game_time": "7:05PM", + "game_date": "Jul 16th", + "start_time_utc": "2026-07-16T23:05:00+00:00", + "status_text": "Final/OT", + "is_live": false, + "is_final": true, + "is_upcoming": false, + "is_halftime": false, + "is_period_break": false, + "home_abbr": "COL", + "home_id": "19", + "home_score": "3", + "home_logo_path": "src/skin_system/fixtures/placeholder_home.png", + "home_logo_url": null, + "home_record": "58-33", + "away_abbr": "VGK", + "away_id": "26", + "away_score": "2", + "away_logo_path": "src/skin_system/fixtures/placeholder_away.png", + "away_logo_url": null, + "away_record": "49-42", + "is_within_window": true, + "period": 5, + "period_text": "Final/OT", + "clock": "0:00", + "power_play": false, + "penalties": [], + "home_shots": 35, + "away_shots": 33 +} \ No newline at end of file diff --git a/src/skin_system/fixtures/hockey_upcoming.json b/src/skin_system/fixtures/hockey_upcoming.json new file mode 100644 index 00000000..64caf089 --- /dev/null +++ b/src/skin_system/fixtures/hockey_upcoming.json @@ -0,0 +1,32 @@ +{ + "id": "401570001", + "game_time": "7:05PM", + "game_date": "Jul 16th", + "start_time_utc": "2026-07-16T23:05:00+00:00", + "status_text": "7:05 PM", + "is_live": false, + "is_final": false, + "is_upcoming": true, + "is_halftime": false, + "is_period_break": false, + "home_abbr": "COL", + "home_id": "19", + "home_score": "0", + "home_logo_path": "src/skin_system/fixtures/placeholder_home.png", + "home_logo_url": null, + "home_record": "58-33", + "away_abbr": "VGK", + "away_id": "26", + "away_score": "0", + "away_logo_path": "src/skin_system/fixtures/placeholder_away.png", + "away_logo_url": null, + "away_record": "49-42", + "is_within_window": true, + "period": 0, + "period_text": "", + "clock": "0:00", + "power_play": false, + "penalties": [], + "home_shots": 0, + "away_shots": 0 +} \ No newline at end of file diff --git a/src/skin_system/fixtures/placeholder_away.png b/src/skin_system/fixtures/placeholder_away.png new file mode 100644 index 00000000..fc9ec4ee Binary files /dev/null and b/src/skin_system/fixtures/placeholder_away.png differ diff --git a/src/skin_system/fixtures/placeholder_home.png b/src/skin_system/fixtures/placeholder_home.png new file mode 100644 index 00000000..7533f054 Binary files /dev/null and b/src/skin_system/fixtures/placeholder_home.png differ diff --git a/src/skin_system/skin_base.py b/src/skin_system/skin_base.py new file mode 100644 index 00000000..e687014b --- /dev/null +++ b/src/skin_system/skin_base.py @@ -0,0 +1,171 @@ +""" +Skin API: the classes a skin author works with. + +A skin is a directory under skins// containing a skin.json +manifest and a Python module exposing a ScoreboardSkin subclass. The +host (a sports scoreboard's base classes) builds a SkinContext per +render and calls render_live / render_recent / render_upcoming with the +game view model. The skin draws onto ctx.canvas and returns True; the +host composites the canvas onto the display. A skin never talks to the +display, the network, or the plugin directly. + +Skin API Version: 1.0.0 +View Model Version: 1.0 +""" + +from abc import ABC +from dataclasses import dataclass, field +from typing import Any, Callable, Dict, Optional, Tuple, Union + +from PIL import Image, ImageDraw + +try: + import freetype +except ImportError: # pragma: no cover - freetype ships with the project deps + freetype = None + +from src.adaptive_layout import FitResult, LayoutContext, Region + +# Major must match a skin manifest's skin_api_version major or the skin +# is refused at load time (renames/removals bump major; additions minor). +SKIN_API_VERSION = "1.0.0" + +# Version of the guaranteed `game` dict keys (see docs/CREATING_SKINS.md). +VIEW_MODEL_VERSION = "1.0" + + +def _draw_bdf_text_on(draw: ImageDraw.ImageDraw, text: str, x: int, y: int, + color: Tuple[int, int, int], face: Any, + clip_w: int, clip_h: int) -> None: + """Render a freetype BDF face glyph-by-glyph onto an arbitrary canvas. + + DisplayManager._draw_bdf_text only draws onto the panel image; skins + draw onto their own canvas, so the fitted-font path (fit_text can + return freetype faces) needs this standalone equivalent. + """ + try: + ascender_px = face.size.ascender >> 6 + except Exception: + ascender_px = 0 + baseline_y = y + ascender_px + for char in text: + face.load_char(char) + bitmap = face.glyph.bitmap + glyph_left = face.glyph.bitmap_left + glyph_top = face.glyph.bitmap_top + for i in range(bitmap.rows): + for j in range(bitmap.width): + byte_index = i * bitmap.pitch + (j // 8) + if byte_index < len(bitmap.buffer) and \ + bitmap.buffer[byte_index] & (1 << (7 - (j % 8))): + px = x + glyph_left + j + py = baseline_y - glyph_top + i + if 0 <= px < clip_w and 0 <= py < clip_h: + draw.point((px, py), fill=color) + x += face.glyph.advance.x >> 6 + + +@dataclass +class SkinContext: + """Everything a skin may touch during one render call. + + The canvas is a fresh RGB image sized to the current display (or + vegas card). Draw onto it via the helpers below or raw ``draw``; + never call display/update methods — the host composites the canvas. + """ + + canvas: Image.Image + draw: ImageDraw.ImageDraw + layout: LayoutContext + width: int + height: int + fonts: Dict[str, Any] + options: Dict[str, Any] + logger: Any + sport: Optional[str] = None + view_model_version: str = VIEW_MODEL_VERSION + # load_logo("home") / load_logo("away") -> RGBA PIL image or None. + # Bound to the current game; hits the host's logo cache (never loads + # from disk twice), downloads missing logos like the built-in layout. + load_logo: Callable[[str], Optional[Image.Image]] = field(default=lambda side: None) + # draw_text_outlined(text, (x, y), font, fill=..., outline_color=...) + # — the classic scorebug outlined text, drawn onto this canvas. + # TTF fonts only (ctx.fonts values are TTF); for ladder-fitted fonts + # use draw_fit / draw_text, which handle BDF faces too. + draw_text_outlined: Callable[..., None] = field(default=lambda *a, **k: None) + + def draw_text(self, text: str, x: int, y: int, + color: Tuple[int, int, int] = (255, 255, 255), + font: Any = None) -> None: + """Draw text at a top-left position, handling both PIL fonts and + the freetype BDF faces that layout.fit_text can return.""" + if font is None: + font = self.fonts.get('time') + if freetype is not None and isinstance(font, freetype.Face): + _draw_bdf_text_on(self.draw, text, int(x), int(y), color, font, + self.width, self.height) + else: + self.draw.text((int(x), int(y)), text, font=font, fill=color) + + def draw_fit(self, fit: FitResult, box: Union[Region, Tuple[int, int]], + color: Tuple[int, int, int] = (255, 255, 255), + align: str = "center", valign: str = "center") -> None: + """Draw a layout.fit_text() result aligned within a Region — the + canvas-local equivalent of adaptive_layout.draw_fitted_text.""" + region = box if isinstance(box, Region) else Region(0, 0, box[0], box[1]) + x, y = region.align_xy(fit.width, fit.height, align, valign) + self.draw_text(fit.text, x, y - fit.y_offset, color=color, font=fit.font) + + def draw_image(self, img: Optional[Image.Image], + box: Union[Region, Tuple[int, int]], *, + mode: str = "contain", align: str = "center", + valign: str = "center", cache_key: Any = None) -> None: + """Fit an image (a logo, art) into a Region and paste it, honoring + alpha. Silently no-ops on None so `ctx.draw_image(ctx.load_logo( + 'home'), ...)` stays safe when a logo is missing.""" + if img is None: + return + region = box if isinstance(box, Region) else Region(0, 0, box[0], box[1]) + fitted = self.layout.fit_image(img, region, mode=mode, + cache_key=cache_key) + result = fitted.image # fit_image returns an ImageFitResult (always RGBA) + if result is None: + return + x, y = region.align_xy(result.width, result.height, align, valign) + self.canvas.paste(result, (int(x), int(y)), result) + + +class ScoreboardSkin(ABC): + """Base class for scoreboard skins. + + Override only the modes you want to restyle; any mode you leave + unimplemented (or return False from) falls back to the plugin's + built-in renderer, so a live-only skin still gets recent/upcoming + screens for free. + + Skins should be stateless: three host instances (live, recent, + upcoming) each hold their own skin instance, and a render must be + derivable from (ctx, game) alone. + """ + + SKIN_API_VERSION = SKIN_API_VERSION + + def __init__(self, manifest: Dict[str, Any], options: Dict[str, Any]): + self.manifest = manifest + self.options = options or {} + + def render_live(self, ctx: SkinContext, game: Dict[str, Any]) -> bool: + return False + + def render_recent(self, ctx: SkinContext, game: Dict[str, Any]) -> bool: + return False + + def render_upcoming(self, ctx: SkinContext, game: Dict[str, Any]) -> bool: + return False + + def render_vegas_card(self, ctx: SkinContext, + game: Dict[str, Any]) -> Optional[Image.Image]: + """Render one vegas scroll card at ctx.width x ctx.height. Return + the finished image, or None to let the host use its default vegas + rendering (which captures the regular display output).""" + return None diff --git a/src/skin_system/skin_runtime.py b/src/skin_system/skin_runtime.py new file mode 100644 index 00000000..6c923eb0 --- /dev/null +++ b/src/skin_system/skin_runtime.py @@ -0,0 +1,352 @@ +""" +Skin runtime: discovery, validation, loading, and context building. + +Deliberately generic — this module knows nothing about sports beyond +passing a `sport` label through; the sports flavor lives in skin_base +(ScoreboardSkin) and in the hosts that call build_context. + +Every failure path here logs and returns None: a broken or missing skin +must never take down the plugin that references it — the host falls +back to its built-in renderer. +""" + +import importlib.util +import json +import sys +import threading +from pathlib import Path +from typing import Any, Dict, Optional, Tuple + +from PIL import Image, ImageDraw + +from src.adaptive_layout import LayoutContext +from src.logging_config import get_logger +from src.skin_system.skin_base import ( + SKIN_API_VERSION, + ScoreboardSkin, + SkinContext, +) + +logger = get_logger(__name__) + +_REQUIRED_MANIFEST_FIELDS = ("id", "name", "version", "skin_api_version", "class_name") +_DEFAULT_ENTRY_POINT = "skin.py" + +_lock = threading.RLock() +# skins_dir -> (fingerprint, {skin_id: manifest+path}) +_discovery_cache: Dict[str, Tuple[Tuple, Dict[str, Dict[str, Any]]]] = {} + +_shared_layout_font_manager: Optional[Any] = None + + +def _get_font_manager() -> Any: + """Shared FontManager for skin LayoutContexts. SportsCore hosts don't + carry a plugin_manager, so skins share one module-level FontManager — + the same shape as base_plugin._fallback_font_manager, constructed + directly so rendering never has to import the whole plugin system.""" + global _shared_layout_font_manager + if _shared_layout_font_manager is None: + from src.font_manager import FontManager + _shared_layout_font_manager = FontManager({}) + return _shared_layout_font_manager + + +def get_skins_directory() -> Path: + """Central skins directory: /skins. Lives outside the + plugin directories on purpose — plugin reinstall/update deletes the + whole plugin directory, and a skin must survive that.""" + return Path(__file__).resolve().parents[2] / "skins" + + +def _major(version: str) -> Optional[int]: + try: + return int(str(version).split(".")[0]) + except (ValueError, AttributeError, IndexError): + return None + + +def _read_manifest(skin_dir: Path) -> Optional[Dict[str, Any]]: + manifest_path = skin_dir / "skin.json" + if not manifest_path.is_file(): + return None + try: + with open(manifest_path, "r", encoding="utf-8") as f: + manifest = json.load(f) + except (OSError, json.JSONDecodeError) as e: + logger.error("Skin manifest %s is unreadable: %s", manifest_path, e) + return None + missing = [k for k in _REQUIRED_MANIFEST_FIELDS if not manifest.get(k)] + if missing: + logger.error("Skin manifest %s missing required fields: %s", + manifest_path, ", ".join(missing)) + return None + if manifest["id"] != skin_dir.name: + logger.warning("Skin manifest id %r does not match directory name %r", + manifest["id"], skin_dir.name) + manifest["_skin_dir"] = str(skin_dir) + return manifest + + +def _discovery_fingerprint(skins_dir: Path) -> Optional[Tuple]: + """Cache key for a skins directory: its mtime plus every skin.json's + (path, mtime). The directory mtime alone misses in-place manifest edits + (a skin updated without adding/removing entries).""" + try: + parts = [skins_dir.stat().st_mtime] + for manifest_path in sorted(skins_dir.glob("*/skin.json")): + parts.append((str(manifest_path), manifest_path.stat().st_mtime)) + return tuple(parts) + except OSError: + return None + + +def discover_skins(skins_dir: Optional[Path] = None, + force_refresh: bool = False) -> Dict[str, Dict[str, Any]]: + """Return {skin_id: manifest} for every valid skin package installed. + + Cached per directory and invalidated when the directory or any + skin.json changes; pass force_refresh to bypass. + """ + skins_dir = Path(skins_dir) if skins_dir else get_skins_directory() + cache_key = str(skins_dir) + fingerprint = _discovery_fingerprint(skins_dir) + if fingerprint is None: + return {} + + with _lock: + cached = _discovery_cache.get(cache_key) + if cached and not force_refresh and cached[0] == fingerprint: + return dict(cached[1]) + + skins: Dict[str, Dict[str, Any]] = {} + for entry in sorted(skins_dir.iterdir()): + if not entry.is_dir() or entry.name.startswith((".", "_")): + continue + manifest = _read_manifest(entry) + if manifest: + skins[manifest["id"]] = manifest + _discovery_cache[cache_key] = (fingerprint, skins) + return dict(skins) + + +def skin_targets(manifest: Dict[str, Any]) -> Tuple[list, list]: + """(sports, sport_keys) a skin declares it supports.""" + targets = manifest.get("targets") or {} + return (list(targets.get("sports") or []), + list(targets.get("sport_keys") or [])) + + +def skin_matches_target(manifest: Dict[str, Any], sport: Optional[str], + sport_key: Optional[str]) -> bool: + """True when the skin declares support for this sport family or exact + sport key. A skin with no targets at all matches everything.""" + sports, sport_keys = skin_targets(manifest) + if not sports and not sport_keys: + return True + if sport and sport in sports: + return True + if sport_key and sport_key in sport_keys: + return True + return False + + +def skins_for_plugin(plugin_id: str, + skins: Optional[Dict[str, Dict[str, Any]]] = None) -> Dict[str, Dict[str, Any]]: + """Installed skins that plausibly apply to a plugin, for UI dropdowns. + + A skin matches when the plugin id is listed in targets.plugins, or any + declared sport / sport_key appears as a token of the plugin id (so a + skin targeting sports=["baseball"] matches "baseball-scoreboard", and + sport_keys=["milb"] matches "milb-scoreboard").""" + if skins is None: + skins = discover_skins() + tokens = set(str(plugin_id).lower().replace("-", "_").split("_")) + matched = {} + for skin_id, manifest in skins.items(): + targets = manifest.get("targets") or {} + if plugin_id in (targets.get("plugins") or []): + matched[skin_id] = manifest + continue + sports, sport_keys = skin_targets(manifest) + if any(str(t).lower() in tokens for t in sports + sport_keys): + matched[skin_id] = manifest + return matched + + +def _load_skin_module(skin_id: str, skin_dir: Path, entry_point: str) -> Optional[Any]: + """Import the skin's entry module under a namespaced sys.modules key, + namespacing its sibling .py files the same way — the collision- + avoidance scheme plugins use (plugin_loader._namespace_plugin_modules), + so two skins can both ship a helpers.py. + + The entry module is cached: the live/recent/upcoming hosts all load + the same skin, and only the first load executes any code. (A skin + whose *code* changed on disk needs a service restart to take effect — + Python modules can't be safely hot-swapped.) + """ + entry_path = skin_dir / entry_point + if not entry_path.is_file(): + logger.error("Skin '%s' entry point not found: %s", skin_id, entry_path) + return None + + module_name = f"_skin_{skin_id}_{Path(entry_point).stem}" + with _lock: + cached_entry = sys.modules.get(module_name) + if cached_entry is not None: + return cached_entry + + # Import siblings under their namespaced alias, and *bind* the bare + # name (cached or fresh) so `import helpers` inside the entry module + # resolves to this skin's copy. The bare bindings are transient — + # restored below so another skin's identically-named sibling can't + # be shadowed by ours. + replaced_bare: Dict[str, Any] = {} + try: + for sibling in skin_dir.glob("*.py"): + if sibling.name == entry_point: + continue + alias = f"_skin_{skin_id}_{sibling.stem}" + module = sys.modules.get(alias) + if module is None: + spec = importlib.util.spec_from_file_location(alias, sibling) + if not spec or not spec.loader: + continue + module = importlib.util.module_from_spec(spec) + sys.modules[alias] = module + replaced_bare.setdefault(sibling.stem, sys.modules.get(sibling.stem)) + sys.modules[sibling.stem] = module + try: + spec.loader.exec_module(module) + except Exception as e: + logger.error("Skin '%s' sibling module %s failed to import: %s", + skin_id, sibling.name, e, exc_info=True) + sys.modules.pop(alias, None) + return None + else: + replaced_bare.setdefault(sibling.stem, sys.modules.get(sibling.stem)) + sys.modules[sibling.stem] = module + + try: + spec = importlib.util.spec_from_file_location(module_name, entry_path) + if not spec or not spec.loader: + logger.error("Skin '%s': could not create import spec for %s", + skin_id, entry_path) + return None + module = importlib.util.module_from_spec(spec) + sys.modules[module_name] = module + spec.loader.exec_module(module) + return module + except Exception as e: + sys.modules.pop(module_name, None) + logger.error("Skin '%s' failed to import: %s", skin_id, e, exc_info=True) + return None + finally: + for bare_name, previous in replaced_bare.items(): + if previous is None: + sys.modules.pop(bare_name, None) + else: + sys.modules[bare_name] = previous + + +def load_skin(skin_id: str, sport: Optional[str] = None, + sport_key: Optional[str] = None, + options: Optional[Dict[str, Any]] = None, + skins_dir: Optional[Path] = None) -> Optional[ScoreboardSkin]: + """Load and instantiate a skin. Returns None (after logging why) on + any failure — callers treat None as 'use the built-in renderer'.""" + skins = discover_skins(skins_dir) + manifest = skins.get(skin_id) + if manifest is None: + logger.warning("Skin '%s' is configured but not installed under %s; " + "using built-in renderer", + skin_id, skins_dir or get_skins_directory()) + return None + + manifest_major = _major(manifest.get("skin_api_version")) + api_major = _major(SKIN_API_VERSION) + if manifest_major != api_major: + logger.error("Skin '%s' targets skin API %s but this LEDMatrix " + "provides %s — the skin needs an update; using " + "built-in renderer", + skin_id, manifest.get("skin_api_version"), SKIN_API_VERSION) + return None + + if not skin_matches_target(manifest, sport, sport_key): + # Soft: the user explicitly configured it, so warn but load anyway + # (a baseball skin may render an acceptable generic scoreboard). + logger.warning("Skin '%s' does not declare support for sport=%r / " + "sport_key=%r; loading anyway", skin_id, sport, sport_key) + + skin_dir = Path(manifest["_skin_dir"]) + module = _load_skin_module(skin_id, skin_dir, + manifest.get("entry_point", _DEFAULT_ENTRY_POINT)) + if module is None: + return None + + class_name = manifest["class_name"] + skin_class = getattr(module, class_name, None) + if skin_class is None or not isinstance(skin_class, type) or \ + not issubclass(skin_class, ScoreboardSkin): + logger.error("Skin '%s': %s is missing or not a ScoreboardSkin subclass", + skin_id, class_name) + return None + + try: + return skin_class(manifest, options or {}) + except Exception as e: + logger.error("Skin '%s' failed to instantiate: %s", skin_id, e, exc_info=True) + return None + + +def build_context(host: Any, game: Dict[str, Any], + size: Optional[Tuple[int, int]] = None) -> SkinContext: + """Build a SkinContext for one render call. + + `host` is a SportsCore-style object: display_manager, fonts, logger, + sport, skin_options, _load_and_resize_logo, _draw_text_with_outline. + `size` overrides the canvas size (vegas cards); default is the + current display size read live from the display manager. + """ + if size is not None: + width, height = int(size[0]), int(size[1]) + else: + dm = host.display_manager + width = getattr(dm, "width", None) or dm.matrix.width + height = getattr(dm, "height", None) or dm.matrix.height + + canvas = Image.new("RGB", (width, height), (0, 0, 0)) + draw = ImageDraw.Draw(canvas) + layout = LayoutContext(width, height, _get_font_manager()) + + def load_logo(side: str) -> Optional[Image.Image]: + if side not in ("home", "away"): + return None + try: + logo_path = game.get(f"{side}_logo_path") + if logo_path is not None and not isinstance(logo_path, Path): + logo_path = Path(logo_path) + return host._load_and_resize_logo( + game.get(f"{side}_id"), game.get(f"{side}_abbr"), + logo_path, game.get(f"{side}_logo_url")) + except Exception as e: + host.logger.warning("Skin logo load failed for %s: %s", side, e) + return None + + def draw_text_outlined(text, position, font, fill=(255, 255, 255), + outline_color=(0, 0, 0)): + host._draw_text_with_outline(draw, text, position, font, + fill=fill, outline_color=outline_color) + + return SkinContext( + canvas=canvas, + draw=draw, + layout=layout, + width=width, + height=height, + fonts=dict(host.fonts), + options=dict(getattr(host, "skin_options", {}) or {}), + logger=host.logger, + sport=getattr(host, "sport", None), + load_logo=load_logo, + draw_text_outlined=draw_text_outlined, + ) diff --git a/src/startup_validator.py b/src/startup_validator.py index 86c703f6..79275dc2 100644 --- a/src/startup_validator.py +++ b/src/startup_validator.py @@ -15,16 +15,23 @@ from src.logging_config import get_logger class StartupValidator: """Validates system state on startup.""" - def __init__(self, config_manager: Any, plugin_manager: Optional[Any] = None) -> None: + def __init__(self, config_manager: Any, plugin_manager: Optional[Any] = None, + cache_manager: Optional[Any] = None) -> None: """ Initialize the startup validator. - + Args: config_manager: ConfigManager instance plugin_manager: Optional PluginManager instance + cache_manager: The CacheManager the application will actually use. + Pass it. Without one this validator builds its own just to read + a directory path, which reports on a cache the app does not + use and leaves behind a cleanup thread that nothing stops -- + validation runs twice per startup, so that was two of them. """ self.config_manager = config_manager self.plugin_manager = plugin_manager + self.cache_manager = cache_manager self.logger = get_logger(__name__) self.errors: List[str] = [] self.warnings: List[str] = [] @@ -37,7 +44,12 @@ class StartupValidator: Tuple of (is_valid, errors, warnings) """ self.logger.info("Starting startup validation...") - + + # Fresh lists each run — without this, calling validate_all() twice + # duplicated every message. + self.errors = [] + self.warnings = [] + # Validate configuration self._validate_config() @@ -50,6 +62,9 @@ class StartupValidator: # Validate plugins if plugin manager is available if self.plugin_manager: self._validate_plugins() + + # Warn when the running systemd unit no longer matches the repo's + self._validate_systemd_units() is_valid = len(self.errors) == 0 @@ -62,6 +77,80 @@ class StartupValidator: return (is_valid, self.errors.copy(), self.warnings.copy()) + #: Units this project installs, and where each is installed to. + _UNITS = ( + ("systemd/ledmatrix.service", "/etc/systemd/system/ledmatrix.service"), + ("systemd/ledmatrix-web.service", "/etc/systemd/system/ledmatrix-web.service"), + ) + + def _validate_systemd_units(self) -> None: + """Warn when an installed unit has drifted from the repo's template. + + Nothing re-applies these after the first install. `git pull` -- which is + what the web UI's update button runs -- brings a new template into the + checkout, but nothing copies it to /etc/systemd/system and nothing runs + `systemctl daemon-reload`, so the unit that actually runs is whatever + first_time_install.sh wrote on day one. + + That makes every hardening added to a unit inert on existing installs. + Measured on one rig: the installed unit was thirteen days older than the + repo's and differed in content, so a MemoryMax the repo had specified + was not being enforced at all -- `systemctl show` reported + MemoryMax=infinity. + + A warning rather than an error, and certainly not a silent rewrite: + editing files under /etc and restarting services is the installer's job, + not something a display process should do to a machine while it boots. + The remedy is to re-run scripts/install/install_service.sh. + """ + try: + project_root = Path(__file__).resolve().parent.parent + for template_rel, installed_path in self._UNITS: + template = project_root / template_rel + installed = Path(installed_path) + if not template.is_file() or not installed.is_file(): + continue + + # The template carries placeholders the installer substitutes, + # so compare the substituted form rather than the raw file. + expected = template.read_text(encoding="utf-8") + expected = expected.replace("__PROJECT_ROOT_DIR__", str(project_root)) + expected = expected.replace("__USER__", "root") + + try: + actual = installed.read_text(encoding="utf-8") + except PermissionError: + continue + + if self._unit_body(expected) != self._unit_body(actual): + self.warnings.append( + f"{installed.name} differs from {template_rel}; the " + "installed unit is not refreshed by an update, so " + "settings added to the template are not in effect. " + "Re-run scripts/install/install_service.sh to apply them." + ) + except OSError as e: + self.logger.debug("Could not compare systemd units: %s", e) + + @staticmethod + def _unit_body(text: str) -> str: + """A unit's meaningful lines, in order: no comments, no blanks. + + Order is preserved deliberately. This used to sort, which made the + comparison insensitive to two changes that matter in a systemd unit: + repeated directives such as ExecStartPre= and ExecStartPost= run in + the order they appear, and a directive that moves between [Unit], + [Service] and [Install] means something different -- or nothing -- + where it lands. A drift check that normalises those away reports no + drift for a unit that has genuinely changed. + """ + lines = [] + for line in text.splitlines(): + line = line.strip() + if line and not line.startswith("#"): + lines.append(line) + return "\n".join(lines) + def _validate_config(self) -> None: """Validate configuration files.""" try: @@ -86,9 +175,21 @@ class StartupValidator: def _validate_cache_directory(self) -> None: """Validate cache directory permissions.""" try: - from src.cache_manager import CacheManager - cache_manager = CacheManager() - cache_dir = cache_manager.get_cache_dir() + cache_manager = self.cache_manager + if cache_manager is None: + # No caller supplied one (older embedders, direct use in a + # script). Build one, but do not leave its cleanup thread + # running behind us -- this instance is discarded on the next + # line but the thread is a closure over it, so it would never + # be collected. + from src.cache_manager import CacheManager + cache_manager = CacheManager() + try: + cache_dir = cache_manager.get_cache_dir() + finally: + cache_manager.stop_cleanup_thread() + else: + cache_dir = cache_manager.get_cache_dir() if not cache_dir: self.warnings.append("Cache directory not available - caching will be disabled") diff --git a/src/vegas_mode/config.py b/src/vegas_mode/config.py index 9c930c1b..f1039f5b 100644 --- a/src/vegas_mode/config.py +++ b/src/vegas_mode/config.py @@ -21,10 +21,136 @@ class VegasModeConfig: scroll_speed: float = 50.0 # Pixels per second separator_width: int = 32 # Gap between plugins (pixels) + # Fraction of the panel width a plugin is told it has while rendering for + # the ticker, as a percentage. Trimming can only remove blank margins; it + # cannot compact a layout that genuinely spans the display — a five-column + # forecast, a full-width progress bar, a centred stat block with the panel's + # whole width between its elements. Rendering at a narrower size makes the + # plugin choose a tighter layout instead. 100 disables it. + render_width_pct: int = 100 + + # Minimum blank columns guaranteed between adjacent content, measured from + # actual ink rather than added blindly. A flat additive gap leaves + # card-style content nearly touching when the cards are drawn flush to their + # own edges, while padding out content that already has wide margins. + min_content_separation: int = 24 + + # Gap between rows contributed by the *same* plugin. separator_width marks + # the handoff from one plugin to the next; applying it between every image + # forced a 32px chasm between each row of a per-row ticker (the F1 + # scoreboard renders its own rows 4px apart), which both looked wrong and + # silently inflated the width that plugin occupied. + intra_plugin_gap: int = 8 + + # Content density + # + # Plugins that render onto a full-display canvas contribute that whole + # canvas to the ticker, blank margins included. On a wide panel that is the + # dominant source of dead air: a plugin drawing 35px of text on a 512px + # canvas otherwise buys 9.5s of black at 50px/s. Trimming reclaims it. + auto_trim: bool = True + trim_threshold: int = 10 # Per-channel value a pixel must exceed to be "ink" + content_padding: int = 8 # Blank columns kept either side of trimmed content + min_plugin_width: int = 8 # Segments narrower than this after trim are dropped + + # Columns of blank lead-in before the first item of a cycle. ScrollHelper + # defaults this to a full display width, which reads as the display being + # switched off at the start of every cycle. + lead_in_width: int = 0 + + # Blend between neighbouring pixel positions so motion happens at the frame + # rate rather than the scroll speed. With integer positioning the number of + # distinct frames per second equals scroll_speed, so at 50px/s the motion is + # 50 discrete 1px steps however fast the loop runs. The trade is a slight + # horizontal softening of text, since each frame is a blend of two positions. + smooth_scroll: bool = True + + # Keep one continuous strip, extending it with the next group of plugins as + # the scroll approaches the end, instead of composing a fresh strip and + # swapping it in. A swap stops the motion, substitutes every pixel at once + # and restarts with the viewport already full — read as a freeze, a flash + # and a jump. Extending means the next group simply scrolls in from the + # right. Set false to restore the swap behaviour. + continuous_scroll: bool = True + + # Extend once the unscrolled remainder falls below this many screen widths. + # Needs to be more than one so the join is prepared before it is on screen. + extend_threshold_screens: float = 2.0 + + # How many plugins are composed into one scroll cycle. Kept separate from + # buffer_ahead (which is only a prefetch low-water mark) because the two + # were previously the same number: a buffer_ahead of 2 meant just 3 plugins + # per cycle, so a 20-plugin install took seven cycles to come around. + plugins_per_cycle: int = 6 + + # Minimum run of blank columns that counts as a boundary between items when + # an oversized segment has to be narrowed. Measured on rendered text, the + # gaps between characters are a single column while gaps between items are + # 8px and up, so anything above 1 stops a cut landing inside a word. Cutting + # mid-word orphaned the tail into the next cycle, which showed up as a lone + # letter floating between two unrelated plugins. + min_cut_gap: int = 6 + + # What to do when a plugin's content exceeds its width budget. + # + # "rotate" — advance a window each cycle so everything is seen eventually. + # Right for interchangeable items: news headlines, odds, stocks. + # "truncate" — always show the start. Right for ordered content, where a + # window into the middle is meaningless: a league table that + # shows ranks 1-6 then resumes at 7 two rotations later reads + # as out of order and out of context. + # + # Override per plugin with vegas_overflow. + overflow_mode: str = "rotate" + + # Cap on one plugin's share of a cycle, as a multiple of display width. + # 0 (the default) disables the cap, so every plugin contributes all of its + # content and is always entered at its beginning. + # + # Capping was the default until it proved to cost more than it bought. + # Measured over a 17-plugin fleet on a 512px panel, only four plugins were + # ever wide enough to hit a 3.0 cap; for those four it produced two visible + # faults. Content resumed mid-item on each appearance (a news ticker entered + # at column 6027 of its own strip), and the final window of a rotation was + # whatever happened to be left — 348px of a 1840px stocks ticker, seven + # seconds of panel time. Both read as the display being broken rather than + # as deferral working. + # + # A wide plugin does hold the panel for a long time uncapped: set the cap + # per plugin with vegas_max_width_screens where that matters, rather than + # globally where it mostly hurts plugins that were never the problem. + max_plugin_width_ratio: float = 0.0 + # Plugin management plugin_order: List[str] = field(default_factory=list) excluded_plugins: Set[str] = field(default_factory=set) + # --- Live content in the ticker ------------------------------------- + # + # By default a live game preempts Vegas entirely: the display controller + # refuses to run the ticker while any plugin reports live priority, and you + # get the full-screen scoreboard instead. Set live_in_ticker to keep the + # marquee running and let live content take extra turns within it. + # + # The rotation is otherwise a strict round robin -- every plugin appears + # exactly once per cycle -- so with a dozen plugins enabled a live score + # comes round once a lap and can be minutes old on screen. Weighting lets a + # plugin claim several slots per cycle instead. + # + # Weights are per plugin, not per game: a scoreboard showing four live + # games still occupies one slot at a time, and rotates its own games within + # that slot using its own favorite_live_boost. + live_in_ticker: bool = False + + # Slots per cycle for a plugin reporting live content. 1 disables the boost + # and restores the plain round robin. + live_weight: int = 3 + + # Slots per cycle for a plugin whose live content involves a favorite team. + # Only plugins implementing get_vegas_priority_weight() can claim this -- + # the core cannot tell whose game is on, so the plugin reports it. + favorite_live_weight: int = 5 + # Performance settings target_fps: int = 125 # Target frame rate buffer_ahead: int = 2 # Number of plugins to buffer ahead @@ -55,8 +181,32 @@ class VegasModeConfig: enabled=vegas_config.get('enabled', False), scroll_speed=float(vegas_config.get('scroll_speed', 50.0)), separator_width=int(vegas_config.get('separator_width', 32)), + intra_plugin_gap=int(vegas_config.get('intra_plugin_gap', 8)), + render_width_pct=int(vegas_config.get('render_width_pct', 100)), + min_content_separation=int( + vegas_config.get('min_content_separation', 24)), + min_cut_gap=int(vegas_config.get('min_cut_gap', 6)), + smooth_scroll=vegas_config.get('smooth_scroll', True), + continuous_scroll=vegas_config.get('continuous_scroll', True), + extend_threshold_screens=float( + vegas_config.get('extend_threshold_screens', 2.0)), + auto_trim=vegas_config.get('auto_trim', True), + trim_threshold=int(vegas_config.get('trim_threshold', 10)), + content_padding=int(vegas_config.get('content_padding', 8)), + min_plugin_width=int(vegas_config.get('min_plugin_width', 8)), + lead_in_width=int(vegas_config.get('lead_in_width', 0)), + plugins_per_cycle=int(vegas_config.get('plugins_per_cycle', 6)), + max_plugin_width_ratio=float( + vegas_config.get('max_plugin_width_ratio', 0.0)), + overflow_mode=str(vegas_config.get('overflow_mode', 'rotate')), plugin_order=list(vegas_config.get('plugin_order', [])), excluded_plugins=set(vegas_config.get('excluded_plugins', [])), + live_in_ticker=bool(vegas_config.get('live_in_ticker', False)), + # Clamped: a weight below 1 would drop the plugin from the rotation + # entirely, and a very large one starves everything else. + live_weight=max(1, min(10, int(vegas_config.get('live_weight', 3)))), + favorite_live_weight=max( + 1, min(10, int(vegas_config.get('favorite_live_weight', 5)))), target_fps=int(vegas_config.get('target_fps', 125)), buffer_ahead=int(vegas_config.get('buffer_ahead', 2)), frame_based_scrolling=vegas_config.get('frame_based_scrolling', True), @@ -72,6 +222,24 @@ class VegasModeConfig: 'enabled': self.enabled, 'scroll_speed': self.scroll_speed, 'separator_width': self.separator_width, + 'intra_plugin_gap': self.intra_plugin_gap, + 'render_width_pct': self.render_width_pct, + 'min_content_separation': self.min_content_separation, + 'min_cut_gap': self.min_cut_gap, + 'smooth_scroll': self.smooth_scroll, + 'continuous_scroll': self.continuous_scroll, + 'extend_threshold_screens': self.extend_threshold_screens, + 'auto_trim': self.auto_trim, + 'trim_threshold': self.trim_threshold, + 'content_padding': self.content_padding, + 'min_plugin_width': self.min_plugin_width, + 'lead_in_width': self.lead_in_width, + 'plugins_per_cycle': self.plugins_per_cycle, + 'max_plugin_width_ratio': self.max_plugin_width_ratio, + 'live_in_ticker': self.live_in_ticker, + 'live_weight': self.live_weight, + 'favorite_live_weight': self.favorite_live_weight, + 'overflow_mode': self.overflow_mode, 'plugin_order': self.plugin_order, 'excluded_plugins': list(self.excluded_plugins), 'target_fps': self.target_fps, @@ -157,6 +325,74 @@ class VegasModeConfig: if self.buffer_ahead > 5: errors.append(f"buffer_ahead must be <= 5, got {self.buffer_ahead}") + if not 10 <= self.render_width_pct <= 100: + errors.append( + "render_width_pct must be between 10 and 100, " + f"got {self.render_width_pct}") + + if not 0 <= self.min_content_separation <= 256: + errors.append( + "min_content_separation must be between 0 and 256, " + f"got {self.min_content_separation}") + + if not 1.0 <= self.extend_threshold_screens <= 10.0: + errors.append( + "extend_threshold_screens must be between 1.0 and 10.0, " + f"got {self.extend_threshold_screens}") + + if not 1 <= self.min_cut_gap <= 128: + errors.append( + "min_cut_gap must be between 1 and 128, " + f"got {self.min_cut_gap}") + + if self.intra_plugin_gap < 0: + errors.append( + f"intra_plugin_gap must be >= 0, got {self.intra_plugin_gap}") + if self.intra_plugin_gap > 128: + errors.append( + f"intra_plugin_gap must be <= 128, got {self.intra_plugin_gap}") + + if not 0 <= self.trim_threshold <= 254: + errors.append( + f"trim_threshold must be between 0 and 254, got {self.trim_threshold}") + + if self.content_padding < 0: + errors.append( + f"content_padding must be >= 0, got {self.content_padding}") + if self.content_padding > 128: + errors.append( + f"content_padding must be <= 128, got {self.content_padding}") + + if self.min_plugin_width < 0: + errors.append( + f"min_plugin_width must be >= 0, got {self.min_plugin_width}") + # Bounded because every segment narrower than this is dropped — an + # unbounded value would discard every plugin and leave a blank ticker. + if self.min_plugin_width > 512: + errors.append( + f"min_plugin_width must be <= 512, got {self.min_plugin_width}") + + if self.lead_in_width < 0: + errors.append( + f"lead_in_width must be >= 0, got {self.lead_in_width}") + + if self.plugins_per_cycle < 1: + errors.append( + f"plugins_per_cycle must be >= 1, got {self.plugins_per_cycle}") + if self.plugins_per_cycle > 50: + errors.append( + f"plugins_per_cycle must be <= 50, got {self.plugins_per_cycle}") + + if self.overflow_mode not in ('rotate', 'truncate'): + errors.append( + "overflow_mode must be 'rotate' or 'truncate', " + f"got {self.overflow_mode!r}") + + if self.max_plugin_width_ratio < 0: + errors.append( + "max_plugin_width_ratio must be >= 0 " + f"(0 disables the cap), got {self.max_plugin_width_ratio}") + return errors def update(self, new_config: Dict[str, Any]) -> None: @@ -170,10 +406,52 @@ class VegasModeConfig: if 'enabled' in vegas_config: self.enabled = vegas_config['enabled'] + if 'live_in_ticker' in vegas_config: + self.live_in_ticker = bool(vegas_config['live_in_ticker']) + # Clamped exactly as from_config does: a weight below 1 would drop the + # plugin from the rotation, and a huge one starves everything else. + if 'live_weight' in vegas_config: + self.live_weight = max(1, min(10, int(vegas_config['live_weight']))) + if 'favorite_live_weight' in vegas_config: + self.favorite_live_weight = max( + 1, min(10, int(vegas_config['favorite_live_weight']))) if 'scroll_speed' in vegas_config: self.scroll_speed = float(vegas_config['scroll_speed']) if 'separator_width' in vegas_config: self.separator_width = int(vegas_config['separator_width']) + if 'intra_plugin_gap' in vegas_config: + self.intra_plugin_gap = int(vegas_config['intra_plugin_gap']) + if 'render_width_pct' in vegas_config: + self.render_width_pct = int(vegas_config['render_width_pct']) + if 'min_content_separation' in vegas_config: + self.min_content_separation = int( + vegas_config['min_content_separation']) + if 'min_cut_gap' in vegas_config: + self.min_cut_gap = int(vegas_config['min_cut_gap']) + if 'smooth_scroll' in vegas_config: + self.smooth_scroll = vegas_config['smooth_scroll'] + if 'continuous_scroll' in vegas_config: + self.continuous_scroll = vegas_config['continuous_scroll'] + if 'extend_threshold_screens' in vegas_config: + self.extend_threshold_screens = float( + vegas_config['extend_threshold_screens']) + if 'auto_trim' in vegas_config: + self.auto_trim = vegas_config['auto_trim'] + if 'trim_threshold' in vegas_config: + self.trim_threshold = int(vegas_config['trim_threshold']) + if 'content_padding' in vegas_config: + self.content_padding = int(vegas_config['content_padding']) + if 'min_plugin_width' in vegas_config: + self.min_plugin_width = int(vegas_config['min_plugin_width']) + if 'lead_in_width' in vegas_config: + self.lead_in_width = int(vegas_config['lead_in_width']) + if 'plugins_per_cycle' in vegas_config: + self.plugins_per_cycle = int(vegas_config['plugins_per_cycle']) + if 'max_plugin_width_ratio' in vegas_config: + self.max_plugin_width_ratio = float( + vegas_config['max_plugin_width_ratio']) + if 'overflow_mode' in vegas_config: + self.overflow_mode = str(vegas_config['overflow_mode']) if 'plugin_order' in vegas_config: self.plugin_order = list(vegas_config['plugin_order']) if 'excluded_plugins' in vegas_config: diff --git a/src/vegas_mode/coordinator.py b/src/vegas_mode/coordinator.py index 42a3fd15..430cbef6 100644 --- a/src/vegas_mode/coordinator.py +++ b/src/vegas_mode/coordinator.py @@ -12,6 +12,7 @@ Supports three display modes per plugin: """ import logging +import math import time import threading from typing import Optional, Dict, Any, List, Callable, TYPE_CHECKING @@ -30,6 +31,21 @@ if TYPE_CHECKING: logger = logging.getLogger(__name__) +def _percentile(ordered: List[float], fraction: float) -> float: + """Nearest-rank percentile of an already-sorted list. + + Index ceil(n * fraction) - 1, so 100 samples at 0.99 give the 99th-ranked + value. The obvious int(n * fraction) is off by one and, at exactly 100 + samples, lands on the maximum -- which is the number already reported + alongside this one as the worst frame, so the two columns would agree + precisely when the sample was smallest. + """ + if not ordered: + return 0.0 + index = math.ceil(len(ordered) * fraction) - 1 + return ordered[min(len(ordered) - 1, max(0, index))] + + class VegasModeCoordinator: """ Orchestrates Vegas scroll mode operation. @@ -64,7 +80,7 @@ class VegasModeCoordinator: self.plugin_manager = plugin_manager # Initialize components - self.plugin_adapter = PluginAdapter(display_manager) + self.plugin_adapter = PluginAdapter(display_manager, self.vegas_config) self.stream_manager = StreamManager( self.vegas_config, plugin_manager, @@ -233,6 +249,11 @@ class VegasModeCoordinator: self._should_stop = False self._start_time = time.time() + # Line up the next group immediately, so the first extension is already + # warm rather than stalling the scroll to fetch it. + if self.vegas_config.continuous_scroll: + self.render_pipeline.start_prefetch() + logger.info("Vegas mode started") return True @@ -301,16 +322,43 @@ class VegasModeCoordinator: if has_pending_update: self._apply_pending_config() - # Check if we need to start a new cycle - if self.render_pipeline.is_cycle_complete(): - if not self.render_pipeline.start_new_cycle(): - logger.warning("Failed to start new Vegas cycle") - return False - self.stats['cycles_completed'] += 1 + if self.vegas_config.continuous_scroll: + # Drop cached content for plugins whose data just changed, so the + # next time each comes round it is composed from current data. The + # swap path's hot_swap_content() does this via process_updates(), + # but it also rebuilds and repositions the whole strip, which is + # the freeze-and-jump this mode exists to avoid. Without this the + # pending-update flags are never consumed and a segment keeps + # rendering whatever it was first built from — last night's live + # game still shown as live the next morning. + self.render_pipeline.refresh_updated_plugins() - # Check for hot-swap opportunities - if self.render_pipeline.should_recompose(): - self.render_pipeline.hot_swap_content() + # Extend the strip before the scroll can reach its end, so the next + # group arrives from the right and motion never stops. No cycle + # boundary, so no freeze, no substitution and no restart with the + # viewport already full. + # Trickle in the plugins that can only be fetched here, one per + # frame, before considering a further extension. + if self.render_pipeline.has_deferred(): + self.render_pipeline.drain_deferred() + elif self.render_pipeline.needs_extension(): + if self.render_pipeline.extend_scroll_content(): + self.stats['cycles_completed'] += 1 + elif self.render_pipeline.is_cycle_complete(): + # Extension failed and the strip has run out: fall back to + # the swap rather than sitting on a dead frame. + self.render_pipeline.start_new_cycle() + else: + # Check if we need to start a new cycle + if self.render_pipeline.is_cycle_complete(): + if not self.render_pipeline.start_new_cycle(): + logger.warning("Failed to start new Vegas cycle") + return False + self.stats['cycles_completed'] += 1 + + # Check for hot-swap opportunities + if self.render_pipeline.should_recompose(): + self.render_pipeline.hot_swap_content() # Render frame return self.render_pipeline.render_frame() @@ -337,16 +385,31 @@ class VegasModeCoordinator: self._update_static_mode_plugins() frame_interval = self.vegas_config.get_frame_interval() - duration = self.render_pipeline.get_dynamic_duration() + if self.vegas_config.continuous_scroll: + # The strip is continuously extended and trimmed, so its width says + # nothing about how long to run. This is only how often control + # returns to the display controller; interrupts are still checked + # every few frames, so it costs nothing to make it a fixed period. + duration = float(self.vegas_config.max_cycle_duration) + else: + duration = self.render_pipeline.get_dynamic_duration() start_time = time.time() frame_count = 0 fps_log_interval = 5.0 # Log FPS every 5 seconds last_fps_log_time = start_time fps_frame_count = 0 + # A mean hides stutter completely. At 120fps a five-second window is + # ~600 frames, so a 200ms freeze -- plainly visible on a marquee -- + # moves the average from 120.0 to 115.4 and reads as healthy. What a + # viewer actually notices is the worst frame, so track that too. + frame_worst = 0.0 + frame_times: List[float] = [] logger.info("Starting Vegas iteration for %.1fs", duration) while True: + frame_started = time.time() + # Check for STATIC mode plugin that should pause scroll static_plugin = self._check_static_plugin_trigger() if static_plugin: @@ -367,8 +430,19 @@ class VegasModeCoordinator: # Paused for live priority - let caller handle return False - # Sleep for frame interval - time.sleep(frame_interval) + # Sleep only the remainder of the frame budget. This used to sleep + # the whole interval on top of however long the frame took, so at a + # measured 31.6ms per frame a fixed 8ms of that was pure idle — a + # quarter of the budget spent not rendering. Subtracting the work + # already done keeps the pacing target while reclaiming that time, + # and yields the GIL either way so other threads still run. + frame_elapsed = time.time() - frame_started + time.sleep(max(0.0, frame_interval - frame_elapsed)) + + # Measured before the sleep: time spent working, not pacing. + if frame_elapsed > frame_worst: + frame_worst = frame_elapsed + frame_times.append(frame_elapsed) # Increment frame count and check for interrupt periodically frame_count += 1 @@ -378,12 +452,16 @@ class VegasModeCoordinator: current_time = time.time() if current_time - last_fps_log_time >= fps_log_interval: fps = fps_frame_count / (current_time - last_fps_log_time) + p99 = _percentile(sorted(frame_times), 0.99) logger.info( - "Vegas FPS: %.1f (target: %d, frames: %d)", - fps, self.vegas_config.target_fps, fps_frame_count + "Vegas FPS: %.1f (target: %d, frames: %d) p99 %.1fms worst %.1fms", + fps, self.vegas_config.target_fps, fps_frame_count, + p99 * 1000.0, frame_worst * 1000.0 ) last_fps_log_time = current_time fps_frame_count = 0 + frame_worst = 0.0 + frame_times.clear() if (self._interrupt_check and frame_count % self._interrupt_check_interval == 0): @@ -450,6 +528,12 @@ class VegasModeCoordinator: if not self._live_priority_check: return False + if self.vegas_config.live_in_ticker: + # The ticker keeps live content rather than yielding to it; the + # extra turns are arranged in the rotation itself, so there is + # nothing to pause for. + return False + try: live_mode = self._live_priority_check() if live_mode: @@ -505,6 +589,10 @@ class VegasModeCoordinator: # Update components self.render_pipeline.update_config(new_vegas_config) self.stream_manager.config = new_vegas_config + self.plugin_adapter.config = new_vegas_config + # Cached segments were trimmed under the old settings, so drop them + # or a changed trim/padding value would not visibly take effect. + self.plugin_adapter.invalidate_cache() # Force refresh of stream manager to pick up plugin_order/buffer changes self.stream_manager._last_refresh = 0 diff --git a/src/vegas_mode/geometry.py b/src/vegas_mode/geometry.py new file mode 100644 index 00000000..2d6e3b8e --- /dev/null +++ b/src/vegas_mode/geometry.py @@ -0,0 +1,474 @@ +""" +Geometry primitives for Vegas Mode. + +Pure, side-effect-free measurements over PIL images. Two consumers: + +- ``PluginAdapter`` trims the blank margins plugins bake into their content + before it enters the ticker (see ``trim_to_content``). +- ``scripts/dev/vegas_audit.py`` reports how much of the composed ticker is + dead space (see ``dead_window_stats``). + +Keeping both on the same primitives means the number the audit reports is the +number the trimmer acted on. + +All column scans go through numpy: a Python-level per-column loop over a +17,000px-wide ticker image takes seconds, which is far too slow for the render +path. +""" + +from typing import List, NamedTuple, Optional, Tuple + +import numpy as np +from PIL import Image + +# A pixel counts as "ink" when any channel exceeds this. Chosen to ignore the +# 1-2/255 noise that JPEG-sourced logos and alpha compositing leave behind in +# nominally black areas, while still treating any deliberately drawn dark grey +# as real content. +DEFAULT_INK_THRESHOLD = 10 + +# A window counts as "dead" when this fraction of its columns carry no ink. +DEFAULT_DEAD_WINDOW_RATIO = 0.95 + + +def column_has_ink(img: Image.Image, threshold: int = DEFAULT_INK_THRESHOLD) -> np.ndarray: + """ + Return a boolean array, one entry per image column, True where the column + contains at least one pixel brighter than ``threshold`` in any channel. + + Args: + img: Image to scan (converted to RGB internally) + threshold: Per-channel value a pixel must exceed to count as ink + + Returns: + Bool array of shape (width,) + """ + arr = np.asarray(img if img.mode == 'RGB' else img.convert('RGB')) + if arr.ndim != 3: + # Degenerate/empty image — treat every column as blank. + return np.zeros(img.width, dtype=bool) + # Collapse rows and channels: a column is ink if any pixel in it is bright. + return arr.max(axis=(0, 2)) > threshold + + +def content_bounds( + img: Image.Image, threshold: int = DEFAULT_INK_THRESHOLD +) -> Optional[Tuple[int, int]]: + """ + Find the first and last columns containing ink. + + Args: + img: Image to measure + threshold: Ink threshold + + Returns: + (first_col, last_col) inclusive, or None if the image is entirely blank + """ + ink = column_has_ink(img, threshold) + if not ink.any(): + return None + first = int(ink.argmax()) + last = len(ink) - 1 - int(ink[::-1].argmax()) + return first, last + + +class TrimResult(NamedTuple): + """Outcome of a ``trim_to_content`` call.""" + + image: Optional[Image.Image] # None when the source was entirely blank + original_width: int + trimmed_left: int + trimmed_right: int + + @property + def is_blank(self) -> bool: + """True when the source image carried no ink at all.""" + return self.image is None + + @property + def width(self) -> int: + """Width after trimming (0 for a blank source).""" + return 0 if self.image is None else self.image.width + + @property + def removed(self) -> int: + """Total columns removed.""" + return self.trimmed_left + self.trimmed_right + + +def trim_to_content( + img: Image.Image, + threshold: int = DEFAULT_INK_THRESHOLD, + padding: int = 0, +) -> TrimResult: + """ + Crop blank columns off the left and right edges of an image. + + Only the outer edges are considered. Blank columns *between* two pieces of + content are deliberately preserved — those are the plugin's own layout + (e.g. a logo on the left and a score on the right), and closing them up + would corrupt the design rather than reclaim dead space. + + A plugin drawing on a non-black background is unaffected: every column of a + filled background carries ink, so there is nothing to trim. + + Args: + img: Image to trim + threshold: Ink threshold + padding: Columns of the original blank margin to keep on each side, as + breathing room. Capped at what the margin actually contains, so + this never widens the image beyond its original bounds. + + Returns: + TrimResult. When the image is entirely blank, ``image`` is None and the + caller decides whether to skip the plugin. + """ + bounds = content_bounds(img, threshold) + if bounds is None: + return TrimResult(None, img.width, 0, 0) + + first, last = bounds + pad = max(0, padding) + left = max(0, first - pad) + right = min(img.width, last + 1 + pad) + + if left == 0 and right == img.width: + return TrimResult(img, img.width, 0, 0) + + cropped = img.crop((left, 0, right, img.height)) + return TrimResult(cropped, img.width, left, img.width - right) + + +def edge_blank( + img: Image.Image, threshold: int = DEFAULT_INK_THRESHOLD +) -> Tuple[int, int]: + """ + Blank column counts at the left and right edges of an image. + + Used to space items by *measured* separation rather than a flat added gap. + A fixed gap gets this wrong in both directions at once: card-style content + drawn flush to its own edges ends up nearly touching its neighbour, while + content that already carries wide margins gets pushed even further apart. + + Args: + img: Image to measure + threshold: Ink threshold + + Returns: + (left_blank, right_blank). For an entirely blank image both are the + full width, since there is no ink to be close to. + """ + bounds = content_bounds(img, threshold) + if bounds is None: + return img.width, img.width + first, last = bounds + return first, img.width - 1 - last + + +def separation_gap( + left_img: Image.Image, + right_img: Image.Image, + target: int, + minimum: int = 0, + threshold: int = DEFAULT_INK_THRESHOLD, +) -> int: + """ + Columns to insert between two images so their ink is ``target`` apart. + + Only the shortfall is added: if the two images already carry enough blank + at the facing edges, nothing (beyond ``minimum``) is inserted. + + Args: + left_img: Image on the left + right_img: Image on the right + target: Desired blank columns between the two pieces of ink + minimum: Floor applied regardless of what the images already have + threshold: Ink threshold + + Returns: + Number of columns to insert, never negative + """ + existing = edge_blank(left_img, threshold)[1] + edge_blank(right_img, threshold)[0] + return max(minimum, target - existing, 0) + + +def blank_runs( + img: Image.Image, + min_run: int, + threshold: int = DEFAULT_INK_THRESHOLD, +) -> List[Tuple[int, int]]: + """ + Find maximal runs of blank columns at least ``min_run`` wide. + + Distinguishes item boundaries from letter spacing. Measured on real + rendered text, the gaps *between characters* are a single column, while the + gaps a plugin puts *between items* are 8px and up (the stocks ticker uses + 32px, baseball 48px). Treating any blank column as a cut point therefore + slices words in half; requiring a run excludes letter spacing. + + Args: + img: Image to scan + min_run: Minimum consecutive blank columns to qualify + threshold: Ink threshold + + Returns: + List of (start, end) half-open column ranges, in left-to-right order + """ + blank = ~column_has_ink(img, threshold) + if not blank.any(): + return [] + + # Vectorised run detection: pad with False so runs touching either edge get + # a boundary, then read starts and ends off the first difference. A Python + # loop here would be far too slow on a 17,000px ticker strip. + padded = np.concatenate(([False], blank, [False])) + diff = np.diff(padded.astype(np.int8)) + starts = np.flatnonzero(diff == 1) + ends = np.flatnonzero(diff == -1) + + long_enough = (ends - starts) >= max(1, min_run) + return list(zip(starts[long_enough].tolist(), ends[long_enough].tolist())) + + +def find_item_boundary( + img: Image.Image, + target: int, + min_run: int, + threshold: int = DEFAULT_INK_THRESHOLD, +) -> Optional[int]: + """ + Find the column nearest ``target`` that sits inside a gap between items. + + Used to narrow an oversized segment without cutting through a word. Only + runs of at least ``min_run`` blank columns are considered, so the + single-column gaps between characters are never chosen — cutting there + orphaned the tail of a word into the following cycle, which is how a lone + "y" from "Wednesday" ended up floating between two unrelated plugins. + + Args: + img: Image to cut + target: Preferred cut column + min_run: Minimum blank-run width that counts as an item boundary + threshold: Ink threshold + + Returns: + A column inside a qualifying gap, or None when the image has no such + gap at all — in which case the caller must not cut it. + """ + runs = blank_runs(img, min_run, threshold) + if not runs: + return None + + # Nearest point of the nearest run. For a run left of target that is its + # end (content resumes just after), for a run right of target its start + # (content stopped just before) — the right choice in both directions. + def clamp_to_run(run: Tuple[int, int]) -> int: + start, end = run + return max(start, min(target, end - 1)) + + return min((clamp_to_run(r) for r in runs), key=lambda c: abs(c - target)) + + +def find_blank_cut( + img: Image.Image, + target: int, + search_radius: int, + threshold: int = DEFAULT_INK_THRESHOLD, +) -> int: + """ + Find a column near ``target`` that carries no ink, so an image can be cut + there without slicing through a glyph or logo. + + Used when a single oversized segment has to be narrowed to fit a width + budget. Cutting at an arbitrary column would leave half a character + hanging at the panel edge; snapping to the nearest gap hides the cut. + + Args: + img: Image to cut + target: Preferred cut column + search_radius: How far either side of ``target`` to look + threshold: Ink threshold + + Returns: + A blank column within the search window, or ``target`` clamped to the + image bounds when the window contains no blank column at all. + """ + width = img.width + target = max(0, min(target, width)) + if search_radius <= 0 or width == 0: + return target + + ink = column_has_ink(img, threshold) + + # target may legitimately equal width (a cut after the last column), but + # there is no column to inspect there, so both bounds stop at width - 1. + lo = max(0, min(target - search_radius, width - 1)) + hi = max(0, min(target + search_radius, width - 1)) + + # Walk outwards from target so the nearest gap wins. + for offset in range(0, search_radius + 1): + right = target + offset + if lo <= right <= hi and not ink[right]: + return right + left = target - offset + if lo <= left <= hi and not ink[left]: + return left + + return target + + +class DeadWindowStats(NamedTuple): + """How much of a composed ticker reads as blank to a viewer.""" + + total_windows: int + dead_windows: int + longest_dead_run: int # consecutive dead windows (i.e. scroll steps) + + @property + def dead_ratio(self) -> float: + """Fraction of viewport positions that are effectively blank.""" + if self.total_windows <= 0: + return 0.0 + return self.dead_windows / self.total_windows + + +def dead_window_stats( + img: Image.Image, + viewport_width: int, + threshold: int = DEFAULT_INK_THRESHOLD, + dead_ratio: float = DEFAULT_DEAD_WINDOW_RATIO, + step: int = 1, +) -> DeadWindowStats: + """ + Slide a viewport across a composed ticker image and count how many + positions are effectively blank. + + This models what the viewer actually experiences: the ticker is only ever + seen ``viewport_width`` columns at a time, so a stretch of blank wider than + the viewport becomes a period where the panel looks switched off. Measuring + per-window rather than per-column is what makes the result correspond to + perceived dead time. + + Args: + img: Composed ticker image + viewport_width: Display width in pixels + threshold: Ink threshold + dead_ratio: Fraction of blank columns for a window to count as dead + step: Column stride between sampled windows. 1 is exact; larger values + trade precision for speed on very wide images. + + Returns: + DeadWindowStats. ``longest_dead_run`` is in units of ``step`` columns, + so multiply by ``step`` for pixels. + """ + if viewport_width <= 0 or img.width <= 0: + return DeadWindowStats(0, 0, 0) + + ink = column_has_ink(img, threshold) + step = max(1, step) + + # Prefix sum of ink counts lets each window be evaluated in constant time, + # instead of re-summing viewport_width columns per position. + prefix = np.concatenate(([0], np.cumsum(ink))) + + # Only whole windows are sampled; a partial tail window would report + # artificially dead because it has fewer columns to draw ink from. + last_start = img.width - viewport_width + if last_start < 0: + # Image narrower than the viewport — evaluate it as a single window. + blank_cols = len(ink) - int(prefix[-1]) + is_dead = blank_cols >= dead_ratio * len(ink) + return DeadWindowStats(1, 1 if is_dead else 0, 1 if is_dead else 0) + + starts = np.arange(0, last_start + 1, step) + ink_counts = prefix[starts + viewport_width] - prefix[starts] + blank_counts = viewport_width - ink_counts + dead = blank_counts >= dead_ratio * viewport_width + + longest = _longest_true_run(dead) + return DeadWindowStats(len(starts), int(dead.sum()), longest) + + +class CoverageStats(NamedTuple): + """How well-filled the viewport stays as the ticker scrolls past.""" + + total_windows: int + mean_ink_ratio: float # average fraction of the viewport carrying ink + min_ink_ratio: float # worst viewport position in the cycle + sparse_windows: int # positions below the "looks empty" threshold + longest_sparse_run: int # consecutive sparse positions, in steps + + @property + def sparse_ratio(self) -> float: + """Fraction of viewport positions that read as near-empty.""" + if self.total_windows <= 0: + return 0.0 + return self.sparse_windows / self.total_windows + + +def window_coverage_stats( + img: Image.Image, + viewport_width: int, + threshold: int = DEFAULT_INK_THRESHOLD, + sparse_ink_ratio: float = 0.10, + step: int = 1, +) -> CoverageStats: + """ + Measure how full the viewport stays across a whole scroll cycle. + + ``dead_window_stats`` only catches viewport positions that are *entirely* + blank. That misses the more common complaint: a position holding one narrow + sliver of content at the very edge, with the other 90% black. Such a + position is not "dead" by that definition but still looks switched off. + This function grades every position by how much ink it carries, so + "there is always something to see" becomes measurable. + + Args: + img: Composed ticker image + viewport_width: Display width in pixels + threshold: Ink threshold + sparse_ink_ratio: A position with less than this fraction of inked + columns counts as reading near-empty + step: Column stride between sampled positions + + Returns: + CoverageStats + """ + if viewport_width <= 0 or img.width <= 0: + return CoverageStats(0, 0.0, 0.0, 0, 0) + + ink = column_has_ink(img, threshold) + step = max(1, step) + prefix = np.concatenate(([0], np.cumsum(ink))) + + last_start = img.width - viewport_width + if last_start < 0: + ratio = float(prefix[-1]) / viewport_width + sparse = ratio < sparse_ink_ratio + return CoverageStats(1, ratio, ratio, 1 if sparse else 0, 1 if sparse else 0) + + starts = np.arange(0, last_start + 1, step) + ratios = (prefix[starts + viewport_width] - prefix[starts]) / viewport_width + sparse_flags = ratios < sparse_ink_ratio + + return CoverageStats( + total_windows=len(starts), + mean_ink_ratio=float(ratios.mean()), + min_ink_ratio=float(ratios.min()), + sparse_windows=int(sparse_flags.sum()), + longest_sparse_run=_longest_true_run(sparse_flags), + ) + + +def _longest_true_run(flags: np.ndarray) -> int: + """Length of the longest consecutive run of True in a boolean array.""" + if flags.size == 0 or not flags.any(): + return 0 + # Reset a running counter at every False by subtracting the cumulative max + # of the counter's value at the preceding False positions. + idx = np.arange(len(flags)) + not_flag = ~flags + # For each position, the index of the most recent False at or before it. + last_false = np.maximum.accumulate(np.where(not_flag, idx, -1)) + run_lengths = idx - last_false + return int(run_lengths[flags].max()) diff --git a/src/vegas_mode/plugin_adapter.py b/src/vegas_mode/plugin_adapter.py index b50d0946..01ef8cca 100644 --- a/src/vegas_mode/plugin_adapter.py +++ b/src/vegas_mode/plugin_adapter.py @@ -8,9 +8,16 @@ implement get_vegas_content() and fallback capture of display() output. import logging import threading import time +from contextlib import nullcontext from typing import Optional, List, Any, Tuple, Union, TYPE_CHECKING from PIL import Image +from src.vegas_mode.geometry import ( + blank_runs, + separation_gap, + trim_to_content, +) + if TYPE_CHECKING: from src.plugin_system.base_plugin import BasePlugin @@ -26,14 +33,21 @@ class PluginAdapter: 2. Fallback: Capture display_manager.image after calling plugin.display() """ - def __init__(self, display_manager: Any): + def __init__(self, display_manager: Any, config: Optional[Any] = None): """ Initialize the plugin adapter. Args: display_manager: DisplayManager instance for fallback capture + config: VegasModeConfig controlling trim behaviour. When omitted, + trimming runs with the dataclass defaults, so existing callers + and tests keep working unchanged. """ self.display_manager = display_manager + if config is None: + from src.vegas_mode.config import VegasModeConfig + config = VegasModeConfig() + self.config = config # Handle both property and method access patterns self.display_width = ( display_manager.width() if callable(display_manager.width) @@ -49,12 +63,33 @@ class PluginAdapter: self._cache_lock = threading.Lock() self._cache_ttl = 5.0 # Cache for 5 seconds - logger.info( + # Per-plugin rotation offset, so a plugin whose content exceeds its + # width budget shows a different slice on each cycle rather than + # always the same opening items. + self._item_offsets: dict = {} + + # What the matching entry in _item_offsets is an offset *into*, as + # (kind, size). An offset only means anything against the content it + # was derived from, and there are three incompatible kinds: + # + # ('rows', n) index into a list of n images + # ('cuts', n) index into the n item boundaries of one image + # ('cols', w) pixel column in a w-wide image with no item boundaries + # + # Without this the offsets were reused across kinds — a plugin that + # returned one wide image on one fetch and several rows on the next had + # a pixel column of 1400 read back as a row index — and across content + # changes, where a column recorded against a 9,793px news strip pointed + # into unrelated headlines once the strip refreshed to 9,505px. + self._offset_shapes: dict = {} + + logger.debug( "PluginAdapter initialized: display=%dx%d", self.display_width, self.display_height ) - def get_content(self, plugin: 'BasePlugin', plugin_id: str) -> Optional[List[Image.Image]]: + def get_content(self, plugin: 'BasePlugin', plugin_id: str, + offscreen_only: bool = False) -> Optional[List[Image.Image]]: """ Get scrollable content from a plugin. @@ -63,11 +98,18 @@ class PluginAdapter: Args: plugin: Plugin instance to get content from plugin_id: Plugin identifier for logging + offscreen_only: Skip every path that touches the shared display + canvas, for callers running off the render thread. The canvas + and the matrix proxy are process-wide mutable state, so + narrowing or capturing through them from another thread would + corrupt the frame the render loop is pushing. Returns None when + the plugin can only be served that way, leaving the caller to + fetch it on the render thread. Returns: List of PIL Images representing plugin content, or None if no content """ - logger.info( + logger.debug( "[%s] Getting content (class=%s)", plugin_id, plugin.__class__.__name__ ) @@ -76,7 +118,7 @@ class PluginAdapter: cached = self._get_cached(plugin_id) if cached is not None: total_width = sum(img.width for img in cached) - logger.info( + logger.debug( "[%s] Using cached content: %d images, %dpx total", plugin_id, len(cached), total_width ) @@ -84,45 +126,50 @@ class PluginAdapter: # Try native Vegas content method first has_native = hasattr(plugin, 'get_vegas_content') - logger.info("[%s] Has get_vegas_content: %s", plugin_id, has_native) + logger.debug("[%s] Has get_vegas_content: %s", plugin_id, has_native) if has_native: - content = self._get_native_content(plugin, plugin_id) + content = self._get_native_content(plugin, plugin_id, offscreen_only) if content: total_width = sum(img.width for img in content) - logger.info( + logger.debug( "[%s] Native content SUCCESS: %d images, %dpx total", plugin_id, len(content), total_width ) - self._cache_content(plugin_id, content) - return content - logger.info("[%s] Native content returned None", plugin_id) + return self._finalize(content, plugin_id, 'native', plugin) + logger.debug("[%s] Native content returned None", plugin_id) # Try to get scroll_helper's cached image (for scrolling plugins like stocks/odds) has_scroll_helper = hasattr(plugin, 'scroll_helper') - logger.info("[%s] Has scroll_helper: %s", plugin_id, has_scroll_helper) - content = self._get_scroll_helper_content(plugin, plugin_id) + logger.debug("[%s] Has scroll_helper: %s", plugin_id, has_scroll_helper) + content = self._get_scroll_helper_content(plugin, plugin_id, offscreen_only) if content: total_width = sum(img.width for img in content) - logger.info( + logger.debug( "[%s] ScrollHelper content SUCCESS: %d images, %dpx total", plugin_id, len(content), total_width ) - self._cache_content(plugin_id, content) - return content + return self._finalize(content, plugin_id, 'scroll_helper', plugin) if has_scroll_helper: - logger.info("[%s] ScrollHelper content returned None", plugin_id) + logger.debug("[%s] ScrollHelper content returned None", plugin_id) + + if offscreen_only: + # Display capture needs the shared canvas; leave it to the caller. + logger.debug( + "[%s] Needs display capture, deferring to the render thread", + plugin_id + ) + return None # Fall back to display capture - logger.info("[%s] Trying fallback display capture...", plugin_id) + logger.debug("[%s] Trying fallback display capture...", plugin_id) content = self._capture_display_content(plugin, plugin_id) if content: total_width = sum(img.width for img in content) - logger.info( + logger.debug( "[%s] Fallback capture SUCCESS: %d images, %dpx total", plugin_id, len(content), total_width ) - self._cache_content(plugin_id, content) - return content + return self._finalize(content, plugin_id, 'fallback', plugin) logger.warning( "[%s] NO CONTENT from any method (native=%s, scroll_helper=%s, fallback=tried)", @@ -130,8 +177,515 @@ class PluginAdapter: ) return None + def _finalize( + self, images: List[Image.Image], plugin_id: str, source: str, + plugin: Optional['BasePlugin'] = None + ) -> Optional[List[Image.Image]]: + """ + Trim dead space off a segment, then cache it. + + Every content path funnels through here so trimming is applied + uniformly. Previously only the scroll_helper path had its margins + stripped, which left plugins that render onto a full-display canvas + contributing their entire blank canvas to the ticker. + + Each image is trimmed independently because compose_scroll_content() + treats every image as its own item and inserts separator_width between + them — so a per-image trim is what makes that separator the real gap. + + Args: + images: Raw content from one of the fetch paths + plugin_id: Plugin identifier for logging + source: Which path produced the content, for logging + + Returns: + Trimmed image list, or None if nothing worth showing remains + """ + if not self.config.auto_trim: + # Trimming is off, but the width budget is a separate concern — + # turning off margin cropping should not let one plugin hold the + # panel for minutes. Skipping it here previously let a 14,848px + # segment through untouched. + kept = self._apply_width_budget(list(images), plugin_id, plugin) + self._cache_content(plugin_id, kept) + return kept + + original_width = sum(img.width for img in images) + kept: List[Image.Image] = [] + dropped_blank = 0 + + for img in images: + result = trim_to_content( + img, + threshold=self.config.trim_threshold, + padding=self.config.content_padding, + ) + if result.is_blank: + dropped_blank += 1 + continue + kept.append(result.image) + + if not kept: + logger.debug( + "[%s] All %d image(s) from %s were blank — contributing nothing", + plugin_id, len(images), source + ) + return None + + trimmed_width = sum(img.width for img in kept) + + if trimmed_width < self.config.min_plugin_width: + logger.debug( + "[%s] Trimmed content %dpx is below min_plugin_width %dpx — skipping", + plugin_id, trimmed_width, self.config.min_plugin_width + ) + return None + + if trimmed_width != original_width or dropped_blank: + logger.debug( + "[%s] Trimmed %s content: %dpx -> %dpx (%.0f%% reclaimed), " + "%d image(s) kept, %d blank dropped", + plugin_id, source, original_width, trimmed_width, + 100.0 * (original_width - trimmed_width) / original_width + if original_width else 0.0, + len(kept), dropped_blank + ) + + kept = self._apply_width_budget(kept, plugin_id, plugin) + + self._cache_content(plugin_id, kept) + return kept + + def _capture(self): + """ + Context manager suppressing hardware writes while plugin render code runs. + + Degrades to a no-op when the display manager predates capture_mode. As + with _render_at, losing the suppression risks a visible flash, whereas + raising would be swallowed by the broad handlers upstream and drop the + plugin's content entirely — much worse. + """ + capture_mode = getattr(self.display_manager, 'capture_mode', None) + if capture_mode is None: + logger.debug( + "display_manager has no capture_mode(); plugin writes during " + "content capture may reach the panel" + ) + return nullcontext() + return capture_mode() + + def _render_at(self, width: int): + """ + Context manager narrowing the plugin-facing canvas to ``width``. + + Degrades to a no-op when the display manager predates render_size (a + third-party or older test harness). Losing the narrowing is a cosmetic + regression; raising here would be caught by the broad handlers upstream + and silently drop the plugin's content entirely. + """ + render_size = getattr(self.display_manager, 'render_size', None) + if render_size is None: + logger.debug( + "display_manager has no render_size(); Vegas width requests " + "will be ignored" + ) + return nullcontext() + return render_size(width) + + def resolve_render_width(self, plugin: 'BasePlugin', plugin_id: str) -> int: + """ + Width to tell a plugin it has while it renders for the ticker. + + Resolution order, most specific first: + 1. the plugin's own ``vegas_width_pct`` config value + 2. the global ``vegas_scroll.render_width_pct`` + 3. the full panel width + + A percentage rather than an absolute width so one setting travels + across panel sizes. + + Args: + plugin: Plugin instance, consulted for a per-plugin override + plugin_id: Plugin identifier for logging + + Returns: + Target width in pixels, never wider than the panel + """ + pct = self.config.render_width_pct + + plugin_cfg = getattr(plugin, 'config', None) + if isinstance(plugin_cfg, dict): + raw = plugin_cfg.get('vegas_width_pct') + if raw not in (None, ''): + try: + candidate = int(raw) + except (TypeError, ValueError): + logger.warning( + "[%s] Invalid vegas_width_pct %r, ignoring", plugin_id, raw) + else: + if 10 <= candidate <= 100: + pct = candidate + else: + logger.warning( + "[%s] vegas_width_pct %d out of range 10-100, ignoring", + plugin_id, candidate) + + if pct >= 100: + return self.display_width + return max(1, int(self.display_width * pct / 100)) + + def _row_gap(self, left: Image.Image, right: Image.Image) -> int: + """ + Gap the compositor will insert between two of a plugin's rows. + + Mirrors RenderPipeline._join_plugin_rows so the width budget measures + what will actually be rendered. + """ + return separation_gap( + left, right, + target=max(0, self.config.min_content_separation), + minimum=max(0, self.config.intra_plugin_gap), + threshold=self.config.trim_threshold, + ) + + def _plugin_setting(self, plugin: 'BasePlugin', key: str): + """Read a per-plugin config override, or None if absent.""" + plugin_cfg = getattr(plugin, 'config', None) + if not isinstance(plugin_cfg, dict): + return None + value = plugin_cfg.get(key) + return None if value in (None, '') else value + + def resolve_overflow_mode(self, plugin: 'BasePlugin', plugin_id: str) -> str: + """ + How to handle content that exceeds this plugin's width budget. + + 'rotate' advances a window each cycle so everything is seen eventually, + which suits interchangeable items. 'truncate' always shows the start, + which suits ordered content — a league table that shows ranks 1-6 and + then resumes at 7 two rotations later reads as out of order, and nobody + needs rank 23 in a ticker anyway. + + Per-plugin ``vegas_overflow`` wins over the global ``overflow_mode``. + """ + raw = self._plugin_setting(plugin, 'vegas_overflow') + if raw is not None: + candidate = str(raw).strip().lower() + if candidate in ('rotate', 'truncate'): + return candidate + logger.warning( + "[%s] Invalid vegas_overflow %r, expected 'rotate' or 'truncate'", + plugin_id, raw + ) + return self.config.overflow_mode + + def _width_budget(self, plugin: Optional['BasePlugin'] = None, + plugin_id: str = '') -> int: + """ + Maximum columns one plugin may occupy in a cycle. 0 means unlimited. + + A per-plugin ``vegas_max_width_screens`` overrides the global ratio, so + content that has to stay whole can be given room (or uncapped with 0) + without lifting the cap on every ticker. + """ + ratio = self.config.max_plugin_width_ratio + + if plugin is not None: + raw = self._plugin_setting(plugin, 'vegas_max_width_screens') + if raw is not None: + try: + candidate = float(raw) + except (TypeError, ValueError): + logger.warning( + "[%s] Invalid vegas_max_width_screens %r, ignoring", + plugin_id, raw + ) + else: + if candidate >= 0: + ratio = candidate + else: + logger.warning( + "[%s] vegas_max_width_screens must be >= 0, got %s", + plugin_id, candidate + ) + + if ratio <= 0: + return 0 + return int(self.display_width * ratio) + + def _resume_offset(self, plugin_id: str, shape: Tuple[str, int]) -> int: + """ + The plugin's stored rotation offset, if it still applies. + + An offset is only meaningful against content shaped the way it was + when the offset was recorded. When the shape has changed — a different + number of rows, a re-rendered strip with different item boundaries — + the stored value points somewhere arbitrary, so rotation restarts. + + Args: + plugin_id: Plugin identifier + shape: (kind, size) describing what an offset would index into now + + Returns: + The stored offset, or 0 when it no longer applies + """ + if self._offset_shapes.get(plugin_id) != shape: + if plugin_id in self._item_offsets: + logger.debug( + "[%s] Content is %s now, was %s — restarting the rotation " + "rather than resuming at a position that no longer means " + "anything", plugin_id, shape, + self._offset_shapes.get(plugin_id)) + self._item_offsets.pop(plugin_id, None) + self._offset_shapes[plugin_id] = shape + return 0 + return self._item_offsets.get(plugin_id, 0) + + def _record_offset( + self, plugin_id: str, offset: int, shape: Tuple[str, int] + ) -> None: + """Store where the next window should resume, with what it indexes.""" + if offset: + self._item_offsets[plugin_id] = offset + self._offset_shapes[plugin_id] = shape + else: + # A wrapped-to-zero rotation is the same as no state at all, and + # keeping the key would report a window as active when the next + # pass starts from the top anyway. + self._item_offsets.pop(plugin_id, None) + self._offset_shapes.pop(plugin_id, None) + + def _clear_offset(self, plugin_id: str) -> None: + """Forget any rotation state for a plugin.""" + self._item_offsets.pop(plugin_id, None) + self._offset_shapes.pop(plugin_id, None) + + def _merge_trailing_runt(self, end: int, width: int, budget: int) -> int: + """ + Extend a window to the end of the content when what would be left over + is too small to be worth its own pass. + + Windows were placed by walking forward from the last one, which makes + the final window whatever happens to remain. Measured on a live panel + that produced a 1,840px stocks ticker splitting 1,492 + 348 — the + second pass showing seven seconds of content before cutting, which + reads as the display failing rather than as a rotation. + + Absorbing the remainder overruns the budget by less than one window + floor, which is a better trade than a fragment: the budget is a guard + against one plugin holding the panel for minutes, not a hard limit. + + Args: + end: Column the window would otherwise end at + width: Full content width + budget: Width budget being applied + + Returns: + ``end``, or ``width`` when the remainder is below the floor + """ + remainder = width - end + # Measured against the budget rather than the panel: snapping to item + # boundaries means an ordinary window already lands short of the budget + # (a 512px budget over 182px-pitch items yields 348px windows), so an + # absolute floor would merge windows that were never fragments. Half a + # budget separates "a short last pass" from "a sliver", and caps the + # overrun this can cause at 1.5 budgets. + floor = budget // 2 + if 0 < remainder < floor: + return width + return end + + def _apply_width_budget( + self, images: List[Image.Image], plugin_id: str, + plugin: Optional['BasePlugin'] = None + ) -> List[Image.Image]: + """ + Hold one plugin to its share of a cycle. + + A ticker returning 7,000px would otherwise own the panel for over two + minutes, which defeats the point of a rotation. Overflow is deferred + rather than discarded: the starting offset advances each time this + plugin is fetched, so later items appear on subsequent cycles instead + of never being seen. + + Args: + images: Trimmed images for this plugin + plugin_id: Plugin identifier, used to track its rotation offset + + Returns: + Images that fit the budget, starting from the plugin's current + rotation offset. + """ + budget = self._width_budget(plugin, plugin_id) + mode = (self.resolve_overflow_mode(plugin, plugin_id) + if plugin is not None else self.config.overflow_mode) + + # Count the gaps the compositor will actually insert, not just the + # pixels of the rows — otherwise a plugin with many rows quietly + # occupies far more of the panel than its budget allows. These must use + # the same measured rule as RenderPipeline._join_plugin_rows; assuming + # the flat intra_plugin_gap here under-counted by up to + # (min_content_separation - intra_plugin_gap) per row. + total = sum(img.width for img in images) + sum( + self._row_gap(images[i], images[i + 1]) for i in range(len(images) - 1) + ) + + if not budget or total <= budget: + # Fits, so reset rotation — the whole segment is being shown. + self._clear_offset(plugin_id) + return images + + if len(images) == 1: + return [self._crop_to_budget(images[0], budget, plugin_id, mode)] + + shape = ('rows', len(images)) + if mode == 'truncate': + # Ordered content: always show from the top. Deliberately does not + # advance the offset, so the same opening items appear every time + # rather than the viewer being shown the middle of a ranked list. + start = 0 + else: + start = self._resume_offset(plugin_id, shape) % len(images) + selected: List[Image.Image] = [] + used = 0 + consumed = 0 + + # Walk forward from the rotation offset, taking whole items only, so a + # cut never lands in the middle of one. + # + # A window may overrun the budget while it is still shorter than the + # runt floor, for the same reason _merge_trailing_runt exists on the + # single-image path: a pass far shorter than its neighbours reads as + # the display failing rather than as a rotation. Rows of 450, 450 and + # 100 against a 512px budget used to give the 100 a pass of its own -- + # two seconds against nine. Wrapping does not prevent that, because it + # only helps when the row wrapped to actually fits. + floor = budget // 2 + for step in range(len(images)): + img = images[(start + step) % len(images)] + cost = img.width + if selected: + cost += self._row_gap(selected[-1], img) + if selected and used + cost > budget: + # Keep the overrun bounded at the same 1.5 budgets the + # single-image path allows. A next row too wide to absorb + # leaves a short window standing -- better than a window of + # 1.9 budgets, and the same trade the always-take-the-first + # rule below already makes. + if used >= floor or used + cost > budget + floor: + break + selected.append(img) + used += cost + consumed += 1 + + if mode == 'truncate': + logger.debug( + "[%s] Width budget %dpx: showing the first %d of %d row(s) " + "(%dpx incl. gaps); the rest are not shown (overflow=truncate)", + plugin_id, budget, len(selected), len(images), used + ) + else: + self._record_offset( + plugin_id, (start + consumed) % len(images), shape) + logger.debug( + "[%s] Width budget %dpx: showing %d of %d row(s) (%dpx incl. gaps) " + "from offset %d; remainder deferred to a later cycle", + plugin_id, budget, len(selected), len(images), used, start + ) + return selected + + def _crop_to_budget( + self, img: Image.Image, budget: int, plugin_id: str, + mode: str = 'rotate' + ) -> Image.Image: + """ + Narrow a single oversized image to the budget, advancing a window + through it across cycles. + + The cut is snapped to the nearest blank column so it does not slice + through a glyph or logo and leave half a character at the panel edge. + + Rotation is tracked as an index into the strip's item boundaries rather + than as a pixel column, because a ticker re-renders between fetches. A + column recorded against one render points at unrelated content in the + next as soon as anything ahead of it changes width — a digit in a + price, a shorter headline. The Nth boundary stays the Nth boundary. + """ + # Cut only where the plugin left a real gap between items. Snapping to + # any blank column used to pick the single-column gaps between + # characters, splitting a word and orphaning its tail into the next + # cycle — a lone "y" from "Wednesday" floating between two unrelated + # plugins. Overshooting the budget is the lesser evil. + min_run = max(2, self.config.min_cut_gap) + gaps = blank_runs(img, min_run, self.config.trim_threshold) + + if not gaps: + # No internal gaps means continuous content — a map, a chart, a + # photo — where any column is as good as any other, so cut to the + # budget exactly. The gap rule exists to protect discrete items + # (words, ticker entries); it would be wrong to let a solid image + # escape the cap in its name. + # + # With no items to index, the offset here has to stay a column, so + # it is only reusable while the image keeps its width. + shape = ('cols', img.width) + offset = 0 if mode == 'truncate' else self._resume_offset( + plugin_id, shape) + end = self._merge_trailing_runt( + min(offset + budget, img.width), img.width, budget) + if mode != 'truncate': + self._record_offset( + plugin_id, 0 if end >= img.width else end, shape) + logger.debug( + "[%s] Width budget %dpx: cropped continuous %dpx image to " + "[%d:%d] (no item gaps of %dpx+ to align to)%s", + plugin_id, budget, img.width, offset, end, min_run, + "" if mode != 'truncate' else "; showing the start only" + ) + return img.crop((offset, 0, end, img.height)) + + # Cut mid-gap so the content either side keeps some breathing room. + cuts = sorted({0, img.width} | {(a + b) // 2 for a, b in gaps}) + + shape = ('cuts', len(cuts)) + index = 0 if mode == 'truncate' else self._resume_offset( + plugin_id, shape) + # Clamped rather than wrapped: a stale index past the end means the + # strip shrank, and restarting reads better than landing near the end. + start_index = index if 0 <= index < len(cuts) - 1 else 0 + start = cuts[start_index] + + later = cuts[start_index + 1:] + if not later: + end = img.width + else: + within = [c for c in later if c <= start + budget] + # No boundary inside the budget: take the next one and overrun, + # because the alternative is cutting through an item. + end = max(within) if within else min(later) + end = self._merge_trailing_runt(end, img.width, budget) + # Every candidate for `end` came from `cuts` (which includes img.width), + # so this always resolves; the fallback is defensive only. + end_index = cuts.index(end) if end in cuts else len(cuts) - 1 + + if mode != 'truncate': + # Next cycle resumes at the boundary this one stopped on; wrap when + # the strip ends. + self._record_offset( + plugin_id, 0 if end >= img.width else end_index, shape) + + logger.debug( + "[%s] Width budget %dpx: cropped single %dpx image to [%d:%d] " + "(%dpx) at item boundaries %d-%d of %d, %s", + plugin_id, budget, img.width, start, end, end - start, + start_index, end_index, len(cuts) - 1, + "showing the start only (overflow=truncate)" + if mode == 'truncate' else "window advances next cycle" + ) + return img.crop((start, 0, end, img.height)) + def _get_native_content( - self, plugin: 'BasePlugin', plugin_id: str + self, plugin: 'BasePlugin', plugin_id: str, offscreen_only: bool = False ) -> Optional[List[Image.Image]]: """ Get content via plugin's native get_vegas_content() method. @@ -144,23 +698,56 @@ class PluginAdapter: List of images or None """ try: - logger.info("[%s] Native: calling get_vegas_content()", plugin_id) - result = plugin.get_vegas_content() + logger.debug("[%s] Native: calling get_vegas_content()", plugin_id) + + # Tell the plugin how much width the ticker wants it to use, and + # narrow the canvas for the duration of the call. A plugin that + # sizes its own images from display_manager.matrix.width picks up + # the narrower value with no changes of its own; one that wants to + # be explicit can read get_vegas_render_width(). + render_width = self.resolve_render_width(plugin, plugin_id) + if render_width != self.display_width: + logger.debug( + "[%s] Native: requesting %dpx instead of %dpx", + plugin_id, render_width, self.display_width + ) + + plugin._vegas_render_width = render_width + try: + # capture_mode unconditionally, even at full width. Building + # Vegas content is an off-screen operation, but a plugin is free + # to call update_display() while doing it — and outside + # capture_mode that write lands on the hardware, flashing the + # panel mid-scroll. The narrowing context is separate because it + # is a no-op at full width. + if offscreen_only: + # _render_at swaps the shared canvas, so it is unsafe here. + # _vegas_render_width is set regardless: a plugin reading + # get_vegas_render_width() still gets its narrow size, and + # one that only reads matrix.width renders full width and is + # trimmed instead. + with self._capture(): + result = plugin.get_vegas_content() + else: + with self._capture(), self._render_at(render_width): + result = plugin.get_vegas_content() + finally: + plugin._vegas_render_width = None if result is None: - logger.info("[%s] Native: get_vegas_content() returned None", plugin_id) + logger.debug("[%s] Native: get_vegas_content() returned None", plugin_id) return None # Normalize to list if isinstance(result, Image.Image): images = [result] - logger.info( + logger.debug( "[%s] Native: got single Image %dx%d", plugin_id, result.width, result.height ) elif isinstance(result, (list, tuple)): images = list(result) - logger.info( + logger.debug( "[%s] Native: got %d items in list/tuple", plugin_id, len(images) ) @@ -181,14 +768,14 @@ class PluginAdapter: ) continue - logger.info( + logger.debug( "[%s] Native: item[%d] is %dx%d, mode=%s", plugin_id, i, img.width, img.height, img.mode ) # Ensure correct height if img.height != self.display_height: - logger.info( + logger.debug( "[%s] Native: resizing item[%d]: %dx%d -> %dx%d", plugin_id, i, img.width, img.height, img.width, self.display_height @@ -206,13 +793,13 @@ class PluginAdapter: if valid_images: total_width = sum(img.width for img in valid_images) - logger.info( + logger.debug( "[%s] Native: SUCCESS - %d images, %dpx total width", plugin_id, len(valid_images), total_width ) return valid_images - logger.info("[%s] Native: no valid images after validation", plugin_id) + logger.debug("[%s] Native: no valid images after validation", plugin_id) return None except (AttributeError, TypeError, ValueError, OSError) as e: @@ -223,7 +810,7 @@ class PluginAdapter: return None def _get_scroll_helper_content( - self, plugin: 'BasePlugin', plugin_id: str + self, plugin: 'BasePlugin', plugin_id: str, offscreen_only: bool = False ) -> Optional[List[Image.Image]]: """ Get content from plugin's scroll_helper if available. @@ -246,17 +833,24 @@ class PluginAdapter: logger.debug("[%s] No scroll_helper attribute", plugin_id) return None - logger.info( + logger.debug( "[%s] Found scroll_helper: %s", plugin_id, type(scroll_helper).__name__ ) cached_image = getattr(scroll_helper, 'cached_image', None) if cached_image is None: - logger.info( + logger.debug( "[%s] scroll_helper.cached_image is None, triggering content generation", plugin_id ) + if offscreen_only: + # Generating it calls display(), which needs the canvas. + logger.debug( + "[%s] scroll_helper cache empty; deferring generation " + "to the render thread", plugin_id + ) + return None # Try to trigger scroll content generation cached_image = self._trigger_scroll_content_generation( plugin, plugin_id, scroll_helper @@ -265,13 +859,13 @@ class PluginAdapter: return None if not isinstance(cached_image, Image.Image): - logger.info( + logger.debug( "[%s] scroll_helper.cached_image is not an Image: %s", plugin_id, type(cached_image).__name__ ) return None - logger.info( + logger.debug( "[%s] scroll_helper.cached_image found: %dx%d, mode=%s", plugin_id, cached_image.width, cached_image.height, cached_image.mode ) @@ -294,7 +888,7 @@ class PluginAdapter: # Ensure correct height if img.height != self.display_height: - logger.info( + logger.debug( "[%s] Resizing scroll_helper content: %dx%d -> %dx%d", plugin_id, img.width, img.height, img.width, self.display_height @@ -308,7 +902,7 @@ class PluginAdapter: if img.mode != 'RGB': img = img.convert('RGB') - logger.info( + logger.debug( "[%s] ScrollHelper content ready: %dx%d", plugin_id, img.width, img.height ) @@ -405,10 +999,10 @@ class PluginAdapter: # Save display state to restore after original_image = self.display_manager.image.copy() - with self.display_manager.capture_mode(): + with self._capture(): # Method 1: Try _create_scrolling_display (stocks pattern) if hasattr(plugin, '_create_scrolling_display'): - logger.info( + logger.debug( "[%s] Triggering via _create_scrolling_display()", plugin_id ) @@ -416,7 +1010,7 @@ class PluginAdapter: plugin._create_scrolling_display() cached_image = getattr(scroll_helper, 'cached_image', None) if cached_image is not None and isinstance(cached_image, Image.Image): - logger.info( + logger.debug( "[%s] _create_scrolling_display() SUCCESS: %dx%d", plugin_id, cached_image.width, cached_image.height ) @@ -428,7 +1022,7 @@ class PluginAdapter: # Method 2: Try display(force_clear=True) which typically builds scroll content if hasattr(plugin, 'display'): - logger.info( + logger.debug( "[%s] Triggering via display(force_clear=True)", plugin_id ) @@ -437,12 +1031,12 @@ class PluginAdapter: plugin.display(force_clear=True) cached_image = getattr(scroll_helper, 'cached_image', None) if cached_image is not None and isinstance(cached_image, Image.Image): - logger.info( + logger.debug( "[%s] display(force_clear=True) SUCCESS: %dx%d", plugin_id, cached_image.width, cached_image.height ) return cached_image - logger.info( + logger.debug( "[%s] display(force_clear=True) did not populate cached_image", plugin_id ) @@ -451,7 +1045,7 @@ class PluginAdapter: "[%s] display(force_clear=True) failed", plugin_id ) - logger.info( + logger.debug( "[%s] Could not trigger scroll content generation", plugin_id ) @@ -483,61 +1077,72 @@ class PluginAdapter: try: # Save current display state original_image = self.display_manager.image.copy() - logger.info("[%s] Fallback: saved original display state", plugin_id) + logger.debug("[%s] Fallback: saved original display state", plugin_id) # Ensure plugin has fresh data before capturing has_update_data = hasattr(plugin, 'update_data') - logger.info("[%s] Fallback: has update_data=%s", plugin_id, has_update_data) + logger.debug("[%s] Fallback: has update_data=%s", plugin_id, has_update_data) if has_update_data: try: plugin.update_data() - logger.info("[%s] Fallback: update_data() called", plugin_id) + logger.debug("[%s] Fallback: update_data() called", plugin_id) except (AttributeError, RuntimeError, OSError): logger.exception("[%s] Fallback: update_data() failed", plugin_id) # Clear and call plugin display — use capture_mode to suppress hardware writes # that plugins may trigger internally via update_display(). - with self.display_manager.capture_mode(): + # + # render_size narrows the canvas the plugin lays out against, so a + # plugin that spreads across the whole panel produces a compact + # arrangement rather than one that has to be cropped afterwards. + render_width = self.resolve_render_width(plugin, plugin_id) + if render_width != self.display_width: + logger.debug( + "[%s] Fallback: rendering at %dpx instead of %dpx", + plugin_id, render_width, self.display_width + ) + + with self._capture(), self._render_at(render_width): self.display_manager.clear() - logger.info("[%s] Fallback: display cleared, calling display()", plugin_id) + logger.debug("[%s] Fallback: display cleared, calling display()", plugin_id) # First try without force_clear (some plugins behave better this way) try: plugin.display() - logger.info("[%s] Fallback: display() called successfully", plugin_id) + logger.debug("[%s] Fallback: display() called successfully", plugin_id) except TypeError: # Plugin may require force_clear argument - logger.info("[%s] Fallback: display() failed, trying with force_clear=True", plugin_id) + logger.debug("[%s] Fallback: display() failed, trying with force_clear=True", plugin_id) plugin.display(force_clear=True) # Capture the result captured = self.display_manager.image.copy() - logger.info( + logger.debug( "[%s] Fallback: captured frame %dx%d, mode=%s", plugin_id, captured.width, captured.height, captured.mode ) # Check if captured image has content (not all black) is_blank, bright_ratio = self._is_blank_image(captured, return_ratio=True) - logger.info( + logger.debug( "[%s] Fallback: brightness check - %.3f%% bright pixels (threshold=0.5%%)", plugin_id, bright_ratio * 100 ) if is_blank: - logger.info( + logger.debug( "[%s] Fallback: first capture blank, retrying with force_clear", plugin_id ) # Try once more with force_clear=True - with self.display_manager.capture_mode(): + with self._capture(), self._render_at(render_width): self.display_manager.clear() plugin.display(force_clear=True) captured = self.display_manager.image.copy() is_blank, bright_ratio = self._is_blank_image(captured, return_ratio=True) - logger.info( + logger.debug( "[%s] Fallback: retry brightness - %.3f%% bright pixels", plugin_id, bright_ratio * 100 ) @@ -554,7 +1159,7 @@ class PluginAdapter: if captured.mode != 'RGB': captured = captured.convert('RGB') - logger.info( + logger.debug( "[%s] Fallback: SUCCESS - captured %dx%d", plugin_id, captured.width, captured.height ) @@ -663,6 +1268,53 @@ class PluginAdapter: else: self._content_cache.clear() + def invalidate_plugin_scroll_cache( + self, plugin: 'BasePlugin', plugin_id: str + ) -> bool: + """ + Drop a plugin's own cached scroll image so its visual is rebuilt. + + Invalidating only this adapter's cache is not enough. A plugin that + composes a scroll strip hands back the *same* image every time until its + own cache is cleared — the sports plugins' ``get_vegas_content()`` + regenerates only "if the cache is empty" — so without this a segment + keeps rendering whatever data it was first built from. That is how a + game that was live last night can still be displayed as live the next + morning. + + Two layouts to cover: a helper directly on the plugin (stocks, news, + odds-ticker) and one owned by a scroll-display manager (the sports + scoreboards). ``cached_image`` and ``cached_array`` must be cleared + together, since the array is the image's numpy mirror and code paths + read whichever is convenient. + + Returns: + True if a cache was found and cleared. + """ + cleared = False + for owner in (plugin, getattr(plugin, '_scroll_manager', None), + getattr(plugin, 'scroll_manager', None)): + if owner is None: + continue + helper = getattr(owner, 'scroll_helper', None) + if helper is None: + continue + try: + if getattr(helper, 'cached_image', None) is not None: + helper.cached_image = None + cleared = True + if getattr(helper, 'cached_array', None) is not None: + helper.cached_array = None + cleared = True + except Exception: # pylint: disable=broad-except + logger.exception( + "[%s] Could not clear scroll cache on %s", + plugin_id, type(owner).__name__ + ) + if cleared: + logger.debug("[%s] Cleared plugin scroll cache", plugin_id) + return cleared + def get_content_type(self, plugin: 'BasePlugin', plugin_id: str) -> str: """ Get the type of content a plugin provides. diff --git a/src/vegas_mode/render_pipeline.py b/src/vegas_mode/render_pipeline.py index 7c9b7c90..8aceabe1 100644 --- a/src/vegas_mode/render_pipeline.py +++ b/src/vegas_mode/render_pipeline.py @@ -6,6 +6,7 @@ Uses the existing ScrollHelper for numpy-optimized scroll operations. """ import logging +import os import time import threading from collections import deque @@ -14,6 +15,7 @@ from PIL import Image from src.common.scroll_helper import ScrollHelper from src.vegas_mode.config import VegasModeConfig +from src.vegas_mode.geometry import separation_gap from src.vegas_mode.stream_manager import StreamManager if TYPE_CHECKING: @@ -34,6 +36,10 @@ class RenderPipeline: - Track scroll cycle completion """ + # Minimum gap between fetches of canvas-bound plugins, so their individual + # stalls land in separate moments rather than one run of hitches. + DEFERRED_DRAIN_INTERVAL = 2.0 + def __init__( self, config: VegasModeConfig, @@ -66,10 +72,6 @@ class RenderPipeline: else display_manager.height ) - # Reusable blank frame for cycle-end pushes (allocated lazily, - # re-blacked before each reuse) - self._blank_frame = None - # ScrollHelper for optimized scrolling self.scroll_helper = ScrollHelper( self.display_width, @@ -85,6 +87,14 @@ class RenderPipeline: self._staging_scroll_image: Optional[Image.Image] = None self._buffer_lock = threading.Lock() + # Group prepared off the render thread, waiting to be appended. + self._prepared_group = None + # Plugins that need the shared canvas, appended one at a time. + self._deferred_queue: List[str] = [] + self._last_drain_time = 0.0 + self._prefetch_thread: Optional[threading.Thread] = None + self._prefetch_lock = threading.Lock() + # Render state self._is_rendering = False self._cycle_complete = False @@ -114,6 +124,7 @@ class RenderPipeline: """Configure ScrollHelper with current settings.""" self.scroll_helper.set_frame_based_scrolling(self.config.frame_based_scrolling) self.scroll_helper.set_scroll_delay(self.config.scroll_delay) + self.scroll_helper.set_sub_pixel_scrolling(self.config.smooth_scroll) # Config scroll_speed is always pixels per second, but ScrollHelper # interprets it differently based on frame_based_scrolling mode: @@ -141,23 +152,37 @@ class RenderPipeline: True if composition successful """ try: - # Get all buffered content - images = self.stream_manager.get_all_content_for_composition() + # Content grouped by plugin, so a separator can be placed at the + # plugin boundaries only. + grouped = self.stream_manager.get_grouped_content_for_composition() - if not images: + if not grouped: logger.warning("No content available for composition") return False - # Add separator gaps between images - content_with_gaps = [] - for i, img in enumerate(images): - content_with_gaps.append(img) + # Collapse each plugin's rows into a single block, joined by + # intra_plugin_gap. ScrollHelper applies one uniform gap between the + # items it is given, so handing it one item per plugin is what makes + # separator_width mean "between plugins" instead of "between every + # row". Without this, a per-row ticker such as the F1 scoreboard got + # the full separator between each of its ~116 rows. + blocks = [] + total_rows = 0 + for plugin_id, images in grouped: + total_rows += len(images) + blocks.append(self._join_plugin_rows(images)) - # Create scrolling image via ScrollHelper + # Create scrolling image via ScrollHelper. + # + # lead_gap is explicit because ScrollHelper otherwise prepends a + # full display width of black — appropriate for a standalone ticker + # scrolling in from off-screen, but in Vegas mode it is charged + # once per cycle and reads as the panel switching off. self.scroll_helper.create_scrolling_image( - content_items=content_with_gaps, + content_items=blocks, item_gap=self.config.separator_width, - element_gap=0 + element_gap=0, + lead_gap=self.config.lead_in_width ) # Verify scroll image was created successfully @@ -177,11 +202,16 @@ class RenderPipeline: self._cycle_complete = False logger.info( - "Composed scroll image: %dx%d, %d plugins, %d items", + "Composed scroll image: %dx%d, %d plugin block(s), %d rows, " + "separator=%dpx between plugins, rows spaced to %dpx of ink " + "(min added %dpx)", self.scroll_helper.cached_image.width if self.scroll_helper.cached_image else 0, self.display_height, - len(self._segments_in_scroll), - len(images) + len(blocks), + total_rows, + self.config.separator_width, + self.config.min_content_separation, + self.config.intra_plugin_gap, ) return True @@ -191,6 +221,264 @@ class RenderPipeline: logger.exception("Error composing scroll content") return False + def needs_extension(self) -> bool: + """ + Whether the strip should be extended with the next group of plugins. + + Cheap enough to call every frame: it is arithmetic over cached state. + """ + if not self.config.continuous_scroll or not self.scroll_helper.cached_image: + return False + threshold = int(self.display_width * self.config.extend_threshold_screens) + return self.scroll_helper.remaining_unscrolled() <= threshold + + def start_prefetch(self) -> None: + """ + Begin preparing the next group in the background, if not already doing so. + + This is what makes the join seamless rather than merely continuous: + fetching a group costs 0.5-4.8s (rendering leaderboard and baseball cards + dominates), and doing it on the render thread stalls the scroll for that + long. Off the render thread there is a whole group's scroll time to work + in, so by the time the strip needs extending the content is already sat + waiting. + + Only paths that avoid the shared display canvas run here; anything + needing it is marked and picked up on the render thread, where it is + safe. Those are the cheap ones — display capture measured 12-14ms + against seconds for the native renders. + """ + if not self.config.continuous_scroll: + return + + with self._prefetch_lock: + if self._prefetch_thread is not None and self._prefetch_thread.is_alive(): + return + if self._prepared_group is not None: + return # already have one waiting + + def _work(): + # Deprioritise against the render loop. Linux applies nice + # per-thread, and the heavy lifting here is PIL and numpy work + # that releases the GIL, so the scheduler can actually act on + # it — without this the prefetch competes for the same cores and + # costs frames. + try: + os.nice(10) + except (OSError, AttributeError): + pass + try: + group = self.stream_manager.take_next_group(offscreen_only=True) + except Exception: + logger.exception("Background prefetch failed") + group = [] + with self._prefetch_lock: + self._prepared_group = group + + self._prefetch_thread = threading.Thread( + target=_work, daemon=True, name="vegas-strip-prefetch") + self._prefetch_thread.start() + + def drain_deferred(self) -> bool: + """ + Fetch one queued canvas-bound plugin and append it to the strip. + + Called once per frame. These plugins cannot be prepared off the render + thread — display capture and scroll-content generation both need the + shared canvas — so each costs roughly 290ms here. Doing one at a time + spreads that out instead of stalling for the whole group at once, and the + strip's lookahead means nothing runs dry while they arrive. + + The cost is that a deferred plugin appears slightly after the group it + came with, which is a fair trade for a smooth scroll. + + Returns: + True if a plugin was appended + """ + if not self._deferred_queue: + return False + + # Space the drains out. Each costs 40-600ms, and taking them back to + # back turns one long stall into a train of short ones — barely better. + # With a healthy lookahead there is no hurry, so wait a beat between + # them; when the strip is actually running short, fetch immediately. + threshold = int(self.display_width * self.config.extend_threshold_screens) + urgent = self.scroll_helper.remaining_unscrolled() <= threshold + if not urgent: + now = time.time() + if now - self._last_drain_time < self.DEFERRED_DRAIN_INTERVAL: + return False + self._last_drain_time = now + else: + self._last_drain_time = time.time() + + plugin_id = self._deferred_queue.pop(0) + plugins = getattr(self.stream_manager.plugin_manager, 'plugins', {}) + plugin = plugins.get(plugin_id) + if plugin is None: + return False + + try: + images = self.stream_manager.plugin_adapter.get_content(plugin, plugin_id) + except Exception: + logger.exception("[%s] Error fetching deferred content", plugin_id) + return False + + if not images: + return False + + appended = self.scroll_helper.append_content( + content_items=[self._join_plugin_rows(images)], + item_gap=self.config.separator_width, + element_gap=0, + ) + if appended: + with self._buffer_lock: + self._active_scroll_image = self.scroll_helper.cached_image + logger.info( + "[%s] Appended deferred content: strip now %dpx, %dpx ahead", + plugin_id, self.scroll_helper.total_scroll_width, + self.scroll_helper.remaining_unscrolled() + ) + return appended + + def has_deferred(self) -> bool: + """Whether any canvas-bound plugins are still queued.""" + return bool(self._deferred_queue) + + def _claim_prepared_group(self): + """Take the prefetched group, if one is ready.""" + with self._prefetch_lock: + group = self._prepared_group + self._prepared_group = None + return group + + def extend_scroll_content(self) -> bool: + """ + Append the next group of plugins to the strip, without interrupting motion. + + This is what replaces the swap. Scroll position is untouched, so the new + content simply arrives from the right; there is no substitution to see + and no restart with the viewport already full. + + Consumed columns behind the viewport are then released, keeping the strip + bounded however long Vegas runs. + + Returns: + True if the strip was extended + """ + try: + grouped = self._claim_prepared_group() + if grouped is None: + # Nothing prepared (first extension, or prefetch still running). + # Fetch inline; the scroll hitches, but content keeps flowing. + logger.info("No prepared group ready; fetching inline") + grouped = self.stream_manager.take_next_group() + + if not grouped: + logger.warning("No content available to extend the scroll strip") + return False + + # Plugins the background thread had to defer need the shared canvas, + # so they can only be fetched here. Queue them rather than doing all + # of them now: measured, six in one go held the render thread for + # 1.75s. They are trickled in one per frame by drain_deferred(), + # which the strip's lookahead comfortably absorbs. + deferred = [pid for pid, images in grouped if images is None] + if deferred: + self._deferred_queue.extend(deferred) + logger.info( + "Queued %d plugin(s) needing the render thread: %s", + len(deferred), ', '.join(deferred) + ) + + grouped = [(pid, imgs) for pid, imgs in grouped if imgs] + + if not grouped: + # Everything in this group is queued; the queue will extend the + # strip as it drains, so this is not a failure. + logger.info("Whole group deferred; strip will extend as it drains") + self.start_prefetch() + return bool(deferred) + + blocks = [] + total_rows = 0 + for _plugin_id, images in grouped: + total_rows += len(images) + blocks.append(self._join_plugin_rows(images)) + + appended = self.scroll_helper.append_content( + content_items=blocks, + item_gap=self.config.separator_width, + element_gap=0, + ) + if not appended: + return False + + # Keep a screen's worth behind the viewport as a safety margin. + self.scroll_helper.drop_scrolled_prefix(keep_before=self.display_width) + + with self._buffer_lock: + self._active_scroll_image = self.scroll_helper.cached_image + + self._segments_in_scroll = [pid for pid, _ in grouped] + self.stats['composition_count'] += 1 + self.stats['extensions'] = self.stats.get('extensions', 0) + 1 + + logger.info( + "Extended scroll strip with %d plugin block(s), %d rows: " + "strip now %dpx, %dpx still ahead of the viewport", + len(blocks), total_rows, self.scroll_helper.total_scroll_width, + self.scroll_helper.remaining_unscrolled() + ) + + # Line up the group after this one straight away, so it is ready + # well before the strip runs short again. + self.start_prefetch() + return True + + except (ValueError, TypeError, OSError, RuntimeError): + logger.exception("Error extending scroll content") + return False + + def _join_plugin_rows(self, images: List[Image.Image]) -> Image.Image: + """ + Concatenate one plugin's images into a single block. + + Args: + images: That plugin's content, in order + + Returns: + A single image with the rows laid out left to right, separated by + ``intra_plugin_gap``. Returned unchanged when there is only one row, + which is the common case and avoids a pointless copy. + """ + if len(images) == 1: + return images[0] + + floor = max(0, self.config.intra_plugin_gap) + target = max(0, self.config.min_content_separation) + threshold = self.config.trim_threshold + + # Space by measured separation, not a flat gap. Rows drawn flush to + # their own edges (sports score cards) would otherwise end up nearly + # touching, while rows that already carry wide margins would be pushed + # needlessly further apart. + gaps = [ + separation_gap(images[i], images[i + 1], target, floor, threshold) + for i in range(len(images) - 1) + ] + + width = sum(img.width for img in images) + sum(gaps) + height = max(img.height for img in images) + + block = Image.new('RGB', (width, height), (0, 0, 0)) + x = 0 + for i, img in enumerate(images): + block.paste(img, (x, 0)) + x += img.width + (gaps[i] if i < len(gaps) else 0) + return block + def render_frame(self) -> bool: """ Render a single frame to the display. @@ -211,21 +499,33 @@ class RenderPipeline: # Determine if the cycle is done. # - # scroll_helper considers a cycle complete only after - # total_distance_scrolled >= total_scroll_width + display_width. - # That extra display_width of travel causes a "wrap-around" phase - # where scroll_position resets to ~0 and the first plugin's content - # re-enters from the right — the user sees this 2-3 s of re-entry - # as "a plugin partially displaying before the next one starts." + # get_visible_portion wraps: once scroll_position + display_width + # passes the end of the strip it fills the right-hand side of the + # frame from the *head* of the same strip. So the last + # display_width of travel shows the cycle's first plugin re-entering + # on the right while its last plugin exits on the left, and the + # recompose that follows then replaces both at once. That reads as + # the ticker "switching mid-scroll". # - # We end the cycle as soon as total_distance_scrolled reaches - # total_scroll_width (the wrap-around point), before any second-pass - # content becomes visible. The scroll_helper's own is_scroll_complete() - # check is kept as a fallback for any edge-cases where that threshold - # is never hit. + # This used to be hidden because the strip began with a full + # display_width of blank, so the wrapped-in region was black. + # lead_in_width now defaults to 0 (that blank was 10s of dead panel + # at 50px/s), which exposed the wrap — so the cycle has to end + # before it, one display width earlier. + # + # A strip no wider than the display never wraps, and subtracting + # would make the cycle complete instantly, so clamp in that case. + # In continuous mode there is no cycle to complete: the strip is + # extended before the scroll can reach its end, so the wrap is never + # entered and motion never stops. The completion path below stays for + # the swap behaviour and as a backstop if an extension fails. + wrap_point = self.scroll_helper.total_scroll_width + if wrap_point > self.display_width: + wrap_point -= self.display_width + at_wrap_point = ( not self._cycle_complete and - self.scroll_helper.total_distance_scrolled >= self.scroll_helper.total_scroll_width + self.scroll_helper.total_distance_scrolled >= wrap_point ) if at_wrap_point or self.scroll_helper.is_scroll_complete(): @@ -236,24 +536,17 @@ class RenderPipeline: "Scroll cycle complete after %.1fs", time.time() - self._cycle_start_time ) - # Push blank immediately so the hardware never shows any - # post-wrap content while the coordinator recomposes the - # next cycle (~100 ms). The blank is allocated once and - # reused across cycle wraps (fresh paste each time in case - # a consumer drew on the previous one). - try: - if self._blank_frame is None or self._blank_frame.size != ( - self.display_width, self.display_height): - self._blank_frame = Image.new( - 'RGB', (self.display_width, self.display_height)) - else: - self._blank_frame.paste( - (0, 0, 0), - (0, 0, self.display_width, self.display_height)) - self.display_manager.image = self._blank_frame - self.display_manager.update_display() - except Exception: - logger.exception("Failed to write blank frame to display at cycle end") + # Deliberately leave the last rendered frame on the panel. + # + # This used to push a blank frame so no post-wrap content + # could be seen while the next cycle was composed. But + # recomposing is synchronous and fetches plugin content: + # measured 84ms at best and 4.8s at worst on a 512px panel, + # and every millisecond of it was black. Holding the last + # frame instead turns that into a brief freeze, which reads + # as far less broken than the display switching off. The + # frame is already past the end of the content, so there is + # no second-pass content to leak. return True # Cycle done; coordinator starts new cycle next frame # Get visible portion @@ -336,6 +629,25 @@ class RenderPipeline: return False + def refresh_updated_plugins(self) -> bool: + """ + Let changed plugin data reach the strip without interrupting motion. + + Used instead of :meth:`hot_swap_content` when scrolling continuously. + The swap rebuilds the whole image and repositions the scroll, which is + visible as a freeze and a jump; the strip is extended here rather than + replaced, so it is enough to drop the stale caches and let the plugin + recompose when it next comes round. + + Returns: + True if any plugin's cached content was dropped. + """ + try: + return bool(self.stream_manager.invalidate_pending_updates()) + except Exception: # pylint: disable=broad-except + logger.exception("Failed to refresh updated plugins") + return False + def hot_swap_content(self) -> bool: """ Hot-swap to new composed content. @@ -415,11 +727,12 @@ class RenderPipeline: result = self.compose_scroll_content() if result and self.sync_manager: - # When sync is active, start the leader at display_width instead of 0. - # This skips the initial black gap so the leader immediately shows content. - # The follower starts at position 0 (the gap) which looks like a clean - # blank transition rather than near-end content wrapping around. - self.scroll_helper.scroll_position = float(self.display_width) + # When sync is active, start the leader past the lead-in gap so it + # immediately shows content, leaving the follower on the blank gap + # for a clean transition rather than near-end content wrapping + # around. This tracks lead_in_width rather than assuming a full + # display width of gap, which is no longer the default. + self.scroll_helper.scroll_position = float(self.config.lead_in_width) if result and self.sync_manager: # Signal follower that a new cycle started (triggers its own rebuild) diff --git a/src/vegas_mode/stream_manager.py b/src/vegas_mode/stream_manager.py index 85d5abdd..59bd1134 100644 --- a/src/vegas_mode/stream_manager.py +++ b/src/vegas_mode/stream_manager.py @@ -14,7 +14,7 @@ Supports three display modes: import logging import threading import time -from typing import Optional, List, Dict, Any, Deque, TYPE_CHECKING +from typing import Optional, List, Dict, Any, Deque, Tuple, TYPE_CHECKING from collections import deque from dataclasses import dataclass, field from PIL import Image @@ -116,8 +116,11 @@ class StreamManager: logger.warning("No plugins available for Vegas scroll") return False - # Prefetch initial content - self._prefetch_content(count=min(self.config.buffer_ahead + 1, len(self._ordered_plugins))) + # Fill the buffer to a whole cycle's worth of plugins. This used to be + # buffer_ahead + 1, which conflated prefetch depth with cycle size and + # meant a 20-plugin install only showed 3 plugins before recomposing. + self._prefetch_content( + count=min(self.config.plugins_per_cycle, len(self._ordered_plugins))) logger.info( "StreamManager initialized with %d plugins, %d segments buffered", @@ -198,6 +201,47 @@ class StreamManager: logger.debug("Plugin %s marked for update", plugin_id) + def invalidate_pending_updates(self) -> List[str]: + """ + Drop cached content for plugins whose data changed, without refetching. + + The continuous-scroll counterpart to :meth:`process_updates`. That method + belongs to the swap path: it refetches immediately and merges into the + active buffer, which continuous mode bypasses entirely, and doing that + work on the render thread would hitch the scroll. + + Here it is enough to clear the caches and let the plugin come round in + the rotation, which recomposes it from current data a moment later. Left + uncalled, ``_pending_updates`` simply accumulates and no visual ever + refreshes — a game that was live last night keeps being drawn as live. + + Returns: + The plugin ids whose caches were dropped. + """ + with self._buffer_lock: + if not self._pending_updates: + return [] + updated = list(self._pending_updates.keys()) + self._pending_updates.clear() + + plugins = getattr(self.plugin_manager, 'plugins', {}) + for plugin_id in updated: + try: + self.plugin_adapter.invalidate_cache(plugin_id) + plugin = plugins.get(plugin_id) + if plugin is not None: + self.plugin_adapter.invalidate_plugin_scroll_cache( + plugin, plugin_id) + except Exception: # pylint: disable=broad-except + logger.exception( + "[%s] Could not invalidate cached content", plugin_id) + + logger.info( + "Vegas: dropped cached content for %d updated plugin(s): %s", + len(updated), ', '.join(updated) + ) + return updated + def has_pending_updates(self) -> bool: """Check if any plugins have pending updates awaiting processing.""" with self._buffer_lock: @@ -362,6 +406,8 @@ class StreamManager: ) logger.info("Ordered plugins: %s", ordered_plugins) + ordered_plugins = self._apply_priority_weights(ordered_plugins) + # Atomically update shared state under lock to avoid races with prefetchers with self._buffer_lock: self._ordered_plugins = ordered_plugins @@ -373,6 +419,143 @@ class StreamManager: logger.info("=" * 60) + def _plugin_weight(self, plugin_id: str) -> int: + """Slots per cycle for one plugin. + + A plugin may answer for itself via get_vegas_priority_weight() -- the + only way favorite-team awareness can reach here, since the core can see + that a game is live but not whose. When it declines (returns None, the + default), live content earns ``live_weight`` and everything else 1. + """ + plugin = None + try: + plugin = self.plugin_manager.plugins.get(plugin_id) + except (AttributeError, TypeError): + return 1 + if plugin is None: + return 1 + + try: + if hasattr(plugin, 'get_vegas_priority_weight'): + declared = plugin.get_vegas_priority_weight() + if declared is not None: + return max(1, min(10, int(declared))) + except Exception: + # Deliberately falls through to the core's own live check rather + # than demoting to 1. The plugin's weight calculation is broken, + # but has_live_priority() and has_live_content() are separate + # methods guarded separately below -- a plugin that genuinely has + # a live game should still get live_weight for it. + logger.exception("[%s] get_vegas_priority_weight() failed", plugin_id) + + try: + if (hasattr(plugin, 'has_live_priority') + and hasattr(plugin, 'has_live_content') + and plugin.has_live_priority() + and plugin.has_live_content()): + return self.config.live_weight + except Exception: + logger.exception("[%s] live-content check failed", plugin_id) + return 1 + + def _apply_priority_weights(self, ordered: List[str]) -> List[str]: + """Expand the rotation so weighted plugins take several turns per cycle. + + Smooth Weighted Round-Robin, the same scheduler the sports plugins use + to rotate their own games: a plugin of weight N appears N times per + cycle, and the repeats are spaced through the cycle rather than + clumped, so a live score is never three-in-a-row followed by a long + silence. + + Returns the input unchanged when nothing is weighted, which is both the + common case and the pre-existing behaviour. + """ + if not ordered or not self.config.live_in_ticker: + return ordered + + weights = {pid: self._plugin_weight(pid) for pid in ordered} + total = sum(weights.values()) + if total <= len(ordered): + return ordered # nothing boosted; plain round robin + + current = {pid: 0 for pid in ordered} + schedule: List[str] = [] + for _ in range(total): + for pid in ordered: + current[pid] += weights[pid] + picked = max(current, key=lambda p: current[p]) + current[picked] -= total + schedule.append(picked) + + schedule = self._unclump_seam(schedule) + + boosted = {p: w for p, w in weights.items() if w > 1} + logger.info( + "Vegas rotation weighted: %d slots for %d plugins (boosted: %s)", + len(schedule), len(ordered), boosted) + return schedule + + @staticmethod + def _unclump_seam(schedule: List[str]) -> List[str]: + """Stop the heaviest plugin sitting on both ends of the cycle. + + Smooth Weighted Round-Robin spaces repeats well *within* a pass, but + it schedules the heaviest item first and often last too. The strip + loops, so those two are neighbours: the one place the marquee shows + the same plugin twice running is the seam between cycles. + + Rotating the list cannot fix this. Rotation preserves the cyclic order + exactly, so it only moves where the seam is drawn, not the adjacency + itself. The trailing entry has to be swapped with one from the middle + whose neighbours differ from it, which breaks the pair without + creating another. + + Left alone when no such position exists -- a rotation short enough or + lopsided enough to have none is one where the plugin is unavoidably + adjacent to itself anyway. + """ + if len(schedule) < 3 or schedule[0] != schedule[-1]: + return schedule + + repeated = schedule[-1] + size = len(schedule) + + def cyclic_doubles(seq) -> int: + return sum(1 for i in range(size) if seq[i] == seq[(i + 1) % size]) + + def clearance(seq, value) -> int: + """Smallest cyclic gap between appearances of `value`.""" + at = [i for i, v in enumerate(seq) if v == value] + if len(at) < 2: + return size + return min(min((b - a) % size, (a - b) % size) + for i, a in enumerate(at) for b in at[i + 1:]) + + # Try each swap and judge the result, rather than reasoning about which + # neighbours the two moved elements will end up with. That reasoning is + # where the first version went wrong: it guarded the slot `repeated` + # moves into but not the one the displaced element lands in, so + # ['a','b','c','d','x','y','x','a'] came back ending ['x','x'] -- the + # seam duplicate traded for a fresh one. + best = None + best_clearance = -1 + for j in range(1, size - 1): + candidate = list(schedule) + candidate[j], candidate[-1] = candidate[-1], candidate[j] + if cyclic_doubles(candidate): + continue + # Among the repairs that work, prefer the one that leaves the + # boosted plugin most evenly spread; taking the first that merely + # fits moved a repeat from a gap of 7 into a gap of 2. + spread = clearance(candidate, repeated) + if spread > best_clearance: + best, best_clearance = candidate, spread + + # None exists when the value is unavoidably adjacent to itself -- a + # plugin holding most of the slots has to be. Schedule it as it is + # rather than refuse. + return best if best is not None else schedule + def _prefetch_content(self, count: int = 1) -> None: """ Prefetch content for upcoming plugins. @@ -385,7 +568,7 @@ class StreamManager: return for _ in range(count): - if len(self._active_buffer) >= self.config.buffer_ahead + 1: + if len(self._active_buffer) >= self.config.plugins_per_cycle: break # Ensure index is valid (guard against empty list) @@ -521,28 +704,117 @@ class StreamManager: logger.debug("Refreshed content for %s in staging buffer", plugin_id) def _ensure_buffer_filled(self) -> None: - """Ensure buffer has enough content prefetched.""" - if len(self._active_buffer) < self.config.buffer_ahead: - needed = self.config.buffer_ahead - len(self._active_buffer) - self._prefetch_content(count=needed) + """ + Top the buffer back up after segments have been served. + + buffer_ahead is the low-water mark only; plugins_per_cycle is the + ceiling and is enforced inside _prefetch_content. + """ + low_water = min(self.config.buffer_ahead, self.config.plugins_per_cycle) + if len(self._active_buffer) < low_water: + self._prefetch_content(count=low_water - len(self._active_buffer)) def get_all_content_for_composition(self) -> List[Image.Image]: """ Get all buffered content as a flat list of images. - Used when composing the full scroll image. Skips STATIC segments as they don't have images to compose. + Prefer get_grouped_content_for_composition(): flattening loses the + plugin boundaries, which is what tells the compositor where a + separator belongs and where it does not. + Returns: List of all images in buffer order """ all_images = [] + for _plugin_id, images in self.get_grouped_content_for_composition(): + all_images.extend(images) + return all_images + + def get_grouped_content_for_composition(self) -> List[Tuple[str, List[Image.Image]]]: + """ + Get buffered content grouped by the plugin that produced it. + + The grouping matters: separator_width is meant to mark the handoff from + one plugin to the next, not to sit between every row a single plugin + contributes. A per-row ticker like the F1 scoreboard returns over a + hundred images that it renders 4px apart internally, so flattening them + into one list and applying a uniform gap forced 32px between each of + its rows — both inconsistent with how the plugin looks standalone, and + a large hidden addition to the width it occupies. + + Skips STATIC segments, which trigger a pause rather than contributing + scroll content, and segments left with no images. + + Returns: + List of (plugin_id, images) in buffer order + """ + grouped: List[Tuple[str, List[Image.Image]]] = [] with self._buffer_lock: for segment in self._active_buffer: - # Skip STATIC segments - they trigger pauses, not scroll content - if segment.display_mode != VegasDisplayMode.STATIC: - all_images.extend(segment.images) - return all_images + if segment.display_mode == VegasDisplayMode.STATIC: + continue + if not segment.images: + continue + grouped.append((segment.plugin_id, list(segment.images))) + return grouped + + def take_next_group( + self, count: Optional[int] = None, offscreen_only: bool = False + ) -> List[Tuple[str, Optional[List[Image.Image]]]]: + """ + Fetch and hand over the next slice of the rotation. + + For continuous scrolling, where the strip is extended rather than + replaced. Advances the rotation index so plugins come round in order + across an unbroken strip, and bypasses the active buffer entirely — that + buffer exists to stage a *replacement* cycle, which continuous mode has + no use for. + + Args: + count: Number of plugins to gather, defaulting to plugins_per_cycle + offscreen_only: Only use content paths that avoid the shared display + canvas, for use off the render thread + + Returns: + Ordered list of (plugin_id, images). ``images`` is None when the + plugin could not be served under ``offscreen_only``, so the caller + can fetch just those on the render thread while keeping the order. + """ + if count is None: + count = self.config.plugins_per_cycle + + self.refresh() + + with self._buffer_lock: + if not self._ordered_plugins: + return [] + total = len(self._ordered_plugins) + ids = [] + for _ in range(min(max(1, count), total)): + ids.append(self._ordered_plugins[self._prefetch_index]) + self._prefetch_index = (self._prefetch_index + 1) % total + + plugins = getattr(self.plugin_manager, 'plugins', {}) + group: List[Tuple[str, Optional[List[Image.Image]]]] = [] + + for plugin_id in ids: + plugin = plugins.get(plugin_id) + if not plugin: + continue + try: + images = self.plugin_adapter.get_content( + plugin, plugin_id, offscreen_only=offscreen_only) + except Exception: + logger.exception("[%s] ERROR fetching content", plugin_id) + self.stats['fetch_errors'] += 1 + continue + if images: + self.stats['segments_fetched'] += 1 + group.append((plugin_id, images if images else None)) + + return group def advance_cycle(self) -> None: """ diff --git a/src/web_interface/api_helpers.py b/src/web_interface/api_helpers.py index 7ba6567a..3cff5293 100644 --- a/src/web_interface/api_helpers.py +++ b/src/web_interface/api_helpers.py @@ -29,18 +29,16 @@ def success_response( Flask jsonify response """ response_data = create_success_response(data, message, metadata) - - # Add request metadata if available - if metadata is None: - metadata = {} - - # Add timing if request start time is available + + # Timing is merged into whatever the caller passed, without inventing a + # metadata block for responses that have neither. + enriched = dict(metadata) if metadata is not None else {} if hasattr(request, 'start_time'): - metadata['response_time_ms'] = int((time.time() - request.start_time) * 1000) - - if metadata: - response_data['metadata'] = metadata - + enriched['response_time_ms'] = int((time.time() - request.start_time) * 1000) + + if metadata is not None or enriched: + response_data['metadata'] = enriched + return jsonify(response_data) diff --git a/src/web_interface/error_handler.py b/src/web_interface/error_handler.py index c15d373c..509c5679 100644 --- a/src/web_interface/error_handler.py +++ b/src/web_interface/error_handler.py @@ -1,11 +1,11 @@ """ Centralized error handling for web interface. -Provides decorators and helpers for consistent error handling across API endpoints. +Provides helpers for consistent error responses across API endpoints. """ -import functools -from typing import Callable, Any, Optional +import re +from typing import Any, Optional from flask import jsonify from src.web_interface.errors import ( @@ -17,68 +17,95 @@ from src.logging_config import get_logger logger = get_logger(__name__) -def handle_errors( - default_error_code: Optional[ErrorCode] = None, - default_category: Optional[ErrorCategory] = None, - log_error: bool = True -): +# Credentials that turn up inside exception text. A requests error quotes the +# URL it failed on, and plugins that authenticate by query string put their key +# there, so echoing an exception verbatim can hand out an API key. Redact the +# value, keep the parameter name -- knowing *which* credential was involved is +# part of the diagnosis. +_REDACT_CREDENTIAL = re.compile( + r'((?:api[_-]?key|access[_-]?token|auth|apikey|key|passwd|password|pwd|' + r'secret|sig|signature|token)["\']?\s*[=:]\s*["\']?)([^\s&"\'<>,}]+)', + re.IGNORECASE, +) + +# `Authorization: `. The scheme name is kept because it +# says which kind of credential failed; the credential goes. Any scheme +# matches, not a fixed list: ApiKey, Negotiate, NTLM, AWS4-HMAC-SHA256 and +# whatever a plugin's API invents next are all credentials, and a list would +# silently leak the ones nobody thought of. Not covered by the generic pattern +# above, whose value part stops at whitespace and so would keep the credential +# once a space follows the scheme. +_REDACT_AUTH_HEADER = re.compile( + r'((?:proxy-)?authorization["\']?\s*[=:]\s*["\']?\s*' + r'(?:[A-Za-z][\w.+-]*[ \t]+)?)' # optional scheme name, kept + r'([^\s,"\'<>}]+)', # the credential, redacted + re.IGNORECASE, +) + +# Credentials embedded in a URL: https://user:password@host. requests quotes +# the full URL in its exceptions, so this is a realistic leak. The username is +# kept -- it identifies which account failed without being the secret. +_REDACT_URL_USERINFO = re.compile(r'([a-z][a-z0-9+.-]*://[^/\s:@]+:)([^/\s@]+)(@)', + re.IGNORECASE) + +# Long enough for an errno string with a path, short enough not to dump a +# parser's worth of context into a JSON field. +_MAX_DETAIL_LENGTH = 400 + + +def describe_exception(exc: BaseException, + max_length: int = _MAX_DETAIL_LENGTH) -> str: """ - Decorator to handle errors in API endpoints. - - Catches exceptions and converts them to structured error responses. - + One-line, safe-to-return description of an exception. + + The generic "an error occurred; see logs for details" tells a user nothing + and, when the failure is bad enough, the logs are unreachable too: a device + whose storage was failing returned that message from every endpoint + *including* the log viewer, because journalctl could not be executed. The + underlying `[Errno 5] Input/output error` named the fault immediately. + + Returns "TypeName: message", credentials redacted and length capped. The + type alone is worth carrying -- a bare PermissionError says more than any + generic sentence. + Args: - default_error_code: Default error code if exception doesn't match known types - default_category: Default error category - log_error: Whether to log the error + exc: The exception to describe + max_length: Truncate beyond this many characters + + Returns: + A single-line description, never empty """ - def decorator(func: Callable) -> Callable: - @functools.wraps(func) - def wrapper(*args, **kwargs): - try: - return func(*args, **kwargs) - except WebInterfaceError as e: - # Already a structured error - if log_error: - logger.error( - f"Error in {func.__name__}: {e.message}", - extra={ - 'error_code': e.error_code.value, - 'category': e.category.value, - 'context': e.context - } - ) - return jsonify(e.to_dict()), 500 - - except Exception as e: - # Convert to structured error - web_error = WebInterfaceError.from_exception( - e, - error_code=default_error_code, - context={ - 'function': func.__name__, - 'endpoint': getattr(func, '__name__', 'unknown') - } - ) - - if default_category: - web_error.category = default_category - - if log_error: - logger.error( - f"Unhandled error in {func.__name__}: {e}", - exc_info=True, - extra={ - 'error_code': web_error.error_code.value, - 'category': web_error.category.value, - 'context': web_error.context - } - ) - - return jsonify(web_error.to_dict()), 500 - - return wrapper - return decorator + message = str(exc).strip() + text = f"{type(exc).__name__}: {message}" if message else type(exc).__name__ + return redact_text(text, max_length) + + +def redact_text(text: str, max_length: int = _MAX_DETAIL_LENGTH) -> str: + """Make arbitrary text safe to hand back over HTTP. + + Split out of describe_exception because exceptions are not the only thing + worth returning: a subprocess's stderr, or a message a helper script + printed, is just as useful to a user and just as capable of carrying a + token or a password in it. + + Args: + text: The text to redact + max_length: Truncate beyond this many characters + + Returns: + A single line, credentials replaced, length capped. + """ + text = text or '' + # Order matters: the URL and header forms are more specific than the + # generic key=value pattern, which would otherwise chew the scheme. + text = _REDACT_URL_USERINFO.sub(r'\1\3', text) + text = _REDACT_AUTH_HEADER.sub(r'\1', text) + text = _REDACT_CREDENTIAL.sub(r'\1', text) + # Collapse newlines/tabs so the detail stays one line in a JSON field. + text = ' '.join(text.split()) + if len(text) > max_length: + text = text[:max_length - 1].rstrip() + '…' + return text def create_error_response( @@ -134,14 +161,17 @@ def create_success_response( "status": "success" } + # All three use `is not None` rather than truthiness: "" and {} are + # values a caller chose to send, and dropping them silently would make + # the response shape depend on the data. if data is not None: response["data"] = data - - if message: + + if message is not None: response["message"] = message - - if metadata: + + if metadata is not None: response["metadata"] = metadata - + return response diff --git a/src/web_interface/errors.py b/src/web_interface/errors.py index 11397892..bb7c6e07 100644 --- a/src/web_interface/errors.py +++ b/src/web_interface/errors.py @@ -89,7 +89,11 @@ class WebInterfaceError: self.category = category or self._infer_category(error_code) self.details = details self.context = context or {} - self.suggested_fixes = suggested_fixes or self._get_default_suggestions(error_code) + # `is None`, not truthiness: an explicit [] means "this caller has + # no suggestions to offer", which the default list would override. + self.suggested_fixes = ( + suggested_fixes if suggested_fixes is not None + else self._get_default_suggestions(error_code)) self.original_error = original_error def _infer_category(self, error_code: ErrorCode) -> ErrorCategory: diff --git a/src/web_interface/logging_config.py b/src/web_interface/logging_config.py deleted file mode 100644 index 010130d1..00000000 --- a/src/web_interface/logging_config.py +++ /dev/null @@ -1,160 +0,0 @@ -""" -Structured logging configuration for web interface. - -Provides JSON-formatted structured logging for better debugging and monitoring. -""" - -import json -import logging -import sys -from datetime import datetime -from typing import Dict, Any, Optional - - -class StructuredFormatter(logging.Formatter): - """ - JSON formatter for structured logging. - - Formats log records as JSON for easy parsing and analysis. - """ - - def format(self, record: logging.LogRecord) -> str: - """Format log record as JSON.""" - log_data = { - 'timestamp': datetime.utcnow().isoformat(), - 'level': record.levelname, - 'logger': record.name, - 'message': record.getMessage(), - 'module': record.module, - 'function': record.funcName, - 'line': record.lineno - } - - # Add exception info if present - if record.exc_info: - log_data['exception'] = self.formatException(record.exc_info) - - # Add extra fields from record - if hasattr(record, 'extra'): - log_data.update(record.extra) - - # Add context from record - if hasattr(record, 'context'): - log_data['context'] = record.context - - return json.dumps(log_data) - - def formatException(self, exc_info) -> Dict[str, Any]: - """Format exception as structured data.""" - import traceback - return { - 'type': exc_info[0].__name__ if exc_info[0] else None, - 'message': str(exc_info[1]) if exc_info[1] else None, - 'traceback': traceback.format_exception(*exc_info) - } - - -def setup_structured_logging( - level: int = logging.INFO, - use_json: bool = False, - output_stream = sys.stdout -) -> None: - """ - Set up structured logging for web interface. - - Args: - level: Logging level - use_json: Whether to use JSON formatting - output_stream: Output stream for logs - """ - root_logger = logging.getLogger() - root_logger.setLevel(level) - - # Remove existing handlers - for handler in root_logger.handlers[:]: - root_logger.removeHandler(handler) - - # Create handler - handler = logging.StreamHandler(output_stream) - handler.setLevel(level) - - # Set formatter - if use_json: - formatter = StructuredFormatter() - else: - formatter = logging.Formatter( - '%(asctime)s - %(name)s - %(levelname)s - %(message)s' - ) - - handler.setFormatter(formatter) - root_logger.addHandler(handler) - - -def log_plugin_operation( - logger: logging.Logger, - operation: str, - plugin_id: str, - status: str, - context: Optional[Dict[str, Any]] = None -) -> None: - """ - Log a plugin operation with structured data. - - Args: - logger: Logger instance - operation: Operation name (install, update, uninstall, etc.) - plugin_id: Plugin identifier - status: Operation status (success, failed, etc.) - context: Optional additional context - """ - extra = { - 'operation': operation, - 'plugin_id': plugin_id, - 'status': status - } - - if context: - extra['context'] = context - - logger.info( - f"Plugin operation: {operation} for {plugin_id} - {status}", - extra=extra - ) - - -def log_config_change( - logger: logging.Logger, - config_key: str, - action: str, - before: Optional[Dict[str, Any]] = None, - after: Optional[Dict[str, Any]] = None, - context: Optional[Dict[str, Any]] = None -) -> None: - """ - Log a configuration change with before/after values. - - Args: - logger: Logger instance - config_key: Configuration key that changed - action: Action performed (save, update, delete, etc.) - before: Configuration before change - after: Configuration after change - context: Optional additional context - """ - extra = { - 'config_key': config_key, - 'action': action - } - - if before: - extra['before'] = before - if after: - extra['after'] = after - if context: - extra['context'] = context - - logger.info( - f"Config change: {action} on {config_key}", - extra=extra - ) - diff --git a/src/web_interface/secret_helpers.py b/src/web_interface/secret_helpers.py index 310d9848..6a00cb94 100644 --- a/src/web_interface/secret_helpers.py +++ b/src/web_interface/secret_helpers.py @@ -143,6 +143,12 @@ def mask_secret_fields(config: Dict[str, Any], schema_properties: Dict[str, Any] return result +#: What a masked secret looks like on the wire. Named because the write path +#: has to recognise it coming back: a client that renders the mask and posts +#: it unchanged must not store the mask as if it were the secret. +SECRET_MASK = '\u2022' * 8 + + def mask_all_secret_values(config: Dict[str, Any]) -> Dict[str, Any]: """Blanket-mask every non-empty value in a secrets config dict. @@ -156,15 +162,25 @@ def mask_all_secret_values(config: Dict[str, Any]) -> Dict[str, Any]: Returns: A copy with all real values replaced by ``'••••••••'``. """ - masked: Dict[str, Any] = {} - for k, v in config.items(): - if isinstance(v, dict): - masked[k] = mask_all_secret_values(v) - elif v not in (None, '') and not (isinstance(v, str) and v.startswith('YOUR_')): - masked[k] = '••••••••' - else: - masked[k] = v - return masked + return {k: _mask_value(v) for k, v in config.items()} + + +def _mask_value(value: Any) -> Any: + """Mask one value, recursing through dicts and lists. + + A list used to be masked as though it were a scalar, so + ``accounts: [{"name": "a", "token": "..."}]`` came back as a single + ``'••••••••'``. Nothing leaked, but the caller could no longer see how + many entries there were or any of their non-secret fields, and the raw + editor was shown a string where the file holds an array. + """ + if isinstance(value, dict): + return {k: _mask_value(v) for k, v in value.items()} + if isinstance(value, list): + return [_mask_value(item) for item in value] + if value in (None, '') or (isinstance(value, str) and value.startswith('YOUR_')): + return value + return SECRET_MASK def remove_empty_secrets(secrets: Dict[str, Any]) -> Dict[str, Any]: @@ -189,3 +205,52 @@ def remove_empty_secrets(secrets: Dict[str, Any]) -> Dict[str, Any]: elif v is not None and not (isinstance(v, str) and v.strip() == ''): result[k] = v return result + + +def strip_masked_values(secrets: Dict[str, Any]) -> Dict[str, Any]: + """Remove values a client echoed back rather than changed. + + The counterpart to :func:`mask_all_secret_values`. A client that GETs the + masked secrets, edits one field and POSTs the whole object back is sending + ``SECRET_MASK`` for every field it did not touch. Storing those would + replace each untouched credential with eight bullet characters. + + Drops the mask and, like :func:`remove_empty_secrets`, blank values -- so + the caller can merge the result onto what is already stored and have + "unchanged" mean unchanged. Empty nested dicts are pruned. + """ + result: Dict[str, Any] = {} + for k, v in secrets.items(): + if isinstance(v, dict): + nested = strip_masked_values(v) + if nested: + result[k] = nested + elif isinstance(v, list): + # A list is merged by replacement, not element by element -- there + # is no identity to match entries on -- so a list that still holds + # a mask cannot be merged safely: keeping it would store bullets, + # and keeping the submitted entries alone would drop whichever the + # client did not send back. Dropping the key leaves the stored + # list untouched, which is what an untouched list should do. + # + # The consequence, deliberately: editing one secret inside a list + # through this endpoint requires sending real values for all of + # them. Sending some masks leaves the whole list as it was. + if not _contains_mask(v): + result[k] = v + elif v is None: + continue + elif isinstance(v, str) and (v.strip() == '' or v == SECRET_MASK): + continue + else: + result[k] = v + return result + + +def _contains_mask(value: Any) -> bool: + """True when a mask sentinel survives anywhere inside ``value``.""" + if isinstance(value, dict): + return any(_contains_mask(v) for v in value.values()) + if isinstance(value, list): + return any(_contains_mask(item) for item in value) + return value == SECRET_MASK diff --git a/src/web_interface/validators.py b/src/web_interface/validators.py index e383ae99..fa1772ef 100644 --- a/src/web_interface/validators.py +++ b/src/web_interface/validators.py @@ -43,10 +43,15 @@ def validate_image_url(url: str) -> Tuple[bool, Optional[str]]: if any(handler in url_lower for handler in ['onerror=', 'onload=', 'onclick=']): return False, "Event handlers not allowed in URLs" + # Reject directory traversal anywhere, not only in relative paths: + # http://host/../secret is as much a traversal attempt as /../secret. + if '..' in url: + return False, "Invalid path: directory traversal not allowed" + # Allow relative paths starting with / if url.startswith('/'): - # Validate it's a safe relative path (no directory traversal) - if '..' in url or url.startswith('//'): + # // would be a protocol-relative URL, not a local path + if url.startswith('//'): return False, "Invalid relative path" return True, None @@ -104,10 +109,11 @@ def validate_file_upload(filename: str, max_size_mb: int = 10, if '..' in filename or '/' in filename or '\\' in filename: return False, "Filename contains invalid characters" - # Check extension if specified + # Check extension if specified. Both sides are lowercased: the caller's + # list is as likely to hold '.TTF' as the filename is. if allowed_extensions: file_ext = Path(filename).suffix.lower() - if file_ext not in allowed_extensions: + if file_ext not in [ext.lower() for ext in allowed_extensions]: return False, f"File extension must be one of: {', '.join(allowed_extensions)}" return True, None @@ -147,7 +153,8 @@ def validate_numeric_range(value: float, min_val: Optional[float] = None, Returns: Tuple of (is_valid, error_message) """ - if not isinstance(value, (int, float)): + # bool is an int subclass, so True would otherwise validate as 1. + if not isinstance(value, (int, float)) or isinstance(value, bool): return False, "Value must be a number" if min_val is not None and value < min_val: @@ -183,11 +190,19 @@ def validate_string_length(text: str, min_length: Optional[int] = None, def sanitize_plugin_config(config: dict) -> dict: """ - Sanitize plugin configuration input to prevent injection. - + Restrict a plugin config to safe key names and value types. + + Drops keys that are not plain identifiers and values that are not + JSON-ish scalars, lists, or dicts, recursing into the latter two. + + String values are returned **unescaped**: output escaping is the + template layer's job, and escaping here would store the escaped form + in config.json. Do not read this function as XSS protection for + rendered output. + Args: config: Configuration dictionary - + Returns: Sanitized configuration dictionary """ diff --git a/start_display.sh b/start_display.sh old mode 100644 new mode 100755 diff --git a/stop_display.sh b/stop_display.sh old mode 100644 new mode 100755 diff --git a/systemd/ledmatrix-wifi-monitor.service b/systemd/ledmatrix-wifi-monitor.service index 2f08fde0..46b69bfe 100644 --- a/systemd/ledmatrix-wifi-monitor.service +++ b/systemd/ledmatrix-wifi-monitor.service @@ -10,8 +10,8 @@ WorkingDirectory=__PROJECT_ROOT_DIR__ ExecStart=/usr/bin/python3 __PROJECT_ROOT_DIR__/scripts/utils/wifi_monitor_daemon.py --interval 30 Restart=on-failure RestartSec=10 -StandardOutput=syslog -StandardError=syslog +StandardOutput=journal +StandardError=journal SyslogIdentifier=ledmatrix-wifi-monitor [Install] diff --git a/systemd/ledmatrix.service b/systemd/ledmatrix.service index f3bc9e28..ab5ddf4d 100644 --- a/systemd/ledmatrix.service +++ b/systemd/ledmatrix.service @@ -8,9 +8,38 @@ Type=simple User=root WorkingDirectory=__PROJECT_ROOT_DIR__ Environment=PYTHONDONTWRITEBYTECODE=1 +# glibc gives each allocating thread its own malloc arena, up to 8 x CPU count, +# and an arena that has grown is never handed back to the OS. This process runs +# 9 threads on a 3-core Pi, so the ceiling is 24 arenas -- and a rig measured at +# 1030 MB resident held 23 large anonymous mappings on 64 MB-aligned addresses, +# 920 MB of them, while the live data it was actually holding (widest scroll +# strip seen: 35,746 x 64) accounts for roughly 15 MB. That gap is arena bloat, +# not leaked objects: RSS was flat across repeated sampling, not climbing. +# +# Capping the arenas trades a little allocator concurrency for a large amount of +# resident memory on a device that has neither to spare. 2 is the usual value; +# raise it if frame times regress. +Environment=MALLOC_ARENA_MAX=2 ExecStart=/usr/bin/python3 __PROJECT_ROOT_DIR__/run.py -Restart=on-failure +# Restart=always, not on-failure: run.py exiting 0 (a clean shutdown path taken +# for a reason that no longer applies, e.g. a config reload) would otherwise leave +# the service stopped and the panel dark indefinitely, with systemd considering +# that a successful outcome and never bringing it back. +Restart=always RestartSec=10 +# Memory ceiling as a share of physical RAM, so one unit file suits a 512 MB +# Pi Zero 2 W and an 8 GB Pi 5 alike. This is a backstop, not a tuning knob: it +# turns "the board runs out of memory, stops being able to fork, and takes sshd +# and the panel down together until someone pulls the plug" into "this one +# service restarts". +# +# NOTE: Raspberry Pi firmware boots the kernel with cgroup_disable=memory, and +# systemd accepts this setting and then silently ignores it. Verify with: +# grep memory /sys/fs/cgroup/cgroup.controllers +# If that prints nothing, add "cgroup_enable=memory cgroup_memory=1" to +# /boot/firmware/cmdline.txt (all on line 1) and reboot. first_time_install.sh +# does this for you. +MemoryMax=85% StandardOutput=journal StandardError=journal SyslogIdentifier=ledmatrix diff --git a/test/_api_v3_test_helpers.py b/test/_api_v3_test_helpers.py new file mode 100644 index 00000000..9bcb7984 --- /dev/null +++ b/test/_api_v3_test_helpers.py @@ -0,0 +1,75 @@ +""" +Shared scaffolding for api_v3 blueprint tests. + +Not a test module (the leading underscore keeps pytest from collecting +it). It is the pytest-fixture equivalent of ``_make_client()`` in +test_uninstall_and_reconcile_endpoint.py, which is unittest-style and +requires ``self.addCleanup``. + +The api_v3 blueprint keeps its managers as attributes on a module-level +singleton, not in Flask app state, so replacing them with mocks leaks +into every later test that imports api_v3 unless the originals are put +back. ``api_v3_client`` snapshots and restores them around each test. +""" + +from unittest.mock import MagicMock + +import pytest +from flask import Flask + + +# Every manager attribute the blueprint reads. Anything missing here keeps +# whatever a previously-run test left on the singleton. +API_V3_MANAGER_ATTRS = ( + 'config_manager', 'plugin_manager', 'plugin_store_manager', + 'plugin_state_manager', 'saved_repositories_manager', 'schema_manager', + 'operation_queue', 'operation_history', 'cache_manager', +) + +_SENTINEL = object() + + +def build_app(blueprint): + app = Flask(__name__) + app.config['TESTING'] = True + app.config['SECRET_KEY'] = 'test' + app.register_blueprint(blueprint, url_prefix='/api/v3') + return app + + +@pytest.fixture +def api_v3_module(): + """The api_v3 module with every manager replaced by a MagicMock. + + Restores the original attributes afterwards. Tests point individual + managers at real objects (a ConfigManager over tmp_path, say) or set + them to None to exercise the not-initialized branches. + """ + from web_interface.blueprints import api_v3 as module + + originals = { + name: getattr(module.api_v3, name, _SENTINEL) + for name in API_V3_MANAGER_ATTRS + } + for name in API_V3_MANAGER_ATTRS: + setattr(module.api_v3, name, MagicMock()) + # Default to the direct path; queue tests opt in explicitly. + module.api_v3.operation_queue = None + + yield module + + for name, original in originals.items(): + if original is _SENTINEL: + if hasattr(module.api_v3, name): + try: + delattr(module.api_v3, name) + except AttributeError: + pass + else: + setattr(module.api_v3, name, original) + + +@pytest.fixture +def api_v3_client(api_v3_module): + """Flask test client wired to the mocked blueprint.""" + return build_app(api_v3_module.api_v3).test_client() diff --git a/test/debug_nba_api.py b/test/debug_nba_api.py deleted file mode 100644 index 5840f5e1..00000000 --- a/test/debug_nba_api.py +++ /dev/null @@ -1,166 +0,0 @@ -#!/usr/bin/env python3 -""" -Diagnostic script to examine NBA API data structure and identify the missing 'id' field issue. -""" -import requests -import logging -from typing import Dict, Any - -# Set up logging -logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s') -logger = logging.getLogger(__name__) - -def fetch_nba_teams_data() -> Dict[str, Any]: - """Fetch NBA teams data from ESPN API.""" - teams_url = "https://site.api.espn.com/apis/site/v2/sports/basketball/nba/teams" - - try: - logger.info(f"Fetching NBA teams data from: {teams_url}") - response = requests.get(teams_url, timeout=30) - response.raise_for_status() - data = response.json() - - logger.info(f"Successfully fetched NBA teams data") - logger.info(f"Response structure keys: {list(data.keys())}") - - # Examine the structure - sports = data.get('sports', []) - if sports: - logger.info(f"Number of sports: {len(sports)}") - sport = sports[0] - logger.info(f"Sport keys: {list(sport.keys())}") - - leagues = sport.get('leagues', []) - if leagues: - league = leagues[0] - logger.info(f"League keys: {list(league.keys())}") - - teams = league.get('teams', []) - logger.info(f"Number of teams: {len(teams)}") - - if teams: - # Examine first team structure - first_team = teams[0] - logger.info(f"First team keys: {list(first_team.keys())}") - - team_data = first_team.get('team', {}) - logger.info(f"Team data keys: {list(team_data.keys())}") - - # Check for id field - team_id = team_data.get('id') - team_abbr = team_data.get('abbreviation') - team_name = team_data.get('name') - - logger.info(f"Sample team: ID={team_id}, ABBR={team_abbr}, NAME={team_name}") - - if team_id: - logger.info(f"Team ID field exists: {team_id}") - else: - logger.error("Team ID field is missing!") - - # Check a few more teams to confirm structure - for i in range(min(5, len(teams))): - team = teams[i].get('team', {}) - logger.info(f"Team {i+1}: ID={team.get('id')}, ABBR={team.get('abbreviation')}") - - return data - - except Exception as e: - logger.error(f"Error fetching NBA teams data: {e}") - return {} - -def fetch_nba_standings_data() -> Dict[str, Any]: - """Fetch NBA standings data from ESPN API.""" - standings_url = "https://site.api.espn.com/apis/v2/sports/basketball/nba/standings" - - try: - logger.info(f"Fetching NBA standings data from: {standings_url}") - response = requests.get(standings_url, timeout=30) - response.raise_for_status() - data = response.json() - - logger.info(f"Successfully fetched NBA standings data") - logger.info(f"Response structure keys: {list(data.keys())}") - - # Check if standings has entries (direct structure) - if 'standings' in data and 'entries' in data['standings']: - entries = data['standings']['entries'] - logger.info(f"Number of standings entries (direct): {len(entries)}") - - if entries: - # Examine first entry structure - first_entry = entries[0] - logger.info(f"First entry keys: {list(first_entry.keys())}") - - team_data = first_entry.get('team', {}) - logger.info(f"Team data keys: {list(team_data.keys())}") - - # Check for id field - team_id = team_data.get('id') - team_abbr = team_data.get('abbreviation') - team_name = team_data.get('displayName') - - logger.info(f"Sample standings team: ID={team_id}, ABBR={team_abbr}, NAME={team_name}") - - if team_id: - logger.info(f"Standings team ID field exists: {team_id}") - else: - logger.error("Standings team ID field is missing!") - - # Check children structure (divisions/conferences) - if 'children' in data: - children = data.get('children', []) - logger.info(f"Number of children (divisions/conferences): {len(children)}") - - for i, child in enumerate(children): - logger.info(f"Child {i+1} keys: {list(child.keys())}") - child_name = child.get('displayName', 'Unknown') - logger.info(f"Child {i+1} name: {child_name}") - - if 'standings' in child and 'entries' in child['standings']: - entries = child['standings']['entries'] - logger.info(f"Child {i+1} has {len(entries)} entries") - - if entries: - # Examine first entry in this child - first_entry = entries[0] - logger.info(f"Child {i+1} first entry keys: {list(first_entry.keys())}") - - team_data = first_entry.get('team', {}) - logger.info(f"Child {i+1} team data keys: {list(team_data.keys())}") - - # Check for id field - team_id = team_data.get('id') - team_abbr = team_data.get('abbreviation') - team_name = team_data.get('displayName') - - logger.info(f"Child {i+1} sample team: ID={team_id}, ABBR={team_abbr}, NAME={team_name}") - - if team_id: - logger.info(f"Child {i+1} team ID field exists: {team_id}") - else: - logger.error(f"Child {i+1} team ID field is missing!") - - return data - - except Exception as e: - logger.error(f"Error fetching NBA standings data: {e}") - return {} - -def main(): - """Main diagnostic function.""" - logger.info("Starting NBA API data structure diagnosis") - - # Fetch teams data - teams_data = fetch_nba_teams_data() - - # Fetch standings data - standings_data = fetch_nba_standings_data() - - # Summary - logger.info("Diagnosis complete") - logger.info("Check the logs above to see if team 'id' fields are present") - logger.info("The leaderboard manager needs team 'id' fields for logo fetching") - -if __name__ == "__main__": - main() diff --git a/test/fixtures/plugins/ci-fixture-plugin/config_schema.json b/test/fixtures/plugins/ci-fixture-plugin/config_schema.json new file mode 100644 index 00000000..9410d90a --- /dev/null +++ b/test/fixtures/plugins/ci-fixture-plugin/config_schema.json @@ -0,0 +1,31 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "title": "CI Fixture Plugin", + "type": "object", + "properties": { + "enabled": { + "type": "boolean", + "default": true + }, + "display_duration": { + "type": "number", + "default": 5 + }, + "border_color": { + "type": "array", + "items": {"type": "integer", "minimum": 0, "maximum": 255}, + "minItems": 3, + "maxItems": 3, + "default": [0, 255, 0], + "description": "RGB color of the border rectangle." + }, + "diagonal_color": { + "type": "array", + "items": {"type": "integer", "minimum": 0, "maximum": 255}, + "minItems": 3, + "maxItems": 3, + "default": [255, 0, 0], + "description": "RGB color of the diagonals." + } + } +} diff --git a/test/fixtures/plugins/ci-fixture-plugin/manager.py b/test/fixtures/plugins/ci-fixture-plugin/manager.py new file mode 100644 index 00000000..02805f60 --- /dev/null +++ b/test/fixtures/plugins/ci-fixture-plugin/manager.py @@ -0,0 +1,44 @@ +""" +CI fixture plugin. + +Exists so the plugin safety harness (test/plugins/test_plugin_matrix.py and +the plugin-safety CI job) always has at least one real plugin to load and +render — without it, an empty plugins/ directory turns the whole job into a +green no-op. The render is deliberately trivial and fully deterministic: +a border rectangle plus both diagonals, sized from the display manager's +declared dimensions. No fonts, no network, no time dependence, so golden +images are stable across platforms. +""" + +from PIL import ImageDraw + +from src.plugin_system.base_plugin import BasePlugin + + +class CIFixturePlugin(BasePlugin): + """Deterministic CI-only fixture plugin: renders a border + diagonals + pattern sized from the display's declared dimensions. Never shipped to + devices; exists solely so the plugin safety harness has a real plugin + to exercise in CI.""" + + def update(self) -> None: + """Nothing to fetch — the render is self-contained.""" + + def display(self, force_clear: bool = False) -> None: + self.display_manager.clear() + width = self.display_manager.matrix.width + height = self.display_manager.matrix.height + border = tuple(self.config.get("border_color", [0, 255, 0])) + diagonal = tuple(self.config.get("diagonal_color", [255, 0, 0])) + + image = self.display_manager.image + draw = ImageDraw.Draw(image) + # Blank only the declared panel area, then draw edge-to-edge content: + # the border proves the plugin reads dynamic dimensions (any overflow + # or underfill at any size is a harness bug or a dimensions bug), the + # diagonals make golden comparisons sensitive to size/offset drift. + draw.rectangle([0, 0, width - 1, height - 1], fill=(0, 0, 0)) + draw.rectangle([0, 0, width - 1, height - 1], outline=border) + draw.line([0, 0, width - 1, height - 1], fill=diagonal) + draw.line([0, height - 1, width - 1, 0], fill=diagonal) + self.display_manager.update_display() diff --git a/test/fixtures/plugins/ci-fixture-plugin/manifest.json b/test/fixtures/plugins/ci-fixture-plugin/manifest.json new file mode 100644 index 00000000..40d188cb --- /dev/null +++ b/test/fixtures/plugins/ci-fixture-plugin/manifest.json @@ -0,0 +1,13 @@ +{ + "id": "ci-fixture-plugin", + "name": "CI Fixture Plugin", + "version": "1.0.0", + "description": "Bundled test fixture so the plugin safety harness always has at least one real plugin to render in CI. Draws a deterministic border + diagonals pattern at any panel size. Not installable from the store and never shipped to devices.", + "author": "LEDMatrix", + "entry_point": "manager.py", + "class_name": "CIFixturePlugin", + "display_modes": ["ci-fixture"], + "update_interval": 3600, + "min_ledmatrix_version": "2.0.0", + "compatible_versions": [">=2.0.0"] +} diff --git a/test/fixtures/plugins/ci-fixture-plugin/requirements.txt b/test/fixtures/plugins/ci-fixture-plugin/requirements.txt new file mode 100644 index 00000000..7fefb0f5 --- /dev/null +++ b/test/fixtures/plugins/ci-fixture-plugin/requirements.txt @@ -0,0 +1,7 @@ +# No dependencies — the fixture must load in any environment. +# +# Pillow is deliberately NOT pinned here even though manager.py imports +# PIL: it is a core LEDMatrix dependency (see the repo-root +# requirements.txt), so it is always present wherever the harness runs, +# and the harness loads plugins with install_deps=False anyway. Pinning +# it here would only invite a needless pip install during test runs. diff --git a/test/fixtures/plugins/ci-fixture-plugin/test/golden/128x32/ci-fixture.png b/test/fixtures/plugins/ci-fixture-plugin/test/golden/128x32/ci-fixture.png new file mode 100644 index 00000000..32608a8c Binary files /dev/null and b/test/fixtures/plugins/ci-fixture-plugin/test/golden/128x32/ci-fixture.png differ diff --git a/test/fixtures/plugins/ci-fixture-plugin/test/golden/128x64/ci-fixture.png b/test/fixtures/plugins/ci-fixture-plugin/test/golden/128x64/ci-fixture.png new file mode 100644 index 00000000..de738059 Binary files /dev/null and b/test/fixtures/plugins/ci-fixture-plugin/test/golden/128x64/ci-fixture.png differ diff --git a/test/fixtures/plugins/ci-fixture-plugin/test/golden/128x96/ci-fixture.png b/test/fixtures/plugins/ci-fixture-plugin/test/golden/128x96/ci-fixture.png new file mode 100644 index 00000000..3107e6da Binary files /dev/null and b/test/fixtures/plugins/ci-fixture-plugin/test/golden/128x96/ci-fixture.png differ diff --git a/test/fixtures/plugins/ci-fixture-plugin/test/golden/256x128/ci-fixture.png b/test/fixtures/plugins/ci-fixture-plugin/test/golden/256x128/ci-fixture.png new file mode 100644 index 00000000..dd43ab76 Binary files /dev/null and b/test/fixtures/plugins/ci-fixture-plugin/test/golden/256x128/ci-fixture.png differ diff --git a/test/fixtures/plugins/ci-fixture-plugin/test/golden/256x32/ci-fixture.png b/test/fixtures/plugins/ci-fixture-plugin/test/golden/256x32/ci-fixture.png new file mode 100644 index 00000000..01638b37 Binary files /dev/null and b/test/fixtures/plugins/ci-fixture-plugin/test/golden/256x32/ci-fixture.png differ diff --git a/test/fixtures/plugins/ci-fixture-plugin/test/golden/64x32/ci-fixture.png b/test/fixtures/plugins/ci-fixture-plugin/test/golden/64x32/ci-fixture.png new file mode 100644 index 00000000..9312505b Binary files /dev/null and b/test/fixtures/plugins/ci-fixture-plugin/test/golden/64x32/ci-fixture.png differ diff --git a/test/fixtures/plugins/ci-fixture-plugin/test/golden/64x64/ci-fixture.png b/test/fixtures/plugins/ci-fixture-plugin/test/golden/64x64/ci-fixture.png new file mode 100644 index 00000000..7095ed8e Binary files /dev/null and b/test/fixtures/plugins/ci-fixture-plugin/test/golden/64x64/ci-fixture.png differ diff --git a/test/fixtures/plugins/ci-fixture-plugin/test/golden/96x48/ci-fixture.png b/test/fixtures/plugins/ci-fixture-plugin/test/golden/96x48/ci-fixture.png new file mode 100644 index 00000000..239561e4 Binary files /dev/null and b/test/fixtures/plugins/ci-fixture-plugin/test/golden/96x48/ci-fixture.png differ diff --git a/test/plugins/conftest.py b/test/plugins/conftest.py index 27bb6718..cff6dd42 100644 --- a/test/plugins/conftest.py +++ b/test/plugins/conftest.py @@ -23,9 +23,17 @@ os.environ['EMULATOR'] = 'true' def plugins_dir() -> Path: """Get the plugins directory path. - Checks plugins/ first, then falls back to plugin-repos/ - for monorepo development environments. + Honors LEDMATRIX_PLUGINS_DIR (first entry) when set — the same override + test_plugin_matrix.py uses, so CI can point every plugin suite at the + bundled fixture plugins. Otherwise checks plugins/ first, then falls + back to plugin-repos/ for monorepo development environments. """ + env = os.environ.get('LEDMATRIX_PLUGINS_DIR') + if env: + first = env.split(os.pathsep)[0] + if first: + return Path(first) + plugins_path = project_root / 'plugins' plugin_repos_path = project_root / 'plugin-repos' diff --git a/test/plugins/test_basketball_scoreboard.py b/test/plugins/test_basketball_scoreboard.py index 0fbd8b79..b93fb89f 100644 --- a/test/plugins/test_basketball_scoreboard.py +++ b/test/plugins/test_basketball_scoreboard.py @@ -1,5 +1,10 @@ """ Integration tests for basketball-scoreboard plugin. + +Requires the real plugin to be installed (plugins/ or plugin-repos/, +or the dir named by LEDMATRIX_PLUGINS_DIR) — on machines without it, +every test here skips by design. CI covers plugin safety with the +bundled fixture plugin via test_plugin_matrix.py instead. """ import pytest diff --git a/test/plugins/test_calendar.py b/test/plugins/test_calendar.py index 18528fdb..4d874940 100644 --- a/test/plugins/test_calendar.py +++ b/test/plugins/test_calendar.py @@ -1,5 +1,10 @@ """ Integration tests for calendar plugin. + +Requires the real plugin to be installed (plugins/ or plugin-repos/, +or the dir named by LEDMATRIX_PLUGINS_DIR) — on machines without it, +every test here skips by design. CI covers plugin safety with the +bundled fixture plugin via test_plugin_matrix.py instead. """ import pytest diff --git a/test/plugins/test_clock_simple.py b/test/plugins/test_clock_simple.py index 507feec9..9b25a59e 100644 --- a/test/plugins/test_clock_simple.py +++ b/test/plugins/test_clock_simple.py @@ -1,5 +1,10 @@ """ Integration tests for clock-simple plugin. + +Requires the real plugin to be installed (plugins/ or plugin-repos/, +or the dir named by LEDMATRIX_PLUGINS_DIR) — on machines without it, +every test here skips by design. CI covers plugin safety with the +bundled fixture plugin via test_plugin_matrix.py instead. """ import pytest diff --git a/test/plugins/test_odds_ticker.py b/test/plugins/test_odds_ticker.py index 90209548..231ed7de 100644 --- a/test/plugins/test_odds_ticker.py +++ b/test/plugins/test_odds_ticker.py @@ -1,5 +1,10 @@ """ Integration tests for odds-ticker plugin. + +Requires the real plugin to be installed (plugins/ or plugin-repos/, +or the dir named by LEDMATRIX_PLUGINS_DIR) — on machines without it, +every test here skips by design. CI covers plugin safety with the +bundled fixture plugin via test_plugin_matrix.py instead. """ import pytest diff --git a/test/plugins/test_soccer_scoreboard.py b/test/plugins/test_soccer_scoreboard.py index 36212bfd..d5526fb8 100644 --- a/test/plugins/test_soccer_scoreboard.py +++ b/test/plugins/test_soccer_scoreboard.py @@ -1,5 +1,10 @@ """ Integration tests for soccer-scoreboard plugin. + +Requires the real plugin to be installed (plugins/ or plugin-repos/, +or the dir named by LEDMATRIX_PLUGINS_DIR) — on machines without it, +every test here skips by design. CI covers plugin safety with the +bundled fixture plugin via test_plugin_matrix.py instead. """ import pytest diff --git a/test/plugins/test_text_display.py b/test/plugins/test_text_display.py index a43815ea..34adfa49 100644 --- a/test/plugins/test_text_display.py +++ b/test/plugins/test_text_display.py @@ -1,5 +1,10 @@ """ Integration tests for text-display plugin. + +Requires the real plugin to be installed (plugins/ or plugin-repos/, +or the dir named by LEDMATRIX_PLUGINS_DIR) — on machines without it, +every test here skips by design. CI covers plugin safety with the +bundled fixture plugin via test_plugin_matrix.py instead. """ import pytest diff --git a/test/test_api_helper.py b/test/test_api_helper.py new file mode 100644 index 00000000..b3c24e99 --- /dev/null +++ b/test/test_api_helper.py @@ -0,0 +1,275 @@ +""" +Tests for src/common/api_helper.py (APIHelper). + +Covers rate limiting, cached GETs, ESPN URL/cache-key construction, +session header defaults and per-call merging, the retry adapter, and the +fixed clear_cache() behavior (real CacheManager surface: clear_cache / +delete / list_cache_files, with safe no-ops elsewhere). + +No real network: helper.session.get/post are always replaced with mocks. +""" + +import types +from unittest.mock import MagicMock, Mock + +import pytest +import requests +from freezegun import freeze_time + +import src.common.api_helper as api_helper_module +from src.common.api_helper import APIHelper + + +def _make_response(payload): + response = MagicMock() + response.json.return_value = payload + response.raise_for_status.return_value = None + return response + + +@pytest.fixture +def cache(): + cache = MagicMock() + cache.get.return_value = None + return cache + + +@pytest.fixture +def helper(cache): + helper = APIHelper(cache_manager=cache) + # Default min interval is 1.0s and would really sleep between requests. + helper.set_rate_limit(0) + return helper + + +# --------------------------------------------------------------------------- +# Rate limiting +# --------------------------------------------------------------------------- + +class TestRateLimiting: + def test_sleeps_for_remaining_interval(self, helper, monkeypatch): + fake_time = MagicMock() + fake_time.time.side_effect = [102.0, 105.0] + monkeypatch.setattr(api_helper_module, 'time', fake_time) + + helper.set_rate_limit(5) + helper._last_request_time = 100.0 + helper._enforce_rate_limit() + + # 2s elapsed of a 5s interval -> sleep the remaining 3s. + fake_time.sleep.assert_called_once() + assert fake_time.sleep.call_args[0][0] == pytest.approx(3.0) + assert helper._last_request_time == 105.0 + + def test_no_sleep_when_interval_elapsed(self, helper, monkeypatch): + fake_time = MagicMock() + fake_time.time.side_effect = [200.0, 201.0] + monkeypatch.setattr(api_helper_module, 'time', fake_time) + + helper.set_rate_limit(5) + helper._last_request_time = 100.0 + helper._enforce_rate_limit() + + fake_time.sleep.assert_not_called() + assert helper._last_request_time == 201.0 + + +# --------------------------------------------------------------------------- +# get() +# --------------------------------------------------------------------------- + +class TestGet: + def test_cache_hit_skips_request_and_rate_limit(self, helper, cache): + cache.get.return_value = {'cached': True} + helper.session.get = Mock() + rate_spy = Mock() + helper._enforce_rate_limit = rate_spy + + result = helper.get('https://example.com/api', cache_key='k') + + assert result == {'cached': True} + helper.session.get.assert_not_called() + rate_spy.assert_not_called() + + def test_cache_miss_fetches_and_caches_without_ttl(self, helper, cache): + cache.get.return_value = None + helper.session.get = Mock(return_value=_make_response({'a': 1})) + + result = helper.get('https://example.com/api', cache_key='k', + cache_ttl=999) + + assert result == {'a': 1} + # Pin the ttl-dropped contract: CacheManager.set is called with + # (key, data) only — the cache_ttl argument is discarded. + cache.set.assert_called_once_with('k', {'a': 1}) + + def test_request_exception_returns_none_and_caches_nothing( + self, helper, cache): + helper.session.get = Mock( + side_effect=requests.exceptions.RequestException('boom')) + + result = helper.get('https://example.com/api', cache_key='k') + + assert result is None + cache.set.assert_not_called() + + def test_timeout_zero_falls_back_to_default(self, helper): + # Quirk pin: `timeout or self.default_timeout` treats an explicit + # timeout=0 as falsy, so the default (30) is used instead. + helper.session.get = Mock(return_value=_make_response({})) + + helper.get('https://example.com/api', timeout=0) + + assert helper.session.get.call_args.kwargs['timeout'] == 30 + + def test_per_call_headers_merge_over_session_headers(self, helper): + helper.session.get = Mock(return_value=_make_response({})) + + helper.get('https://example.com/api', headers={'X-Custom': 'yes'}) + + sent = helper.session.get.call_args.kwargs['headers'] + # Merged, not replaced: session defaults survive alongside the + # per-call header. + assert sent['X-Custom'] == 'yes' + assert sent['User-Agent'] == ( + 'LEDMatrix/1.0 (+https://github.com/ChuckBuilds/LEDMatrix)') + assert sent['Accept'] == 'application/json' + # The session's own headers are not polluted by the per-call ones. + assert 'X-Custom' not in helper.session.headers + + +# --------------------------------------------------------------------------- +# ESPN helpers +# --------------------------------------------------------------------------- + +class TestEspnHelpers: + @freeze_time('2026-08-07') + def test_fetch_espn_scoreboard_url_params_and_cache_key(self, helper): + helper.get = Mock(return_value={'ok': 1}) + + result = helper.fetch_espn_scoreboard('football', 'nfl') + + assert result == {'ok': 1} + helper.get.assert_called_once_with( + 'https://site.api.espn.com/apis/site/v2/sports/football/nfl/scoreboard', + params={'dates': '20260807', 'limit': 1000}, + cache_key='espn_football_nfl_20260807', + cache_ttl=300, + ) + + def test_fetch_espn_scoreboard_explicit_date(self, helper): + helper.get = Mock(return_value=None) + + helper.fetch_espn_scoreboard('basketball', 'nba', date='20250115') + + kwargs = helper.get.call_args.kwargs + assert kwargs['params'] == {'dates': '20250115', 'limit': 1000} + assert kwargs['cache_key'] == 'espn_basketball_nba_20250115' + + def test_fetch_espn_standings_url_and_cache_key(self, helper): + helper.get = Mock(return_value={'ok': 1}) + + helper.fetch_espn_standings('football', 'nfl') + + helper.get.assert_called_once_with( + 'https://site.api.espn.com/apis/site/v2/sports/football/nfl/standings', + cache_key='espn_standings_football_nfl', + cache_ttl=3600, + ) + + def test_fetch_espn_rankings_url_and_cache_key(self, helper): + helper.get = Mock(return_value={'ok': 1}) + + helper.fetch_espn_rankings('football', 'college-football') + + helper.get.assert_called_once_with( + 'https://site.api.espn.com/apis/site/v2/sports/football/college-football/rankings', + cache_key='espn_rankings_football_college-football', + cache_ttl=3600, + ) + + +# --------------------------------------------------------------------------- +# Session setup +# --------------------------------------------------------------------------- + +class TestSessionSetup: + def test_user_agent_exact(self, helper): + # Regression guard: ESPN began 403ing other user agents; this exact + # string must be sent on every request. + assert helper.session.headers['User-Agent'] == ( + 'LEDMatrix/1.0 (+https://github.com/ChuckBuilds/LEDMatrix)') + + def test_retry_adapter_configuration(self): + helper = APIHelper(cache_manager=None, max_retries=7) + + retries = helper.session.get_adapter('https://x').max_retries + assert retries.total == 7 + assert {429, 500, 502, 503, 504} <= set(retries.status_forcelist) + + +# --------------------------------------------------------------------------- +# clear_cache (fixed behavior: real CacheManager surface) +# --------------------------------------------------------------------------- + +class TestClearCache: + def test_no_pattern_uses_clear_cache_method(self): + manager = types.SimpleNamespace(clear_cache=Mock()) + helper = APIHelper(cache_manager=manager) + helper.set_rate_limit(0) + + helper.clear_cache() + + manager.clear_cache.assert_called_once_with() + + def test_no_pattern_falls_back_to_clear(self): + manager = types.SimpleNamespace(clear=Mock()) + helper = APIHelper(cache_manager=manager) + helper.set_rate_limit(0) + + helper.clear_cache() + + manager.clear.assert_called_once_with() + + def test_no_pattern_manager_without_any_clear_is_noop(self): + helper = APIHelper(cache_manager=object()) + helper.set_rate_limit(0) + + helper.clear_cache() # must not raise + + def test_pattern_deletes_only_matching_keys(self): + manager = types.SimpleNamespace( + list_cache_files=Mock(return_value=[ + {'key': 'espn_nfl_x'}, + {'key': 'other'}, + ]), + delete=Mock(), + ) + helper = APIHelper(cache_manager=manager) + helper.set_rate_limit(0) + + helper.clear_cache(pattern='espn') + + manager.delete.assert_called_once_with('espn_nfl_x') + + def test_pattern_manager_without_list_cache_files_is_noop(self): + helper = APIHelper(cache_manager=object()) + helper.set_rate_limit(0) + + helper.clear_cache(pattern='espn') # must not raise + + +# --------------------------------------------------------------------------- +# No cache manager +# --------------------------------------------------------------------------- + +class TestNoCacheManager: + def test_all_cache_operations_safe_without_manager(self): + helper = APIHelper(cache_manager=None) + helper.set_rate_limit(0) + + assert helper.get_cache('k') is None + assert helper._get_from_cache('k') is None + assert helper.set_cache('k', {'a': 1}) is None + assert helper.clear_cache() is None + assert helper.clear_cache(pattern='espn') is None diff --git a/test/test_api_v3_calendar_credentials.py b/test/test_api_v3_calendar_credentials.py new file mode 100644 index 00000000..48274e01 --- /dev/null +++ b/test/test_api_v3_calendar_credentials.py @@ -0,0 +1,226 @@ +""" +Endpoint tests for POST /plugins/calendar/upload-credentials. + +The endpoint takes an uploaded Google OAuth credentials file, writes it +into the calendar plugin's directory as credentials.json at mode 0600, and +copies any previous file aside first. It had no tests. + +Regression coverage for two fixed bugs: +- The OAuth-shape check sat inside `except Exception: pass`, so a valid + JSON document that is not an object — a bare `42`, a list, a string — + raised TypeError on the membership test, was swallowed, and got saved + as credentials.json anyway. +- Each overwrite created a timestamped backup and nothing ever removed + them, so every re-upload left another complete copy of the user's OAuth + client credentials in the plugin directory, indefinitely. +""" + +import io +import json +import os +import stat +import sys +import time +from pathlib import Path +from types import SimpleNamespace + +import pytest + +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from test._api_v3_test_helpers import api_v3_client, api_v3_module # noqa: F401,E402 + +URL = "/api/v3/plugins/calendar/upload-credentials" + +VALID_CREDENTIALS = { + "installed": { + "client_id": "abc.apps.googleusercontent.com", + "client_secret": "shh", + "redirect_uris": ["http://localhost"], + } +} + + +@pytest.fixture +def plugin_dir(tmp_path, api_v3_module): + directory = tmp_path / "plugins" / "calendar" + directory.mkdir(parents=True) + api_v3_module.api_v3.plugin_manager.get_plugin_directory.return_value = str(directory) + return directory + + +def upload(client, content, filename="credentials.json"): + # bytes are sent verbatim (to exercise malformed input); anything else + # is serialized, so None becomes the JSON literal null rather than an + # empty body. + payload = content if isinstance(content, bytes) else json.dumps(content).encode() + return client.post( + URL, + data={"file": (io.BytesIO(payload), filename)}, + content_type="multipart/form-data", + ) + + +def backups(plugin_dir): + return sorted(plugin_dir.glob("credentials.json.backup.*")) + + +class TestRequestValidation: + def test_no_file_part_is_a_400(self, api_v3_client, plugin_dir): + response = api_v3_client.post(URL, data={}, content_type="multipart/form-data") + assert response.status_code == 400 + assert "No file provided" in response.get_json()["message"] + + def test_empty_filename_is_a_400(self, api_v3_client, plugin_dir): + response = upload(api_v3_client, VALID_CREDENTIALS, filename="") + assert response.status_code == 400 + + @pytest.mark.parametrize("filename", ["creds.txt", "creds.pem", "creds"]) + def test_non_json_extension_is_a_400(self, api_v3_client, plugin_dir, filename): + response = upload(api_v3_client, VALID_CREDENTIALS, filename=filename) + assert response.status_code == 400 + assert "JSON file" in response.get_json()["message"] + + def test_uppercase_json_extension_accepted(self, api_v3_client, plugin_dir): + assert upload(api_v3_client, VALID_CREDENTIALS, + filename="CREDENTIALS.JSON").status_code == 200 + + def test_oversized_file_is_a_400(self, api_v3_client, plugin_dir): + response = upload(api_v3_client, b"x" * (1024 * 1024 + 1)) + assert response.status_code == 400 + assert "1MB" in response.get_json()["message"] + assert not (plugin_dir / "credentials.json").exists() + + def test_invalid_json_is_a_400(self, api_v3_client, plugin_dir): + response = upload(api_v3_client, b"{not json") + assert response.status_code == 400 + assert "not valid JSON" in response.get_json()["message"] + assert not (plugin_dir / "credentials.json").exists() + + def test_missing_plugin_directory_is_a_404(self, api_v3_client, api_v3_module, tmp_path): + api_v3_module.api_v3.plugin_manager.get_plugin_directory.return_value = str( + tmp_path / "not-installed") + assert upload(api_v3_client, VALID_CREDENTIALS).status_code == 404 + + +class TestOAuthShapeValidation: + def test_installed_key_accepted(self, api_v3_client, plugin_dir): + assert upload(api_v3_client, VALID_CREDENTIALS).status_code == 200 + + def test_web_key_accepted(self, api_v3_client, plugin_dir): + assert upload(api_v3_client, {"web": {"client_id": "x"}}).status_code == 200 + + def test_object_without_oauth_keys_is_a_400(self, api_v3_client, plugin_dir): + response = upload(api_v3_client, {"something": "else"}) + assert response.status_code == 400 + assert "valid Google OAuth" in response.get_json()["message"] + assert not (plugin_dir / "credentials.json").exists() + + @pytest.mark.parametrize("content", [42, "a string", [1, 2, 3], True, None]) + def test_valid_json_that_is_not_an_object_is_rejected( + self, api_v3_client, plugin_dir, content): + # Regression: `'installed' not in 42` raises TypeError, which the + # bare `except Exception: pass` swallowed — the file was then saved + # as credentials.json despite being unusable as credentials. + response = upload(api_v3_client, content) + assert response.status_code == 400 + assert "valid Google OAuth" in response.get_json()["message"] + assert not (plugin_dir / "credentials.json").exists() + + +class TestSaving: + def test_file_written_with_contents_intact(self, api_v3_client, plugin_dir): + response = upload(api_v3_client, VALID_CREDENTIALS) + assert response.status_code == 200 + saved = json.loads((plugin_dir / "credentials.json").read_text()) + assert saved == VALID_CREDENTIALS + + def test_response_reports_the_path(self, api_v3_client, plugin_dir): + body = upload(api_v3_client, VALID_CREDENTIALS).get_json() + assert body["path"].endswith("credentials.json") + + def test_permissions_are_owner_only(self, api_v3_client, plugin_dir): + upload(api_v3_client, VALID_CREDENTIALS) + mode = stat.S_IMODE((plugin_dir / "credentials.json").stat().st_mode) + assert mode == 0o600 + + def test_first_upload_creates_no_backup(self, api_v3_client, plugin_dir): + upload(api_v3_client, VALID_CREDENTIALS) + assert backups(plugin_dir) == [] + + def test_overwrite_backs_up_the_previous_file(self, api_v3_client, plugin_dir): + (plugin_dir / "credentials.json").write_text(json.dumps({"installed": {"old": 1}})) + upload(api_v3_client, VALID_CREDENTIALS) + assert len(backups(plugin_dir)) == 1 + assert json.loads(backups(plugin_dir)[0].read_text()) == {"installed": {"old": 1}} + assert json.loads((plugin_dir / "credentials.json").read_text()) == VALID_CREDENTIALS + + +class TestBackupPruning: + def _seed(self, plugin_dir, count): + """Create `count` backups with distinct, increasing mtimes.""" + now = int(time.time()) + for i in range(count): + path = plugin_dir / f"credentials.json.backup.{now - (count - i) * 10}" + path.write_text(json.dumps({"installed": {"gen": i}})) + os.utime(path, (now - (count - i) * 10, now - (count - i) * 10)) + + def test_old_backups_are_pruned(self, api_v3_client, plugin_dir): + # Regression: nothing ever removed these, so a plugin directory + # accumulated one full copy of the user's OAuth credentials per + # re-upload, forever. + (plugin_dir / "credentials.json").write_text(json.dumps({"installed": {"cur": 1}})) + self._seed(plugin_dir, 7) + assert len(backups(plugin_dir)) == 7 + + upload(api_v3_client, VALID_CREDENTIALS) + assert len(backups(plugin_dir)) == 5 + + def test_the_newest_backups_are_the_ones_kept(self, api_v3_client, plugin_dir): + (plugin_dir / "credentials.json").write_text(json.dumps({"installed": {"cur": 1}})) + self._seed(plugin_dir, 7) + + upload(api_v3_client, VALID_CREDENTIALS) + remaining = backups(plugin_dir) + # The just-created backup (of "cur") plus the four newest seeds. + contents = [json.loads(p.read_text()) for p in remaining] + assert {"installed": {"cur": 1}} in contents + assert {"installed": {"gen": 0}} not in contents # oldest seed gone + + def test_under_the_limit_nothing_is_removed(self, api_v3_client, plugin_dir): + (plugin_dir / "credentials.json").write_text(json.dumps({"installed": {"cur": 1}})) + self._seed(plugin_dir, 2) + upload(api_v3_client, VALID_CREDENTIALS) + assert len(backups(plugin_dir)) == 3 # 2 seeded + 1 new + + def test_repeated_uploads_stay_bounded( + self, api_v3_client, plugin_dir, api_v3_module, monkeypatch): + # The backup filename carries int(time.time()), so uploads inside + # the same second all write the same name and overwrite each other. + # Advance a fake clock a second per round — otherwise this never + # reaches six backups and the bound holds for the wrong reason. + clock = {"now": int(time.time())} + monkeypatch.setattr( + api_v3_module, "time", SimpleNamespace(time=lambda: clock["now"])) + for i in range(10): + clock["now"] += 1 + upload(api_v3_client, {"installed": {"round": i}}) + os.utime(plugin_dir / "credentials.json", + (clock["now"], clock["now"])) + remaining = backups(plugin_dir) + assert len(remaining) == 5 + # And they are the five most recent rounds, not an arbitrary five. + kept = sorted(int(p.name.rsplit(".", 1)[1]) for p in remaining) + assert kept == [clock["now"] - 4 + i for i in range(5)] + + def test_unremovable_backup_does_not_fail_the_upload( + self, api_v3_client, plugin_dir, monkeypatch): + (plugin_dir / "credentials.json").write_text(json.dumps({"installed": {"cur": 1}})) + self._seed(plugin_dir, 7) + + def refuse(self): + raise OSError("read-only filesystem") + monkeypatch.setattr(Path, "unlink", refuse) + + # Pruning is housekeeping; failing it must not lose the upload. + assert upload(api_v3_client, VALID_CREDENTIALS).status_code == 200 diff --git a/test/test_api_v3_music_auth_endpoints.py b/test/test_api_v3_music_auth_endpoints.py new file mode 100644 index 00000000..f7c709af --- /dev/null +++ b/test/test_api_v3_music_auth_endpoints.py @@ -0,0 +1,302 @@ +""" +Endpoint tests for /plugins/authenticate/spotify and .../ytm. + +The Spotify step-2 handler writes a Python wrapper script to a temp file +with the user's redirect URL embedded in it, then runs that file through +subprocess. That is the most dangerous shape in the blueprint and had no +tests: the URL is user input reaching generated source code. + +The two endpoints are NOT symmetrical, despite the matching names. Only +Spotify has a two-step flow, a wrapper script, and a redirect_url; YTM +just runs its script directly. + +Regression coverage for one fixed bug: the wrapper file was unlinked in +the success/failure branch and again in the TimeoutExpired handler, so +any other failure from subprocess.run — the interpreter missing, a fork +failure, an interrupted call — left a temp file containing the user's +redirect URL behind. +""" + +import ast +import json +import os +import subprocess +import sys +from pathlib import Path +from unittest.mock import MagicMock, patch + +import pytest + +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from test._api_v3_test_helpers import api_v3_client, api_v3_module # noqa: F401,E402 + + +@pytest.fixture +def plugin_dir(tmp_path, api_v3_module): + """A plugin directory containing both auth scripts.""" + directory = tmp_path / "plugins" / "ledmatrix-music" + directory.mkdir(parents=True) + (directory / "authenticate_spotify.py").write_text("print('spotify')\n") + (directory / "authenticate_ytm.py").write_text("print('ytm')\n") + api_v3_module.api_v3.plugin_manager.get_plugin_directory.return_value = str(directory) + return directory + + +def completed(returncode=0, stdout="ok", stderr=""): + return subprocess.CompletedProcess( + args=["python3"], returncode=returncode, stdout=stdout, stderr=stderr) + + +class TestSpotifyPreconditions: + URL = "/api/v3/plugins/authenticate/spotify" + + def test_missing_plugin_directory_is_404(self, api_v3_client, api_v3_module, tmp_path): + api_v3_module.api_v3.plugin_manager.get_plugin_directory.return_value = str( + tmp_path / "not-installed") + response = api_v3_client.post(self.URL, json={}) + assert response.status_code == 404 + assert response.get_json()["message"] == "Plugin not found" + + def test_none_plugin_directory_is_404(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_manager.get_plugin_directory.return_value = None + assert api_v3_client.post(self.URL, json={}).status_code == 404 + + def test_missing_auth_script_is_404(self, api_v3_client, plugin_dir): + (plugin_dir / "authenticate_spotify.py").unlink() + response = api_v3_client.post(self.URL, json={}) + assert response.status_code == 404 + assert "script not found" in response.get_json()["message"] + + +class TestSpotifyStepTwo: + """redirect_url present — the wrapper-script path.""" + + URL = "/api/v3/plugins/authenticate/spotify" + + def test_success(self, api_v3_client, plugin_dir): + with patch.object(subprocess, "run", return_value=completed(0, "done")): + response = api_v3_client.post(self.URL, json={"redirect_url": "http://cb/?code=x"}) + assert response.status_code == 200 + body = response.get_json() + assert body["status"] == "success" + assert body["output"] == "done" + + def test_script_failure_is_a_400_with_combined_output(self, api_v3_client, plugin_dir): + with patch.object(subprocess, "run", return_value=completed(1, "out", "err")): + response = api_v3_client.post(self.URL, json={"redirect_url": "http://cb/?code=x"}) + assert response.status_code == 400 + assert response.get_json()["output"] == "outerr" + + def test_timeout_is_a_408(self, api_v3_client, plugin_dir): + with patch.object(subprocess, "run", + side_effect=subprocess.TimeoutExpired("python3", 120)): + response = api_v3_client.post(self.URL, json={"redirect_url": "http://cb/?code=x"}) + assert response.status_code == 408 + assert "timed out" in response.get_json()["message"] + + def test_runs_a_list_argv_never_a_shell(self, api_v3_client, plugin_dir): + with patch.object(subprocess, "run", return_value=completed()) as run: + api_v3_client.post(self.URL, json={"redirect_url": "http://cb/?code=x"}) + args, kwargs = run.call_args + assert isinstance(args[0], list) + assert args[0][0] == "python3" + assert kwargs.get("shell") in (None, False) + + def test_timeout_is_bounded(self, api_v3_client, plugin_dir): + with patch.object(subprocess, "run", return_value=completed()) as run: + api_v3_client.post(self.URL, json={"redirect_url": "http://cb/?code=x"}) + assert run.call_args.kwargs["timeout"] == 120 + + +class TestSpotifyWrapperCleanup: + URL = "/api/v3/plugins/authenticate/spotify" + + def _wrapper_paths_after(self, api_v3_client, run_mock): + """Run the endpoint and return the wrapper path subprocess saw.""" + seen = {} + + def capture(args, **kwargs): + seen["path"] = args[1] + return run_mock(args, **kwargs) + + with patch.object(subprocess, "run", side_effect=capture): + api_v3_client.post(self.URL, json={"redirect_url": "http://cb/?code=x"}) + return seen["path"] + + def test_removed_after_success(self, api_v3_client, plugin_dir): + path = self._wrapper_paths_after(api_v3_client, lambda *a, **kw: completed()) + assert not os.path.exists(path) + + def test_removed_after_script_failure(self, api_v3_client, plugin_dir): + path = self._wrapper_paths_after( + api_v3_client, lambda *a, **kw: completed(1, "out", "err")) + assert not os.path.exists(path) + + def test_removed_after_timeout(self, api_v3_client, plugin_dir): + def raise_timeout(*a, **kw): + raise subprocess.TimeoutExpired("python3", 120) + path = self._wrapper_paths_after(api_v3_client, raise_timeout) + assert not os.path.exists(path) + + def test_removed_when_subprocess_cannot_start(self, api_v3_client, plugin_dir): + # Regression: cleanup lived in the success/failure branch and in the + # TimeoutExpired handler only. An OSError from subprocess.run itself + # — no interpreter, fork failure — skipped both and left the wrapper, + # which contains the user's redirect URL, on disk. + def raise_oserror(*a, **kw): + raise OSError("[Errno 12] Cannot allocate memory") + path = self._wrapper_paths_after(api_v3_client, raise_oserror) + assert not os.path.exists(path) + + +class TestSpotifyRedirectUrlIsNotInjectable: + """The wrapper embeds redirect_url into generated Python source.""" + + URL = "/api/v3/plugins/authenticate/spotify" + + ADVERSARIAL = [ + '''http://cb/?code=x"''', + """http://cb/?code=x'""", + 'http://cb/?code=x\\', + 'http://cb/?code=x\nimport os; os.system("id")', + 'http://cb/?code=x"""\nimport os\n"""', + "http://cb/?code=x'''", + 'http://cb/?code=x\\"\\n', + '"; import os; os.system("id"); "', + ] + + def _wrapper_source(self, api_v3_client, redirect_url): + captured = {} + + def capture(args, **kwargs): + captured["source"] = Path(args[1]).read_text() + return completed() + + with patch.object(subprocess, "run", side_effect=capture): + api_v3_client.post(self.URL, json={"redirect_url": redirect_url}) + return captured["source"] + + @pytest.mark.parametrize("redirect_url", ADVERSARIAL) + def test_wrapper_is_still_valid_python(self, api_v3_client, plugin_dir, redirect_url): + # If escaping failed, the generated file would not parse at all. + source = self._wrapper_source(api_v3_client, redirect_url) + ast.parse(source) + + @pytest.mark.parametrize("redirect_url", ADVERSARIAL) + def test_url_survives_as_one_string_literal( + self, api_v3_client, plugin_dir, redirect_url): + # Stronger than "it parses": the URL must still be a single string + # assigned to redirect_url, not code that escaped into statements. + source = self._wrapper_source(api_v3_client, redirect_url) + tree = ast.parse(source) + assigned = [ + node.value.value for node in ast.walk(tree) + if isinstance(node, ast.Assign) + and isinstance(node.value, ast.Constant) + and any(getattr(t, "id", None) == "redirect_url" for t in node.targets) + ] + assert assigned == [redirect_url.strip()] + + def test_injected_call_does_not_become_a_statement(self, api_v3_client, plugin_dir): + source = self._wrapper_source( + api_v3_client, 'http://cb/\nimport os; os.system("id")') + tree = ast.parse(source) + imported = { + alias.name for node in ast.walk(tree) + if isinstance(node, ast.Import) for alias in node.names + } + # The wrapper legitimately imports sys, subprocess and os; what it + # must not gain is a *call* smuggled in through the URL. + calls = [ + node for node in ast.walk(tree) + if isinstance(node, ast.Call) + and isinstance(node.func, ast.Attribute) + and node.func.attr == "system" + ] + assert calls == [] + + +class TestSpotifyStepOne: + """No redirect_url — the OAuth-URL path, which imports the script.""" + + URL = "/api/v3/plugins/authenticate/spotify" + + def test_script_without_credentials_helper_is_an_error( + self, api_v3_client, plugin_dir): + # The stub script defines neither get_auth_url nor + # load_spotify_credentials, so no URL can be produced. + response = api_v3_client.post(self.URL, json={}) + assert response.status_code in (400, 500) + assert response.get_json()["status"] == "error" + + def test_unusable_credentials_do_not_leak_into_the_response( + self, api_v3_client, plugin_dir): + (plugin_dir / "authenticate_spotify.py").write_text( + "def load_spotify_credentials():\n" + " return ('id-abc', 'super-secret-value', None)\n" + ) + response = api_v3_client.post(self.URL, json={}) + assert "super-secret-value" not in response.get_data(as_text=True) + + def test_script_raising_on_import_is_handled(self, api_v3_client, plugin_dir): + (plugin_dir / "authenticate_spotify.py").write_text("raise RuntimeError('boom')\n") + response = api_v3_client.post(self.URL, json={}) + assert response.status_code == 500 + assert response.get_json()["status"] == "error" + + def test_bodyless_post_reaches_step_one(self, api_v3_client, plugin_dir): + # Covered by the silent=True fix: previously a 500 from body parsing. + response = api_v3_client.post(self.URL) + assert response.status_code in (400, 500) + assert response.get_json()["status"] == "error" + + def test_whitespace_redirect_url_is_treated_as_absent( + self, api_v3_client, plugin_dir): + with patch.object(subprocess, "run", return_value=completed()) as run: + api_v3_client.post(self.URL, json={"redirect_url": " "}) + # Step 2 never runs, so no wrapper is executed. + run.assert_not_called() + + +class TestYouTubeMusic: + """No wrapper script and no redirect_url — deliberately not symmetric.""" + + URL = "/api/v3/plugins/authenticate/ytm" + + def test_missing_plugin_directory_is_404(self, api_v3_client, api_v3_module, tmp_path): + api_v3_module.api_v3.plugin_manager.get_plugin_directory.return_value = str( + tmp_path / "not-installed") + assert api_v3_client.post(self.URL).status_code == 404 + + def test_missing_script_is_404(self, api_v3_client, plugin_dir): + (plugin_dir / "authenticate_ytm.py").unlink() + response = api_v3_client.post(self.URL) + assert response.status_code == 404 + assert "script not found" in response.get_json()["message"] + + def test_success(self, api_v3_client, plugin_dir): + with patch.object(subprocess, "run", return_value=completed(0, "authorized")): + response = api_v3_client.post(self.URL) + assert response.status_code == 200 + assert response.get_json()["output"] == "authorized" + + def test_failure_is_a_400_with_combined_output(self, api_v3_client, plugin_dir): + with patch.object(subprocess, "run", return_value=completed(1, "out", "err")): + response = api_v3_client.post(self.URL) + assert response.status_code == 400 + assert response.get_json()["output"] == "outerr" + + def test_timeout_is_a_408(self, api_v3_client, plugin_dir): + with patch.object(subprocess, "run", + side_effect=subprocess.TimeoutExpired("python3", 60)): + assert api_v3_client.post(self.URL).status_code == 408 + + def test_runs_the_script_directly_without_a_shell(self, api_v3_client, plugin_dir): + with patch.object(subprocess, "run", return_value=completed()) as run: + api_v3_client.post(self.URL) + args, kwargs = run.call_args + assert args[0][0] == "python3" + assert args[0][1].endswith("authenticate_ytm.py") + assert kwargs.get("shell") in (None, False) + assert kwargs["timeout"] == 60 diff --git a/test/test_api_v3_optional_body.py b/test/test_api_v3_optional_body.py new file mode 100644 index 00000000..4b7eb133 --- /dev/null +++ b/test/test_api_v3_optional_body.py @@ -0,0 +1,136 @@ +""" +Regression tests: POST endpoints whose body is optional must accept a +request that has no body at all. + +Six handlers in api_v3 read their body as ``request.get_json() or {}``. +The ``or {}`` states the intent plainly — every field is optional, so a +bodyless POST should fall back to defaults. But ``get_json()`` without +``silent=True`` raises ``UnsupportedMediaType`` when the request carries +no JSON Content-Type, and it raises *before* ``or {}`` is evaluated. Each +handler's catch-all then turned that into a 500. + +So the natural way to call these endpoints — a POST with no body, which +is what curl, a fetch() without options, and most HTTP clients send by +default — failed on every one of them. The shipped UI always sends a JSON +object, which is why this went unnoticed. + +This file covers the endpoints whose bodyless behaviour is not already +tested in their own suite. +""" + +import re +import sys +from pathlib import Path +from unittest.mock import MagicMock + +import pytest + +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from test._api_v3_test_helpers import api_v3_client, api_v3_module # noqa: F401,E402 + + +class TestOnDemandStart: + URL = "/api/v3/display/on-demand/start" + + def test_bodyless_post_is_not_a_server_error(self, api_v3_client, api_v3_module): + response = api_v3_client.post(self.URL) + # The endpoint may still reject the request on its own terms (no + # plugin_id, nothing to display); what it must not do is fail with + # a 500 raised out of body parsing. + assert response.status_code != 500 + + def test_json_body_still_works(self, api_v3_client, api_v3_module): + assert api_v3_client.post(self.URL, json={}).status_code != 500 + + +class TestResetPluginConfig: + URL = "/api/v3/plugins/config/reset" + + def test_bodyless_post_is_not_a_server_error(self, api_v3_client, api_v3_module): + assert api_v3_client.post(self.URL).status_code != 500 + + def test_json_body_still_works(self, api_v3_client, api_v3_module): + assert api_v3_client.post(self.URL, json={}).status_code != 500 + + +class TestDeleteOfTheDayJson: + URL = "/api/v3/plugins/of-the-day/json/delete" + + def test_bodyless_post_is_not_a_server_error(self, api_v3_client, api_v3_module): + assert api_v3_client.post(self.URL).status_code != 500 + + def test_json_body_still_works(self, api_v3_client, api_v3_module): + assert api_v3_client.post(self.URL, json={}).status_code != 500 + + +class TestPluginLimits: + URL = "/api/v3/plugins/clock/limits" + + def test_bodyless_post_is_not_a_server_error(self, api_v3_client, api_v3_module): + assert api_v3_client.post(self.URL).status_code != 500 + + +class TestMissingBodyGivesTheDeclaredError: + """Handlers that answer "No data provided" must actually be able to. + + A second group of handlers reads `data = request.get_json()` and then + guards with `if not data: return 400`. That guard is unreachable for a + request with no JSON body, because get_json() raises first — so the + caller got a 500 "an error occurred; see logs for details" instead of + the 400 the handler plainly intends to send. + """ + + @pytest.mark.parametrize("url", [ + "/api/v3/plugins/install", + "/api/v3/plugins/install-from-url", + "/api/v3/plugins/registry-from-url", + "/api/v3/config/raw/main", + "/api/v3/config/raw/secrets", + "/api/v3/cache/delete", + ]) + def test_bodyless_post_gets_a_400_not_a_500(self, api_v3_client, api_v3_module, url): + response = api_v3_client.post(url) + assert response.status_code == 400, ( + f"{url} answered {response.status_code}: " + f"{response.get_data(as_text=True)[:200]}") + + @pytest.mark.parametrize("url", [ + "/api/v3/plugins/install", + "/api/v3/config/raw/main", + ]) + def test_malformed_json_gets_a_400_not_a_500(self, api_v3_client, api_v3_module, url): + response = api_v3_client.post( + url, data="{not json", content_type="application/json") + assert response.status_code == 400 + + +class TestNoBodyReadContradictsItsOwnGuard: + SOURCE = Path(__file__).parent.parent / "web_interface/blueprints/api_v3.py" + + def test_no_or_default_read_is_unguarded(self): + """`get_json() or ` is a contradiction without silent=True. + + Writing `or {}` declares the body optional; omitting silent=True + means the call raises before the default can apply. + """ + offenders = [ + line.strip() for line in self.SOURCE.read_text().splitlines() + if "request.get_json()" in line and " or " in line + ] + assert offenders == [], ( + "these reads declare a default but raise before reaching it; " + f"use get_json(silent=True): {offenders}") + + def test_no_not_data_guard_is_unreachable(self): + """A `if not data:` guard needs a read that can actually return None.""" + lines = self.SOURCE.read_text().splitlines() + offenders = [] + for i, line in enumerate(lines): + if re.search(r"=\s*request\.get_json\(\)\s*$", line): + window = "\n".join(lines[i + 1:i + 3]) + if re.search(r"if\s+(not\s+data\b|data\s+is\s+None)", window): + offenders.append(f"line {i + 1}: {line.strip()}") + assert offenders == [], ( + "these handlers guard on a missing body but raise before the " + f"guard runs; use get_json(silent=True): {offenders}") diff --git a/test/test_api_v3_plugin_install_endpoints.py b/test/test_api_v3_plugin_install_endpoints.py new file mode 100644 index 00000000..29f7f7dc --- /dev/null +++ b/test/test_api_v3_plugin_install_endpoints.py @@ -0,0 +1,302 @@ +""" +Endpoint tests for POST /plugins/install and POST /plugins/install-from-url. + +Both were only ever tested at the PluginStoreManager layer, so the route +logic — the queue-vs-direct branch, schema invalidation, plugin discovery, +state and history recording — was unexercised. + +/plugins/install carries the same install logic twice: once inside the +operation-queue callback and once in the direct fallback. The paired +tests below assert both branches produce the same side effects, so the +duplication cannot quietly drift. +""" + +import sys +from pathlib import Path +from unittest.mock import MagicMock + +import pytest + +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from test._api_v3_test_helpers import api_v3_client, api_v3_module # noqa: F401,E402 + +INSTALL = "/api/v3/plugins/install" +FROM_URL = "/api/v3/plugins/install-from-url" + + +@pytest.fixture +def queued(api_v3_module): + """Enable the operation queue and run its callback synchronously.""" + queue = MagicMock() + + def enqueue(operation_type, plugin_id, operation_callback=None): + queue.callback_result = operation_callback(MagicMock()) + return "op-123" + + queue.enqueue_operation.side_effect = enqueue + api_v3_module.api_v3.operation_queue = queue + return queue + + +def side_effects(module): + """The manager calls a successful install is expected to make.""" + api = module.api_v3 + return { + "schema_invalidated": api.schema_manager.invalidate_cache.call_args_list, + "discovered": api.plugin_manager.discover_plugins.call_count, + "loaded": api.plugin_manager.load_plugin.call_args_list, + "state_set": api.plugin_state_manager.set_plugin_installed.call_args_list, + "history": api.operation_history.record_operation.call_args_list, + } + + +class TestInstallValidation: + def test_uninitialized_store_manager_is_a_500(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager = None + response = api_v3_client.post(INSTALL, json={"plugin_id": "clock"}) + assert response.status_code == 500 + assert "not initialized" in response.get_json()["message"] + + def test_missing_plugin_id_is_a_400(self, api_v3_client, api_v3_module): + response = api_v3_client.post(INSTALL, json={}) + assert response.status_code == 400 + assert "plugin_id required" in response.get_json()["message"] + api_v3_module.api_v3.plugin_store_manager.install_plugin.assert_not_called() + + def test_empty_body_is_a_400(self, api_v3_client, api_v3_module): + assert api_v3_client.post(INSTALL, json=None).status_code == 400 + + +class TestInstallDirectPath: + """operation_queue is None — the fallback branch.""" + + def test_success(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.install_plugin.return_value = True + response = api_v3_client.post(INSTALL, json={"plugin_id": "clock"}) + assert response.status_code == 200 + assert response.get_json()["status"] == "success" + + def test_success_side_effects(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.install_plugin.return_value = True + api_v3_client.post(INSTALL, json={"plugin_id": "clock"}) + effects = side_effects(api_v3_module) + assert effects["schema_invalidated"] == [(("clock",), {})] + assert effects["discovered"] == 1 + assert effects["loaded"] == [(("clock",), {})] + assert effects["state_set"] == [(("clock",), {})] + assert effects["history"][0].kwargs["status"] == "success" + + def test_branch_forwarded_to_the_manager(self, api_v3_client, api_v3_module): + manager = api_v3_module.api_v3.plugin_store_manager + manager.install_plugin.return_value = True + api_v3_client.post(INSTALL, json={"plugin_id": "clock", "branch": "dev"}) + manager.install_plugin.assert_called_once_with("clock", branch="dev") + + def test_branch_named_in_the_message(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.install_plugin.return_value = True + response = api_v3_client.post(INSTALL, json={"plugin_id": "clock", "branch": "dev"}) + assert "(branch: dev)" in response.get_json()["message"] + + def test_failure_is_a_500(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.install_plugin.return_value = False + response = api_v3_client.post(INSTALL, json={"plugin_id": "clock"}) + assert response.status_code == 500 + assert "Failed to install" in response.get_json()["message"] + + def test_failure_mentions_missing_registry_entry(self, api_v3_client, api_v3_module): + manager = api_v3_module.api_v3.plugin_store_manager + manager.install_plugin.return_value = False + manager.get_plugin_info.return_value = None + response = api_v3_client.post(INSTALL, json={"plugin_id": "ghost"}) + assert "not found in registry" in response.get_json()["message"] + + def test_failure_omits_registry_note_when_plugin_is_known( + self, api_v3_client, api_v3_module): + manager = api_v3_module.api_v3.plugin_store_manager + manager.install_plugin.return_value = False + manager.get_plugin_info.return_value = {"id": "clock"} + response = api_v3_client.post(INSTALL, json={"plugin_id": "clock"}) + assert "not found in registry" not in response.get_json()["message"] + + def test_failure_recorded_in_history(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.install_plugin.return_value = False + api_v3_client.post(INSTALL, json={"plugin_id": "clock"}) + record = api_v3_module.api_v3.operation_history.record_operation.call_args + assert record.kwargs["status"] == "failed" + + def test_no_side_effects_on_failure(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.install_plugin.return_value = False + api_v3_client.post(INSTALL, json={"plugin_id": "clock"}) + effects = side_effects(api_v3_module) + assert effects["schema_invalidated"] == [] + assert effects["loaded"] == [] + assert effects["state_set"] == [] + + +class TestInstallQueuedPath: + """operation_queue present — the callback branch.""" + + def test_returns_an_operation_id(self, api_v3_client, api_v3_module, queued): + api_v3_module.api_v3.plugin_store_manager.install_plugin.return_value = True + response = api_v3_client.post(INSTALL, json={"plugin_id": "clock"}) + assert response.status_code == 200 + assert response.get_json()["data"]["operation_id"] == "op-123" + + def test_message_says_queued(self, api_v3_client, api_v3_module, queued): + api_v3_module.api_v3.plugin_store_manager.install_plugin.return_value = True + response = api_v3_client.post(INSTALL, json={"plugin_id": "clock"}) + assert "queued" in response.get_json()["message"] + + def test_callback_success_side_effects(self, api_v3_client, api_v3_module, queued): + api_v3_module.api_v3.plugin_store_manager.install_plugin.return_value = True + api_v3_client.post(INSTALL, json={"plugin_id": "clock"}) + effects = side_effects(api_v3_module) + assert effects["schema_invalidated"] == [(("clock",), {})] + assert effects["discovered"] == 1 + assert effects["loaded"] == [(("clock",), {})] + assert effects["state_set"] == [(("clock",), {})] + assert effects["history"][0].kwargs["status"] == "success" + + def test_callback_reports_success(self, api_v3_client, api_v3_module, queued): + api_v3_module.api_v3.plugin_store_manager.install_plugin.return_value = True + api_v3_client.post(INSTALL, json={"plugin_id": "clock"}) + assert queued.callback_result["success"] is True + + def test_callback_failure_raises_for_the_queue(self, api_v3_client, api_v3_module, queued): + api_v3_module.api_v3.plugin_store_manager.install_plugin.return_value = False + # The callback signals failure by raising, so the queue can mark the + # operation failed; the route's catch-all turns it into a 500. + response = api_v3_client.post(INSTALL, json={"plugin_id": "clock"}) + assert response.status_code == 500 + + def test_callback_failure_recorded_in_history(self, api_v3_client, api_v3_module, queued): + api_v3_module.api_v3.plugin_store_manager.install_plugin.return_value = False + api_v3_client.post(INSTALL, json={"plugin_id": "clock"}) + record = api_v3_module.api_v3.operation_history.record_operation.call_args + assert record.kwargs["status"] == "failed" + + def test_branch_forwarded_from_the_callback(self, api_v3_client, api_v3_module, queued): + manager = api_v3_module.api_v3.plugin_store_manager + manager.install_plugin.return_value = True + api_v3_client.post(INSTALL, json={"plugin_id": "clock", "branch": "dev"}) + manager.install_plugin.assert_called_once_with("clock", branch="dev") + + +class TestInstallPathsAgree: + """The queue callback and the direct fallback duplicate the same logic.""" + + def _run(self, client, module, install_ok, queue): + module.api_v3.plugin_store_manager.install_plugin.return_value = install_ok + client.post(INSTALL, json={"plugin_id": "clock", "branch": "dev"}) + return side_effects(module) + + def test_success_side_effects_match(self, api_v3_client, api_v3_module): + direct = self._run(api_v3_client, api_v3_module, True, None) + + # Reset and re-run through the queue. + for mock in (api_v3_module.api_v3.schema_manager, + api_v3_module.api_v3.plugin_manager, + api_v3_module.api_v3.plugin_state_manager, + api_v3_module.api_v3.operation_history): + mock.reset_mock() + queue = MagicMock() + queue.enqueue_operation.side_effect = ( + lambda t, p, operation_callback=None: operation_callback(MagicMock()) and "op") + api_v3_module.api_v3.operation_queue = queue + queued = self._run(api_v3_client, api_v3_module, True, queue) + + assert direct["schema_invalidated"] == queued["schema_invalidated"] + assert direct["discovered"] == queued["discovered"] + assert direct["loaded"] == queued["loaded"] + assert direct["state_set"] == queued["state_set"] + assert (direct["history"][0].kwargs["status"] + == queued["history"][0].kwargs["status"]) + assert (direct["history"][0].kwargs["details"] + == queued["history"][0].kwargs["details"]) + + def test_only_the_message_wording_differs(self, api_v3_client, api_v3_module): + # Characterized: the direct path says "Plugin installed + # successfully" while the queue callback says "Plugin clock + # installed successfully". Cosmetic, and the queue's text is + # internal to the operation record rather than the HTTP response. + api_v3_module.api_v3.plugin_store_manager.install_plugin.return_value = True + direct = api_v3_client.post(INSTALL, json={"plugin_id": "clock"}).get_json() + assert direct["message"] == "Plugin installed successfully" + + +class TestInstallFromUrl: + def test_uninitialized_store_manager_is_a_500(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager = None + assert api_v3_client.post(FROM_URL, json={"repo_url": "http://x"}).status_code == 500 + + def test_missing_repo_url_is_a_400(self, api_v3_client, api_v3_module): + response = api_v3_client.post(FROM_URL, json={}) + assert response.status_code == 400 + assert "repo_url required" in response.get_json()["message"] + + def test_success(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.install_from_url.return_value = { + "success": True, "plugin_id": "clock", "name": "Clock"} + response = api_v3_client.post(FROM_URL, json={"repo_url": "https://github.com/o/r"}) + assert response.status_code == 200 + body = response.get_json() + assert body["plugin_id"] == "clock" + assert body["name"] == "Clock" + + def test_all_optional_arguments_forwarded(self, api_v3_client, api_v3_module): + manager = api_v3_module.api_v3.plugin_store_manager + manager.install_from_url.return_value = {"success": True, "plugin_id": "clock"} + api_v3_client.post(FROM_URL, json={ + "repo_url": " https://github.com/o/r ", + "plugin_id": "clock", + "plugin_path": "plugins/clock", + "branch": "dev", + }) + manager.install_from_url.assert_called_once_with( + repo_url="https://github.com/o/r", + plugin_id="clock", + plugin_path="plugins/clock", + branch="dev", + ) + + def test_success_invalidates_schema_and_loads_plugin(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.install_from_url.return_value = { + "success": True, "plugin_id": "clock"} + api_v3_client.post(FROM_URL, json={"repo_url": "http://x"}) + api_v3_module.api_v3.schema_manager.invalidate_cache.assert_called_once_with("clock") + api_v3_module.api_v3.plugin_manager.load_plugin.assert_called_once_with("clock") + + def test_success_without_plugin_id_skips_discovery(self, api_v3_client, api_v3_module): + # install_from_url can succeed without naming the plugin; there is + # then nothing to invalidate or load. + api_v3_module.api_v3.plugin_store_manager.install_from_url.return_value = { + "success": True, "plugin_id": None} + api_v3_client.post(FROM_URL, json={"repo_url": "http://x"}) + api_v3_module.api_v3.schema_manager.invalidate_cache.assert_not_called() + api_v3_module.api_v3.plugin_manager.load_plugin.assert_not_called() + + def test_branch_from_result_included(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.install_from_url.return_value = { + "success": True, "plugin_id": "clock", "branch": "dev"} + body = api_v3_client.post(FROM_URL, json={"repo_url": "http://x"}).get_json() + assert body["branch"] == "dev" + assert "(branch: dev)" in body["message"] + + def test_failure_reports_the_managers_error(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.install_from_url.return_value = { + "success": False, "error": "repo not found"} + response = api_v3_client.post(FROM_URL, json={"repo_url": "http://x"}) + assert response.status_code == 500 + assert response.get_json()["message"] == "repo not found" + + def test_failure_without_error_uses_fallback_text(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.install_from_url.return_value = { + "success": False} + response = api_v3_client.post(FROM_URL, json={"repo_url": "http://x"}) + assert "Failed to install plugin from URL" in response.get_json()["message"] + + def test_manager_exception_is_a_500(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.install_from_url.side_effect = ( + RuntimeError("boom")) + assert api_v3_client.post(FROM_URL, json={"repo_url": "http://x"}).status_code == 500 diff --git a/test/test_api_v3_registry_endpoints.py b/test/test_api_v3_registry_endpoints.py new file mode 100644 index 00000000..85f4b377 --- /dev/null +++ b/test/test_api_v3_registry_endpoints.py @@ -0,0 +1,179 @@ +""" +Endpoint tests for the plugin-registry routes in api_v3: +POST /plugins/store/refresh and POST /plugins/registry-from-url. + +Both reach out to the network through PluginStoreManager (mocked here) and +had no endpoint-level coverage; registry-from-url in particular takes a +user-supplied URL and hands it straight to the manager. +""" + +import sys +from pathlib import Path + +import pytest + +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from test._api_v3_test_helpers import api_v3_client, api_v3_module # noqa: F401,E402 + + +class TestRefreshPluginStore: + URL = "/api/v3/plugins/store/refresh" + + def test_uninitialized_manager_is_a_500(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager = None + response = api_v3_client.post(self.URL, json={}) + assert response.status_code == 500 + assert "not initialized" in response.get_json()["message"] + + def test_success_reports_plugin_count(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.fetch_registry.return_value = { + "plugins": [{"id": "a"}, {"id": "b"}, {"id": "c"}]} + response = api_v3_client.post(self.URL, json={}) + assert response.status_code == 200 + assert response.get_json()["plugin_count"] == 3 + + def test_forces_a_refresh_rather_than_using_cache(self, api_v3_client, api_v3_module): + manager = api_v3_module.api_v3.plugin_store_manager + manager.fetch_registry.return_value = {"plugins": []} + api_v3_client.post(self.URL, json={}) + manager.fetch_registry.assert_called_once_with(force_refresh=True) + + def test_empty_registry_reports_zero(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.fetch_registry.return_value = {} + response = api_v3_client.post(self.URL, json={}) + assert response.get_json()["plugin_count"] == 0 + + def test_no_body_is_accepted(self, api_v3_client, api_v3_module): + # Regression: `request.get_json() or {}` says a missing body is + # fine, but get_json() raises UnsupportedMediaType before `or {}` + # is reached, so a bodyless POST — the natural way to call a + # refresh endpoint — came back 500. + api_v3_module.api_v3.plugin_store_manager.fetch_registry.return_value = {"plugins": []} + assert api_v3_client.post(self.URL).status_code == 200 + + def test_body_without_json_content_type_is_accepted( + self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.fetch_registry.return_value = {"plugins": []} + response = api_v3_client.post(self.URL, data="", content_type="text/plain") + assert response.status_code == 200 + + def test_malformed_json_body_falls_back_to_defaults( + self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.fetch_registry.return_value = {"plugins": []} + response = api_v3_client.post( + self.URL, data="{not json", content_type="application/json") + assert response.status_code == 200 + + @pytest.mark.parametrize("key", ["fetch_commit_info", "fetch_latest_versions"]) + def test_either_commit_info_key_extends_the_message( + self, api_v3_client, api_v3_module, key): + # fetch_latest_versions is the older spelling; both must work. + api_v3_module.api_v3.plugin_store_manager.fetch_registry.return_value = {"plugins": []} + response = api_v3_client.post(self.URL, json={key: True}) + assert "commit metadata" in response.get_json()["message"] + + def test_message_stays_plain_without_the_flag(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.fetch_registry.return_value = {"plugins": []} + response = api_v3_client.post(self.URL, json={}) + assert response.get_json()["message"] == "Plugin store refreshed" + + def test_network_failure_is_a_500(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.fetch_registry.side_effect = ( + ConnectionError("github unreachable")) + response = api_v3_client.post(self.URL, json={}) + assert response.status_code == 500 + assert response.get_json()["message"] == "An error occurred; see logs for details" + + def test_failure_body_carries_no_traceback_or_paths( + self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.fetch_registry.side_effect = ( + RuntimeError("failed at /home/user/LEDMatrix/src/secret.py line 42")) + body = api_v3_client.post(self.URL, json={}).get_json() + assert "Traceback" not in str(body) + # `details` is describe_exception output: one line, type-named, + # credential-redacted. It may quote the message, but never a stack. + assert body["details"].startswith("RuntimeError:") + assert "\n" not in body["details"] + + +class TestRegistryFromUrl: + URL = "/api/v3/plugins/registry-from-url" + + def test_uninitialized_manager_is_a_500(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager = None + response = api_v3_client.post(self.URL, json={"repo_url": "http://x"}) + assert response.status_code == 500 + + def test_missing_repo_url_is_a_400(self, api_v3_client, api_v3_module): + response = api_v3_client.post(self.URL, json={}) + assert response.status_code == 400 + assert "repo_url required" in response.get_json()["message"] + api_v3_module.api_v3.plugin_store_manager.fetch_registry_from_url.assert_not_called() + + def test_success_returns_the_plugin_list(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.fetch_registry_from_url.return_value = { + "plugins": [{"id": "clock"}]} + response = api_v3_client.post( + self.URL, json={"repo_url": "https://github.com/o/r"}) + assert response.status_code == 200 + body = response.get_json() + assert body["plugins"] == [{"id": "clock"}] + assert body["registry_url"] == "https://github.com/o/r" + + def test_url_is_trimmed_before_use(self, api_v3_client, api_v3_module): + manager = api_v3_module.api_v3.plugin_store_manager + manager.fetch_registry_from_url.return_value = {"plugins": []} + api_v3_client.post(self.URL, json={"repo_url": " https://github.com/o/r "}) + manager.fetch_registry_from_url.assert_called_once_with("https://github.com/o/r") + + def test_registry_without_plugins_key_returns_empty_list( + self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.fetch_registry_from_url.return_value = { + "other": 1} + response = api_v3_client.post(self.URL, json={"repo_url": "http://x"}) + assert response.get_json()["plugins"] == [] + + def test_no_registry_found_is_a_400(self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.fetch_registry_from_url.return_value = None + response = api_v3_client.post(self.URL, json={"repo_url": "http://x/not-a-registry"}) + assert response.status_code == 400 + assert "Failed to fetch registry" in response.get_json()["message"] + + @pytest.mark.parametrize("url", [ + "not a url", + "javascript:alert(1)", + "file:///etc/passwd", + "http://localhost:8080/admin", + ]) + def test_unusable_urls_fail_cleanly(self, api_v3_client, api_v3_module, url): + # Characterization: the handler performs no URL validation of its + # own — whatever the manager makes of the URL decides the outcome. + # What is pinned here is that a rejected URL produces a clean 400 + # rather than a traceback or a 500. + api_v3_module.api_v3.plugin_store_manager.fetch_registry_from_url.return_value = None + response = api_v3_client.post(self.URL, json={"repo_url": url}) + assert response.status_code == 400 + assert "Traceback" not in str(response.get_json()) + + def test_fetch_exception_is_a_500_without_internals( + self, api_v3_client, api_v3_module): + api_v3_module.api_v3.plugin_store_manager.fetch_registry_from_url.side_effect = ( + ValueError("parse failed in /srv/app/internal.py")) + response = api_v3_client.post(self.URL, json={"repo_url": "http://x"}) + assert response.status_code == 500 + body = response.get_json() + assert body["message"] == "An error occurred; see logs for details" + assert "Traceback" not in str(body) + + def test_non_string_repo_url_is_rejected(self, api_v3_client, api_v3_module): + # Regression: .strip() on a non-string raised, and the catch-all + # reported the caller's own mistake as a server fault. + response = api_v3_client.post(self.URL, json={"repo_url": 12345}) + assert response.status_code == 400 + api_v3_module.api_v3.plugin_store_manager.fetch_registry_from_url.assert_not_called() + + def test_blank_repo_url_is_rejected(self, api_v3_client, api_v3_module): + response = api_v3_client.post(self.URL, json={"repo_url": " "}) + assert response.status_code == 400 + api_v3_module.api_v3.plugin_store_manager.fetch_registry_from_url.assert_not_called() diff --git a/test/test_api_v3_wifi_endpoints.py b/test/test_api_v3_wifi_endpoints.py new file mode 100644 index 00000000..0c713ae2 --- /dev/null +++ b/test/test_api_v3_wifi_endpoints.py @@ -0,0 +1,240 @@ +""" +Endpoint tests for the /wifi/* routes in api_v3. + +These routes drive the host's actual networking — connecting, dropping a +connection, switching the radio off — and had no endpoint-level tests at +all. WiFiManager is mocked throughout; nothing here may touch real +networking. + +Each handler does `from src.wifi_manager import WiFiManager` inside the +function body, so the patch target is the class at its definition site. +""" + +import sys +from pathlib import Path +from unittest.mock import MagicMock, patch + +import pytest + +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from test._api_v3_test_helpers import api_v3_client, api_v3_module # noqa: F401,E402 + + +@pytest.fixture +def wifi_manager(): + """Patch WiFiManager where it is defined; yield the instance mock.""" + with patch("src.wifi_manager.WiFiManager") as cls: + instance = MagicMock() + cls.return_value = instance + yield instance + + +class TestConnect: + URL = "/api/v3/wifi/connect" + + def test_success(self, api_v3_client, wifi_manager): + wifi_manager.connect_to_network.return_value = (True, "Connected to HomeNet") + response = api_v3_client.post(self.URL, json={"ssid": "HomeNet", "password": "pw"}) + assert response.status_code == 200 + assert response.get_json()["message"] == "Connected to HomeNet" + wifi_manager.connect_to_network.assert_called_once_with("HomeNet", "pw") + + def test_missing_body_rejected(self, api_v3_client, wifi_manager): + response = api_v3_client.post(self.URL, json={}) + assert response.status_code == 400 + wifi_manager.connect_to_network.assert_not_called() + + def test_missing_ssid_rejected(self, api_v3_client, wifi_manager): + response = api_v3_client.post(self.URL, json={"password": "pw"}) + assert response.status_code == 400 + assert "SSID is required" in response.get_json()["message"] + wifi_manager.connect_to_network.assert_not_called() + + @pytest.mark.parametrize("ssid", ["", " ", "\t"]) + def test_blank_ssid_rejected(self, api_v3_client, wifi_manager, ssid): + response = api_v3_client.post(self.URL, json={"ssid": ssid}) + assert response.status_code == 400 + wifi_manager.connect_to_network.assert_not_called() + + def test_ssid_is_trimmed(self, api_v3_client, wifi_manager): + wifi_manager.connect_to_network.return_value = (True, "ok") + api_v3_client.post(self.URL, json={"ssid": " HomeNet "}) + wifi_manager.connect_to_network.assert_called_once_with("HomeNet", "") + + def test_missing_password_becomes_empty_string(self, api_v3_client, wifi_manager): + wifi_manager.connect_to_network.return_value = (True, "ok") + api_v3_client.post(self.URL, json={"ssid": "OpenNet"}) + wifi_manager.connect_to_network.assert_called_once_with("OpenNet", "") + + def test_null_password_becomes_empty_string(self, api_v3_client, wifi_manager): + wifi_manager.connect_to_network.return_value = (True, "ok") + api_v3_client.post(self.URL, json={"ssid": "OpenNet", "password": None}) + wifi_manager.connect_to_network.assert_called_once_with("OpenNet", "") + + def test_failure_reports_the_managers_reason(self, api_v3_client, wifi_manager): + wifi_manager.connect_to_network.return_value = (False, "Bad password") + response = api_v3_client.post(self.URL, json={"ssid": "HomeNet"}) + assert response.status_code == 400 + assert response.get_json()["message"] == "Bad password" + + def test_failure_without_reason_uses_fallback_text(self, api_v3_client, wifi_manager): + wifi_manager.connect_to_network.return_value = (False, None) + response = api_v3_client.post(self.URL, json={"ssid": "HomeNet"}) + assert response.status_code == 400 + assert response.get_json()["message"] == "Failed to connect to network" + + def test_manager_exception_is_a_500_without_leaking_internals( + self, api_v3_client, wifi_manager): + wifi_manager.connect_to_network.side_effect = RuntimeError( + "/usr/lib/secret/path blew up") + response = api_v3_client.post(self.URL, json={"ssid": "HomeNet"}) + assert response.status_code == 500 + body = response.get_json() + assert body["message"] == "An error occurred; see logs for details" + # `details` comes from describe_exception, which is deliberately + # safe to return (redacted, capped) — it names the type. + assert "RuntimeError" in body["details"] + + +class TestDisconnect: + URL = "/api/v3/wifi/disconnect" + + def test_success(self, api_v3_client, wifi_manager): + wifi_manager.disconnect_from_network.return_value = (True, "Disconnected") + response = api_v3_client.post(self.URL) + assert response.status_code == 200 + assert response.get_json()["message"] == "Disconnected" + + def test_failure(self, api_v3_client, wifi_manager): + wifi_manager.disconnect_from_network.return_value = (False, "Not connected") + response = api_v3_client.post(self.URL) + assert response.status_code == 400 + assert response.get_json()["message"] == "Not connected" + + def test_failure_without_reason_uses_fallback(self, api_v3_client, wifi_manager): + wifi_manager.disconnect_from_network.return_value = (False, "") + response = api_v3_client.post(self.URL) + assert response.get_json()["message"] == "Failed to disconnect from network" + + def test_exception_is_a_500(self, api_v3_client, wifi_manager): + wifi_manager.disconnect_from_network.side_effect = OSError("nmcli missing") + assert api_v3_client.post(self.URL).status_code == 500 + + +class TestApMode: + ENABLE = "/api/v3/wifi/ap/enable" + DISABLE = "/api/v3/wifi/ap/disable" + + def test_enable_success(self, api_v3_client, wifi_manager): + wifi_manager.enable_ap_mode.return_value = (True, "AP enabled") + response = api_v3_client.post(self.ENABLE, json={}) + assert response.status_code == 200 + wifi_manager.enable_ap_mode.assert_called_once_with(force=False) + + @pytest.mark.parametrize("raw,expected", [ + (True, True), (False, False), + ("true", True), ("TRUE", True), ("1", True), + ("false", False), ("no", False), ("yes", False), + (1, False), # only real True or the listed strings count + ]) + def test_force_coercion(self, api_v3_client, wifi_manager, raw, expected): + wifi_manager.enable_ap_mode.return_value = (True, "ok") + api_v3_client.post(self.ENABLE, json={"force": raw}) + wifi_manager.enable_ap_mode.assert_called_once_with(force=expected) + + def test_enable_without_body(self, api_v3_client, wifi_manager): + wifi_manager.enable_ap_mode.return_value = (True, "ok") + assert api_v3_client.post(self.ENABLE).status_code == 200 + + def test_enable_failure(self, api_v3_client, wifi_manager): + wifi_manager.enable_ap_mode.return_value = (False, "hostapd missing") + response = api_v3_client.post(self.ENABLE, json={}) + assert response.status_code == 400 + assert response.get_json()["message"] == "hostapd missing" + + def test_disable_success(self, api_v3_client, wifi_manager): + wifi_manager.disable_ap_mode.return_value = (True, "AP disabled") + assert api_v3_client.post(self.DISABLE).status_code == 200 + + def test_disable_failure(self, api_v3_client, wifi_manager): + wifi_manager.disable_ap_mode.return_value = (False, "not running") + assert api_v3_client.post(self.DISABLE).status_code == 400 + + def test_enable_exception_is_a_500(self, api_v3_client, wifi_manager): + wifi_manager.enable_ap_mode.side_effect = RuntimeError("boom") + assert api_v3_client.post(self.ENABLE, json={}).status_code == 500 + + +class TestRadio: + URL = "/api/v3/wifi/radio" + + def test_get_state(self, api_v3_client, wifi_manager): + wifi_manager.get_wifi_radio_state.return_value = { + "enabled": True, "ethernet_connected": False} + response = api_v3_client.get(self.URL) + assert response.status_code == 200 + assert response.get_json()["data"]["enabled"] is True + + def test_get_state_exception_is_a_500(self, api_v3_client, wifi_manager): + wifi_manager.get_wifi_radio_state.side_effect = OSError("rfkill missing") + assert api_v3_client.get(self.URL).status_code == 500 + + def test_enabled_is_required(self, api_v3_client, wifi_manager): + response = api_v3_client.post(self.URL, json={}) + assert response.status_code == 400 + assert "enabled is required" in response.get_json()["message"] + wifi_manager.set_wifi_radio.assert_not_called() + + def test_enable_success(self, api_v3_client, wifi_manager): + wifi_manager.set_wifi_radio.return_value = (True, "Radio on", None) + wifi_manager.get_wifi_radio_state.return_value = {"enabled": True} + response = api_v3_client.post(self.URL, json={"enabled": True}) + assert response.status_code == 200 + wifi_manager.set_wifi_radio.assert_called_once_with(True, force=False) + + @pytest.mark.parametrize("raw,expected", [ + (True, True), ("true", True), ("1", True), ("yes", True), + (False, False), ("false", False), ("off", False), (0, False), + ]) + def test_enabled_coercion_is_string_aware( + self, api_v3_client, wifi_manager, raw, expected): + # bool("false") is True, so the endpoint parses strings explicitly + # rather than trusting truthiness — it is a public contract, not + # only the shipped UI which always sends real JSON booleans. + wifi_manager.set_wifi_radio.return_value = (True, "ok", None) + wifi_manager.get_wifi_radio_state.return_value = {} + api_v3_client.post(self.URL, json={"enabled": raw}) + wifi_manager.set_wifi_radio.assert_called_once_with(expected, force=False) + + def test_force_passed_through(self, api_v3_client, wifi_manager): + wifi_manager.set_wifi_radio.return_value = (True, "ok", None) + wifi_manager.get_wifi_radio_state.return_value = {} + api_v3_client.post(self.URL, json={"enabled": False, "force": "true"}) + wifi_manager.set_wifi_radio.assert_called_once_with(False, force=True) + + def test_refusal_reports_reason(self, api_v3_client, wifi_manager): + # Disabling the radio without Ethernet would lock the user out of + # this very interface, so the manager can refuse with a reason. + wifi_manager.set_wifi_radio.return_value = ( + False, "Refusing: no wired fallback", "no_ethernet") + response = api_v3_client.post(self.URL, json={"enabled": False}) + assert response.status_code == 400 + body = response.get_json() + assert body["reason"] == "no_ethernet" + assert "Refusing" in body["message"] + + def test_exception_is_a_500(self, api_v3_client, wifi_manager): + wifi_manager.set_wifi_radio.side_effect = RuntimeError("boom") + assert api_v3_client.post(self.URL, json={"enabled": True}).status_code == 500 + + +class TestNoRealNetworking: + def test_wifi_manager_is_never_constructed_for_real(self, api_v3_client): + # Guard against a future refactor moving the import to module level, + # where the fixture's patch of the definition site would stop + # applying and the tests would start driving real networking. + with patch("src.wifi_manager.WiFiManager") as cls: + cls.return_value.disconnect_from_network.return_value = (True, "ok") + api_v3_client.post("/api/v3/wifi/disconnect") + assert cls.called diff --git a/test/test_backup_manager.py b/test/test_backup_manager.py index fef10d82..e6533d69 100644 --- a/test/test_backup_manager.py +++ b/test/test_backup_manager.py @@ -3,6 +3,7 @@ from __future__ import annotations import json +import stat import zipfile from pathlib import Path @@ -41,6 +42,13 @@ def _make_project(root: Path) -> Path: json.dumps({"ap_mode": {"ssid": "LEDMatrix"}}), encoding="utf-8", ) + # Device-local auth that lives in config/ like the three above. It was + # omitted from backups, so a restore silently signed the user out of + # YouTube Music and they had to re-authenticate by hand. + (root / "config" / "ytm_auth.json").write_text( + json.dumps({"token": "YTM-TOKEN"}), + encoding="utf-8", + ) fonts = root / "assets" / "fonts" fonts.mkdir(parents=True) @@ -240,6 +248,10 @@ def test_restore_roundtrip(project: Path, empty_project: Path, tmp_path: Path) - restored_secrets = json.loads((empty_project / "config" / "config_secrets.json").read_text()) assert restored_secrets["ledmatrix-weather"]["api_key"] == "SECRET" + assert "ytm_auth" in result.restored + restored_ytm = json.loads((empty_project / "config" / "ytm_auth.json").read_text()) + assert restored_ytm["token"] == "YTM-TOKEN" + # User font restored, bundled font untouched. assert (empty_project / "assets" / "fonts" / "my-custom-font.ttf").read_bytes() == b"\x00\x01USER" assert (empty_project / "assets" / "fonts" / "5x7.bdf").read_text() == "BUNDLED" @@ -271,6 +283,10 @@ def test_restore_honors_options(project: Path, empty_project: Path, tmp_path: Pa assert result.plugins_to_install == [] assert "secrets" in result.skipped assert "wifi" in result.skipped + # ytm_auth rides on restore_wifi rather than its own flag -- disabling + # wifi restore must not leave a stale session token behind. + assert "ytm_auth" in result.skipped + assert not (empty_project / "config" / "ytm_auth.json").exists() def test_restore_rejects_malicious_zip(empty_project: Path, tmp_path: Path) -> None: @@ -282,3 +298,39 @@ def test_restore_rejects_malicious_zip(empty_project: Path, tmp_path: Path) -> N # validate_backup catches it before extraction. assert not result.success assert any("unsafe" in e.lower() for e in result.errors) + + +def test_restore_over_a_file_the_user_cannot_write( + project: Path, empty_project: Path, tmp_path: Path +) -> None: + """Restore must not need write permission on the destination *file*. + + Reproduces what a fresh install leaves behind: config files owned by root + and only group-readable, while the web interface that performs the restore + runs as a non-root user. shutil.copy2 opens the destination for writing and + failed with EACCES; writing alongside and renaming needs only directory + permission, which that account has. + + Simulated here by making the destination read-only — the owner cannot + open it for writing either, but can still replace it within its directory. + """ + zip_path = create_backup(project, output_dir=tmp_path / "exports") + + # Pre-existing, read-only destinations. + (empty_project / "config").mkdir(parents=True, exist_ok=True) + for name in ("config.json", "config_secrets.json", "wifi_config.json", "ytm_auth.json"): + target = empty_project / "config" / name + target.write_text("{}", encoding="utf-8") + target.chmod(0o444) + + result = restore_backup(zip_path, empty_project, RestoreOptions()) + + assert result.success, result.errors + for section in ("config", "secrets", "wifi", "ytm_auth"): + assert section in result.restored, f"{section} not restored: {result.errors}" + + restored = json.loads((empty_project / "config" / "config.json").read_text()) + assert restored["my-plugin"]["favorites"] == ["A", "B"] + + # The destination's mode is preserved rather than widened to the umask. + assert stat.S_IMODE((empty_project / "config" / "config_secrets.json").stat().st_mode) == 0o444 diff --git a/test/test_base_odds_manager.py b/test/test_base_odds_manager.py new file mode 100644 index 00000000..e53a5036 --- /dev/null +++ b/test/test_base_odds_manager.py @@ -0,0 +1,365 @@ +""" +Tests for src/base_odds_manager.py (BaseOddsManager). + +Covers get_odds validation/caching/URL construction, the null-safe +_extract_espn_data fix (ESPN sends explicit JSON nulls for absent sides), +the no_odds sentinel, stale-cache fallback on request failure, +is_odds_available's ML-blind truth table, the fixed format_odds_summary +gate (money-line-only odds now format), get_odds_for_games, and +configuration loading. + +No real network: requests.Session.get is always patched. The odds path sends +its requests through a session so it can identify itself to ESPN, so patching +the module-level requests.get would no longer intercept anything. +""" + +from unittest.mock import MagicMock, patch + +import pytest +import requests + +from src.base_odds_manager import BaseOddsManager + + +FULL_ITEM = { + 'details': 'DAL -3.5', + 'overUnder': 47.5, + 'spread': -3.5, + 'homeTeamOdds': {'moneyLine': -150, 'current': {'pointSpread': {'value': -3.5}}}, + 'awayTeamOdds': {'moneyLine': 130, 'current': {'pointSpread': {'value': 3.5}}}, +} + +FULL_EXTRACTED = { + 'details': 'DAL -3.5', + 'over_under': 47.5, + 'spread': -3.5, + 'home_team_odds': {'money_line': -150, 'spread_odds': -3.5}, + 'away_team_odds': {'money_line': 130, 'spread_odds': 3.5}, +} + + +def _make_response(payload): + response = MagicMock() + response.json.return_value = payload + response.raise_for_status.return_value = None + return response + + +@pytest.fixture +def cache_manager(): + cm = MagicMock() + # A bare MagicMock returns truthy Mocks from every call, so every + # get_odds() would look like a cache hit. Explicitly wire a miss. + cm.get_with_auto_strategy.return_value = None + return cm + + +@pytest.fixture +def manager(cache_manager): + return BaseOddsManager(cache_manager) + + +@pytest.fixture +def mock_get(): + with patch('src.base_odds_manager.requests.Session.get') as m: + m.return_value = _make_response({'items': [dict(FULL_ITEM)]}) + yield m + + +# --------------------------------------------------------------------------- +# get_odds +# --------------------------------------------------------------------------- + +class TestGetOdds: + def test_none_sport_raises(self, manager): + with pytest.raises(ValueError): + manager.get_odds(None, 'nfl', '1') + + def test_none_league_raises(self, manager): + with pytest.raises(ValueError): + manager.get_odds('football', None, '1') + + def test_cache_key_and_url(self, manager, cache_manager, mock_get): + manager.get_odds('football', 'nfl', '401') + + cache_manager.get_with_auto_strategy.assert_called_once_with( + 'odds_espn_football_nfl_401') + url = mock_get.call_args[0][0] + # Event id appears twice: /events//competitions//odds + assert '/events/401/competitions/401/odds' in url + assert url == ('https://sports.core.api.espn.com/v2/sports/football/' + 'leagues/nfl/events/401/competitions/401/odds') + # The number matters less than the property: a single stalled request + # must not be able to consume the plugin executor's 30s operation + # budget, since odds are fetched per live game inside update(). + assert mock_get.call_args.kwargs['timeout'] == 5 + assert mock_get.call_args.kwargs['timeout'] < 30 + + def test_ncaa_fb_maps_to_college_football(self, manager, mock_get): + manager.get_odds('football', 'ncaa_fb', '401') + + url = mock_get.call_args[0][0] + assert '/leagues/college-football/' in url + + def test_unknown_league_passes_through(self, manager, mock_get): + manager.get_odds('football', 'xfl', '401') + + assert '/leagues/xfl/' in mock_get.call_args[0][0] + + def test_cache_hit_skips_http(self, manager, cache_manager, mock_get): + cache_manager.get_with_auto_strategy.return_value = {'spread': -3.0} + + result = manager.get_odds('football', 'nfl', '401') + + assert result == {'spread': -3.0} + mock_get.assert_not_called() + + def test_cached_no_odds_sentinel_returned_verbatim( + self, manager, cache_manager, mock_get): + cache_manager.get_with_auto_strategy.return_value = {'no_odds': True} + + result = manager.get_odds('football', 'nfl', '401') + + assert result == {'no_odds': True} + mock_get.assert_not_called() + assert manager.is_odds_available(result) is False + + def test_success_caches_extracted_data_with_interval_ttl( + self, manager, cache_manager, mock_get): + result = manager.get_odds('football', 'nfl', '401', + update_interval_seconds=100) + + assert result == FULL_EXTRACTED + cache_manager.set.assert_called_once_with( + 'odds_espn_football_nfl_401', FULL_EXTRACTED, ttl=100) + + def test_no_odds_caches_sentinel(self, manager, cache_manager, mock_get): + mock_get.return_value = _make_response({'count': 0, 'items': []}) + + result = manager.get_odds('football', 'nfl', '401') + + assert result is None + cache_manager.set.assert_called_once_with( + 'odds_espn_football_nfl_401', {'no_odds': True}, ttl=3600) + + def test_zero_interval_falls_back_to_default( + self, manager, cache_manager, mock_get): + # Quirk pin: `update_interval_seconds or self.update_interval` + # treats an explicit 0 as falsy, so the 3600 default wins. + manager.get_odds('football', 'nfl', '401', update_interval_seconds=0) + + assert cache_manager.set.call_args.kwargs['ttl'] == 3600 + + def test_request_exception_falls_back_to_stale_cache( + self, manager, cache_manager, mock_get): + cache_manager.get_with_auto_strategy.side_effect = [ + None, {'stale': True}] + mock_get.side_effect = requests.exceptions.RequestException('boom') + + result = manager.get_odds('football', 'nfl', '401') + + assert result == {'stale': True} + assert cache_manager.get_with_auto_strategy.call_count == 2 + + +# --------------------------------------------------------------------------- +# _extract_espn_data +# --------------------------------------------------------------------------- + +class TestExtractEspnData: + def test_full_item_extracts_all_fields(self, manager): + result = manager._extract_espn_data({'items': [dict(FULL_ITEM)]}) + assert result == FULL_EXTRACTED + + def test_explicit_nulls_do_not_raise(self, manager): + # Post-fix: ESPN sends explicit JSON nulls for absent sides + # ("homeTeamOdds": null, "current": null); extraction must not + # raise and yields None fields. + payload = {'items': [{ + 'homeTeamOdds': None, + 'awayTeamOdds': {'moneyLine': 150, 'current': None}, + }]} + + result = manager._extract_espn_data(payload) + + assert result is not None + assert result['home_team_odds']['money_line'] is None + assert result['home_team_odds']['spread_odds'] is None + assert result['away_team_odds']['money_line'] == 150 + assert result['away_team_odds']['spread_odds'] is None + + def test_valid_empty_response_returns_none(self, manager): + assert manager._extract_espn_data({'count': 0, 'items': []}) is None + + def test_unexpected_structure_returns_none(self, manager): + assert manager._extract_espn_data({'unexpected': True}) is None + + def test_item_without_odds_fields_cached_as_data_not_sentinel( + self, manager, cache_manager, mock_get): + # Characterization pin: an item with no odds fields still extracts + # to a truthy dict of all-None values, so get_odds caches it as + # real data (NOT the no_odds sentinel) — but is_odds_available + # correctly reports False for it. + mock_get.return_value = _make_response({'items': [{}]}) + + result = manager.get_odds('football', 'nfl', '401') + + assert result == { + 'details': None, + 'over_under': None, + 'spread': None, + 'home_team_odds': {'money_line': None, 'spread_odds': None}, + 'away_team_odds': {'money_line': None, 'spread_odds': None}, + } + cache_manager.set.assert_called_once_with( + 'odds_espn_football_nfl_401', result, ttl=3600) + assert manager.is_odds_available(result) is False + + +# --------------------------------------------------------------------------- +# is_odds_available +# --------------------------------------------------------------------------- + +class TestIsOddsAvailable: + def test_none_is_false(self, manager): + assert manager.is_odds_available(None) is False + + def test_empty_dict_is_false(self, manager): + assert manager.is_odds_available({}) is False + + def test_no_odds_sentinel_is_false(self, manager): + assert manager.is_odds_available({'no_odds': True}) is False + + def test_spread_is_true(self, manager): + assert manager.is_odds_available({'spread': -3.5}) is True + + def test_over_under_is_true(self, manager): + assert manager.is_odds_available({'over_under': 47.5}) is True + + def test_nested_home_spread_odds_is_true(self, manager): + assert manager.is_odds_available( + {'home_team_odds': {'spread_odds': -3.5}}) is True + + def test_nested_away_spread_odds_is_true(self, manager): + assert manager.is_odds_available( + {'away_team_odds': {'spread_odds': 3.5}}) is True + + def test_moneyline_only_is_false(self, manager): + # Pinned ML-blind contract: is_odds_available ignores money lines + # (its callers decide whether to render an odds widget). Note that + # format_odds_summary deliberately uses a DIFFERENT gate — it will + # still format money-line-only odds (see TestFormatOddsSummary). + ml_only = { + 'home_team_odds': {'money_line': -120}, + 'away_team_odds': {'money_line': 100}, + } + assert manager.is_odds_available(ml_only) is False + + +# --------------------------------------------------------------------------- +# format_odds_summary (fixed gate: empty / no_odds only) +# --------------------------------------------------------------------------- + +class TestFormatOddsSummary: + def test_moneyline_only_formats(self, manager): + result = manager.format_odds_summary({ + 'home_team_odds': {'money_line': -120}, + 'away_team_odds': {'money_line': 100}, + }) + assert result == 'Home ML: -120 | Away ML: 100' + + def test_full_data_formats_all_parts(self, manager): + result = manager.format_odds_summary(FULL_EXTRACTED) + assert result == 'Spread: -3.5 | O/U: 47.5 | Home ML: -150 | Away ML: 130' + + def test_none_is_no_odds(self, manager): + assert manager.format_odds_summary(None) == 'No odds available' + + def test_empty_dict_is_no_odds(self, manager): + assert manager.format_odds_summary({}) == 'No odds available' + + def test_no_odds_sentinel_is_no_odds(self, manager): + assert manager.format_odds_summary( + {'no_odds': True}) == 'No odds available' + + +# --------------------------------------------------------------------------- +# get_odds_for_games +# --------------------------------------------------------------------------- + +class TestGetOddsForGames: + def test_missing_fields_get_none_odds_without_http(self, manager, mock_get): + games = [ + {'sport': 'football'}, + {'league': 'nfl'}, + {'id': '9'}, + {}, + ] + + result = manager.get_odds_for_games(games) + + assert all(g['odds'] is None for g in result) + mock_get.assert_not_called() + + def test_per_game_exception_continues_loop(self, manager, monkeypatch): + def fake_get_odds(sport, league, event_id, + update_interval_seconds=None): + if event_id == 'bad': + raise RuntimeError('boom') + return {'spread': -1.0} + + monkeypatch.setattr(manager, 'get_odds', fake_get_odds) + games = [ + {'sport': 'football', 'league': 'nfl', 'id': 'bad'}, + {'sport': 'football', 'league': 'nfl', 'id': 'ok'}, + ] + + result = manager.get_odds_for_games(games) + + assert len(result) == 2 + assert result[0]['odds'] is None + assert result[1]['odds'] == {'spread': -1.0} + + def test_input_dicts_mutated_in_place_and_returned(self, manager, mock_get): + # Pin: get_odds_for_games mutates the caller's game dicts in place + # and returns the same objects, not copies. + game = {'sport': 'football', 'league': 'nfl', 'id': '401'} + + result = manager.get_odds_for_games([game]) + + assert result[0] is game + assert game['odds'] == FULL_EXTRACTED + + +# --------------------------------------------------------------------------- +# _load_configuration +# --------------------------------------------------------------------------- + +class TestLoadConfiguration: + def test_loads_values_from_config(self, cache_manager): + config_manager = MagicMock() + config_manager.get_config.return_value = { + 'base_odds_manager': { + 'update_interval': 100, + 'timeout': 5, + 'cache_ttl': 42, + } + } + + manager = BaseOddsManager(cache_manager, config_manager=config_manager) + + assert manager.update_interval == 100 + # Key/attr mismatch pin: the config key is 'timeout' but the + # attribute is request_timeout. + assert manager.request_timeout == 5 + assert manager.cache_ttl == 42 + + def test_get_config_raising_keeps_defaults(self, cache_manager): + config_manager = MagicMock() + config_manager.get_config.side_effect = RuntimeError('boom') + + manager = BaseOddsManager(cache_manager, config_manager=config_manager) + + assert manager.update_interval == 3600 + assert manager.request_timeout == 5 + assert manager.cache_ttl == 1800 diff --git a/test/test_base_plugin_duration.py b/test/test_base_plugin_duration.py new file mode 100644 index 00000000..af7fe976 --- /dev/null +++ b/test/test_base_plugin_duration.py @@ -0,0 +1,146 @@ +""" +Tests for BasePlugin.get_display_duration — ~100 lines of type coercion that +every plugin's rotation slot depends on, previously untested. + +The contract: a positive number wins wherever it comes from; everything else +falls through instance attr → config → the 15.0 default, logging on the way. +""" + +from unittest.mock import MagicMock + +import pytest + +from src.plugin_system.base_plugin import BasePlugin + + +class _MinimalPlugin(BasePlugin): + def update(self): + pass + + def display(self, force_clear=False): + pass + + +def make_plugin(config=None, instance_duration="__unset__"): + plugin = _MinimalPlugin( + plugin_id="duration-test", + config=config or {}, + display_manager=MagicMock(), + cache_manager=MagicMock(), + plugin_manager=MagicMock(), + ) + if instance_duration != "__unset__": + plugin.display_duration = instance_duration + return plugin + + +class TestInstanceVariable: + def test_positive_int_wins(self): + assert make_plugin(instance_duration=30).get_display_duration() == 30.0 + + def test_positive_float_wins(self): + assert make_plugin(instance_duration=12.5).get_display_duration() == 12.5 + + def test_returns_float_type(self): + result = make_plugin(instance_duration=30).get_display_duration() + assert isinstance(result, float) + + def test_numeric_string_wins(self): + assert make_plugin(instance_duration="25").get_display_duration() == 25.0 + + def test_zero_falls_through_to_config(self): + plugin = make_plugin(config={"display_duration": 20}, + instance_duration=0) + assert plugin.get_display_duration() == 20.0 + + def test_negative_falls_through_to_config(self): + plugin = make_plugin(config={"display_duration": 20}, + instance_duration=-5) + assert plugin.get_display_duration() == 20.0 + + def test_none_falls_through_to_config(self): + plugin = make_plugin(config={"display_duration": 20}, + instance_duration=None) + assert plugin.get_display_duration() == 20.0 + + def test_garbage_string_falls_through(self): + plugin = make_plugin(config={"display_duration": 20}, + instance_duration="abc") + assert plugin.get_display_duration() == 20.0 + + def test_non_positive_string_falls_through(self): + plugin = make_plugin(config={"display_duration": 20}, + instance_duration="0") + assert plugin.get_display_duration() == 20.0 + + def test_unexpected_type_falls_through(self): + plugin = make_plugin(config={"display_duration": 20}, + instance_duration=[30]) + assert plugin.get_display_duration() == 20.0 + + def test_bool_true_falls_through_like_any_non_number(self): + # bool is an int subclass, but a boolean is not a duration: True + # must NOT read as 1 second — it falls through to config/default. + assert make_plugin(instance_duration=True).get_display_duration() == 15.0 + + def test_bool_true_falls_through_to_config(self): + plugin = make_plugin(config={"display_duration": 20}, + instance_duration=True) + assert plugin.get_display_duration() == 20.0 + + def test_bool_false_still_falls_through(self): + plugin = make_plugin(config={"display_duration": 20}, + instance_duration=False) + assert plugin.get_display_duration() == 20.0 + + +class TestConfigFallback: + def test_config_number(self): + assert make_plugin({"display_duration": 20}).get_display_duration() == 20.0 + + def test_config_numeric_string(self): + assert make_plugin({"display_duration": "12.5"}).get_display_duration() == 12.5 + + def test_missing_config_uses_default(self): + assert make_plugin({}).get_display_duration() == 15.0 + + def test_config_zero_uses_default(self): + assert make_plugin({"display_duration": 0}).get_display_duration() == 15.0 + + def test_config_negative_uses_default(self): + assert make_plugin({"display_duration": -10}).get_display_duration() == 15.0 + + def test_config_garbage_string_uses_default(self): + assert make_plugin({"display_duration": "soon"}).get_display_duration() == 15.0 + + def test_config_unexpected_type_uses_default(self): + assert make_plugin({"display_duration": {"s": 5}}).get_display_duration() == 15.0 + + def test_config_none_uses_default(self): + assert make_plugin({"display_duration": None}).get_display_duration() == 15.0 + + def test_config_bool_uses_default(self): + assert make_plugin({"display_duration": True}).get_display_duration() == 15.0 + assert make_plugin({"display_duration": False}).get_display_duration() == 15.0 + + +class TestValidateConfigDuration: + # validate_config must agree with get_display_duration about what a + # valid duration is — a config it accepts must not then be rejected + # (or silently defaulted) when the duration is actually read. + + def test_positive_number_valid(self): + assert make_plugin({"display_duration": 20}).validate_config() is True + + def test_zero_and_negative_invalid(self): + assert make_plugin({"display_duration": 0}).validate_config() is False + assert make_plugin({"display_duration": -5}).validate_config() is False + + def test_bool_invalid(self): + # bool is an int subclass; True would otherwise pass as "positive + # number" here while get_display_duration rejects it. + assert make_plugin({"display_duration": True}).validate_config() is False + assert make_plugin({"display_duration": False}).validate_config() is False + + def test_missing_duration_valid(self): + assert make_plugin({}).validate_config() is True diff --git a/test/test_cache_cleanup_thread_ownership.py b/test/test_cache_cleanup_thread_ownership.py new file mode 100644 index 00000000..2efa38c6 --- /dev/null +++ b/test/test_cache_cleanup_thread_ownership.py @@ -0,0 +1,146 @@ +"""Tests that one cache directory gets one cleanup thread per process. + +The sweep lists a directory and deletes from it, so a second thread over the +same directory only duplicates the scan. Nothing enforced that: every +CacheManager started its own, and since the loop closes over `self`, a +discarded manager could never be collected -- its thread stayed alive and +re-scanned the same directory every 24 hours for the life of the process. + +On the dev rig a display process carried three, for one cache directory: + + 14:22:59.954 display_controller (the real one) + 14:22:59.973 startup validation, run 1 (discarded) + 14:23:01.055 startup validation, run 2 (discarded) + +Startup validation runs twice and built a throwaway manager each time, purely +to read a directory path. +""" + +import threading + +import pytest + +from src.cache_manager import CacheManager + + +@pytest.fixture(autouse=True) +def _clean_registry(): + CacheManager._cleanup_owners.clear() + yield + for owner in list(CacheManager._cleanup_owners.values()): + owner.stop_cleanup_thread() + CacheManager._cleanup_owners.clear() + + +def _live_cleanup_threads(): + return [t for t in threading.enumerate() + if t.name == 'DiskCacheCleanup' and t.is_alive()] + + +@pytest.fixture +def manager(tmp_path, monkeypatch): + """A CacheManager pinned to a temp dir, so tests never touch the real one.""" + monkeypatch.setattr(CacheManager, '_get_writable_cache_dir', + lambda self: str(tmp_path)) + return CacheManager + + +class TestOneThreadPerDirectory: + def test_a_single_manager_starts_one(self, manager): + before = len(_live_cleanup_threads()) + m = manager() + try: + assert len(_live_cleanup_threads()) == before + 1 + finally: + m.stop_cleanup_thread() + + def test_three_managers_still_start_one(self, manager): + # Exactly the rig's shape: the real manager plus two throwaways. + before = len(_live_cleanup_threads()) + managers = [manager() for _ in range(3)] + try: + assert len(_live_cleanup_threads()) == before + 1 + finally: + for m in managers: + m.stop_cleanup_thread() + + def test_the_first_one_owns_it(self, manager): + first, second = manager(), manager() + try: + assert CacheManager._cleanup_owners[first.cache_dir] is first + assert second._cleanup_thread is None + finally: + first.stop_cleanup_thread() + second.stop_cleanup_thread() + + def test_the_survivor_can_take_over(self, manager): + first = manager() + first.stop_cleanup_thread() + assert not _live_cleanup_threads() + + second = manager() + try: + # Ownership was released, so the directory is swept again rather + # than being left permanently unclaimed by a dead owner. + assert len(_live_cleanup_threads()) == 1 + assert CacheManager._cleanup_owners[second.cache_dir] is second + finally: + second.stop_cleanup_thread() + + def test_stopping_a_non_owner_does_not_unclaim_the_directory(self, manager): + first, second = manager(), manager() + try: + second.stop_cleanup_thread() # never owned it + assert CacheManager._cleanup_owners[first.cache_dir] is first + assert len(_live_cleanup_threads()) == 1 + finally: + first.stop_cleanup_thread() + + def test_separate_directories_get_separate_threads(self, tmp_path, monkeypatch): + a, b = tmp_path / 'a', tmp_path / 'b' + a.mkdir() + b.mkdir() + dirs = iter([str(a), str(b)]) + monkeypatch.setattr(CacheManager, '_get_writable_cache_dir', + lambda self: next(dirs)) + first, second = CacheManager(), CacheManager() + try: + assert first.cache_dir != second.cache_dir + assert len(_live_cleanup_threads()) == 2 + finally: + first.stop_cleanup_thread() + second.stop_cleanup_thread() + + def test_no_thread_leaks_across_many_constructions(self, manager): + before = len(_live_cleanup_threads()) + made = [manager() for _ in range(12)] + try: + assert len(_live_cleanup_threads()) == before + 1 + finally: + for m in made: + m.stop_cleanup_thread() + assert len(_live_cleanup_threads()) == before + + +class TestValidatorDoesNotBuildItsOwn: + def test_it_uses_the_cache_manager_it_is_given(self, manager): + from src.startup_validator import StartupValidator + + shared = manager() + try: + before = len(_live_cleanup_threads()) + v = StartupValidator(config_manager=object(), cache_manager=shared) + v._validate_cache_directory() + assert len(_live_cleanup_threads()) == before, ( + "validation started another cleanup thread") + finally: + shared.stop_cleanup_thread() + + def test_without_one_it_cleans_up_after_itself(self, manager): + from src.startup_validator import StartupValidator + + before = len(_live_cleanup_threads()) + v = StartupValidator(config_manager=object()) + v._validate_cache_directory() + assert len(_live_cleanup_threads()) == before, ( + "the fallback manager left its cleanup thread running") diff --git a/test/test_cache_manager.py b/test/test_cache_manager.py index 8a93a39d..9d1195a8 100644 --- a/test/test_cache_manager.py +++ b/test/test_cache_manager.py @@ -458,3 +458,26 @@ class TestDiskCacheWriteEconomy: cache = DiskCache(cache_dir=str(tmp_path)) cache.set("k", {"when": datetime(2026, 7, 12, 10, 30)}) assert cache.get("k") == {"when": "2026-07-12T10:30:00"} + + +# --- the ceiling has to hold between cleanup sweeps --------------------------- + +def test_memory_cache_enforces_ceiling_on_every_write(): + """_cleanup_memory_cache only runs every cleanup_interval seconds (300 by + default). If set() accepted entries without bound in between, a burst could + take the cache far past max_size -- which is the unbounded growth the limit + exists to prevent, and on a 1GB board the difference between a bounded cache + and a Pi that cannot fork. + """ + from src.cache.memory_cache import MemoryCache + + cache = MemoryCache(max_size=150, cleanup_interval=300.0) + for i in range(1000): + cache.set(f"k{i}", {"v": i}) + + assert len(cache._cache) <= 150 + # The timestamp map has to be evicted alongside the values, or it becomes + # the leak instead. + assert len(cache._timestamps) <= 150 + assert cache.get("k999") is not None, "the newest write must survive" + assert cache.get("k0") is None, "the oldest must be the one evicted" diff --git a/test/test_cache_orphan_temp_sweep.py b/test/test_cache_orphan_temp_sweep.py new file mode 100644 index 00000000..f35579d6 --- /dev/null +++ b/test/test_cache_orphan_temp_sweep.py @@ -0,0 +1,188 @@ +"""Tests that abandoned cache temp files get collected. + +DiskCache.set() writes through tempfile.mkstemp and os.replace, removing its +own temp file in a finally. That covers a failed write, but not a process that +dies between the two -- a SIGKILL, a lost restart race, a power cut, all +ordinary on a Pi. Nothing collected what was left behind: the temp names are +"..json.", and the expiry sweep only listed names ending in +.json, so they accumulated for as long as the card had been in service. + +Measured on a live rig before this fix: 76 orphans totalling 1,050 MB -- 81% +of the entire cache directory -- the oldest six months old. + +The predicate that decides what to delete is tested harder than the sweep +itself, because a false positive here destroys real data. +""" + +import os +import time + +import pytest + +from src.cache.disk_cache import DiskCache, _ORPHAN_TEMP_MAX_AGE_SECONDS + + +class FakeStrategy: + @staticmethod + def get_data_type_from_key(key): + return 'default' + + +POLICIES = {'default': 30} + + +@pytest.fixture +def cache(tmp_path): + return DiskCache(str(tmp_path)) + + +def _age(path, seconds): + old = time.time() - seconds + os.utime(path, (old, old)) + + +def _write(tmp_path, name, body='{}'): + p = tmp_path / name + p.write_text(body, encoding='utf-8') + return p + + +class TestWhatCountsAsAnOrphan: + @pytest.mark.parametrize('name', [ + '.weather.json.a1b2c3d4', + '.odds_espn_football_nfl_401.json.xyz00000', + '.a.json.b', + ]) + def test_our_temp_files_are_orphans(self, name): + assert DiskCache._is_orphaned_temp(name) + + @pytest.mark.parametrize('name', [ + 'weather.json', # real data + '.weather.json', # a dotted key that completed + '.gitignore', # not ours + '.hidden', # not ours + 'weather.json.bak', # no leading dot: someone else's + '.json.abc', # no key between the dot and .json. + '.weather.json.', # no random component + 'notes.txt', + ]) + def test_everything_else_is_left_alone(self, name): + assert not DiskCache._is_orphaned_temp(name) + + def test_the_names_set_actually_creates_are_matched(self, cache, tmp_path): + """Guard against the predicate and the writer drifting apart.""" + created = [] + real = os.replace + + def capture(src, dst): + created.append(os.path.basename(src)) + return real(src, dst) + + import src.cache.disk_cache as mod + mod.os.replace = capture + try: + cache.set('weather', {'v': 1}) + finally: + mod.os.replace = real + + assert created, "set() did not go through the temp-file path" + assert all(DiskCache._is_orphaned_temp(n) for n in created), created + + +class TestTheSweep: + def test_an_old_orphan_is_removed(self, cache, tmp_path): + p = _write(tmp_path, '.weather.json.a1b2c3d4', 'x' * 5000) + _age(p, _ORPHAN_TEMP_MAX_AGE_SECONDS + 60) + + stats = cache.cleanup_expired_files(FakeStrategy(), POLICIES) + + assert not p.exists() + assert stats['orphan_temp_files_deleted'] == 1 + assert stats['space_freed_bytes'] >= 5000 + + def test_an_in_flight_write_is_not_snatched_away(self, cache, tmp_path): + # The whole risk of this sweep: deleting a temp file another thread is + # about to os.replace into place. + p = _write(tmp_path, '.weather.json.inflight') + + cache.cleanup_expired_files(FakeStrategy(), POLICIES) + + assert p.exists() + + def test_real_cache_files_survive(self, cache, tmp_path): + fresh = _write(tmp_path, 'weather.json') + dotted = _write(tmp_path, '.weather.json') + _age(dotted, _ORPHAN_TEMP_MAX_AGE_SECONDS + 60) + + cache.cleanup_expired_files(FakeStrategy(), POLICIES) + + assert fresh.exists() + assert dotted.exists(), "a completed .json was treated as a temp file" + + def test_unrelated_dotfiles_survive(self, cache, tmp_path): + keep = _write(tmp_path, '.gitignore') + _age(keep, 400 * 86400) + + cache.cleanup_expired_files(FakeStrategy(), POLICIES) + + assert keep.exists() + + def test_expiry_still_works_alongside_it(self, cache, tmp_path): + stale = _write(tmp_path, 'old.json') + _age(stale, 40 * 86400) # past the 30-day default + orphan = _write(tmp_path, '.old.json.zz999999') + _age(orphan, _ORPHAN_TEMP_MAX_AGE_SECONDS + 60) + + stats = cache.cleanup_expired_files(FakeStrategy(), POLICIES) + + assert not stale.exists() + assert not orphan.exists() + assert stats['files_deleted'] == 2 + assert stats['orphan_temp_files_deleted'] == 1 + + def test_the_rig_scenario(self, cache, tmp_path): + """76 orphans of assorted ages, none of them reachable before.""" + for i in range(76): + p = _write(tmp_path, '.sched_%d.json.r%06d' % (i, i), 'x' * 1000) + _age(p, (i + 2) * 86400) + keep = _write(tmp_path, 'sched.json') + + stats = cache.cleanup_expired_files(FakeStrategy(), POLICIES) + + assert stats['orphan_temp_files_deleted'] == 76 + assert keep.exists() + assert not list(tmp_path.glob('.sched_*')) + # The summary line is "/", so an orphan that is + # deleted but never counted as scanned renders as "76/1". + assert stats['files_scanned'] == 77 + assert stats['files_deleted'] <= stats['files_scanned'] + + def test_deleted_never_exceeds_scanned(self, cache, tmp_path): + p = _write(tmp_path, '.only.json.a1b2c3d4') + _age(p, _ORPHAN_TEMP_MAX_AGE_SECONDS + 60) + + stats = cache.cleanup_expired_files(FakeStrategy(), POLICIES) + + assert stats['files_deleted'] == 1 + assert stats['files_scanned'] == 1 + + def test_a_missing_file_mid_sweep_is_not_an_error(self, cache, tmp_path): + p = _write(tmp_path, '.weather.json.a1b2c3d4') + _age(p, _ORPHAN_TEMP_MAX_AGE_SECONDS + 60) + + import src.cache.disk_cache as mod + real = mod.os.path.getsize + + def vanish(path): + if path.endswith('.a1b2c3d4'): + os.remove(path) + raise FileNotFoundError(path) + return real(path) + + mod.os.path.getsize = vanish + try: + stats = cache.cleanup_expired_files(FakeStrategy(), POLICIES) + finally: + mod.os.path.getsize = real + + assert stats['errors'] == 0 diff --git a/test/test_cache_ttl_honoured.py b/test/test_cache_ttl_honoured.py new file mode 100644 index 00000000..080cbe2e --- /dev/null +++ b/test/test_cache_ttl_honoured.py @@ -0,0 +1,130 @@ +"""Tests that a per-entry ttl actually controls expiry. + +Regression under test: `CacheManager.set(key, data, ttl=...)` stored the value +and no read path ever consulted it. Expiry came from a `max_age` inferred from +substrings in the key ("live", "odds", "stock"), so every caller passing `ttl=` +-- 48 sites across the plugins and 4 in the core -- was writing a number that +did nothing. The old docstring admitted as much: "stored for compatibility but +expiration is still controlled via max_age when reading". + +Measured against a real device's cache (8,873 entries carrying a ttl), the +inferred value and the intended one disagreed almost everywhere: + + stocks max_age 600 vs ttl 1800 4903 entries + news max_age 3600 vs ttl 600 1770 entries + odds max_age 1800 vs ttl 3600 1301 entries + images max_age 300 vs ttl 2592000 20 entries + +No `sports_live` entry carries a ttl, so live scores keep their inferred +30-second freshness either way. +""" + +import time + +import pytest + +from src.cache.memory_cache import MemoryCache +from src.cache.disk_cache import DiskCache + + +@pytest.fixture +def disk(tmp_path): + return DiskCache(cache_dir=str(tmp_path)) + + +def _record(ttl=None, age=0.0): + rec = {"data": {"v": 1}, "timestamp": time.time() - age} + if ttl is not None: + rec["ttl"] = ttl + return rec + + +class TestDiskCacheHonoursTtl: + def test_ttl_longer_than_max_age_keeps_the_entry(self, disk): + # The odds case: written wanting an hour, expired at 30 minutes. + disk.set("odds_espn_football_nfl_401", _record(ttl=3600, age=1900)) + assert disk.get("odds_espn_football_nfl_401", max_age=1800) is not None + + def test_ttl_shorter_than_max_age_expires_the_entry(self, disk): + # The news case: written wanting 10 minutes, kept for an hour. + disk.set("news_NHL_1", _record(ttl=600, age=900)) + assert disk.get("news_NHL_1", max_age=3600) is None + + def test_without_a_ttl_max_age_still_applies(self, disk): + disk.set("plain_key", _record(age=400)) + assert disk.get("plain_key", max_age=300) is None + disk.set("plain_key2", _record(age=100)) + assert disk.get("plain_key2", max_age=300) is not None + + def test_a_fresh_entry_within_its_ttl_survives(self, disk): + disk.set("k", _record(ttl=600, age=10)) + assert disk.get("k", max_age=30) is not None + + def test_ttl_zero_expires_immediately(self, disk): + # 0 means zero seconds, not "forever" -- max_age=None is how a caller + # asks for no expiry. + disk.set("k", _record(ttl=0, age=1)) + assert disk.get("k", max_age=99999) is None + + @pytest.mark.parametrize("bad", ["600", None, True, False, -5, {"a": 1}]) + def test_a_nonsense_ttl_falls_back_to_max_age(self, disk, bad): + # Including bools: True is an int in Python and must not become a 1s ttl. + rec = _record(age=400) + rec["ttl"] = bad + disk.set("k_%s" % type(bad).__name__, rec) + assert disk.get("k_%s" % type(bad).__name__, max_age=300) is None + + +class TestMemoryCacheHonoursTtl: + def test_ttl_longer_than_max_age_keeps_the_entry(self): + m = MemoryCache() + m.set("k", _record(ttl=3600)) + m._timestamps["k"] = time.time() - 1900 + assert m.get("k", max_age=1800) is not None + + def test_ttl_shorter_than_max_age_expires_the_entry(self): + m = MemoryCache() + m.set("k", _record(ttl=600)) + m._timestamps["k"] = time.time() - 900 + assert m.get("k", max_age=3600) is None + + def test_without_a_ttl_max_age_still_applies(self): + m = MemoryCache() + m.set("k", _record()) + m._timestamps["k"] = time.time() - 400 + assert m.get("k", max_age=300) is None + + def test_both_layers_agree(self, tmp_path): + """A record must not be live in one layer and expired in the other.""" + rec = _record(ttl=3600, age=1900) + d = DiskCache(cache_dir=str(tmp_path)) + d.set("k", rec) + m = MemoryCache() + m.set("k", rec) + m._timestamps["k"] = rec["timestamp"] + assert (d.get("k", max_age=1800) is not None) == (m.get("k", max_age=1800) is not None) + + +class TestEndToEnd: + def test_set_then_get_respects_the_ttl(self, tmp_path, monkeypatch): + """The behaviour a caller of CacheManager.set(ttl=...) expects.""" + from src.cache_manager import CacheManager + + cm = CacheManager() + cm._disk_cache_component = DiskCache(cache_dir=str(tmp_path)) + cm._memory_cache_component = MemoryCache() + + cm.set("odds_espn_football_nfl_401", {"spread": 6.5}, ttl=3600) + + # Age the stored record past the inferred max_age for odds (1800s) but + # within the ttl the caller asked for. + path = cm._disk_cache_component.get_cache_path("odds_espn_football_nfl_401") + import json + rec = json.load(open(path)) + rec["timestamp"] = time.time() - 1900 + json.dump(rec, open(path, "w")) + cm._memory_cache_component.clear() if hasattr( + cm._memory_cache_component, "clear") else None + + got = cm.get_with_auto_strategy("odds_espn_football_nfl_401") + assert got is not None, "the ttl the caller asked for was ignored" diff --git a/test/test_checkbox_group_stale_values.py b/test/test_checkbox_group_stale_values.py new file mode 100644 index 00000000..c852372a --- /dev/null +++ b/test/test_checkbox_group_stale_values.py @@ -0,0 +1,113 @@ +"""A checkbox group must not post back options it cannot show. + +The enum that lets the widget draw checkboxes is also what validates the +saved value. When a league retires a team code -- OAK for the Athletics, ARI +for the Coyotes -- or a schema drops an option, a config that still holds the +old value has nothing to render for it. The value stayed in the hidden +``_data`` input regardless, because that input is seeded from the stored array +and only rebuilt by ``updateCheckboxGroupData()`` on change. Editing any other +field on that plugin therefore posted the stale value back, the schema +rejected it, and the save endpoint returned 400 +``CONFIG_VALIDATION_FAILED`` -- so the whole plugin became uneditable until +the user worked out which invisible entry was at fault. + +Runtime was never affected: plugin loading treats schema violations as +warn/degrade, and the stale code already matched no team. Only the web UI +blocked. + +These tests render the checkbox-group block lifted *out of the shipped +template*, following test_enum_option_labels.py, so they exercise the +production expression rather than a copy that could drift from it. +""" +import json +import re +from pathlib import Path + +from jinja2 import DictLoader, Environment + +PROJECT_ROOT = Path(__file__).resolve().parent.parent +CONFIG_FORM = (PROJECT_ROOT / 'web_interface' / 'templates' / 'v3' / 'partials' + / 'plugin_config.html') + +# The checkbox-group branch: from its `{% elif %}` guard through the sentinel +# hidden input that closes it. Anchored on the guard so the match cannot run on +# into a neighbouring widget branch. +BLOCK_RE = re.compile( + r"\{%\s*elif x_widget == 'checkbox-group'\s*%\}(.*?)" + r"", + re.S, +) + + +def _shipped_block() -> str: + """Return the live checkbox-group block lifted from plugin_config.html.""" + source = CONFIG_FORM.read_text(encoding='utf-8') + match = BLOCK_RE.search(source) + assert match, ( + 'could not find the checkbox-group block in plugin_config.html — the ' + 'template changed shape and this guard needs updating' + ) + block = match.group(1) + assert 'data-option-value' in block, 'extracted the wrong branch' + assert '{% elif' not in block, 'extraction ran past the checkbox-group branch' + return block + + +def _render(prop: dict, value=None) -> str: + env = Environment(loader=DictLoader({'f': _shipped_block()}), autoescape=True) + return env.get_template('f').render( + prop=prop, value=value, field_id='fid', full_key='k' + ) + + +def _submitted(html: str) -> list: + """The array the form will actually post: the hidden _data input.""" + match = re.search(r'id="fid_data"[^>]*\svalue=\'([^\']*)\'', html) + assert match, f'hidden _data input not found in:\n{html}' + return json.loads(match.group(1).replace(''', "'")) + + +def _checked(html: str) -> list: + return re.findall(r'data-option-value="([^"]+)"[^>]*checked', html) + + +MLB = {'type': 'array', 'items': {'type': 'string', 'enum': ['NYY', 'BOS', 'ATH']}, + 'x-widget': 'checkbox-group'} + + +def test_a_retired_code_is_not_posted_back() -> None: + """The regression: OAK became ATH, and OAK used to ride along on save.""" + html = _render(MLB, ['NYY', 'OAK']) + assert _submitted(html) == ['NYY'], 'stale value would still be submitted' + + +def test_the_dropped_value_is_named_rather_than_vanishing() -> None: + html = _render(MLB, ['NYY', 'OAK']) + assert 'OAK' in html + assert 'data-stale-options' in html + + +def test_valid_values_are_untouched_and_still_checked() -> None: + html = _render(MLB, ['NYY', 'ATH']) + assert _submitted(html) == ['NYY', 'ATH'] + assert sorted(_checked(html)) == ['ATH', 'NYY'] + assert 'data-stale-options' not in html + + +def test_an_all_stale_selection_clears_rather_than_blocking() -> None: + html = _render(MLB, ['OAK', 'SD']) + assert _submitted(html) == [] + + +def test_an_empty_enum_leaves_the_value_alone() -> None: + """No options means nothing to validate against — filtering would wipe it.""" + prop = {'type': 'array', 'items': {'type': 'string'}, 'x-widget': 'checkbox-group'} + html = _render(prop, ['ANYTHING', 'GOES']) + assert _submitted(html) == ['ANYTHING', 'GOES'] + + +def test_unset_value_falls_back_to_the_default() -> None: + prop = dict(MLB, default=['BOS']) + html = _render(prop, None) + assert _submitted(html) == ['BOS'] + assert _checked(html) == ['BOS'] diff --git a/test/test_compatibility.py b/test/test_compatibility.py new file mode 100644 index 00000000..34f88869 --- /dev/null +++ b/test/test_compatibility.py @@ -0,0 +1,271 @@ +""" +Tests for src/plugin_system/compatibility.py — the "can this plugin run on +this core?" gate used by both the plugin loader (advisory) and the store +manager (blocking at install/update time). + +This module had zero direct test coverage despite guarding every install. +These tests pin the documented contract: refuse only on evidence, resolve +every uncertain case (unparseable versions, missing fields, untrustworthy +core) to compatible. +""" + +import pytest + +from src.plugin_system.compatibility import ( + TRUSTWORTHY_FLOOR, + parse_semver, + _parse_strict, + _satisfies_range, + satisfies_compatible_versions, + declared_min_version, + check, +) + + +class TestParseSemver: + def test_plain_triplet(self): + assert parse_semver("1.2.3") == (1, 2, 3) + + def test_leading_v_tolerated(self): + assert parse_semver("v3.2.1") == (3, 2, 1) + + def test_prerelease_suffix_stripped(self): + # "3.2.0-rc1" must NOT parse as (3, 2, 1) — a release candidate must + # not rank above its own release. + assert parse_semver("3.2.0-rc1") == (3, 2, 0) + + def test_build_suffix_stripped(self): + # "3.2.0+build42" must NOT parse as (3, 2, 42). + assert parse_semver("3.2.0+build42") == (3, 2, 0) + + def test_two_part_version_pads_zero(self): + assert parse_semver("1.2") == (1, 2, 0) + + def test_one_part_version_pads_zeros(self): + assert parse_semver("2") == (2, 0, 0) + + def test_extra_parts_ignored(self): + assert parse_semver("1.2.3.4") == (1, 2, 3) + + def test_non_string_returns_none(self): + assert parse_semver(None) is None + assert parse_semver(123) is None + assert parse_semver((1, 2, 3)) is None + + def test_garbage_with_no_digits_is_lenient_zero(self): + # Documented leniency: digit-scraping yields (0, 0, 0) for pure + # garbage. Fine for a floor (0.0.0 never blocks), wrong for ranges — + # which is why ranges go through _parse_strict instead. + assert parse_semver("garbage") == (0, 0, 0) + + def test_whitespace_stripped(self): + assert parse_semver(" 1.2.3 ") == (1, 2, 3) + + +class TestParseStrict: + def test_accepts_real_versions(self): + assert _parse_strict("1.2.3") == (1, 2, 3) + assert _parse_strict("v1.2.3-rc1") == (1, 2, 3) + assert _parse_strict("2.0") == (2, 0, 0) + + def test_rejects_garbage(self): + assert _parse_strict("not-a-version") is None + assert _parse_strict("") is None + + def test_rejects_non_string(self): + assert _parse_strict(None) is None + + +class TestSatisfiesRange: + CORE = (3, 1, 0) + + @pytest.mark.parametrize("spec,expected", [ + (">=3.0.0", True), + (">=3.1.0", True), + (">=3.2.0", False), + ("<=3.1.0", True), + ("<=3.0.9", False), + (">3.0.9", True), + (">3.1.0", False), + ("<3.2.0", True), + ("<3.1.0", False), + ]) + def test_comparison_operators(self, spec, expected): + assert _satisfies_range(self.CORE, spec) is expected + + def test_tilde_allows_patch_only(self): + # ~3.1.0 means >=3.1.0, <3.2.0 + assert _satisfies_range((3, 1, 5), "~3.1.0") is True + assert _satisfies_range((3, 2, 0), "~3.1.0") is False + assert _satisfies_range((3, 0, 9), "~3.1.0") is False + + def test_caret_allows_minor_and_patch(self): + # ^3.1.0 means >=3.1.0, <4.0.0 + assert _satisfies_range((3, 9, 9), "^3.1.0") is True + assert _satisfies_range((4, 0, 0), "^3.1.0") is False + assert _satisfies_range((3, 0, 0), "^3.1.0") is False + + def test_bare_exact_version(self): + assert _satisfies_range((3, 1, 0), "3.1.0") is True + assert _satisfies_range((3, 1, 1), "3.1.0") is False + + def test_inclusive_dash_range(self): + assert _satisfies_range((2, 5, 0), "2.0.0 - 3.1.0") is True + assert _satisfies_range((2, 0, 0), "2.0.0 - 3.1.0") is True + assert _satisfies_range((3, 1, 0), "2.0.0 - 3.1.0") is True + assert _satisfies_range((3, 1, 1), "2.0.0 - 3.1.0") is False + + def test_unparseable_spec_returns_none_not_false(self): + # Garbage must read as "no evidence", never as a refusal — an + # unrecognised spelling must not cost a user a working install. + assert _satisfies_range(self.CORE, "banana") is None + assert _satisfies_range(self.CORE, ">=banana") is None + assert _satisfies_range(self.CORE, "") is None + assert _satisfies_range(self.CORE, "banana - 3.0.0") is None + + +class TestSatisfiesCompatibleVersions: + def test_any_entry_satisfying_wins(self): + manifest = {"compatible_versions": ["<1.0.0", ">=3.0.0"]} + assert satisfies_compatible_versions(manifest, (3, 1, 0)) is True + + def test_all_entries_failing_is_false(self): + manifest = {"compatible_versions": ["<1.0.0", "2.0.0 - 2.9.9"]} + assert satisfies_compatible_versions(manifest, (3, 1, 0)) is False + + def test_absent_field_returns_none(self): + assert satisfies_compatible_versions({}, (3, 1, 0)) is None + + def test_empty_list_returns_none(self): + assert satisfies_compatible_versions( + {"compatible_versions": []}, (3, 1, 0)) is None + + def test_non_list_returns_none(self): + assert satisfies_compatible_versions( + {"compatible_versions": ">=2.0.0"}, (3, 1, 0)) is None + + def test_all_unparseable_entries_returns_none(self): + manifest = {"compatible_versions": ["banana", 42, None]} + assert satisfies_compatible_versions(manifest, (3, 1, 0)) is None + + def test_mixed_parseable_and_garbage_uses_parseable(self): + manifest = {"compatible_versions": ["banana", ">=3.0.0"]} + assert satisfies_compatible_versions(manifest, (3, 1, 0)) is True + + +class TestDeclaredMinVersion: + def test_top_level_field(self): + assert declared_min_version({"min_ledmatrix_version": "2.1.0"}) == "2.1.0" + + def test_requires_dict_fallback(self): + manifest = {"requires": {"min_ledmatrix_version": "2.2.0"}} + assert declared_min_version(manifest) == "2.2.0" + + def test_versions_array_fallback(self): + manifest = {"versions": [{"ledmatrix_min_version": "2.3.0"}]} + assert declared_min_version(manifest) == "2.3.0" + + def test_versions_array_deprecated_spelling(self): + manifest = {"versions": [{"ledmatrix_min": "2.4.0"}]} + assert declared_min_version(manifest) == "2.4.0" + + def test_top_level_wins_over_versions_array(self): + manifest = { + "min_ledmatrix_version": "2.1.0", + "versions": [{"ledmatrix_min_version": "9.9.9"}], + } + assert declared_min_version(manifest) == "2.1.0" + + def test_requires_as_list_does_not_raise(self): + # A hand-edited manifest can carry `requires` as a list; this used to + # raise AttributeError and one malformed manifest would take down the + # whole install path. + assert declared_min_version({"requires": ["something"]}) is None + + def test_versions_as_dict_does_not_raise(self): + # Same for `versions` as a mapping (used to raise KeyError). + assert declared_min_version({"versions": {"0": {}}}) is None + + def test_nothing_declared_returns_none(self): + assert declared_min_version({}) is None + + +class TestCheck: + def test_compatible_when_nothing_declared(self): + assert check({}, "3.1.0") == (True, None) + + def test_min_version_blocks_older_core(self): + manifest = {"name": "Test Plugin", "min_ledmatrix_version": "3.2.0"} + ok, reason = check(manifest, "3.1.0") + assert ok is False + assert "3.2.0" in reason and "3.1.0" in reason + + def test_min_version_allows_equal_core(self): + manifest = {"min_ledmatrix_version": "3.1.0"} + assert check(manifest, "3.1.0") == (True, None) + + def test_compatible_versions_upper_bound_blocks(self): + # A range is the only field that can express "not compatible with + # newer cores" — it must win even when the floor passes. + manifest = { + "name": "Old Plugin", + "min_ledmatrix_version": "2.0.0", + "compatible_versions": ["2.0.0 - 2.9.9"], + } + ok, reason = check(manifest, "3.1.0") + assert ok is False + assert "2.0.0 - 2.9.9" in reason + + def test_unparseable_core_with_high_floor_is_blocked(self): + manifest = {"min_ledmatrix_version": "3.2.0", + "compatible_versions": [">=3.2.0"]} + # An unparseable core version is "unknown", not "old"... but note + # parse_semver("garbage") == (0,0,0) which is below TRUSTWORTHY_FLOOR, + # so this rides the untrustworthy-core branch: floor > 2.0.0 blocks. + ok, reason = check(manifest, "garbage") + assert ok is False + assert "too old to identify reliably" in reason + + def test_untrustworthy_core_allows_ecosystem_baseline_floor(self): + # A core reporting 1.0.0 may really be v3.1.0 (which shipped with a + # wrong __version__). Floors at or below TRUSTWORTHY_FLOOR must not + # block, or that population could install nothing. + manifest = {"min_ledmatrix_version": "2.0.0", + "compatible_versions": [">=2.0.0"]} + assert check(manifest, "1.0.0") == (True, None) + + def test_untrustworthy_core_blocks_floor_above_baseline(self): + # But a floor above 2.0.0 needs modules that no core reporting below + # the floor can have — the one refusal on that branch. + manifest = {"name": "New Plugin", "min_ledmatrix_version": "3.2.0"} + ok, reason = check(manifest, "1.0.0") + assert ok is False + assert "too old to identify reliably" in reason + + def test_untrustworthy_core_ignores_compatible_versions(self): + # On the untrustworthy branch only the declared floor is consulted; + # ranges cannot be evaluated against a version that isn't evidence. + manifest = {"compatible_versions": ["2.0.0 - 2.9.9"]} + assert check(manifest, "1.0.0") == (True, None) + + def test_floor_exactly_at_trustworthy_floor_is_allowed(self): + floor = ".".join(str(n) for n in TRUSTWORTHY_FLOOR) + manifest = {"min_ledmatrix_version": floor} + assert check(manifest, "1.0.0") == (True, None) + + def test_reason_uses_manifest_name(self): + manifest = {"name": "Fancy Clock", "min_ledmatrix_version": "9.0.0"} + ok, reason = check(manifest, "3.1.0") + assert ok is False + assert reason.startswith("Fancy Clock") + + def test_reason_falls_back_to_id(self): + manifest = {"id": "fancy-clock", "min_ledmatrix_version": "9.0.0"} + ok, reason = check(manifest, "3.1.0") + assert ok is False + assert reason.startswith("fancy-clock") + + def test_prerelease_core_compares_equal_to_release(self): + # Documented: prereleases compare equal to their release. + manifest = {"min_ledmatrix_version": "3.2.0"} + assert check(manifest, "3.2.0-rc1") == (True, None) diff --git a/test/test_composer_code_injection.py b/test/test_composer_code_injection.py index 54a5b77b..c09e8797 100644 --- a/test/test_composer_code_injection.py +++ b/test/test_composer_code_injection.py @@ -164,8 +164,34 @@ def test_the_generated_config_assignment_does_not_precede_super_init(): """Guards the reasoning behind the reserved list, not just the list.""" src = _generated(_with_key("brightness")) body = src.splitlines() - super_at = next(i for i, l in enumerate(body) if "super().__init__(" in l) - assign_at = next(i for i, l in enumerate(body) if "self.brightness = config.get(" in l) + super_at = next(i for i, line in enumerate(body) if "super().__init__(" in line) + assign_at = next(i for i, line in enumerate(body) + if "self.brightness = config.get(" in line) assert assign_at > super_at, ( "config vars are assigned before super().__init__(); the reserved-name " "list assumes they land after it") + + +# --- optional keys ---------------------------------------------------------- + +@pytest.mark.parametrize("el_type,missing", [ + ("text", "text"), ("text", "text2"), ("clock", "format"), +]) +def test_an_element_missing_an_optional_key_does_not_500(el_type, missing): + """`p` is a copy of the raw element, so an absent key stays absent. + + The defaults were applied to locals only, so manager.py.j2 rendered + `{{ el.text | tojson }}` over a jinja2.Undefined and tojson raised + TypeError -- which no handler catches, making a missing key a 500 rather + than a validation error or a sensible default. + """ + el = {"type": el_type, "id": "e1", "x": 0, "y": 0, "font": "press_start"} + src = _generated(_payload(elements=[el])) + ast.parse(src) # must still be valid Python + assert "Undefined" not in src + + +def test_a_clock_without_a_format_uses_the_documented_default(): + el = {"type": "clock", "id": "c1", "x": 0, "y": 0, "font": "press_start"} + src = _generated(_payload(elements=[el])) + assert '"%H:%M"' in src, "the %H:%M default did not reach the generated source" diff --git a/test/test_config_helper.py b/test/test_config_helper.py new file mode 100644 index 00000000..4931fccc --- /dev/null +++ b/test/test_config_helper.py @@ -0,0 +1,253 @@ +""" +Tests for src/common/config_helper.py — pins the ConfigHelper contract. + +Covers: load/save round trips (missing/malformed files return {} rather +than raising, non-ASCII preserved via ensure_ascii=False, top-level JSON +lists returned as-is), dot-notation get/set including the silent-failure +contract when an intermediate key holds a non-dict, merge_configs deep +semantics with NO aliasing of the base config (the fixed bug — the old +shallow copy let mutations of the merged result leak into base's nested +dicts), simplified schema validation including the caught-TypeError path +when a schema 'type' is given as a string, plugin config key conventions +('{plugin_id}_config', enabled defaults True), and required-key checks +where a key present with value None counts as present. +""" + +import json + +import pytest + +from src.common.config_helper import ConfigHelper + + +@pytest.fixture +def helper(): + return ConfigHelper() + + +class TestLoadConfig: + def test_missing_file_returns_empty_dict(self, helper, tmp_path): + assert helper.load_config(tmp_path / "nope.json") == {} + + def test_malformed_json_returns_empty_dict(self, helper, tmp_path): + path = tmp_path / "bad.json" + path.write_text("{ this is not json", encoding="utf-8") + assert helper.load_config(path) == {} + + def test_top_level_list_returned_as_is(self, helper, tmp_path): + # load_config does not enforce a dict shape: a JSON list comes + # straight back. Pinned as a characterization of current behavior. + path = tmp_path / "list.json" + path.write_text("[1, 2, 3]", encoding="utf-8") + assert helper.load_config(path) == [1, 2, 3] + + +class TestSaveConfig: + def test_round_trip(self, helper, tmp_path): + path = tmp_path / "config.json" + config = {'display': {'hardware': {'rows': 32}}, 'timezone': 'UTC'} + assert helper.save_config(config, path) is True + assert helper.load_config(path) == config + + def test_creates_parent_directories(self, helper, tmp_path): + path = tmp_path / "deep" / "nested" / "config.json" + assert helper.save_config({'a': 1}, path) is True + assert path.exists() + assert helper.load_config(path) == {'a': 1} + + def test_non_ascii_survives_round_trip(self, helper, tmp_path): + path = tmp_path / "config.json" + config = {'city': 'Zürich', 'note': 'météo ☀'} + assert helper.save_config(config, path) is True + assert helper.load_config(path) == config + # ensure_ascii=False: characters are written raw, not \u-escaped + assert 'Zürich' in path.read_text(encoding='utf-8') + + def test_directory_path_returns_false_not_raise(self, helper, tmp_path): + assert helper.save_config({'a': 1}, tmp_path) is False + + +class TestGetConfigValue: + def test_dot_notation_hit(self, helper): + config = {'display': {'hardware': {'rows': 32}}} + assert helper.get_config_value(config, 'display.hardware.rows') == 32 + + def test_missing_returns_default(self, helper): + sentinel = object() + assert helper.get_config_value({}, 'display.rows', default=sentinel) is sentinel + + def test_intermediate_non_dict_returns_default(self, helper): + config = {'display': 'not-a-dict'} + assert helper.get_config_value(config, 'display.hardware.rows', default=64) == 64 + + def test_required_missing_raises_keyerror(self, helper): + with pytest.raises(KeyError): + helper.get_config_value({}, 'display.rows', required=True) + + +class TestSetConfigValue: + def test_sets_top_level(self, helper): + config = {} + helper.set_config_value(config, 'timezone', 'UTC') + assert config == {'timezone': 'UTC'} + + def test_auto_creates_intermediates(self, helper): + config = {} + helper.set_config_value(config, 'display.hardware.rows', 32) + assert config == {'display': {'hardware': {'rows': 32}}} + + def test_silent_failure_on_non_dict_intermediate(self, helper): + # 'a' exists but holds an int; the assignment attempt raises + # TypeError internally, which set_config_value swallows and logs. + # The config is left unchanged — pinned silent-failure contract. + config = {'a': 5} + helper.set_config_value(config, 'a.b', 1) + assert config == {'a': 5} + + +class TestMergeConfigs: + def test_nested_dicts_merge_recursively(self, helper): + base = {'display': {'rows': 32, 'cols': 64}, 'timezone': 'UTC'} + override = {'display': {'cols': 128, 'brightness': 90}} + merged = helper.merge_configs(base, override) + assert merged == { + 'display': {'rows': 32, 'cols': 128, 'brightness': 90}, + 'timezone': 'UTC', + } + + def test_scalar_override_wins_over_dict(self, helper): + merged = helper.merge_configs({'display': {'rows': 32}}, {'display': 7}) + assert merged['display'] == 7 + + def test_dict_override_wins_over_scalar(self, helper): + merged = helper.merge_configs({'display': 7}, {'display': {'rows': 32}}) + assert merged['display'] == {'rows': 32} + + def test_no_aliasing_of_base(self, helper): + # Post-fix: merge deep-copies base, so mutating the result never + # leaks back into the caller's base config. + base = {'display': {'x': 1}} + merged = helper.merge_configs(base, {}) + assert merged['display'] is not base['display'] + merged['display']['x'] = 99 + assert base['display']['x'] == 1 + + def test_inputs_unchanged(self, helper): + base = {'a': {'b': 1}} + override = {'a': {'c': 2}} + helper.merge_configs(base, override) + assert base == {'a': {'b': 1}} + assert override == {'a': {'c': 2}} + + def test_no_aliasing_of_override_values(self, helper): + # The non-recursive branch must deep-copy the override value too: + # mutating a merged-in list or dict must not reach back into + # override_config. + override = {'teams': ['A', 'B'], 'nested': {'x': [1]}} + merged = helper.merge_configs({}, override) + merged['teams'].append('C') + merged['nested']['x'].append(2) + assert override == {'teams': ['A', 'B'], 'nested': {'x': [1]}} + + +class TestValidateConfig: + def test_no_schema_dict_is_valid(self, helper): + assert helper.validate_config({'a': 1}) is True + + def test_no_schema_list_is_invalid(self, helper): + assert helper.validate_config([1, 2]) is False + + def test_required_key_missing_is_invalid(self, helper): + schema = {'rows': {'required': True, 'type': int}} + assert helper.validate_config({}, schema) is False + + def test_optional_key_missing_is_valid(self, helper): + schema = {'rows': {'required': False, 'type': int}} + assert helper.validate_config({}, schema) is True + + def test_wrong_type_is_invalid(self, helper): + schema = {'rows': {'type': int}} + assert helper.validate_config({'rows': 'thirty-two'}, schema) is False + assert helper.validate_config({'rows': 32}, schema) is True + + def test_allowed_values_violation_is_invalid(self, helper): + schema = {'mode': {'allowed_values': ['clock', 'weather']}} + assert helper.validate_config({'mode': 'stocks'}, schema) is False + assert helper.validate_config({'mode': 'clock'}, schema) is True + + def test_string_type_in_schema_is_invalid_via_typeerror(self, helper): + # 'type' given as the STRING "int" makes isinstance() raise + # TypeError; validate_config catches it and returns False rather + # than raising. Pinned characterization. + schema = {'rows': {'type': 'int'}} + assert helper.validate_config({'rows': 32}, schema) is False + + +class TestPluginConfigHelpers: + def test_get_plugin_config_uses_suffixed_key(self, helper): + plugin_cfg = {'enabled': True, 'display_duration': 30} + assert helper.get_plugin_config({'clock_config': plugin_cfg}, 'clock') == plugin_cfg + + def test_get_plugin_config_bare_id_key_not_found(self, helper): + # Only '{plugin_id}_config' is consulted — a bare 'clock' section + # is invisible to this helper. Pinned key contract. + assert helper.get_plugin_config({'clock': {'enabled': True}}, 'clock') == {} + + def test_create_default_config_wraps_in_suffixed_key(self, helper): + defaults = {'enabled': True} + assert helper.create_default_config('clock', defaults) == {'clock_config': defaults} + + def test_is_plugin_enabled_defaults_true_for_unknown(self, helper): + assert helper.is_plugin_enabled({}, 'clock') is True + + def test_is_plugin_enabled_false_when_disabled(self, helper): + config = {'clock_config': {'enabled': False}} + assert helper.is_plugin_enabled(config, 'clock') is False + + def test_is_plugin_enabled_ignores_bare_id_key(self, helper): + # Disabled under the wrong key -> still reported enabled (default). + config = {'clock': {'enabled': False}} + assert helper.is_plugin_enabled(config, 'clock') is True + + +class TestSportsAndDisplayHelpers: + def test_get_display_config(self, helper): + display = {'hardware': {'rows': 32}} + assert helper.get_display_config({'display': display}) == display + assert helper.get_display_config({}) == {} + + def test_get_sports_config_uses_scoreboard_suffix(self, helper): + sport_cfg = {'favorite_teams': ['TB']} + config = {'football_scoreboard': sport_cfg} + assert helper.get_sports_config(config, 'football') == sport_cfg + assert helper.get_sports_config(config, 'hockey') == {} + + def test_get_favorite_teams(self, helper): + config = {'football_scoreboard': {'favorite_teams': ['TB', 'DAL']}} + assert helper.get_favorite_teams(config, 'football') == ['TB', 'DAL'] + assert helper.get_favorite_teams({}, 'football') == [] + + def test_get_display_modes(self, helper): + modes = {'live': True, 'recent': False} + config = {'football_scoreboard': {'display_modes': modes}} + assert helper.get_display_modes(config, 'football') == modes + assert helper.get_display_modes({}, 'football') == {} + + +class TestValidateRequiredKeys: + def test_returns_missing_subset(self, helper): + config = {'a': 1, 'c': {'d': 2}} + missing = helper.validate_required_keys(config, ['a', 'b', 'c.d', 'c.e']) + assert missing == ['b', 'c.e'] + + def test_dot_notation_present(self, helper): + config = {'display': {'hardware': {'rows': 32}}} + assert helper.validate_required_keys(config, ['display.hardware.rows']) == [] + + def test_empty_requirements(self, helper): + assert helper.validate_required_keys({'a': 1}, []) == [] + + def test_present_with_none_counts_as_present(self, helper): + # _has_key checks key membership, not truthiness — a key set to + # None is NOT reported missing. Pinned semantics. + assert helper.validate_required_keys({'a': None}, ['a']) == [] diff --git a/test/test_config_main_redacts_secrets.py b/test/test_config_main_redacts_secrets.py new file mode 100644 index 00000000..15172df2 --- /dev/null +++ b/test/test_config_main_redacts_secrets.py @@ -0,0 +1,143 @@ +"""GET /config/main must not hand out credentials. + +The endpoint returned the raw config to anyone who could reach the port, and +this web interface has no authentication of any kind. Measured against a live +rig, an unauthenticated request returned: + + github.api_token 40 chars + incoming-packages.ha_token 183 chars + jellyfin-now-playing.api_key 32 chars + ledmatrix-weather.api_key 32 chars + on-air.mqtt_password 8 chars + youtube.api_key 20 chars + youtube-stats.api_key 39 chars + +A GitHub token and a Home Assistant long-lived token among them. + +The x-secret masking the plugin config endpoints use does not apply here: this +endpoint never consults a schema, and core keys such as github.api_token have +no schema to carry the marker. Several of those fields *are* tagged x-secret in +their plugin's schema and were still returned in full, which is what makes the +schema route the wrong one to rely on for this endpoint. + +Matching on field name is blunt. For a whole-config dump it is the right +default: anything named like a credential should not leave the process, and a +new plugin that adds a differently-shaped secret is covered without anyone +remembering to tag it. +""" +import pytest + +from web_interface.blueprints.api_v3 import ( + _looks_like_a_credential, + _redact_credentials, +) + + +@pytest.mark.parametrize("name", [ + "password", "mqtt_password", "opensky_password", "passwd", + "api_key", "apikey", "API_KEY", "flightaware_api_key", + "token", "ha_token", "api_token", "access_token", + "secret", "client_secret", "spotify_client_secret", + "access_key", "private_key", +]) +def test_credential_names_are_recognised(name): + assert _looks_like_a_credential(name) + + +@pytest.mark.parametrize("name", [ + "timezone", "city", "brightness", "enabled", "update_interval", + "favorite_teams", "display_duration", "keyword", +]) +def test_ordinary_names_are_left_alone(name): + assert not _looks_like_a_credential(name) + + +def test_the_measured_leak_is_closed(): + """The exact shape taken off the rig.""" + config = { + "github": {"api_token": "ghp_" + "x" * 36}, + "incoming-packages": {"ha_token": "y" * 183, "enabled": True}, + "jellyfin-now-playing": {"api_key": "z" * 32}, + "on-air": {"mqtt_password": "hunter22"}, + "youtube": {"api_key": "k" * 20}, + "timezone": "America/New_York", + } + out = _redact_credentials(config) + assert out["github"]["api_token"] == "" + assert out["incoming-packages"]["ha_token"] == "" + assert out["jellyfin-now-playing"]["api_key"] == "" + assert out["on-air"]["mqtt_password"] == "" + assert out["youtube"]["api_key"] == "" + # Everything else survives, or the config editor breaks. + assert out["timezone"] == "America/New_York" + assert out["incoming-packages"]["enabled"] is True + + +def test_nested_and_listed_credentials_are_reached(): + config = {"a": {"b": {"c": {"password": "p"}}}, + "feeds": [{"name": "x", "api_key": "k"}, {"name": "y"}]} + out = _redact_credentials(config) + assert out["a"]["b"]["c"]["password"] == "" + assert out["feeds"][0]["api_key"] == "" + assert out["feeds"][0]["name"] == "x" + + +def test_the_original_is_not_mutated(): + """The caller holds the live config; redaction must not edit it in place.""" + config = {"github": {"api_token": "keepme"}} + _redact_credentials(config) + assert config["github"]["api_token"] == "keepme" + + +def test_a_credential_shaped_container_is_still_walked(): + """`secrets: {...}` is a section name, not a value to blank.""" + config = {"secrets": {"api_key": "k", "note": "keep"}} + out = _redact_credentials(config) + assert out["secrets"]["api_key"] == "" + assert out["secrets"]["note"] == "keep" + + +def test_non_dict_input_passes_through(): + assert _redact_credentials("plain") == "plain" + assert _redact_credentials(7) == 7 + assert _redact_credentials(None) is None + + +def test_the_endpoint_itself_redacts(): + """Through the view function, not the helper. + + The helper tests above all passed with the route still returning + `config` -- reverting the one line that calls the redactor changed + nothing, because nothing exercised the route. A property asserted on a + helper is not a property asserted on the endpoint, and it is the endpoint + that is exposed to the network. + """ + import json as _json + from unittest.mock import MagicMock + + import flask + + from web_interface.blueprints import api_v3 as mod + + raw = {"github": {"api_token": "ghp_secret_value"}, + "timezone": "America/New_York"} + + manager = MagicMock() + manager.load_config.return_value = raw + previous = getattr(mod.api_v3, "config_manager", None) + mod.api_v3.config_manager = manager + + app = flask.Flask(__name__) + try: + with app.test_request_context("/config/main"): + response = mod.get_main_config() + payload = response.get_json() if hasattr(response, "get_json") else _json.loads(response[0].data) + finally: + mod.api_v3.config_manager = previous + + data = payload["data"] + assert data["github"]["api_token"] == "", ( + "the endpoint returned the token; the redactor is not wired in") + assert data["timezone"] == "America/New_York" + # And the config the manager handed over is untouched. + assert raw["github"]["api_token"] == "ghp_secret_value" diff --git a/test/test_config_manager.py b/test/test_config_manager.py index 29705f36..68c411e9 100644 --- a/test/test_config_manager.py +++ b/test/test_config_manager.py @@ -383,19 +383,6 @@ class TestConfigHelpers: display_config = manager.get_display_config() assert display_config["hardware"]["rows"] == 32 - def test_get_clock_config(self, tmp_path): - """Test getting clock config.""" - config_file = tmp_path / "config.json" - config_data = {"clock": {"format": "12h"}} - - with open(config_file, 'w') as f: - json.dump(config_data, f) - - manager = ConfigManager(config_path=str(config_file)) - manager.load_config() - - clock_config = manager.get_clock_config() - assert clock_config["format"] == "12h" class TestPluginConfigManagement: diff --git a/test/test_config_manager_secrets.py b/test/test_config_manager_secrets.py new file mode 100644 index 00000000..f7070c9c --- /dev/null +++ b/test/test_config_manager_secrets.py @@ -0,0 +1,335 @@ +""" +Tests for the ConfigManager secrets round-trip and the load_config fast path. + +The contract under test: config_secrets.json values are deep-merged INTO the +in-memory config at load time, and stripped back OUT before anything is +written to config.json — so secrets live in exactly one file on disk. This +suite pins that round-trip plus its sharp edges, including the guard that a +save REFUSES (ConfigError) when the secrets file exists but can't be loaded, +rather than leaking merged secrets into config.json in plaintext. + +Complements test_config_manager.py, which covers loading/migration/validation. +""" + +import json +import os + +import pytest + +from src.config_manager import ConfigManager +from src.exceptions import ConfigError + + +def make_manager(tmp_path, config=None, secrets=None): + """A ConfigManager over tmp_path files, template migration neutralized.""" + config_file = tmp_path / "config.json" + secrets_file = tmp_path / "config_secrets.json" + config_file.write_text(json.dumps(config if config is not None else {})) + if secrets is not None: + secrets_file.write_text(json.dumps(secrets)) + manager = ConfigManager(config_path=str(config_file), + secrets_path=str(secrets_file)) + # Point the (CWD-relative) template at nothing so migration never runs — + # these tests assert exact on-disk contents. + manager.template_path = str(tmp_path / "no-template.json") + return manager + + +class TestLoadMergesSecrets: + def test_secrets_deep_merged_into_config(self, tmp_path): + manager = make_manager( + tmp_path, + config={"weather": {"city": "Austin"}, "timezone": "UTC"}, + secrets={"weather": {"api_key": "s3cret"}}, + ) + loaded = manager.load_config() + assert loaded["weather"] == {"city": "Austin", "api_key": "s3cret"} + assert loaded["timezone"] == "UTC" + + def test_secret_scalar_overrides_config_value(self, tmp_path): + manager = make_manager( + tmp_path, + config={"weather": {"api_key": "YOUR_API_KEY"}}, + secrets={"weather": {"api_key": "real-key"}}, + ) + assert manager.load_config()["weather"]["api_key"] == "real-key" + + def test_missing_secrets_file_loads_config_fine(self, tmp_path): + manager = make_manager(tmp_path, config={"timezone": "UTC"}) + assert manager.load_config() == {"timezone": "UTC"} + + def test_corrupt_secrets_file_loads_config_without_secrets(self, tmp_path): + manager = make_manager(tmp_path, config={"timezone": "UTC"}) + (tmp_path / "config_secrets.json").write_text("{not json") + loaded = manager.load_config() + assert loaded["timezone"] == "UTC" + + +class TestSaveStripsSecrets: + def test_round_trip_keeps_secrets_out_of_config_json(self, tmp_path): + manager = make_manager( + tmp_path, + config={"weather": {"city": "Austin"}}, + secrets={"weather": {"api_key": "s3cret"}}, + ) + loaded = manager.load_config() + assert loaded["weather"]["api_key"] == "s3cret" # merged in memory + + manager.save_config(loaded) + + on_disk = json.loads((tmp_path / "config.json").read_text()) + assert "api_key" not in on_disk.get("weather", {}) + assert on_disk["weather"]["city"] == "Austin" + # In-memory config still carries the secret for runtime use. + assert manager.config["weather"]["api_key"] == "s3cret" + + def test_group_dropped_when_only_secrets_remain(self, tmp_path): + # _strip_secrets_recursive drops a group entirely when nothing + # non-secret is left in it. + manager = make_manager( + tmp_path, + config={}, + secrets={"weather": {"api_key": "s3cret"}}, + ) + manager.save_config({"weather": {"api_key": "s3cret"}, "timezone": "UTC"}) + on_disk = json.loads((tmp_path / "config.json").read_text()) + assert on_disk == {"timezone": "UTC"} + + def test_scalar_secret_key_stripped_at_top_level(self, tmp_path): + manager = make_manager(tmp_path, config={}, secrets={"token": "t"}) + manager.save_config({"token": "t", "timezone": "UTC"}) + on_disk = json.loads((tmp_path / "config.json").read_text()) + assert on_disk == {"timezone": "UTC"} + + def test_corrupt_secrets_file_refuses_save_no_plaintext_leak(self, tmp_path): + # Regression guard: when the secrets file exists but is corrupt at + # save time, stripping is impossible — the save must raise instead of + # writing the merged secrets into config.json in plaintext (the + # historical behavior). + manager = make_manager( + tmp_path, + config={"weather": {"city": "Austin"}}, + secrets={"weather": {"api_key": "s3cret"}}, + ) + loaded = manager.load_config() + (tmp_path / "config_secrets.json").write_text("{corrupt") + + with pytest.raises(ConfigError): + manager.save_config(loaded) + + # On-disk config untouched: no secret leaked. + on_disk = json.loads((tmp_path / "config.json").read_text()) + assert "api_key" not in on_disk.get("weather", {}) + + def test_corrupt_secrets_file_refuses_atomic_save_too(self, tmp_path): + # Same refusal on the atomic save path, which shared the leak. + manager = make_manager( + tmp_path, + config={"weather": {"city": "Austin"}}, + secrets={"weather": {"api_key": "s3cret"}}, + ) + loaded = manager.load_config() + (tmp_path / "config_secrets.json").write_text("{corrupt") + + with pytest.raises(ConfigError): + manager.save_config_atomic(loaded) + + on_disk = json.loads((tmp_path / "config.json").read_text()) + assert "api_key" not in on_disk.get("weather", {}) + + +class TestLoadFastPath: + def test_unchanged_files_return_cached_dict(self, tmp_path): + manager = make_manager(tmp_path, config={"timezone": "UTC"}) + first = manager.load_config() + second = manager.load_config() + assert second is first # same aliased dict, no re-read + + def test_touching_secrets_file_invalidates_cache(self, tmp_path): + manager = make_manager( + tmp_path, + config={"weather": {}}, + secrets={"weather": {"api_key": "old"}}, + ) + assert manager.load_config()["weather"]["api_key"] == "old" + + secrets_file = tmp_path / "config_secrets.json" + secrets_file.write_text(json.dumps({"weather": {"api_key": "new"}})) + # Force a different mtime_ns in case the write landed within the + # filesystem's timestamp granularity. + os.utime(secrets_file, ns=(1, 1)) + + assert manager.load_config()["weather"]["api_key"] == "new" + + def test_same_mtime_same_size_change_served_stale(self, tmp_path): + # Characterized fast-path blind spot: the signature is (mtime_ns, + # size) only, so a same-length content swap with a forged identical + # mtime is not detected. Real writes bump mtime_ns, so this is + # acceptable — but it is a contract worth pinning. + manager = make_manager(tmp_path, config={"timezone": "AAA"}) + config_file = tmp_path / "config.json" + os.utime(config_file, ns=(1_000_000_000, 1_000_000_000)) + manager._loaded_sig = None + first = manager.load_config() + assert first["timezone"] == "AAA" + + config_file.write_text(json.dumps({"timezone": "BBB"})) # same length + os.utime(config_file, ns=(1_000_000_000, 1_000_000_000)) + + assert manager.load_config()["timezone"] == "AAA" # stale, by design + + +class TestArraySecretStripAndMerge: + """Array-item secrets round-trip (parallel-placeholder lists). + + secret_helpers.separate_secrets emits array secrets as a list parallel + to the regular list, with {} for items that carry no secrets. Strip + must remove the secret fields from config.json while preserving item + indices; load must merge them back into the right items. The regular + list's length is authoritative in both directions. + """ + + def test_strip_removes_array_item_secrets_keeps_indices(self, tmp_path): + manager = make_manager(tmp_path) + data = {"plugin": {"accounts": [ + {"name": "a", "token": "ta"}, + {"name": "b"}, + ]}} + secrets = {"plugin": {"accounts": [{"token": "ta"}, {}]}} + stripped = manager._strip_secrets_recursive(data, secrets) + assert stripped == {"plugin": {"accounts": [{"name": "a"}, {"name": "b"}]}} + + def test_strip_keeps_all_placeholder_items(self, tmp_path): + # Even when every item strips to nothing extra, the list survives + # with its indices — required for merge-on-load alignment. + manager = make_manager(tmp_path) + data = {"accounts": [{"token": "t1"}, {"token": "t2"}]} + secrets = {"accounts": [{"token": "t1"}, {"token": "t2"}]} + stripped = manager._strip_secrets_recursive(data, secrets) + assert stripped == {"accounts": [{}, {}]} + + def test_strip_whole_scalar_array_secret_drops_key(self, tmp_path): + # A list of secret scalars is a whole-key secret, not the parallel + # shape — the key must vanish from config.json entirely. + manager = make_manager(tmp_path) + data = {"recovery_codes": ["a", "b"], "city": "Austin"} + secrets = {"recovery_codes": ["a", "b"]} + stripped = manager._strip_secrets_recursive(data, secrets) + assert stripped == {"city": "Austin"} + + def test_strip_shape_mismatch_drops_key(self, tmp_path): + # Conservative contract: if the shapes disagree, never leak. + manager = make_manager(tmp_path) + data = {"accounts": {"name": "not-a-list"}} + secrets = {"accounts": [{"token": "t"}]} + stripped = manager._strip_secrets_recursive(data, secrets) + assert stripped == {} + + def test_strip_ignores_extra_secrets_entries(self, tmp_path): + # Regular list length is authoritative: a user deleted an item. + manager = make_manager(tmp_path) + data = {"accounts": [{"name": "a", "token": "ta"}]} + secrets = {"accounts": [{"token": "ta"}, {"token": "tb"}]} + stripped = manager._strip_secrets_recursive(data, secrets) + assert stripped == {"accounts": [{"name": "a"}]} + + def test_merge_restores_array_item_secrets(self, tmp_path): + manager = make_manager(tmp_path) + target = {"accounts": [{"name": "a"}, {"name": "b"}]} + manager._deep_merge(target, {"accounts": [{"token": "ta"}, {}]}) + assert target == {"accounts": [ + {"name": "a", "token": "ta"}, + {"name": "b"}, + ]} + + def test_merge_ignores_extra_secrets_entries_with_warning(self, tmp_path, caplog): + manager = make_manager(tmp_path) + target = {"accounts": [{"name": "a"}]} + with caplog.at_level("WARNING"): + manager._deep_merge( + target, {"accounts": [{"token": "ta"}, {"token": "ghost"}]}) + assert target == {"accounts": [{"name": "a", "token": "ta"}]} + assert any("longer than the config list" in r.message for r in caplog.records) + + def test_merge_non_dict_item_replaced_by_secret(self, tmp_path): + # Shape drift inside the list: the secret wins for that index. + manager = make_manager(tmp_path) + target = {"accounts": ["oddball", {"name": "b"}]} + manager._deep_merge(target, {"accounts": [{"token": "ta"}, {}]}) + assert target == {"accounts": [{"token": "ta"}, {"name": "b"}]} + + def test_merge_whole_scalar_array_still_replaces(self, tmp_path): + # Legacy behavior preserved: a non-parallel list replaces wholesale. + manager = make_manager(tmp_path) + target = {"recovery_codes": ["old"]} + manager._deep_merge(target, {"recovery_codes": ["new1", "new2"]}) + assert target == {"recovery_codes": ["new1", "new2"]} + + def test_full_save_load_round_trip(self, tmp_path): + # End to end on real files: save strips array secrets out of + # config.json; load merges them back into the right items. + manager = make_manager( + tmp_path, + config={"plugin": {"accounts": [ + {"name": "a", "token": "s3cret-a"}, + {"name": "b", "token": "s3cret-b"}, + ]}}, + secrets={"plugin": {"accounts": [ + {"token": "s3cret-a"}, {"token": "s3cret-b"}, + ]}}, + ) + loaded = manager.load_config() + assert loaded["plugin"]["accounts"][0]["token"] == "s3cret-a" + + manager.save_config(loaded) + + raw = (tmp_path / "config.json").read_text() + assert "s3cret" not in raw + on_disk = json.loads(raw) + assert on_disk["plugin"]["accounts"] == [{"name": "a"}, {"name": "b"}] + + # A fresh manager (constructed directly — make_manager would + # overwrite the just-saved config.json) re-merges from the secrets + # file on load. + fresh = ConfigManager(config_path=str(tmp_path / "config.json"), + secrets_path=str(tmp_path / "config_secrets.json")) + fresh.template_path = str(tmp_path / "no-template.json") + reloaded = fresh.load_config() + assert reloaded["plugin"]["accounts"] == [ + {"name": "a", "token": "s3cret-a"}, + {"name": "b", "token": "s3cret-b"}, + ] + + def test_whole_item_secret_list_never_leaks_values(self, tmp_path): + # When the ENTIRE array item is secret (schema marks both key[] + # and key[].field), separate_secrets stores the full item dicts in + # the secrets file. That shape also matches the parallel-list + # discriminator — which is safe: strip drops every leaf key that + # appears in the secret item, so only empty {} skeletons (item + # count, no values) can reach config.json, and merge-on-load + # restores the full items from those skeletons. + from src.web_interface.secret_helpers import ( + find_secret_fields, separate_secrets) + schema_props = {"accounts": { + "type": "array", + "items": {"type": "object", "x-secret": True, "properties": { + "id": {"type": "string"}, + "token": {"type": "string", "x-secret": True}, + }}, + }} + paths = find_secret_fields(schema_props) + assert paths == {"accounts[]", "accounts[].token"} + full = {"accounts": [{"id": "i1", "token": "s3cret-a"}, + {"id": "i2", "token": "s3cret-b"}]} + _, secrets = separate_secrets(full, paths) + assert secrets == full # whole items are secret + + manager = make_manager(tmp_path) + stripped = manager._strip_secrets_recursive(full, secrets) + assert stripped == {"accounts": [{}, {}]} + + raw = json.dumps(stripped) + assert "s3cret" not in raw and "i1" not in raw + + manager._deep_merge(stripped, secrets) + assert stripped == full # round trip restores the items diff --git a/test/test_core_owned_config_keys.py b/test/test_core_owned_config_keys.py new file mode 100644 index 00000000..a134fdc8 --- /dev/null +++ b/test/test_core_owned_config_keys.py @@ -0,0 +1,98 @@ +"""The core's own tuning keys must not make a plugin look broken. + +`vegas_width_pct`, `vegas_overflow` and `vegas_max_width_screens` are read by +the *core* out of each plugin's config block — `vegas_mode/plugin_adapter.py` +and `base_plugin.py`. No plugin declares them, and 37 of the 42 published +config schemas set `"additionalProperties": false`, so schema validation +reported them as violations. + +That is not just log noise: `_validate_config_schema_soft` sets `degraded` in +the health tracker, which the web UI surfaces. Measured on a real device, **9 +of 27 installed plugins** were flagged degraded purely for using a documented +core feature — including `baseball-scoreboard` and `f1-scoreboard`. + +The fix strips those keys before validating. It deliberately does *not* match +on a `vegas_` prefix: `vegas_mode` is plugin-owned and declared in schemas, and +a prefix rule would silently stop validating it. +""" + +from unittest.mock import MagicMock + +import pytest + +from src.plugin_system.plugin_manager import PluginManager + + +STRICT_SCHEMA = { + "type": "object", + "additionalProperties": False, + "properties": { + "enabled": {"type": "boolean"}, + "vegas_mode": {"type": "string"}, # plugin-owned, must stay validated + }, +} + + +@pytest.fixture +def manager(): + mgr = PluginManager.__new__(PluginManager) # skip the heavy constructor + mgr.logger = MagicMock() + mgr.schema_manager = MagicMock() + mgr._set_degraded_safe = MagicMock() + return mgr + + +class TestStripCoreOwnedKeys: + def test_removes_every_core_owned_key(self, manager): + cfg = {"enabled": True, "vegas_width_pct": 50, + "vegas_overflow": "wrap", "vegas_max_width_screens": 2} + assert manager._strip_core_owned_keys(cfg) == {"enabled": True} + + def test_leaves_plugin_owned_vegas_mode_alone(self, manager): + """A prefix rule would have eaten this one.""" + cfg = {"enabled": True, "vegas_mode": "scroll"} + assert manager._strip_core_owned_keys(cfg) == cfg + + def test_returns_the_same_object_when_nothing_to_strip(self, manager): + cfg = {"enabled": True} + assert manager._strip_core_owned_keys(cfg) is cfg + + def test_does_not_mutate_the_caller_config(self, manager): + cfg = {"enabled": True, "vegas_width_pct": 50} + manager._strip_core_owned_keys(cfg) + assert "vegas_width_pct" in cfg, "the live plugin config was mutated" + + def test_tolerates_a_non_dict(self, manager): + assert manager._strip_core_owned_keys(None) is None + + +class TestSoftValidation: + def _validate_with(self, manager, config, valid=True, errors=()): + manager.schema_manager.load_schema.return_value = STRICT_SCHEMA + manager.schema_manager.validate_config_against_schema.return_value = ( + valid, list(errors)) + manager._validate_config_schema_soft("baseball-scoreboard", config) + return manager.schema_manager.validate_config_against_schema.call_args + + def test_core_keys_never_reach_the_validator(self, manager): + """The regression: these keys reaching a strict schema is what flagged + 9 of 27 plugins degraded.""" + args = self._validate_with( + manager, {"enabled": True, "vegas_width_pct": 50}) + validated = args[0][0] + assert "vegas_width_pct" not in validated + assert validated == {"enabled": True} + + def test_plugin_owned_keys_still_reach_the_validator(self, manager): + args = self._validate_with( + manager, {"enabled": True, "vegas_mode": "scroll"}) + assert args[0][0]["vegas_mode"] == "scroll" + + def test_a_genuine_violation_is_still_reported(self, manager): + """Stripping core keys must not turn the check into a no-op.""" + self._validate_with( + manager, {"enabled": True, "typo_key": 1}, + valid=False, errors=["Field root: 'typo_key' was unexpected"]) + manager._set_degraded_safe.assert_called() + reason = manager._set_degraded_safe.call_args[0][1] + assert reason and "typo_key" in reason diff --git a/test/test_discovery_path_contract.py b/test/test_discovery_path_contract.py new file mode 100644 index 00000000..7c60b88c --- /dev/null +++ b/test/test_discovery_path_contract.py @@ -0,0 +1,167 @@ +""" +Drift guard: three components independently answer "where is plugin X?" and +their answers must stay coherent — plus the `.standalone-backup-` naming +contract that install/rollback shares with discovery. + +The three resolvers: +1. PluginManager._scan_directory_for_plugins — scans ONLY the configured dir. +2. PluginStoreManager._find_plugin_path — configured dir, then a sibling + `plugins/` fallback derived from the configured dir's parent. +3. SchemaManager.get_schema_path — configured dir, then project-root + `plugins/`, then `plugin-repos/`, then case-insensitive scans. + +The divergence is characterized (a plugin visible to the store/schema +fallbacks but invisible to discovery is a real support-issue shape) so any +change to the fallback chains is a deliberate one. + +The `.standalone-backup-` contract: store_manager renames a plugin dir aside +with that substring during install/rollback; discovery MUST skip such dirs +or a half-finished install would surface a ghost plugin. The substring is +duplicated as a literal in both files — this test breaks if either side +changes it unilaterally. +""" + +import json +import logging +import threading +from pathlib import Path + +import pytest + +from src.plugin_system.plugin_manager import PluginManager +from src.plugin_system.schema_manager import SchemaManager +from src.plugin_system.store_manager import PluginStoreManager + + +def _write_plugin(base: Path, plugin_id: str, dir_name: str = None): + plugin_dir = base / (dir_name or plugin_id) + plugin_dir.mkdir(parents=True) + (plugin_dir / "manifest.json").write_text(json.dumps({ + "id": plugin_id, "name": plugin_id, "version": "1.0.0", + })) + (plugin_dir / "config_schema.json").write_text(json.dumps({ + "type": "object", "properties": {"enabled": {"type": "boolean"}}, + })) + return plugin_dir + + +def _scanner(): + """A PluginManager stripped to just its discovery machinery — the full + constructor wires config/schema/health managers this test doesn't need.""" + pm = object.__new__(PluginManager) + pm.logger = logging.getLogger("test_discovery_path_contract") + pm._discovery_lock = threading.Lock() + pm.plugin_manifests = {} + pm.plugin_directories = {} + return pm + + +class TestResolversAgreeOnConfiguredDir: + def test_all_three_find_a_plugin_in_the_configured_dir(self, tmp_path): + plugins_dir = tmp_path / "plugin-repos" + plugin_dir = _write_plugin(plugins_dir, "demo-plugin") + + found = _scanner()._scan_directory_for_plugins(plugins_dir) + assert found == ["demo-plugin"] + + store = PluginStoreManager( + plugins_dir=str(plugins_dir), + uninstalled_registry_path=str(tmp_path / "uninstalled.json")) + assert store._find_plugin_path("demo-plugin") == plugin_dir + + schema = SchemaManager(plugins_dir=plugins_dir, project_root=tmp_path) + assert schema.get_schema_path("demo-plugin") == \ + plugin_dir / "config_schema.json" + + +class TestFallbackDivergence: + def test_plugin_only_in_plugins_dir_fallback(self, tmp_path): + """Characterized divergence: configured dir is plugin-repos/, but the + plugin sits in a sibling plugins/. The store and schema fallbacks + find it; discovery does NOT — so the plugin is installable/ + configurable but never loads. Pinned so a change to any fallback + chain shows up here.""" + configured = tmp_path / "plugin-repos" + configured.mkdir() + legacy_dir = _write_plugin(tmp_path / "plugins", "legacy-plugin") + + # Discovery: invisible. + assert _scanner()._scan_directory_for_plugins(configured) == [] + + # Store fallback: visible (parent-of-configured / 'plugins'). + store = PluginStoreManager( + plugins_dir=str(configured), + uninstalled_registry_path=str(tmp_path / "uninstalled.json")) + assert store._find_plugin_path("legacy-plugin") == legacy_dir + + # Schema fallback: visible (project_root / 'plugins'). + schema = SchemaManager(plugins_dir=configured, project_root=tmp_path) + assert schema.get_schema_path("legacy-plugin") == \ + legacy_dir / "config_schema.json" + + def test_schema_manager_probes_plugins_before_plugin_repos(self, tmp_path): + # Documented order (also in CLAUDE.md): plugins/ wins over + # plugin-repos/ when the same id exists in both. + in_plugins = _write_plugin(tmp_path / "plugins", "dupe") + _write_plugin(tmp_path / "plugin-repos", "dupe") + schema = SchemaManager(plugins_dir=None, project_root=tmp_path) + assert schema.get_schema_path("dupe") == \ + in_plugins / "config_schema.json" + + def test_schema_manager_case_insensitive_fallback(self, tmp_path): + plugin_dir = _write_plugin(tmp_path / "plugins", "MyPlugin", + dir_name="MyPlugin") + schema = SchemaManager(plugins_dir=None, project_root=tmp_path) + assert schema.get_schema_path("myplugin") == \ + plugin_dir / "config_schema.json" + + +class TestStandaloneBackupContract: + def test_discovery_skips_backup_dirs(self, tmp_path): + plugins_dir = tmp_path / "plugins" + _write_plugin(plugins_dir, "real-plugin") + # A rollback-in-progress dir with a valid manifest must NOT surface. + _write_plugin(plugins_dir, "real-plugin", + dir_name="real-plugin.standalone-backup-migrating") + + found = _scanner()._scan_directory_for_plugins(plugins_dir) + assert found == ["real-plugin"] + + def test_backup_substring_literal_matches_across_files(self): + """The substring is duplicated in plugin_manager (skip check) and + store_manager (rename-aside names). If either side changes it, the + other silently stops honoring the contract — this test is the + tripwire.""" + root = Path(__file__).resolve().parents[1] + pm_text = (root / "src/plugin_system/plugin_manager.py").read_text() + sm_text = (root / "src/plugin_system/store_manager.py").read_text() + assert "'.standalone-backup-'" in pm_text.replace('"', "'") + assert ".standalone-backup-" in sm_text + + +class TestSkinTargetResolution: + def _store(self, tmp_path): + return PluginStoreManager( + plugins_dir=str(tmp_path / "plugins"), + uninstalled_registry_path=str(tmp_path / "uninstalled.json")) + + def test_valid_skin_id_resolves_inside_skins_dir(self, tmp_path): + from src.skin_system import skin_runtime + store = self._store(tmp_path) + target = store._resolve_skin_target("my-skin") + assert target is not None + assert target.parent == skin_runtime.get_skins_directory().resolve() + + @pytest.mark.parametrize("bad_id", [ + "../evil", + "..", + "a/../../etc", + "/etc/passwd", + "skin/../../outside", + "", + None, + 123, + ]) + def test_traversal_and_malformed_ids_rejected(self, tmp_path, bad_id): + store = self._store(tmp_path) + assert store._resolve_skin_target(bad_id) is None diff --git a/test/test_display_controller.py b/test/test_display_controller.py index 782ce69e..da551b93 100644 --- a/test/test_display_controller.py +++ b/test/test_display_controller.py @@ -15,13 +15,6 @@ class TestDisplayControllerInitialization: assert test_display_controller.plugin_manager is not None assert test_display_controller.available_modes == [] - @pytest.mark.skip(reason="No assertions; init logic is covered by test_init_success and fixture setup") - def test_plugin_discovery_and_loading(self, test_display_controller): - """Test plugin discovery and loading during initialization.""" - pm = test_display_controller.plugin_manager - pm.discover_plugins.return_value = ["plugin1", "plugin2"] - pm.get_plugin.return_value = MagicMock() - class TestDisplayControllerModeRotation: """Test display mode rotation logic.""" @@ -345,31 +338,48 @@ class TestDisplayControllerSchedule: """Test schedule management.""" def test_schedule_disabled(self, test_display_controller): - """Test when schedule is disabled.""" + """schedule.enabled=False keeps the display active even outside the + configured window. (This test used to patch config_service, which + _check_schedule never reads — it asserted the init default.)""" controller = test_display_controller - schedule_config = {"schedule": {"enabled": False}} - with patch.object(controller.config_service, 'get_config', return_value=schedule_config): + controller.config['schedule'] = { + "enabled": False, + "start_time": "09:00", + "end_time": "17:00", + } + controller._schedule_checked_minute = None + controller._tz = None + controller.is_display_active = False # prove the method flips it back + + with patch('src.display_controller.datetime') as mock_datetime: + mock_datetime.now.return_value.strftime.return_value.lower.return_value = "monday" + mock_datetime.now.return_value.time.return_value = datetime.strptime("20:00", "%H:%M").time() + mock_datetime.strptime = datetime.strptime + controller._check_schedule() assert controller.is_display_active is True def test_active_hours(self, test_display_controller): - """Test active hours check.""" + """A time inside the window activates the display. (This test used + to patch config_service, which _check_schedule never reads — it + asserted the init default.)""" controller = test_display_controller + controller.config['schedule'] = { + "enabled": True, + "start_time": "09:00", + "end_time": "17:00", + } + controller._schedule_checked_minute = None + controller._tz = None + controller.is_display_active = False # prove the method flips it on + with patch('src.display_controller.datetime') as mock_datetime: mock_datetime.now.return_value.strftime.return_value.lower.return_value = "monday" mock_datetime.now.return_value.time.return_value = datetime.strptime("12:00", "%H:%M").time() mock_datetime.strptime = datetime.strptime - schedule_config = { - "schedule": { - "enabled": True, - "start_time": "09:00", - "end_time": "17:00" - } - } - with patch.object(controller.config_service, 'get_config', return_value=schedule_config): - controller._check_schedule() - assert controller.is_display_active is True + controller._check_schedule() + assert controller.is_display_active is True def test_inactive_hours(self, test_display_controller): """Test inactive hours check.""" diff --git a/test/test_display_controller_schedule.py b/test/test_display_controller_schedule.py new file mode 100644 index 00000000..a7237d4b --- /dev/null +++ b/test/test_display_controller_schedule.py @@ -0,0 +1,277 @@ +""" +Behavioral tests for DisplayController._check_schedule and +_check_dim_schedule — the on/off window and night-dimming logic. + +test_display_controller_optimizations.py::TestScheduleMinuteGate already +covers the once-per-minute gating; this file covers what it doesn't: +midnight-crossing windows, mode selection (global / per-day / legacy +inference), per-day disabled days, invalid time strings, unknown +timezones, boundary equality, and the transition-tracking flags. + +Both methods read only self.config and a handful of instance attributes, +so a bare stub via object.__new__ (the test_display_controller_vegas_tick +pattern) is enough — no managers needed. +""" + +import os +from datetime import datetime + +from unittest.mock import patch + +import pytest + +os.environ.setdefault("EMULATOR", "true") + +from src.display_controller import DisplayController # noqa: E402 + + +def make_controller(config=None, *, normal_brightness=90): + dc = object.__new__(DisplayController) + dc.config = config or {} + dc._tz = None + dc._schedule_checked_minute = None + dc.is_display_active = True + dc._was_display_active = True + dc._normal_brightness = normal_brightness + dc._dim_checked_minute = None + dc._cached_target_brightness = None + dc.is_dimmed = False + dc._was_dimmed = False + return dc + + +def at(time_str, day="monday"): + """Context manager patching the controller module's clock.""" + patcher = patch("src.display_controller.datetime") + mock_dt = patcher.start() + mock_dt.strptime = datetime.strptime + mock_dt.now.return_value.time.return_value = ( + datetime.strptime(time_str, "%H:%M").time()) + mock_dt.now.return_value.strftime.return_value.lower.return_value = day + mock_dt.now.return_value.hour = int(time_str.split(":")[0]) + mock_dt.now.return_value.minute = int(time_str.split(":")[1]) + return patcher + + +@pytest.fixture +def clock(): + patchers = [] + + def _at(time_str, day="monday"): + patchers.append(p := at(time_str, day)) + return p + + yield _at + for p in patchers: + p.stop() + + +def check_at(dc, time_str, day="monday", clock=None): + """Run _check_schedule at a mocked wall time, resetting the minute gate.""" + dc._schedule_checked_minute = None + p = at(time_str, day) + try: + dc._check_schedule() + finally: + p.stop() + return dc.is_display_active + + +def dim_at(dc, time_str, day="monday"): + dc._dim_checked_minute = None + p = at(time_str, day) + try: + return dc._check_dim_schedule() + finally: + p.stop() + + +class TestScheduleWindows: + def _config(self, start, end, **extra): + return {"schedule": {"enabled": True, "start_time": start, + "end_time": end, **extra}, + "timezone": "UTC"} + + def test_same_day_window(self): + dc = make_controller(self._config("09:00", "17:00")) + assert check_at(dc, "12:00") is True + assert check_at(dc, "20:00") is False + assert check_at(dc, "08:59") is False + + def test_boundaries_are_inclusive(self): + dc = make_controller(self._config("09:00", "17:00")) + assert check_at(dc, "09:00") is True # now == start + assert check_at(dc, "17:00") is True # now == end + + def test_midnight_crossing_window(self): + # 21:00 -> 07:00: active late evening AND early morning, inactive + # mid-day. + dc = make_controller(self._config("21:00", "07:00")) + assert check_at(dc, "23:00") is True + assert check_at(dc, "03:00") is True + assert check_at(dc, "12:00") is False + assert check_at(dc, "21:00") is True # boundary + assert check_at(dc, "07:00") is True # boundary + + def test_no_schedule_config_is_always_active(self): + dc = make_controller({"timezone": "UTC"}) + dc.is_display_active = False + dc._check_schedule() + assert dc.is_display_active is True + + def test_invalid_time_string_falls_back_to_active(self): + dc = make_controller(self._config("9 o'clock", "17:00")) + dc.is_display_active = False + assert check_at(dc, "03:00") is True # ValueError -> stay on + + def test_unknown_timezone_falls_back_to_utc(self): + dc = make_controller({"schedule": {"enabled": True, + "start_time": "09:00", + "end_time": "17:00"}, + "timezone": "Mars/Olympus_Mons"}) + assert check_at(dc, "12:00") is True + import pytz + assert dc._tz is pytz.UTC + + +class TestScheduleModes: + DAYS = { + "monday": {"enabled": True, "start_time": "10:00", + "end_time": "18:00"}, + "tuesday": {"enabled": False}, + } + + def test_global_mode_ignores_days(self): + dc = make_controller({"schedule": { + "enabled": True, "mode": "global", + "start_time": "09:00", "end_time": "17:00", + "days": self.DAYS}, "timezone": "UTC"}) + # 09:30 is inside the global window but outside monday's per-day one. + assert check_at(dc, "09:30", day="monday") is True + + def test_per_day_mode_uses_day_window(self): + dc = make_controller({"schedule": { + "enabled": True, "mode": "per-day", + "start_time": "09:00", "end_time": "17:00", + "days": self.DAYS}, "timezone": "UTC"}) + assert check_at(dc, "09:30", day="monday") is False # before 10:00 + assert check_at(dc, "12:00", day="monday") is True + + def test_per_day_underscore_spelling_accepted(self): + dc = make_controller({"schedule": { + "enabled": True, "mode": "per_day", + "days": self.DAYS}, "timezone": "UTC"}) + assert check_at(dc, "12:00", day="monday") is True + + def test_legacy_no_mode_infers_per_day_from_days_config(self): + dc = make_controller({"schedule": { + "enabled": True, + "start_time": "09:00", "end_time": "17:00", + "days": self.DAYS}, "timezone": "UTC"}) + assert check_at(dc, "09:30", day="monday") is False # per-day won + + def test_per_day_disabled_day_turns_display_off(self): + dc = make_controller({"schedule": { + "enabled": True, "mode": "per-day", + "days": self.DAYS}, "timezone": "UTC"}) + assert check_at(dc, "12:00", day="tuesday") is False + + def test_per_day_missing_day_falls_back_to_global(self): + dc = make_controller({"schedule": { + "enabled": True, "mode": "per-day", + "start_time": "09:00", "end_time": "17:00", + "days": self.DAYS}, "timezone": "UTC"}) + # Wednesday has no per-day entry -> global window applies. + assert check_at(dc, "09:30", day="wednesday") is True + + def test_missing_enabled_key_means_enabled(self): + # Backward compat: schedules written before the enabled flag. + dc = make_controller({"schedule": { + "start_time": "09:00", "end_time": "17:00"}, "timezone": "UTC"}) + assert check_at(dc, "20:00") is False + + +class TestScheduleTransitions: + def test_was_display_active_tracks_state(self): + dc = make_controller({"schedule": {"enabled": True, + "start_time": "09:00", + "end_time": "17:00"}, + "timezone": "UTC"}) + check_at(dc, "12:00") + assert dc._was_display_active is True + check_at(dc, "20:00") + assert dc._was_display_active is False + check_at(dc, "12:05") + assert dc._was_display_active is True + + +class TestDimSchedule: + def _config(self, start="20:00", end="07:00", **extra): + return {"dim_schedule": {"enabled": True, "start_time": start, + "end_time": end, "dim_brightness": 25, + **extra}, + "timezone": "UTC"} + + def test_disabled_by_default(self): + dc = make_controller({"dim_schedule": {"start_time": "20:00", + "end_time": "07:00"}, + "timezone": "UTC"}) + # Unlike the on/off schedule, dimming defaults to DISABLED when the + # enabled key is missing. + assert dim_at(dc, "23:00") == 90 + assert dc.is_dimmed is False + + def test_overnight_dim_window(self): + dc = make_controller(self._config()) + assert dim_at(dc, "23:00") == 25 + assert dc.is_dimmed is True + assert dim_at(dc, "03:00") == 25 + assert dim_at(dc, "12:00") == 90 + assert dc.is_dimmed is False + + def test_dim_brightness_defaults_to_30(self): + dc = make_controller({"dim_schedule": {"enabled": True, + "start_time": "20:00", + "end_time": "07:00"}, + "timezone": "UTC"}) + assert dim_at(dc, "23:00") == 30 + + def test_inactive_display_short_circuits_undimmed(self): + dc = make_controller(self._config()) + dc.is_display_active = False + dc.is_dimmed = True + assert dim_at(dc, "23:00") == 90 + assert dc.is_dimmed is False + + def test_per_day_mode(self): + dc = make_controller(self._config(mode="per-day", days={ + "monday": {"enabled": True, "start_time": "22:00", + "end_time": "06:00"}, + "tuesday": {"enabled": False}, + })) + assert dim_at(dc, "23:00", day="monday") == 25 + assert dim_at(dc, "21:00", day="monday") == 90 # before per-day start + assert dim_at(dc, "23:00", day="tuesday") == 90 # day disabled + assert dc.is_dimmed is False + + def test_no_legacy_inference_for_dim(self): + # Unlike _check_schedule, dim mode defaults to GLOBAL even when a + # days config exists — no legacy inference. + dc = make_controller(self._config(days={ + "monday": {"enabled": True, "start_time": "22:00", + "end_time": "06:00"}, + })) + # 21:00 is inside the global 20:00-07:00 window but outside monday's + # per-day 22:00 start; global mode wins. + assert dim_at(dc, "21:00", day="monday") == 25 + + def test_invalid_time_string_returns_normal(self): + dc = make_controller(self._config(start="late")) + assert dim_at(dc, "23:00") == 90 + + def test_was_dimmed_tracks_transitions(self): + dc = make_controller(self._config()) + dim_at(dc, "23:00") + assert dc._was_dimmed is True + dim_at(dc, "12:00") + assert dc._was_dimmed is False diff --git a/test/test_display_controller_vegas_tick.py b/test/test_display_controller_vegas_tick.py index 643359f3..538e9272 100644 --- a/test/test_display_controller_vegas_tick.py +++ b/test/test_display_controller_vegas_tick.py @@ -11,9 +11,18 @@ orphaning VegasModeCoordinator.mark_plugin_updated() -- it has had zero callers since. """ +import os from typing import Dict, List, Optional from unittest.mock import MagicMock +# display_controller imports display_manager, which imports the hardware +# rgbmatrix module unless EMULATOR=true was set before import. Use the +# emulator (same convention as test_display_dirty_tracking.py and +# test/plugins/conftest.py) so this file collects on machines without the +# hardware library — and so display_manager gets the emulator binding no +# matter which test module imports it first. +os.environ.setdefault("EMULATOR", "true") + from src.display_controller import DisplayController diff --git a/test/test_display_helper.py b/test/test_display_helper.py new file mode 100644 index 00000000..b8c9be63 --- /dev/null +++ b/test/test_display_helper.py @@ -0,0 +1,307 @@ +"""Tests for src/common/display_helper.py (DisplayHelper). + +Pure-PIL tests, no hardware or mocks required. Pixel assertions rely on +getbbox()/getpixel() rather than exact text pixel counts, because the +default-font metrics vary across Pillow versions. + +These tests pin the FIXED behaviors on this branch: +- draw_error_message / draw_no_data_message return a rendered image + (they previously crashed with AttributeError), +- draw_scorebug_layout draws period/status/clock as one combined top + line (previously overprinted at the same y), +- draw_ticker_layout draws at x=0 (previously started at + x=display_width, i.e. entirely off-canvas -> blank frames). +""" + +from PIL import Image, ImageDraw, ImageFont + +from src.common.display_helper import DisplayHelper + + +def default_font(): + return ImageFont.load_default() + + +def make_helper(width=128, height=32): + return DisplayHelper(width, height) + + +class TestCreateBaseImage: + def test_default_is_black_rgb_display_sized(self): + helper = make_helper() + img = helper.create_base_image() + assert img.size == (128, 32) + assert img.mode == 'RGB' + assert img.getpixel((0, 0)) == (0, 0, 0) + assert img.getpixel((127, 31)) == (0, 0, 0) + # Entirely black -> no bounding box in luminance + assert img.convert('L').getbbox() is None + + def test_custom_background_color(self): + helper = make_helper() + img = helper.create_base_image(background_color=(10, 20, 30)) + assert img.getpixel((0, 0)) == (10, 20, 30) + assert img.getpixel((64, 16)) == (10, 20, 30) + + def test_mode_rgba_is_honored(self): + helper = make_helper() + img = helper.create_base_image(mode='RGBA') + assert img.mode == 'RGBA' + assert img.size == (128, 32) + + +class TestCreateOverlay: + def test_overlay_is_transparent_rgba(self): + helper = make_helper() + overlay = helper.create_overlay() + assert overlay.mode == 'RGBA' + assert overlay.size == (128, 32) + assert overlay.getpixel((0, 0)) == (0, 0, 0, 0) + assert overlay.getpixel((127, 31)) == (0, 0, 0, 0) + + +class TestCompositeImages: + def test_rgb_inputs_are_upconverted_and_result_is_rgba(self): + helper = make_helper() + base = Image.new('RGB', (128, 32), (0, 0, 0)) + overlay = Image.new('RGB', (128, 32), (255, 0, 0)) + result = helper.composite_images(base, overlay) + assert result.mode == 'RGBA' + assert result.size == base.size + # RGB->RGBA conversion yields a fully opaque overlay + assert result.getpixel((0, 0)) == (255, 0, 0, 255) + + def test_transparent_overlay_leaves_base_visible(self): + helper = make_helper() + base = Image.new('RGB', (128, 32), (5, 6, 7)) + overlay = helper.create_overlay() + result = helper.composite_images(base, overlay) + assert result.mode == 'RGBA' + assert result.getpixel((64, 16)) == (5, 6, 7, 255) + + +class TestScorebugLayout: + def test_full_game_data_renders(self): + helper = make_helper() + font = default_font() + fonts = {'time': font, 'status': font, 'score': font, 'team': font} + game_data = { + 'home_score': 3, 'away_score': 2, + 'home_abbr': 'NYY', 'away_abbr': 'BOS', + 'status_text': 'LIVE', 'period_text': 'T9', 'clock': '2:30', + } + img = helper.draw_scorebug_layout(game_data, fonts) + assert img.mode == 'RGB' + assert img.size == (128, 32) + assert img.convert('L').getbbox() is not None + + def test_empty_game_data_uses_defaults_without_raising(self): + helper = make_helper() + font = default_font() + fonts = {'time': font, 'status': font, 'score': font, 'team': font} + img = helper.draw_scorebug_layout({}, fonts) + assert img.mode == 'RGB' + assert img.size == (128, 32) + # Defaults '0'/'HOME'/'AWAY' actually render something + assert img.convert('L').getbbox() is not None + + def test_empty_fonts_dict_falls_back_to_default_font(self): + # Pin: fonts={} must not raise — PIL falls back to the default + # font when font=None is passed through. + helper = make_helper() + img = helper.draw_scorebug_layout( + {'status_text': 'FINAL', 'period_text': 'Q4', 'clock': '0:00'}, {}) + assert img.size == (128, 32) + assert img.convert('L').getbbox() is not None + + def test_top_line_is_one_combined_centered_draw(self): + # FIXED behavior: period/status/clock are joined into a single + # top line drawn once at y=1 instead of three overprinted draws. + helper = make_helper() + calls = [] + original = helper._draw_centered_text + + def spy(draw, text, font, y_position): + calls.append({'text': text, 'y_position': y_position}) + original(draw, text, font, y_position) + + helper._draw_centered_text = spy + font = default_font() + fonts = {'time': font, 'status': font, 'score': font, 'team': font} + helper.draw_scorebug_layout( + {'period_text': 'Q4', 'status_text': 'LIVE', 'clock': '2:30'}, + fonts) + + top_calls = [c for c in calls if c['y_position'] == 1] + assert len(top_calls) == 1 + text = top_calls[0]['text'] + assert 'Q4' in text + assert 'LIVE' in text + assert '2:30' in text + + def test_no_top_line_when_all_parts_empty(self): + helper = make_helper() + calls = [] + original = helper._draw_centered_text + + def spy(draw, text, font, y_position): + calls.append(y_position) + original(draw, text, font, y_position) + + helper._draw_centered_text = spy + font = default_font() + helper.draw_scorebug_layout({}, {'score': font, 'team': font}) + assert 1 not in calls # no combined top line drawn + + def test_logo_positions_bleed_off_edges(self): + # Home logo pastes at x = width - logo.width + 10 (right edge, + # bleeding off-screen right); away at x = -10 (bleeding left). + helper = make_helper() + home_logo = Image.new('RGBA', (20, 20), (0, 0, 255, 255)) # blue + away_logo = Image.new('RGBA', (20, 20), (255, 0, 0, 255)) # red + # Empty abbrs/status so text can't land on the probed pixels. + game_data = {'home_abbr': '', 'away_abbr': ''} + font = default_font() + img = helper.draw_scorebug_layout(game_data, {'score': font}, + home_logo=home_logo, + away_logo=away_logo) + # center_y = 16; logos span y 6..25 -> probe y=16 at both edges. + assert img.getpixel((0, 16)) == (255, 0, 0) # away (left edge) + assert img.getpixel((127, 16)) == (0, 0, 255) # home (right edge) + # And the off-screen parts are truly clipped: image is still 128 wide + assert img.size == (128, 32) + + +class TestTickerLayout: + def test_frame_is_not_blank(self): + # FIXED behavior: text now starts at x=0. Previously it was drawn + # at x=display_width, entirely off-canvas, so frames were blank. + helper = make_helper() + img = helper.draw_ticker_layout('HELLO WORLD', default_font()) + assert img.size == (128, 32) + assert img.mode == 'RGB' + assert img.convert('L').getbbox() is not None + + def test_text_starts_at_left_edge(self): + helper = make_helper() + img = helper.draw_ticker_layout('HELLO', default_font()) + bbox = img.convert('L').getbbox() + assert bbox is not None + # Text is positioned at x=0 (outline extends 1px left, clipped), + # so ink begins hugging the left edge. Allow a couple of pixels of + # slack for font-dependent left-side bearing. + assert bbox[0] <= 2 + + def test_scroll_speed_does_not_affect_frame(self): + # Pin: scroll_speed is accepted for API compatibility only. + helper = make_helper() + font = default_font() + img1 = helper.draw_ticker_layout('SCROLLING', font, scroll_speed=1) + img5 = helper.draw_ticker_layout('SCROLLING', font, scroll_speed=5) + assert img1.tobytes() == img5.tobytes() + + def test_custom_colors(self): + helper = make_helper() + img = helper.draw_ticker_layout('X', default_font(), + background_color=(0, 0, 40), + text_color=(0, 255, 0)) + assert img.getpixel((127, 0)) == (0, 0, 40) # background corner + colors = {img.getpixel((x, y)) + for x in range(img.width) for y in range(img.height)} + # Text color appears somewhere (anti-aliasing may blend it, so + # check for a green-dominant pixel rather than the exact color). + assert any(g > 150 and r < 100 for (r, g, b) in colors) + + +class TestCenteredText: + def test_renders_centered_text_on_background(self): + helper = make_helper() + img = helper.draw_centered_text('HI', default_font(), + background_color=(0, 0, 60), + text_color=(255, 255, 0)) + assert img.size == (128, 32) + assert img.convert('L').getbbox() is not None + # Corners stay pure background + assert img.getpixel((0, 0)) == (0, 0, 60) + assert img.getpixel((127, 0)) == (0, 0, 60) + assert img.getpixel((0, 31)) == (0, 0, 60) + assert img.getpixel((127, 31)) == (0, 0, 60) + + +class TestErrorAndNoDataMessages: + def test_draw_error_message_returns_rendered_image(self): + # FIXED behavior: used to crash with AttributeError; now returns + # a rendered image on a dark red background. + helper = make_helper() + img = helper.draw_error_message('Boom') + assert img.size == (128, 32) + assert img.mode == 'RGB' + assert img.convert('L').getbbox() is not None + assert img.getpixel((0, 0)) == (50, 0, 0) # dark red background + + def test_draw_error_message_default_text(self): + helper = make_helper() + img = helper.draw_error_message() + assert img.size == (128, 32) + assert img.getpixel((127, 31)) == (50, 0, 0) + + def test_draw_no_data_message_returns_rendered_image(self): + helper = make_helper() + img = helper.draw_no_data_message() + assert img.size == (128, 32) + assert img.mode == 'RGB' + assert img.convert('L').getbbox() is not None + assert img.getpixel((0, 0)) == (0, 0, 0) # black background + + +class TestDrawTextWithOutline: + def test_fill_color_appears_in_output(self): + helper = make_helper() + img = Image.new('RGB', (40, 20), (0, 0, 255)) + draw = ImageDraw.Draw(img) + helper._draw_text_with_outline(draw, 'X', (5, 2), default_font(), + fill=(255, 0, 0)) + pixels = {img.getpixel((x, y)) + for x in range(img.width) for y in range(img.height)} + # Anti-aliased fonts blend edge pixels, so look for red-dominant + # (fill) and near-black (outline) pixels rather than exact colors. + assert any(r > 150 and g < 50 for (r, g, b) in pixels) # fill + assert any(max(p) < 80 for p in pixels) # outline + + def test_default_fill_is_white(self): + helper = make_helper() + img = Image.new('RGB', (40, 20), (0, 0, 255)) + draw = ImageDraw.Draw(img) + helper._draw_text_with_outline(draw, 'X', (5, 2), default_font()) + pixels = {img.getpixel((x, y)) + for x in range(img.width) for y in range(img.height)} + # White-dominant pixel present (exact white may be anti-aliased) + assert any(r > 200 and g > 200 for (r, g, b) in pixels) + + +class TestOrientationAndDimensions: + def test_landscape_display(self): + helper = DisplayHelper(128, 32) + assert helper.is_landscape() is True + assert helper.is_portrait() is False + + def test_portrait_display(self): + helper = DisplayHelper(32, 128) + assert helper.is_portrait() is True + assert helper.is_landscape() is False + + def test_square_display_is_neither(self): + # Pin: a square display is neither portrait nor landscape. + helper = DisplayHelper(64, 64) + assert helper.is_portrait() is False + assert helper.is_landscape() is False + + def test_get_center_position(self): + assert DisplayHelper(128, 32).get_center_position() == (64, 16) + + def test_get_center_position_floors_odd_dimensions(self): + assert DisplayHelper(65, 33).get_center_position() == (32, 16) + + def test_get_display_dimensions(self): + assert DisplayHelper(128, 32).get_display_dimensions() == (128, 32) + assert DisplayHelper(64, 64).get_display_dimensions() == (64, 64) diff --git a/test/test_display_manager.py b/test/test_display_manager.py index 63c49114..9ccb7773 100644 --- a/test/test_display_manager.py +++ b/test/test_display_manager.py @@ -1,6 +1,15 @@ +import os import pytest from unittest.mock import MagicMock, patch from PIL import ImageDraw + +# display_manager imports the hardware rgbmatrix module at import time unless +# EMULATOR=true. Use the emulator (same convention as +# test_display_dirty_tracking.py) so this file collects standalone instead of +# relying on collection order — the tests below patch RGBMatrix/ +# RGBMatrixOptions explicitly, so the underlying binding doesn't matter here. +os.environ.setdefault("EMULATOR", "true") + from src.display_manager import DisplayManager @pytest.fixture @@ -77,18 +86,26 @@ class TestDisplayManagerDrawing: assert dm.matrix.Clear.called def test_draw_text(self, test_config, mock_rgb_matrix): - """Test text drawing.""" + """Text drawn through draw_text must actually light pixels.""" + from PIL import Image, ImageDraw, ImageFont + import src.display_manager as dm_mod with patch.dict('os.environ', {'EMULATOR': 'false'}): - dm = DisplayManager(test_config) - - # Mock font - font = MagicMock() - - dm.draw_text("Test", 0, 0, font) - - # Verify draw_text was called (DisplayManager uses freetype/PIL) - # The actual implementation uses freetype or PIL, not graphics module - assert True # draw_text should execute without error + DisplayManager._instance = None + dm = DisplayManager(test_config, suppress_test_pattern=True) + # The fixture replaces the module's freetype with a MagicMock, + # which breaks draw_text's isinstance(font, freetype.Face) check + # (and silently swallows the draw). Give the mock a real class so + # isinstance works and the PIL path is taken. + dm_mod.freetype.Face = type("_FakeFace", (), {}) + # Start from a known-black canvas so the assertion below can only + # pass if draw_text itself lit something. + dm.image = Image.new('RGB', (dm.width, dm.height)) + dm.draw = ImageDraw.Draw(dm.image) + + dm.draw_text("Test", 0, 0, font=ImageFont.load_default()) + + assert dm.image.convert("L").getbbox() is not None, \ + "draw_text lit no pixels" def test_draw_image(self, test_config, mock_rgb_matrix): """Test image drawing.""" @@ -220,3 +237,45 @@ class TestDisplayManagerDoubleSided: suppress_test_pattern=True) assert dm.set_brightness(70) is True assert mock_rgb_matrix['matrix_instance'].brightness == 70 + + +class TestDisplayManagerOrientation: + """The orientation setting composes onto pixel_mapper_config for panels + mounted upside down, without disturbing a custom pixel_mapper_config.""" + + def _config(self, **hardware_overrides): + config = { + 'display': { + 'hardware': { + 'rows': 32, 'cols': 64, 'chain_length': 2, 'parallel': 1, + 'hardware_mapping': 'adafruit-hat-pwm', 'brightness': 90, + }, + 'runtime': {'gpio_slowdown': 2}, + }, + 'timezone': 'UTC', + 'plugin_system': {'plugins_directory': 'plugins'}, + } + config['display']['hardware'].update(hardware_overrides) + return config + + def test_default_orientation_leaves_pixel_mapper_config_untouched(self, mock_rgb_matrix): + DisplayManager._instance = None + with patch.dict('os.environ', {'EMULATOR': 'false'}): + DisplayManager(self._config(), suppress_test_pattern=True) + options = mock_rgb_matrix['options_class'].return_value + assert options.pixel_mapper_config == '' + + def test_orientation_180_appends_rotate_mapper(self, mock_rgb_matrix): + DisplayManager._instance = None + with patch.dict('os.environ', {'EMULATOR': 'false'}): + DisplayManager(self._config(orientation='180'), suppress_test_pattern=True) + options = mock_rgb_matrix['options_class'].return_value + assert options.pixel_mapper_config == 'Rotate:180' + + def test_orientation_180_composes_with_existing_pixel_mapper_config(self, mock_rgb_matrix): + DisplayManager._instance = None + with patch.dict('os.environ', {'EMULATOR': 'false'}): + DisplayManager(self._config(orientation='180', pixel_mapper_config='U-mapper'), + suppress_test_pattern=True) + options = mock_rgb_matrix['options_class'].return_value + assert options.pixel_mapper_config == 'U-mapper;Rotate:180' diff --git a/test/test_doc_links.py b/test/test_doc_links.py new file mode 100644 index 00000000..08f904af --- /dev/null +++ b/test/test_doc_links.py @@ -0,0 +1,43 @@ +"""Guard: relative markdown links in active docs must resolve. + +Scans repo-root *.md and docs/ (excluding docs/archive/, which is allowed +to rot). External URLs, mailto links, and pure anchors are skipped, as are +links inside fenced code blocks. +""" +import re +from pathlib import Path +from typing import Iterator + +PROJECT_ROOT = Path(__file__).resolve().parent.parent +LINK_RE = re.compile(r'\[[^\]]*\]\(([^)\s]+)\)') +FENCE_RE = re.compile(r'^(```|~~~)') + + +def _md_files() -> Iterator[Path]: + """Yield active markdown files (repo root + docs/, excluding docs/archive/).""" + yield from PROJECT_ROOT.glob('*.md') + for path in PROJECT_ROOT.glob('docs/**/*.md'): + if 'archive' not in path.parts: + yield path + + +def test_relative_markdown_links_resolve() -> None: + """Every relative markdown link outside code fences must resolve on disk.""" + broken = [] + for md in _md_files(): + in_fence = False + for lineno, line in enumerate(md.read_text(encoding='utf-8').splitlines(), 1): + if FENCE_RE.match(line.strip()): + in_fence = not in_fence + continue + if in_fence: + continue + for target in LINK_RE.findall(line): + if target.startswith(('http://', 'https://', 'mailto:', '#')): + continue + resolved = (md.parent / target.split('#')[0]).resolve() + if not resolved.exists(): + broken.append( + f'{md.relative_to(PROJECT_ROOT)}:{lineno} -> {target}' + ) + assert not broken, 'Broken relative markdown links:\n' + '\n'.join(broken) diff --git a/test/test_dynamic_team_resolver.py b/test/test_dynamic_team_resolver.py new file mode 100644 index 00000000..c562d0d9 --- /dev/null +++ b/test/test_dynamic_team_resolver.py @@ -0,0 +1,259 @@ +""" +Tests for src/dynamic_team_resolver.py (DynamicTeamResolver). + +Covers dynamic team expansion (AP_TOP_5/10/25), order-preserving dedup, +unknown dynamic-name dropping, rankings parsing, the fixed genuinely +class-shared rankings cache (fetch and clear_cache write through +DynamicTeamResolver._rankings_cache / _cache_timestamp), TTL expiry, +network-failure resilience, and the resolve_dynamic_teams module function. + +No real network: src.dynamic_team_resolver.requests.get is always patched. +""" + +import types +from unittest.mock import MagicMock, patch + +import pytest +import requests + +import src.dynamic_team_resolver as dtr_module +from src.dynamic_team_resolver import DynamicTeamResolver, resolve_dynamic_teams + + +TOP_TEAMS = ['UGA', 'MICH', 'OSU', 'TEX', 'ALA', 'ORE', 'PSU', 'ND', 'FSU', 'OU'] + + +def _rankings_payload(teams=None): + teams = TOP_TEAMS if teams is None else teams + return { + 'rankings': [{ + 'name': 'AP Top 25', + 'ranks': [ + {'current': i + 1, 'team': {'abbreviation': abbr}} + for i, abbr in enumerate(teams) + ], + }] + } + + +def _make_response(payload): + response = MagicMock() + response.json.return_value = payload + response.raise_for_status.return_value = None + return response + + +@pytest.fixture(autouse=True) +def reset_class_cache(): + """Reset the CLASS-level shared cache between tests.""" + DynamicTeamResolver._rankings_cache = {} + DynamicTeamResolver._cache_timestamp = 0 + yield + DynamicTeamResolver._rankings_cache = {} + DynamicTeamResolver._cache_timestamp = 0 + + +@pytest.fixture +def mock_get(): + with patch('src.dynamic_team_resolver.requests.get') as m: + m.return_value = _make_response(_rankings_payload()) + yield m + + +@pytest.fixture +def resolver(): + return DynamicTeamResolver() + + +# --------------------------------------------------------------------------- +# resolve_teams basics +# --------------------------------------------------------------------------- + +class TestResolveTeamsBasics: + def test_empty_list_returns_empty_no_http(self, resolver, mock_get): + assert resolver.resolve_teams([]) == [] + mock_get.assert_not_called() + + def test_no_dynamic_names_passthrough_no_http(self, resolver, mock_get): + assert resolver.resolve_teams(['UGA', 'AUB', 'LSU']) == [ + 'UGA', 'AUB', 'LSU'] + mock_get.assert_not_called() + + def test_expansion_inserted_in_place_order_preserved( + self, resolver, mock_get): + result = resolver.resolve_teams(['UGA', 'AP_TOP_5', 'AUB']) + + # UGA is also ranked #1, so dedup keeps its first occurrence; the + # top-5 expansion lands where AP_TOP_5 appeared, AUB stays after. + assert result == ['UGA', 'MICH', 'OSU', 'TEX', 'ALA', 'AUB'] + + def test_order_preserving_dedup(self, resolver, mock_get): + result = resolver.resolve_teams(['UGA', 'AP_TOP_5', 'UGA']) + + assert result == ['UGA', 'MICH', 'OSU', 'TEX', 'ALA'] + assert result.count('UGA') == 1 + assert result[0] == 'UGA' + + +# --------------------------------------------------------------------------- +# AP_TOP_N slicing +# --------------------------------------------------------------------------- + +class TestSlicing: + def test_top_n_counts_and_order(self, resolver, mock_get): + teams_25 = [f'T{i:02d}' for i in range(1, 26)] + mock_get.return_value = _make_response(_rankings_payload(teams_25)) + + top5 = resolver.resolve_teams(['AP_TOP_5']) + top10 = resolver.resolve_teams(['AP_TOP_10']) + top25 = resolver.resolve_teams(['AP_TOP_25']) + + assert top5 == teams_25[:5] + assert top10 == teams_25[:10] + assert top25 == teams_25 + + +# --------------------------------------------------------------------------- +# Unknown dynamic-looking names +# --------------------------------------------------------------------------- + +class TestUnknownDynamicNames: + def test_unknown_dynamic_looking_names_dropped(self, resolver, mock_get): + result = resolver.resolve_teams( + ['AP_TOP_100', 'TOP_10', 'RANKED_ALL', 'PLAYOFF_TEAMS']) + + assert result == [] + mock_get.assert_not_called() + + def test_top_substring_hazard(self, resolver, mock_get): + # Hazard pin: _is_potential_dynamic_team matches the substring + # 'TOP_' anywhere in the (upper-cased) name, so a team literally + # named 'TOP_GUN' is dropped as an unknown dynamic team too. + assert resolver.resolve_teams(['TOP_GUN']) == [] + + +# --------------------------------------------------------------------------- +# Rankings parsing +# --------------------------------------------------------------------------- + +class TestRankingsParsing: + def test_drops_zero_rank_and_empty_abbreviation_sorts_ascending( + self, resolver, mock_get): + payload = { + 'rankings': [{ + 'name': 'AP Top 25', + 'ranks': [ + {'current': 3, 'team': {'abbreviation': 'C3'}}, + {'current': 1, 'team': {'abbreviation': 'A1'}}, + {'current': 0, 'team': {'abbreviation': 'ZERO'}}, + {'current': 4, 'team': {'abbreviation': ''}}, + {'current': 2, 'team': {'abbreviation': 'B2'}}, + ], + }] + } + mock_get.return_value = _make_response(payload) + + rankings = resolver._fetch_ncaa_fb_rankings() + + assert list(rankings.keys()) == ['A1', 'B2', 'C3'] + assert list(rankings.values()) == [1, 2, 3] + + def test_empty_rankings_returns_empty_and_caches_nothing( + self, resolver, mock_get): + mock_get.return_value = _make_response({'rankings': []}) + + assert resolver._fetch_ncaa_fb_rankings() == {} + # Nothing was cached, so the next call hits HTTP again. + assert resolver._fetch_ncaa_fb_rankings() == {} + assert mock_get.call_count == 2 + + +# --------------------------------------------------------------------------- +# Shared class cache (fixed behavior) +# --------------------------------------------------------------------------- + +class TestSharedCache: + def test_cache_shared_across_instances(self, mock_get): + resolver1 = DynamicTeamResolver() + resolver1.resolve_teams(['AP_TOP_5']) + assert mock_get.call_count == 1 + + resolver2 = DynamicTeamResolver() + result = resolver2.resolve_teams(['AP_TOP_5']) + + # Post-fix: the class-level cache serves the second instance with + # ZERO additional HTTP calls. + assert result == TOP_TEAMS[:5] + assert mock_get.call_count == 1 + + def test_ttl_expiry_refetches(self, resolver, mock_get, monkeypatch): + resolver.resolve_teams(['AP_TOP_5']) + assert mock_get.call_count == 1 + + stamp = DynamicTeamResolver._cache_timestamp + monkeypatch.setattr( + dtr_module, 'time', types.SimpleNamespace(time=lambda: stamp + 3601)) + + resolver.resolve_teams(['AP_TOP_5']) + assert mock_get.call_count == 2 + + def test_clear_cache_through_one_instance_affects_all(self, mock_get): + resolver1 = DynamicTeamResolver() + resolver1.resolve_teams(['AP_TOP_5']) + assert mock_get.call_count == 1 + + resolver2 = DynamicTeamResolver() + resolver2.clear_cache() + + # Post-fix: clear_cache writes through the class, so resolver1 + # must refetch even though resolver2 did the clearing. + resolver1.resolve_teams(['AP_TOP_5']) + assert mock_get.call_count == 2 + + def test_module_function_benefits_from_class_cache(self, mock_get): + # resolve_dynamic_teams constructs a fresh resolver per call, but + # the class-shared cache means only the first call hits HTTP. + first = resolve_dynamic_teams(['AP_TOP_5']) + second = resolve_dynamic_teams(['AP_TOP_5']) + + assert first == second == TOP_TEAMS[:5] + assert mock_get.call_count == 1 + + +# --------------------------------------------------------------------------- +# Failure handling +# --------------------------------------------------------------------------- + +class TestFailureHandling: + def test_network_failure_drops_dynamic_keeps_static_caches_nothing( + self, resolver, mock_get): + mock_get.side_effect = [ + requests.exceptions.RequestException('boom'), + _make_response(_rankings_payload()), + ] + + result = resolver.resolve_teams(['UGA', 'AP_TOP_5']) + + # Dynamic name silently dropped, static name kept, nothing raises. + assert result == ['UGA'] + + # Nothing was cached on failure: a subsequent call refetches and + # succeeds. + result = resolver.resolve_teams(['UGA', 'AP_TOP_5']) + assert result == ['UGA', 'MICH', 'OSU', 'TEX', 'ALA'] + assert mock_get.call_count == 2 + + +# --------------------------------------------------------------------------- +# sport argument +# --------------------------------------------------------------------------- + +class TestSportArgument: + def test_sport_arg_ignored_for_expansion(self, resolver, mock_get): + # Pin: the sport argument is effectively ignored — each pattern + # carries its own sport ('ncaa_fb'), so passing sport='nfl' still + # expands from the college-football rankings. + result = resolver.resolve_teams(['AP_TOP_5'], sport='nfl') + + assert result == TOP_TEAMS[:5] + assert mock_get.call_count == 1 diff --git a/test/test_element_style.py b/test/test_element_style.py new file mode 100644 index 00000000..8d0535ab --- /dev/null +++ b/test/test_element_style.py @@ -0,0 +1,412 @@ +""" +Tests for src.element_style — the shared per-element style resolver behind +the x-style-elements system. + +The contract under test (defined by the plugin consumers: of-the-day, +ledmatrix-music, football-scoreboard): + +- defaults_from_schema_file parses BOTH declaration forms — the compact + x-style-elements map and hand-written customization blocks. +- expand_style_elements turns an x-style-elements declaration into the full + per-element blocks (plus layout offsets) the web-UI form renders. +- A config value counts as user-forced only when it genuinely differs from + the schema default; untouched (or schema-default-populated) configs + resolve to EXACTLY the classic font/size/color, keeping rendering + byte-identical. +- style() never raises; malformed input degrades to the classic style. +""" + +import json +import os + +import pytest +from PIL import ImageFont + +from src.element_style import ( + ElementStyleResolver, + defaults_from_schema, + defaults_from_schema_file, + expand_style_elements, + load_font, + resolve_font_path, +) + +# --------------------------------------------------------------------------- +# Schema fixtures +# --------------------------------------------------------------------------- + +# Compact declaration form (of-the-day's shape). +STYLE_ELEMENTS_SCHEMA = { + "type": "object", + "properties": { + "enabled": {"type": "boolean", "default": False}, + "customization": { + "type": "object", + "x-style-elements": { + "title_text": { + "title": "Title", + "font": {"default": "PressStart2P-Regular.ttf"}, + "size": {"default": 8, "min": 4, "max": 16}, + "color": {"default": [255, 255, 255]}, + "offsets": True, + }, + "body_text": { + "title": "Body Text", + "font": {"default": "4x6-font.ttf"}, + "size": {"default": 6, "min": 4, "max": 12}, + "color": {"default": [200, 200, 200]}, + "offsets": True, + }, + }, + }, + }, +} + +# Manual declaration form (the scoreboards' / music's shape). +MANUAL_SCHEMA = { + "type": "object", + "properties": { + "customization": { + "type": "object", + "properties": { + "status_text": { + "type": "object", + "properties": { + "font": {"type": "string", + "default": "4x6-font.ttf"}, + "font_size": {"type": "integer", "default": 6}, + }, + }, + "score_text": { + "type": "object", + "properties": { + "font": {"type": "string", + "default": "PressStart2P-Regular.ttf"}, + "font_size": {"type": "integer", "default": 10}, + "text_color": {"type": "array", + "default": [255, 255, 0]}, + }, + }, + "layout": {"type": "object", "properties": {}}, + }, + }, + }, +} + + +@pytest.fixture +def style_schema_path(tmp_path): + path = tmp_path / "config_schema.json" + path.write_text(json.dumps(STYLE_ELEMENTS_SCHEMA)) + return str(path) + + +@pytest.fixture +def manual_schema_path(tmp_path): + path = tmp_path / "config_schema.json" + path.write_text(json.dumps(MANUAL_SCHEMA)) + return str(path) + + +def _resolver(config, schema_path): + return ElementStyleResolver(config, defaults_from_schema_file(schema_path)) + + +# --------------------------------------------------------------------------- +# Schema parsing +# --------------------------------------------------------------------------- + +class TestDefaultsFromSchema: + def test_x_style_elements_defaults(self, style_schema_path): + defaults = defaults_from_schema_file(style_schema_path) + cust = defaults["customization"] + assert cust["title_text"] == {"font": "PressStart2P-Regular.ttf", + "font_size": 8, + "text_color": [255, 255, 255]} + assert cust["body_text"]["font_size"] == 6 + assert cust["body_text"]["text_color"] == [200, 200, 200] + + def test_manual_block_defaults(self, manual_schema_path): + defaults = defaults_from_schema_file(manual_schema_path) + cust = defaults["customization"] + assert cust["status_text"] == {"font": "4x6-font.ttf", "font_size": 6} + assert cust["score_text"]["text_color"] == [255, 255, 0] + assert "layout" not in cust + + def test_missing_file_degrades_to_empty(self, tmp_path): + defaults = defaults_from_schema_file(str(tmp_path / "nope.json")) + assert defaults == {"customization": {}} + + def test_malformed_file_degrades_to_empty(self, tmp_path): + path = tmp_path / "bad.json" + path.write_text("{not json") + assert defaults_from_schema_file(str(path)) == {"customization": {}} + + def test_schema_without_customization(self): + assert defaults_from_schema({"properties": {}}) == {"customization": {}} + + +class TestExpandStyleElements: + def test_expansion_generates_blocks(self): + expanded = expand_style_elements(STYLE_ELEMENTS_SCHEMA) + cust = expanded["properties"]["customization"]["properties"] + title = cust["title_text"] + assert title["x-style-managed"] is True + assert title["properties"]["font"]["default"] == \ + "PressStart2P-Regular.ttf" + assert title["properties"]["font_size"]["default"] == 8 + assert title["properties"]["font_size"]["minimum"] == 4 + assert title["properties"]["font_size"]["maximum"] == 16 + assert cust["body_text"]["properties"]["text_color"]["default"] == \ + [200, 200, 200] + + def test_expansion_generates_layout_offsets(self): + expanded = expand_style_elements(STYLE_ELEMENTS_SCHEMA) + layout = expanded["properties"]["customization"]["properties"]["layout"] + assert "title_text" in layout["properties"] + offsets = layout["properties"]["body_text"]["properties"] + assert offsets["x_offset"]["default"] == 0 + assert offsets["y_offset"]["default"] == 0 + + def test_input_schema_not_mutated(self): + before = json.dumps(STYLE_ELEMENTS_SCHEMA, sort_keys=True) + expand_style_elements(STYLE_ELEMENTS_SCHEMA) + assert json.dumps(STYLE_ELEMENTS_SCHEMA, sort_keys=True) == before + + def test_no_declaration_returns_same_object(self): + assert expand_style_elements(MANUAL_SCHEMA) is MANUAL_SCHEMA + empty = {"properties": {}} + assert expand_style_elements(empty) is empty + + def test_garbage_input_never_raises(self): + bad = {"properties": {"customization": {"x-style-elements": "nope"}}} + assert expand_style_elements(bad) is bad + + +# --------------------------------------------------------------------------- +# Classic identity: untouched configs resolve to the classic style +# --------------------------------------------------------------------------- + +class TestClassicIdentity: + def test_bare_config_resolves_classic(self, style_schema_path): + r = _resolver({}, style_schema_path) + style = r.style("title_text", classic_font="PressStart2P-Regular.ttf", + classic_size=8, classic_color=(255, 255, 255)) + assert style.font_name == "PressStart2P-Regular.ttf" + assert style.font_size == 8 + assert style.color == (255, 255, 255) + assert style.offset == (0, 0) + assert not style.user_forced + assert not style.user_forced_color + assert isinstance(style.font, ImageFont.FreeTypeFont) + assert style.font.size == 8 + + def test_schema_populated_config_is_not_an_override(self, style_schema_path): + # The web UI's save flow writes the full schema defaults into config + # on every save — that must not count as a user override. + config = {"customization": { + "title_text": {"font": "PressStart2P-Regular.ttf", "font_size": 8, + "text_color": [255, 255, 255]}, + "layout": {"title_text": {"x_offset": 0, "y_offset": 0}}, + }} + style = _resolver(config, style_schema_path).style( + "title_text", classic_font="PressStart2P-Regular.ttf", + classic_size=8, classic_color=(255, 255, 255)) + assert not style.user_forced + assert not style.user_forced_color + assert style.font_size == 8 + assert style.color == (255, 255, 255) + assert style.offset == (0, 0) + + def test_schema_default_falls_back_to_classic_not_schema_font( + self, manual_schema_path): + # Classic values and schema defaults can legitimately differ + # (football's status_text: schema says 4x6, classic loader used + # PressStart). A schema-default config value must yield the CLASSIC + # font, byte-identical to the old loader. + config = {"customization": {"status_text": {"font": "4x6-font.ttf", + "font_size": 6}}} + style = _resolver(config, manual_schema_path).style( + "status_text", classic_font="PressStart2P-Regular.ttf", + classic_size=6) + assert not style.user_forced + assert style.font_name == "PressStart2P-Regular.ttf" + assert style.font_size == 6 + + def test_same_font_object_from_cache(self, style_schema_path): + r = _resolver({}, style_schema_path) + s1 = r.style("title_text", classic_font="PressStart2P-Regular.ttf", + classic_size=8) + s2 = ElementStyleResolver({}, {}).style( + "title_text", classic_font="PressStart2P-Regular.ttf", + classic_size=8) + assert s1.font is s2.font + + +# --------------------------------------------------------------------------- +# User overrides engage +# --------------------------------------------------------------------------- + +class TestUserOverrides: + def test_font_override(self, style_schema_path): + config = {"customization": {"title_text": {"font": "4x6-font.ttf"}}} + style = _resolver(config, style_schema_path).style( + "title_text", classic_font="PressStart2P-Regular.ttf", + classic_size=8) + assert style.user_forced + assert style.font_name == "4x6-font.ttf" + assert style.font_size == 8 # size untouched -> classic + + def test_size_override(self, style_schema_path): + config = {"customization": {"title_text": { + "font": "PressStart2P-Regular.ttf", "font_size": 16}}} + style = _resolver(config, style_schema_path).style( + "title_text", classic_font="PressStart2P-Regular.ttf", + classic_size=8) + assert style.user_forced + assert style.font_name == "PressStart2P-Regular.ttf" + assert style.font_size == 16 + assert style.font.size == 16 + + def test_size_override_detected_vs_schema_default(self, manual_schema_path): + # font_size 8 differs from the schema default 6 -> forced. + config = {"customization": {"status_text": {"font": "4x6-font.ttf", + "font_size": 8}}} + style = _resolver(config, manual_schema_path).style( + "status_text", classic_font="PressStart2P-Regular.ttf", + classic_size=6) + assert style.user_forced + assert style.font_size == 8 + + def test_color_override(self, style_schema_path): + config = {"customization": {"title_text": {"text_color": [255, 0, 0]}}} + style = _resolver(config, style_schema_path).style( + "title_text", classic_font="PressStart2P-Regular.ttf", + classic_size=8, classic_color=(255, 255, 255)) + assert style.user_forced_color + assert not style.user_forced + assert style.color == (255, 0, 0) + + def test_offsets(self, style_schema_path): + config = {"customization": {"layout": { + "title_text": {"x_offset": 4, "y_offset": -2}}}} + r = _resolver(config, style_schema_path) + assert r.offset("title_text") == (4, -2) + assert r.offset("body_text") == (0, 0) + style = r.style("title_text", classic_font="PressStart2P-Regular.ttf", + classic_size=8) + assert style.offset == (4, -2) + + def test_offset_value_arbitrary_axis_and_strings(self, style_schema_path): + # The scoreboards read non-standard axes (away_x_offset) and configs + # can carry numeric strings/floats. + config = {"customization": {"layout": {"records": { + "away_x_offset": "3", "home_x_offset": 2.7}}}} + r = _resolver(config, style_schema_path) + assert r.offset_value("records", "away_x_offset", 0) == 3 + assert r.offset_value("records", "home_x_offset", 0) == 2 + assert r.offset_value("records", "missing_axis", 5) == 5 + + +# --------------------------------------------------------------------------- +# Defensive degradation +# --------------------------------------------------------------------------- + +class TestDegradation: + @pytest.mark.parametrize("config", [ + None, + {"customization": "not a dict"}, + {"customization": {"title_text": "not a dict"}}, + {"customization": {"title_text": {"font": 42, "font_size": "huge", + "text_color": "red"}}}, + {"customization": {"layout": {"title_text": {"x_offset": "junk"}}}}, + ]) + def test_bad_config_degrades_to_classic(self, config, style_schema_path): + style = _resolver(config, style_schema_path).style( + "title_text", classic_font="PressStart2P-Regular.ttf", + classic_size=8, classic_color=(10, 20, 30)) + assert not style.user_forced + assert not style.user_forced_color + assert style.font_name == "PressStart2P-Regular.ttf" + assert style.font_size == 8 + assert style.color == (10, 20, 30) + assert style.offset == (0, 0) + + def test_unknown_font_falls_back(self, style_schema_path): + config = {"customization": {"title_text": {"font": "no-such.ttf"}}} + style = _resolver(config, style_schema_path).style( + "title_text", classic_font="PressStart2P-Regular.ttf", + classic_size=8) + # The override IS honored as forced, but the face degrades safely. + assert style.user_forced + assert style.font is not None + + def test_empty_defaults_treats_config_as_reference_to_classic(self): + # No schema defaults at all: a config value equal to the classic + # value is not forced; a different one is. + r = ElementStyleResolver( + {"customization": {"e": {"font": "4x6-font.ttf"}}}, {}) + assert not r.style("e", classic_font="4x6-font.ttf", + classic_size=6).user_forced + assert r.style("e", classic_font="PressStart2P-Regular.ttf", + classic_size=6).user_forced + + +# --------------------------------------------------------------------------- +# Resolver plumbing the consumers rely on +# --------------------------------------------------------------------------- + +class TestResolverPlumbing: + def test_config_identity_exposed(self, style_schema_path): + # Consumers rebuild the resolver when the config dict is swapped: + # `resolver._config is not self.config`. + config = {"customization": {}} + r = _resolver(config, style_schema_path) + assert r._config is config + + def test_font_path_resolution_is_cwd_independent(self, tmp_path, + monkeypatch): + monkeypatch.chdir(tmp_path) # no assets/fonts under cwd + path = resolve_font_path("PressStart2P-Regular.ttf") + assert path is not None and os.path.isfile(path) + font = load_font("PressStart2P-Regular.ttf", 8) + assert isinstance(font, ImageFont.FreeTypeFont) + + def test_bdf_font_loads_as_freetype_face(self): + import freetype + font = load_font("5x7.bdf", 7) + assert isinstance(font, freetype.Face) + + @pytest.mark.parametrize("hostile", [ + "../../config/config.json", + "../secrets.txt", + "sub/dir/font.ttf", + "..", + ]) + def test_relative_font_name_with_path_components_is_rejected(self, hostile): + # font_name comes from plugin config (web-UI writable); a relative name + # carrying path separators would escape assets/fonts/ after os.path.join + # and let a config probe arbitrary paths. Only bare filenames resolve. + assert resolve_font_path(hostile) is None + + def test_bare_filename_still_resolves(self): + # The guard must not reject legitimate bare names. + assert resolve_font_path("PressStart2P-Regular.ttf") is not None + + def test_schema_manager_expands_on_load(self, tmp_path): + # The web-UI form path: SchemaManager.load_schema serves the + # expanded schema so the style blocks actually appear in the UI. + from src.plugin_system.schema_manager import SchemaManager + plugin_dir = tmp_path / "plugins" / "styled" + plugin_dir.mkdir(parents=True) + (plugin_dir / "config_schema.json").write_text( + json.dumps(STYLE_ELEMENTS_SCHEMA)) + (plugin_dir / "manifest.json").write_text(json.dumps({ + "id": "styled", "config_schema": "config_schema.json"})) + manager = SchemaManager(plugins_dir=tmp_path / "plugins", + project_root=tmp_path) + schema = manager.load_schema("styled") + assert schema is not None + cust = schema["properties"]["customization"]["properties"] + assert cust["title_text"]["x-style-managed"] is True + assert "title_text" in cust["layout"]["properties"] diff --git a/test/test_enum_option_labels.py b/test/test_enum_option_labels.py new file mode 100644 index 00000000..1e8d248e --- /dev/null +++ b/test/test_enum_option_labels.py @@ -0,0 +1,134 @@ +"""Guard: enum dropdowns in the plugin config form honour x-options.labels. + +The form derives an option's visible text from its value — underscores +replaced, title case applied ("day_first" -> "Day First"). That cannot +express every label a schema needs: "vs" reads as "Vs", and "abbrev" says +nothing about the "Sep 19" it produces. Schemas can supply x-options.labels +instead, the same convention the checkbox-group widget already uses. + +These tests extract the enum `. +ENUM_BLOCK_RE = re.compile( + r"(\{%\s*set enum_labels\s*=.*?)", re.S +) + + +def _shipped_enum_block() -> str: + """Return the live enum block in plugin_config.html — the ' + 'template changed shape and this guard needs updating' + ) + return match.group(1) + + +def _render(prop: dict, value=None) -> str: + """Render the shipped enum block with a minimal fixture.""" + env = Environment(loader=DictLoader({'f': _shipped_enum_block()}), + autoescape=True) + return env.get_template('f').render( + prop=prop, value=value, field_id='fid', full_key='k' + ) + + +def _option_labels(html: str) -> dict: + """Map each rendered option's value to its visible text.""" + return { + value: text.strip() + for value, text in re.findall( + r'', html, re.S + ) + } + + +def test_labels_are_used_when_supplied() -> None: + html = _render({ + 'enum': ['vs', 'date_time'], + 'x-options': {'labels': {'vs': 'VS', 'date_time': 'Date and time'}}, + }) + assert _option_labels(html) == {'vs': 'VS', 'date_time': 'Date and time'} + + +def test_unlabelled_values_keep_the_humanised_fallback() -> None: + """Schemas without labels must render exactly as they did before.""" + html = _render({'enum': ['day_first', 'weekday']}) + assert _option_labels(html) == {'day_first': 'Day First', 'weekday': 'Weekday'} + + +def test_partial_labels_fall_back_per_value() -> None: + """A labels map covering some values leaves the rest humanised.""" + html = _render({'enum': ['vs', 'day_first'], + 'x-options': {'labels': {'vs': 'VS'}}}) + assert _option_labels(html) == {'vs': 'VS', 'day_first': 'Day First'} + + +def test_option_values_are_unchanged_by_labelling() -> None: + """Labels are display-only: the submitted value stays the enum value.""" + html = _render({'enum': ['abbrev'], + 'x-options': {'labels': {'abbrev': 'Sep 19'}}}) + assert _option_labels(html) == {'abbrev': 'Sep 19'} + + +def test_selected_option_still_tracks_the_current_value() -> None: + """Labelling must not disturb which option is marked selected.""" + html = _render({'enum': ['abbrev', 'numeric'], + 'x-options': {'labels': {'abbrev': 'Sep 19'}}}, + value='numeric') + selected = re.search(r'