rick/LEDMatrix
1
0
Fork 0
mirror of https://github.com/ChuckBuilds/LEDMatrix.git synced 2026-10-04 06:15:09 +00:00
Code Issues Packages Projects Releases Wiki Activity

refactor(api-v3): split the 10,469-line blueprint into a package (#553)

Browse Source 
* refactor(api-v3): split the 10,469-line blueprint into a package

web_interface/blueprints/api_v3.py held 111 routes, 56 helpers and 181
functions in one module -- 9% of the core by line count and three times the
next largest file. It becomes a package of nine route modules grouped by path
segment, plus __init__.py for the shared imports, constants, Blueprint and
helpers.

Every route module decorates the SAME api_v3 Blueprint object, so endpoint
names stay api_v3.<function>, the URL map is unchanged and app.py is untouched.
Verified: 111 routes before, 111 after, byte-identical rules, endpoints and
methods, and every endpoint still on the one blueprint.

  plugins   3,867   config    1,178   starlark  692   system  619
  fonts       452   misc        398   wifi      361   display 326   backup 212
  __init__  1,787 (imports, constants, Blueprint, 56 helpers)

Two things the URL-map check could not catch, both found by running the suite:

1. PROJECT_ROOT = Path(__file__).parent.parent.parent. Moving the code one
   directory deeper made that resolve to web_interface/ instead of the project
   root. Nothing failed at import; it surfaced as ~110 tests failing with 404s
   and "installation script not found", because every path built from it was
   one level too shallow. Now parents[3], and test_api_v3_url_map.py asserts
   PROJECT_ROOT/run.py exists so the next move cannot repeat it.

2. Module-attribute patching. Tests do
   monkeypatch.setattr(api_v3_module, "_BACKUP_EXPORT_DIR", ...) and a route
   module that binds such a name by value never sees the patch. The shared code
   therefore stays in __init__.py rather than moving to a _common submodule --
   it has to live on the module the tests patch -- and the eleven names tests
   patch are read back through the package (_pkg.X) instead of bound by value.
   Those eleven were found by AST-scanning every setattr in the test tree, not
   by guessing; "time" is among them, used to drive a fake clock through the
   second-resolution credential-backup filenames.

Test changes are confined to what genuinely moved: patch targets that now name
the owning route module, imports of helpers, and six tests that scan the api_v3
source as a file and now read the package directory.

Full suite: 4,278 passed, 68 skipped, 0 failed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RRtqXDCnvnY6EQwhT5CV9

* fix(api-v3): address CodeRabbit findings from the blueprint-split review

Fixes to the api_v3 package split (PR #553), one per finding verified
against the actual code:

- __init__.py: _redact_credentials only blanked scalar values under a
  credential-named key; a bare list of secrets under such a key (e.g.
  tokens: ["a", "b"]) passed through untouched, since the list branch
  recursed with no memory that its key looked like a credential. Nested
  dicts still walk normally (a documented, tested behaviour -- a container
  like secrets: {api_key: ..., note: ...} is a section name, not a value to
  blank outright), but any value reached under a credential-shaped key is
  now actually blanked.

- __init__.py: the OAuth helper script's raw stderr/stdout went to
  logger.error unredacted (CWE-532) right next to a comment claiming this
  was deliberate; the HTTP response already used the existing redact_text
  helper. Routed the log line through the same helper.

- __init__.py / starlark.py: the standalone Starlark manifest fallback
  (used when the plugin instance isn't loaded) read-modified-wrote
  manifest.json with no lock, unlike StarlarkAppsPlugin._update_manifest_safe
  (plugin-repos/starlark-apps/manager.py), which already holds an flock for
  the same file when the plugin is loaded. Added _starlark_manifest_lock,
  mirroring that pattern, and wrapped every standalone read-modify-write
  call site in it. The app-config update route also wrote config.json and
  the manifest as two separate, non-transactional writes (a second,
  distinct finding at the same call site); config.json is now rolled back
  if the manifest write that follows it fails.

- backup.py: restore options used bare bool() on values from the request,
  so {"restore_secrets": "false"} restored secrets anyway (bool("false") is
  True). Switched to the existing _coerce_to_bool helper already used for
  this exact purpose elsewhere in the package.

- config.py: an automated import-rewrite mangled four user-facing
  validation strings and their neighbouring comments -- "Invalid start
  time" had become "Invalid start _pkg.time" (and likewise for "end time")
  in both the schedule and dim-schedule per-day validation paths.

- display.py: `import _pkg.time as time_module` -- _pkg is a local alias
  for the package, not a real importable module, so this raised
  ModuleNotFoundError whenever a caller restarted an already-running
  display service via /display/on-demand/start, after the on-demand
  request was already written to cache. Fixed to `import time`. Audited
  the rest of the package for the same `_pkg.<module>` import mistake;
  every other `_pkg.` reference is a legitimate attribute read-through
  (`_pkg.time.time()`, `_pkg._get_starlark_plugin()`, ...), not a broken
  import statement.

- fonts.py: validate_file_upload's max_size_mb parameter is silently
  unused by that helper (it only checks filename/extension) -- the font
  upload route saved arbitrarily large files as a result. Added the same
  seek-and-check pattern already used for the sibling .star upload.

- wifi.py: two ad hoc, inconsistent bool coercions. POST
  /wifi/ap/auto-enable used bare bool(), so a JSON string "false" enabled
  it. POST /wifi/radio's enabled/force parsing recognized real bool and
  some strings but not int 1/0 (1 is True is False in Python). Factored one
  small _parse_bool_ish helper local to this file and used it at all three
  sites.

Not changed: the "unknown/misspelled restore option keys default to True"
half of the backup.py finding -- the file's own comment documents that a
missing key deliberately means "restore everything," matching the
already-existing JSON-parse-failure guard a few lines above it; only the
bool-coercion defect was a real bug.

Added or extended regression tests for every fix, following each area's
existing test conventions. Full suite: 4328 passed, 62 skipped, 2 failed
on both this branch and origin/main (missing tzdata package breaks two
timezone-alias tests in test_onboarding_checklist.py, unrelated to this
change) -- no new failures.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S3bPMESe2TfrGvbs1ef9c5

* fix(api-v3): reject unknown restore option keys

CodeRabbit's review of the blueprint split (#553) asked that
POST /backup/restore reject option keys outside RestoreOptions'
known set. The follow-up commit fixed the bool("false")-is-True
bug with _coerce_to_bool but never added the key check: a typo'd
or renamed key (e.g. "restoreSecrets") is silently ignored by
opts_dict.get(key, True), so the flag stays at its True default
and secrets get restored despite the caller's request saying
otherwise -- with no indication anything was wrong.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vmcwf5vMgYqdt8bJTZtiwb

* fix(api-v3): address CodeRabbit findings on the blueprint split

- _redact_credentials: blank scalar descendants of objects reached
  through a credential-owned list (e.g. tokens: [{"value": "secret"}])
  regardless of field name -- the existing name-based walk only
  protected direct dict values under a credential key, not list items.
- wifi.py: reject enabled/force/auto_enable_ap_mode values
  _parse_bool_ish can't recognize (400) instead of silently treating
  them as False, which could disable Wi-Fi or the radio itself.
- Starlark manifest locking: lock a stable manifest.json.lock sidecar
  instead of manifest.json itself, in both the standalone route path
  (_starlark_manifest_lock) and the plugin path
  (StarlarkAppsPlugin._save_manifest / _update_manifest_safe).
  manifest.json is replaced by an atomic rename on every write, which
  swaps in a fresh inode; a lock held on the old inode does not
  exclude a second locker that opens the path afresh right after the
  rename and gets the new inode, so two writers could race despite
  each holding "a lock". A sidecar that no write ever touches always
  resolves to the same inode for every locker.

Skipped as stale: the "serialize the complete manifest
read-modify-write" finding at api_v3/__init__.py -- every standalone
handler that calls _write_starlark_manifest is already wrapped in
_starlark_manifest_lock() on this branch.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(api-v3): re-check reconciliation findings by the reconciler's own rules

Both CodeRabbit findings on the merge commit, verified against the code first.

Major, plugins.py: the stale-findings filter derived its own notion of "in
config" and "on disk", and both were looser than the reconciliation module's.
set(load_config()) also contains system keys, the secrets-file keys load_config()
merges in, and non-dict values; and any directory holding a manifest.json
counted as installed even when that manifest does not parse. Either looseness
clears a finding that is still true -- and a secrets key read as a plugin is the
precise bug the filter exists to stop reporting, so reintroducing that asymmetry
while re-checking was the wrong way round.

The two extractions now live in state_reconciliation.py as config_plugin_ids()
and disk_plugin_ids(), with ignored_config_keys() and secrets_top_level_keys()
alongside. _get_config_state() and _get_disk_state() use them too, so there is
one definition rather than two that can drift. _get_disk_state() re-reads each
manifest for version/name after taking membership from the shared extractor;
that costs one extra small read per plugin on a path that runs once per boot.

Minor, the new test: the fixture assigned api_v3.config_manager and
api_v3.plugin_manager directly. Those live on a module-level blueprint
singleton, so the mocks leaked into every later test that imports api_v3 --
pointing at a tmp_path already deleted. Both now go through monkeypatch.setattr,
which restores them. This is the same pollution class that made an earlier test
in this session break seven unrelated ones, so it is worth getting right.

Five cases added for the parity itself: a secrets key, a system key and a
non-dict value must not clear an "installed but missing from config" finding,
and neither an unparseable manifest nor a .standalone-backup- directory may
count as installed. All five fail against the looser version.

Linux CI on the preceding commit: Core unit tests, plugin harness, CodeQL and
CodeRabbit all pass. Codacy reads action_required on every commit of this
branch including the first, so it is pre-existing and not from this work.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chuck
2026-09-11 10:07:38 -04:00
committed by GitHub
co-authored by Claude Opus 5
parent 0ab95586fb
commit bdb9a94033
32 changed files with 12151 additions and 10083 deletions
Download Patch File Download Diff File Expand all files Collapse all files
web_interface/blueprints/api_v3/misc.py
+398
View File
@@ -0,0 +1,398 @@
"""Routes with no larger group of their own: errors, integrations,
cache, sync, skins, logs, health and hardware.
Routes decorate the shared `api_v3` Blueprint from ._common, so their
endpoint names are unchanged by living here.
"""
from web_interface.blueprints.api_v3 import (
ErrorCode, Path, _JOURNALCTL, _SUDO, _get_display_service_status, api_v3,
describe_exception, error_response, get_error_aggregator, json, jsonify,
logger, os, request, subprocess, success_response,
)
import web_interface.blueprints.api_v3 as _pkg
# Read through the module rather than bound by value: tests patch these
# as module attributes, and a value binding would not see the patch.
# Several are also called from helpers that live in __init__, so the
# package is the only patch point that covers every caller.
@api_v3.route('/health', methods=['GET'])
def get_health():
"""Get system health status"""
try:
health_status = {
'status': 'healthy',
'timestamp': _pkg.time.time(),
'services': {},
'checks': {}
}
# Check web interface service
# Stamp the start _pkg.time before measuring against it -- reading it with a
# fallback of _pkg.time.time() and only assigning afterwards made the very
# first call subtract two separate clock reads, reporting a small
# negative uptime.
if not hasattr(get_health, '_start_time'):
get_health._start_time = _pkg.time.time()
health_status['services']['web_interface'] = {
'status': 'running',
'uptime_seconds': _pkg.time.time() - get_health._start_time
}
# Check display service
display_service_status = _get_display_service_status()
health_status['services']['display_service'] = {
'status': 'active' if display_service_status.get('active') else 'inactive',
'details': display_service_status
}
# Check config file accessibility
try:
if api_v3.config_manager:
test_config = api_v3.config_manager.load_config()
health_status['checks']['config_file'] = {
'status': 'accessible',
'readable': True
}
else:
health_status['checks']['config_file'] = {
'status': 'unknown',
'readable': False
}
except Exception as e:
health_status['checks']['config_file'] = {
'status': 'error',
'readable': False,
'error': 'see logs for details'
}
# Check plugin system
try:
if api_v3.plugin_manager:
# Try to discover plugins (lightweight check)
plugin_count = len(api_v3.plugin_manager.get_available_plugins()) if hasattr(api_v3.plugin_manager, 'get_available_plugins') else 0
health_status['checks']['plugin_system'] = {
'status': 'operational',
'plugin_count': plugin_count
}
else:
health_status['checks']['plugin_system'] = {
'status': 'not_initialized'
}
except Exception as e:
health_status['checks']['plugin_system'] = {
'status': 'error',
'error': 'see logs for details'
}
# Check hardware connectivity (if display manager available)
try:
snapshot_path = "/tmp/led_matrix_preview.png"
if os.path.exists(snapshot_path):
# Check if snapshot is recent (updated in last 60 seconds)
mtime = os.path.getmtime(snapshot_path)
age_seconds = _pkg.time.time() - mtime
health_status['checks']['hardware'] = {
'status': 'connected' if age_seconds < 60 else 'stale',
'snapshot_age_seconds': round(age_seconds, 1)
}
else:
health_status['checks']['hardware'] = {
'status': 'no_snapshot',
'note': 'Display service may not be running'
}
except Exception as e:
health_status['checks']['hardware'] = {
'status': 'unknown',
'error': 'see logs for details'
}
# Determine overall health
all_healthy = all(
check.get('status') in ['accessible', 'operational', 'connected', 'running', 'active']
for check in health_status['checks'].values()
)
if not all_healthy:
health_status['status'] = 'degraded'
return jsonify({'status': 'success', 'data': health_status})
except Exception as e:
logger.error("%s failed", request.path, exc_info=True)
return jsonify({
'status': 'error',
'message': 'An error occurred; see logs for details',
'details': describe_exception(e),
'data': {'status': 'unhealthy'}
}), 500
@api_v3.route('/hardware/status', methods=['GET'])
def get_hardware_status():
"""Return LED matrix hardware initialization status written by display_manager at startup."""
status_path = "/tmp/led_matrix_hw_status.json" # nosec B108
try:
with open(status_path) as f:
hw_data = json.load(f)
return jsonify({"status": "success", "data": hw_data})
except FileNotFoundError:
return jsonify({"status": "success", "data": {"ok": None, "error": "Display service not yet started"}})
except PermissionError:
logger.warning("Permission denied reading hardware status file; display service may be running as a different user")
return jsonify({"status": "success", "data": {"ok": False, "error": "Hardware status temporarily unavailable"}})
except json.JSONDecodeError:
logger.error("Failed to parse hardware status file", exc_info=True)
return jsonify({"status": "success", "data": {"ok": False, "error": "Hardware status file corrupted"}})
except Exception:
logger.error("Unexpected error reading hardware status", exc_info=True)
return jsonify({"status": "error", "message": "Unable to read hardware status"}), 500
@api_v3.route('/skins', methods=['GET'])
def list_skins():
"""List installed visual skins (docs/SKIN_SYSTEM.md).
Optional ?plugin_id=... filters to skins matching that plugin.
"""
try:
from src.skin_system import skin_runtime
plugin_id = request.args.get('plugin_id')
if plugin_id:
skins = skin_runtime.skins_for_plugin(plugin_id)
else:
# The discovery cache self-invalidates on directory/manifest
# mtime changes, so no force_refresh — keeps Pi disk I/O down.
skins = skin_runtime.discover_skins()
payload = []
for skin_id, manifest in sorted(skins.items()):
skin_dir = Path(manifest['_skin_dir'])
preview = manifest.get('preview')
payload.append({
'id': skin_id,
'name': manifest.get('name', skin_id),
'version': manifest.get('version'),
'author': manifest.get('author'),
'description': manifest.get('description', ''),
'skin_api_version': manifest.get('skin_api_version'),
'targets': manifest.get('targets', {}),
'modes': manifest.get('modes', []),
'has_preview': bool(preview and (skin_dir / preview).is_file()),
})
return jsonify({'status': 'success', 'data': {'skins': payload}})
except Exception as e:
logger.error('Error in list_skins', exc_info=True)
return jsonify({'status': 'error', 'message': 'An error occurred; see logs for details', 'details': describe_exception(e)}), 500
@api_v3.route('/logs', methods=['GET'])
def get_logs():
"""Get system logs from journalctl"""
try:
if not _JOURNALCTL:
return jsonify({'status': 'error', 'message': 'journalctl not found on this system'}), 503
# Get recent logs from journalctl
_cmd = ([_SUDO, _JOURNALCTL] if _SUDO else [_JOURNALCTL]) + [
'-u', 'ledmatrix.service', '-u', 'ledmatrix-web.service',
'-n', '100', '--no-pager', '--output=short-iso']
result = subprocess.run(
_cmd,
capture_output=True,
text=True,
timeout=5
)
if result.returncode == 0:
logs_text = result.stdout.strip()
return jsonify({
'status': 'success',
'data': {
'logs': logs_text if logs_text else 'No logs available from ledmatrix or ledmatrix-web service'
}
})
else:
return jsonify({
'status': 'error',
'message': f'Failed to get logs: {result.stderr}'
}), 500
except subprocess.TimeoutExpired:
return jsonify({
'status': 'error',
'message': 'Timeout while fetching logs'
}), 500
except Exception as e:
logger.error("%s failed", request.path, exc_info=True)
return jsonify({
'status': 'error',
'message': 'An error occurred; see logs for details',
'details': describe_exception(e)
}), 500
# Multi-Display Sync Endpoints
@api_v3.route('/sync/status', methods=['GET'])
def get_sync_status():
"""Return live multi-display sync status written by the display process."""
import os as _os
status_file = "/tmp/led_matrix_sync_status.json"
# Also surface config so the UI can show the configured role even before
# the display process has written a status file.
cfg_role = "standalone"
cfg_port = 5765
if api_v3.config_manager:
try:
cfg = api_v3.config_manager.load_config().get("sync", {})
cfg_role = cfg.get("role", "standalone")
cfg_port = int(cfg.get("port", 5765))
except Exception:
pass
if _os.path.exists(status_file):
try:
with open(status_file) as f:
live = json.load(f)
return jsonify({"status": "success", "data": live})
except Exception:
pass
# Status file not yet written — return config-only placeholder
return jsonify({
"status": "success",
"data": {
"role": cfg_role,
"port": cfg_port,
"state": "starting",
}
})
@api_v3.route('/cache/list', methods=['GET'])
def list_cache_files():
"""List all cache files with metadata"""
try:
if not api_v3.cache_manager:
# Initialize cache manager if not already initialized
from src.cache_manager import CacheManager
api_v3.cache_manager = CacheManager()
cache_files = api_v3.cache_manager.list_cache_files()
cache_dir = api_v3.cache_manager.get_cache_dir()
return jsonify({
'status': 'success',
'data': {
'cache_files': cache_files,
'cache_dir': cache_dir,
'total_files': len(cache_files)
}
})
except Exception as e:
logger.error('Error in list_cache_files', exc_info=True)
return jsonify({'status': 'error', 'message': 'An error occurred; see logs for details', 'details': describe_exception(e)}), 500
@api_v3.route('/cache/delete', methods=['POST'])
def delete_cache_file():
"""Delete a specific cache file by key"""
try:
if not api_v3.cache_manager:
# Initialize cache manager if not already initialized
from src.cache_manager import CacheManager
api_v3.cache_manager = CacheManager()
data = request.get_json(silent=True)
if not data or 'key' not in data:
return jsonify({'status': 'error', 'message': 'cache key is required'}), 400
cache_key = data['key']
# Delete the cache file
api_v3.cache_manager.clear_cache(cache_key)
return jsonify({
'status': 'success',
'message': f'Cache file for key "{cache_key}" deleted successfully'
})
except Exception as e:
logger.error('Error in delete_cache_file', exc_info=True)
return jsonify({'status': 'error', 'message': 'An error occurred; see logs for details', 'details': describe_exception(e)}), 500
@api_v3.route('/errors/summary', methods=['GET'])
def get_error_summary():
"""
Get summary of all errors for monitoring and debugging.
Returns error counts, detected patterns, and recent errors.
"""
try:
aggregator = get_error_aggregator()
summary = aggregator.get_error_summary()
return success_response(data=summary, message="Error summary retrieved")
except Exception as e:
logger.error(f"Error getting error summary: {e}", exc_info=True)
return error_response(
error_code=ErrorCode.SYSTEM_ERROR,
message="Failed to retrieve error summary",
status_code=500
)
@api_v3.route('/errors/plugin/<plugin_id>', methods=['GET'])
def get_plugin_errors(plugin_id):
"""
Get error health status for a specific plugin.
Args:
plugin_id: Plugin identifier
Returns health status and error statistics for the plugin.
"""
try:
aggregator = get_error_aggregator()
health = aggregator.get_plugin_health(plugin_id)
return success_response(data=health, message="Plugin health retrieved")
except Exception as e:
logger.error(f"Error getting plugin health for {plugin_id}: {e}", exc_info=True)
return error_response(
error_code=ErrorCode.SYSTEM_ERROR,
message=f"Failed to retrieve health for plugin {plugin_id}",
status_code=500
)
@api_v3.route('/errors/clear', methods=['POST'])
def clear_old_errors():
"""
Clear error records older than specified age.
Request body (optional):
max_age_hours: Maximum age in hours (default: 24, max: 8760 = 1 year)
"""
try:
data = request.get_json(silent=True) or {}
raw_max_age = data.get('max_age_hours', 24)
# Validate and coerce max_age_hours
try:
max_age_hours = int(raw_max_age)
if max_age_hours < 1:
return error_response(
error_code=ErrorCode.INVALID_INPUT,
message="max_age_hours must be at least 1",
context={'provided_value': raw_max_age},
status_code=400
)
if max_age_hours > 8760: # 1 year max
return error_response(
error_code=ErrorCode.INVALID_INPUT,
message="max_age_hours cannot exceed 8760 (1 year)",
context={'provided_value': raw_max_age},
status_code=400
)
except (ValueError, TypeError, OverflowError):
return error_response(
error_code=ErrorCode.INVALID_INPUT,
message="max_age_hours must be a valid integer",
context={'provided_value': str(raw_max_age)},
status_code=400
)
aggregator = get_error_aggregator()
cleared_count = aggregator.clear_old_records(max_age_hours=max_age_hours)
return success_response(
data={'cleared_count': cleared_count},
message=f"Cleared {cleared_count} error records older than {max_age_hours} hours"
)
except Exception as e:
logger.error(f"Error clearing old errors: {e}", exc_info=True)
return error_response(
error_code=ErrorCode.SYSTEM_ERROR,
message="Failed to clear old errors",
status_code=500
)