mirror of
https://github.com/ChuckBuilds/LEDMatrix.git
synced 2026-10-05 23:05:10 +00:00
feat(tools): MQTT bridge and Pixlet editor, ported onto the api_v3 split (#554)
* feat(tools): manage the MQTT bridge and Pixlet editor from the Tools tab PR #544's change, ported onto the api_v3 package split (#553). Identical behaviour; only the placement of the new code differs. The original added 508 lines to web_interface/blueprints/api_v3.py, which #553 deletes, so every hunk of it would conflict irreconcilably. Ported by AST: 26 new top-level items sorted to where the split puts each kind -- __init__.py 2 imports, 11 constants, 7 helpers starlark.py 4 routes (/starlark/editor/{apps,status,start,stop}) misc.py 2 routes (/integrations/mqtt-bridge{,/config}) Everything outside api_v3.py -- the Tools partial, the installer scripts, the JS tests -- applied unchanged. Routes: 111 from the split plus these 6 = 117, and the url-map snapshot is regenerated to match, which is exactly what test_api_v3_url_map.py is designed to make you do when routes are added. Full Python suite: 4,278 passed, 68 skipped, 0 failed. The JS tests this PR ships could not be run here -- node is not installed on this machine -- so test/js/dom/test_tools_sections.js is unverified. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014RRtqXDCnvnY6EQwhT5CV9 * fix(starlark): don't crash the pixlet editor's start/stop routes, and honor an operator-set PIXLET_EDITOR_HOST The AST-based port of #544 onto the api_v3 package split dropped `time` from starlark.py's import list. start_pixlet_editor() and stop_pixlet_editor() both call time.time()/time.sleep() directly, so every start (NameError building `state['started_at']`) and every stop that has to wait out the EXIT trap crashed with a 500. No test caught it because the route's own tests mock subprocess.Popen but never actually invoked it before now. Also carries over #544's later fix that this port branched before: env['PIXLET_EDITOR_HOST'] = '0.0.0.0' unconditionally overrode an operator who had already pinned PIXLET_EDITOR_HOST to loopback, forcing the unauthenticated `pixlet serve` process onto the LAN regardless (CodeQL CWE-1188). Switched to env.setdefault(...), same as api_v3.starlark.py's siblings already do for _pkg-owned names. Both fixes route the shared _pkg.time reference the rest of the package's route modules already use for anything a test might need to patch, rather than a bare `import time` local to this file. Ported the existing regression test from #544 (TestPixletEditorHostDefaultsButDoesNotOverride) onto this branch's module layout (web_interface.blueprints.api_v3.starlark instead of the old monolithic api_v3 module), which is what caught the NameError. Full suite: 4330 passed, 62 skipped, 2 failed -- identical on this branch and on origin/main (missing tzdata package breaks two timezone-alias tests in test_onboarding_checklist.py, unrelated to this change). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(api-v3): clear the six lint errors this rebase introduced All six were introduced by rebasing this branch onto the merged blueprint split, not by the split itself. Confirmed by diffing pyflakes output against main with line numbers normalised -- everything else it reports is present on main too and is the package's deliberate re-export pattern. starlark.py used _STARLARK_APPS_DIR three times without importing it (F821). The rebase resolved an import-list conflict as a union of both sides, and that symbol was on neither side of the conflict hunk, so it was silently lost. It is defined in __init__.py and is now imported like its neighbours. This was the only one of the six that would fail at runtime rather than merely lint. __init__.py imported contextlib twice (F811): the cherry-pick added one next to the existing import. Removed the duplicate; the original at line 19 is used. __init__.py imported signal purely to re-export it to starlark.py, so pyflakes saw it as unused (F401). signal is stdlib and does not need routing through the blueprint package, so starlark.py imports it directly and __init__.py no longer does. contextlib stays re-exported because this module genuinely uses it. _read_mqtt_bridge_config()'s local `config` shadowed the `config` submodule this module imports at the bottom for its route side effects (F811). Renamed to `settings`, with a comment saying why, since the name is otherwise the obvious one to reach for. Verified: pyflakes now reports nothing on this branch that main does not, the package imports, all nine route modules load, and 117 routes register, matching the pinned URL-map snapshot. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(api-v3): reject MQTT bridge bodies the endpoint cannot apply Two CodeRabbit findings on the bridge settings endpoint, both of which returned 200 while doing something other than what the caller asked. `request.get_json(silent=True) or {}` turned a missing or unparseable body -- and the JSON literals null, [] and false -- into an empty dict, which then satisfied the isinstance(data, dict) guard on the very next line. The guard was there to reject exactly those bodies. Dropping the `or {}` lets None fail it. The same `or {}` on /errors/clear is left alone: its docstring documents the body as optional, so an absent body legitimately means "use the defaults". The difference is that saving settings has nothing sensible to do with no body. `if data.get('clear_password'):` accepted any truthy value, and the string "false" is truthy in Python -- so a client echoing the field back as a string wiped a password it meant to keep. Now coerced through the package's existing _coerce_to_bool, which already maps 'true'/'on'/'1'/'yes' and nothing else. test_mqtt_bridge_config_endpoint.py covers both: five unusable body shapes plus a missing body, and clear_password across truthy and falsy spellings. Verified against the unfixed code -- reverting the body guard fails 5, reverting the coercion fails 3. Not changed here: CodeRabbit also asks this endpoint to reject MQTT credentials when TLS is off (CWE-319). That is a policy decision about the feature rather than a defect -- unencrypted MQTT on a trusted LAN is common and often deliberate -- so it is raised on the PR for a maintainer call instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: work through the remaining review findings on the editor and bridge allow_insecure_mqtt (CWE-319, requested): a password with TLS disabled crosses the network in cleartext. Refused now rather than merely warned about -- but refused, not forbidden, because unencrypted MQTT on a trusted LAN is a normal deliberate setup. allow_insecure_mqtt is the explicit acknowledgement, defaults false, and is coerced like the other booleans so the string "false" cannot switch the guard off. starlark.py:796 -- the supported service runs Flask threaded, so two start requests could each see running=False, each launch an editor, and the second state write replace the first PID, orphaning a process that holds the display down with nothing recording it. The check-launch-write sequence now takes a module-level lock. starlark.py:848 -- if the state write failed the route returned success with an editor running and no PID recorded: status and stop both reported no session while the display stayed down until the timeout expired. It now terminates the process group and returns an error. starlark.py:890 -- SIGKILL gives the script's EXIT trap no chance to run, so nothing hands the display back, yet the response said "the display is restarting". After an escalation the display is now restarted explicitly, and a failure to do so returns an error naming the manual step instead of a success. pixlet_config_editor.sh:184 -- find_pixlet supports Darwin but macOS ships no timeout(1); GNU coreutils installs it as gtimeout. Resolved up front so the failure lands before the display is stopped rather than after. pixlet_config_editor.sh:154 -- wildcard, loopback and an explicit interface address are three cases, not two. Collapsing the last two printed a URL saying "localhost" whenever PIXLET_EDITOR_HOST named a LAN address. tools.html:1254 -- escHtml does not encode single quotes, and the app id was interpolated into an inline onclick="startPixletEditor('...')", so a directory containing an apostrophe could break out of the JS string and run script. The handler binds with addEventListener and reads the id from dataset, where it is only ever parsed as an HTML attribute. Tests: test_mqtt_bridge_config_endpoint.py grows to 23 cases covering the opt-in in both directions. The tools DOM suite gains three guards asserting the edit buttons carry no inline onclick and pass the id via dataset -- those need jsdom and did not run here, so CI verifies them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(api-v3): log the traceback on the editor state-write failure The 848 fix answers 500 when the session state cannot be written, and logged that at error level -- but without exc_info, so the traceback never reached the log. test_web_error_detail.py guards exactly this: a handler returning 5xx must write an error-level record *with* the traceback and return the sanitized detail, because checking that merely something was logged is too weak. Caught by Core unit tests on the previous commit, not locally: the guard parses every module under web_interface/blueprints/api_v3 as one source, so it only fires once the whole package is read together. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chuck
@@ -3,11 +3,21 @@
|
||||
Routes decorate the shared `api_v3` Blueprint from ._common, so their
|
||||
endpoint names are unchanged by living here.
|
||||
"""
|
||||
import signal
|
||||
import threading
|
||||
|
||||
from web_interface.blueprints.api_v3 import (
|
||||
PROJECT_ROOT, Path, _find_pixlet_binary, _install_star_file, _standalone_render_starlark_app,
|
||||
_starlark_github_token, _starlark_manifest_lock, _validate_and_sanitize_app_id,
|
||||
_validate_starlark_app_path, _validate_timing_value, api_v3, describe_exception, json, jsonify,
|
||||
logger, os, request, shutil, subprocess, tempfile,
|
||||
PROJECT_ROOT, Path, _PIXLET_EDITOR_DEFAULT_PORT,
|
||||
_PIXLET_EDITOR_DEFAULT_TIMEOUT, _PIXLET_EDITOR_MAX_TIMEOUT,
|
||||
_PIXLET_EDITOR_SCRIPT, _PIXLET_EDITOR_STATE, _clear_pixlet_editor_state,
|
||||
_find_pixlet_binary, _install_star_file, _pixlet_editor_alive,
|
||||
_pixlet_editor_status, _read_pixlet_editor_state,
|
||||
_STARLARK_APPS_DIR, _standalone_render_starlark_app,
|
||||
_starlark_github_token, _starlark_manifest_lock,
|
||||
_validate_and_sanitize_app_id,
|
||||
_validate_starlark_app_path, _validate_timing_value, api_v3, contextlib,
|
||||
describe_exception, json, jsonify, logger, os, request, shutil,
|
||||
subprocess, tempfile,
|
||||
)
|
||||
import web_interface.blueprints.api_v3 as _pkg
|
||||
# Read through the module rather than bound by value: tests patch these
|
||||
@@ -710,3 +720,239 @@ def get_tronbyte_categories():
|
||||
except Exception as e:
|
||||
logger.exception("[Starlark] get_tronbyte_categories failed")
|
||||
return jsonify({'status': 'error', 'message': 'Failed to fetch categories', 'details': describe_exception(e)}), 500
|
||||
|
||||
|
||||
@api_v3.route('/starlark/editor/apps', methods=['GET'])
|
||||
def list_pixlet_editor_apps():
|
||||
"""Apps on disk that the editor can open.
|
||||
|
||||
Read from the directory rather than the loaded plugin: the editor works on
|
||||
files, and the plugin may not be loaded in this process at all.
|
||||
"""
|
||||
try:
|
||||
apps = []
|
||||
if _STARLARK_APPS_DIR.is_dir():
|
||||
for entry in sorted(_STARLARK_APPS_DIR.iterdir()):
|
||||
if not entry.is_dir():
|
||||
continue
|
||||
star_files = sorted(entry.glob('*.star'))
|
||||
name = entry.name
|
||||
manifest = entry / 'manifest.json'
|
||||
if manifest.is_file():
|
||||
try:
|
||||
with open(manifest, encoding='utf-8') as handle:
|
||||
data = json.load(handle)
|
||||
if isinstance(data, dict):
|
||||
name = data.get('name') or name
|
||||
except (OSError, json.JSONDecodeError):
|
||||
pass
|
||||
apps.append({
|
||||
'id': entry.name,
|
||||
'name': name,
|
||||
'editable': bool(star_files),
|
||||
'has_config': (entry / 'config.json').is_file(),
|
||||
})
|
||||
return jsonify({'status': 'success', 'data': {
|
||||
'apps': apps,
|
||||
'apps_dir': str(_STARLARK_APPS_DIR),
|
||||
'pixlet_available': _find_pixlet_binary() is not None,
|
||||
}})
|
||||
except Exception as e:
|
||||
logger.exception('Error listing editor apps')
|
||||
return jsonify({'status': 'error', 'message': 'Could not list apps',
|
||||
'details': describe_exception(e)}), 500
|
||||
|
||||
@api_v3.route('/starlark/editor/status', methods=['GET'])
|
||||
def get_pixlet_editor_status():
|
||||
"""Whether a session is running, and how long it has left."""
|
||||
try:
|
||||
return jsonify({'status': 'success', 'data': _pixlet_editor_status()})
|
||||
except Exception as e:
|
||||
logger.exception('Error reading editor status')
|
||||
return jsonify({'status': 'error', 'message': 'Could not read editor status',
|
||||
'details': describe_exception(e)}), 500
|
||||
|
||||
#: Flask runs threaded in the supported service, so two start requests can each
|
||||
#: observe running=False, each launch an editor, and the second state write
|
||||
#: replace the first PID -- orphaning a process that holds the display down with
|
||||
#: nothing left recording it. The check-launch-write sequence takes this lock.
|
||||
_EDITOR_START_LOCK = threading.Lock()
|
||||
|
||||
|
||||
def _terminate_editor_process(pid, wait_s=5):
|
||||
"""Signal an editor's process group and wait briefly for it to exit."""
|
||||
try:
|
||||
os.killpg(os.getpgid(pid), signal.SIGTERM)
|
||||
except (ProcessLookupError, PermissionError, OSError):
|
||||
with contextlib.suppress(ProcessLookupError, PermissionError):
|
||||
os.kill(pid, signal.SIGTERM)
|
||||
deadline = _pkg.time.time() + wait_s
|
||||
while _pkg.time.time() < deadline and _pixlet_editor_alive(pid):
|
||||
_pkg.time.sleep(0.25)
|
||||
if _pixlet_editor_alive(pid):
|
||||
with contextlib.suppress(ProcessLookupError, PermissionError, OSError):
|
||||
os.killpg(os.getpgid(pid), signal.SIGKILL)
|
||||
|
||||
|
||||
@api_v3.route('/starlark/editor/start', methods=['POST'])
|
||||
def start_pixlet_editor():
|
||||
"""Start an editing session for one app."""
|
||||
try:
|
||||
data = request.get_json(silent=True) or {}
|
||||
app_id = data.get('app_id')
|
||||
|
||||
app_dir, err = _validate_starlark_app_path(app_id or '')
|
||||
if err or not app_dir:
|
||||
return jsonify({'status': 'error', 'message': err or 'Invalid app_id'}), 400
|
||||
if not app_dir.is_dir():
|
||||
return jsonify({'status': 'error', 'message': f'No such app: {app_id}'}), 404
|
||||
if not any(app_dir.glob('*.star')):
|
||||
return jsonify({'status': 'error',
|
||||
'message': f'{app_id} has no .star file to edit'}), 400
|
||||
if not _PIXLET_EDITOR_SCRIPT.is_file():
|
||||
return jsonify({'status': 'error', 'message': 'Editor script not found'}), 404
|
||||
if _find_pixlet_binary() is None:
|
||||
return jsonify({'status': 'error',
|
||||
'message': 'Pixlet is not installed - install it first'}), 503
|
||||
|
||||
with _EDITOR_START_LOCK:
|
||||
current = _pixlet_editor_status()
|
||||
if current.get('running'):
|
||||
return jsonify({'status': 'error',
|
||||
'message': f"An editor session for '{current.get('app_id')}' is "
|
||||
f"already running; stop it first"}), 409
|
||||
|
||||
try:
|
||||
timeout_s = int(data.get('timeout') or _PIXLET_EDITOR_DEFAULT_TIMEOUT)
|
||||
except (TypeError, ValueError):
|
||||
return jsonify({'status': 'error', 'message': 'timeout must be a whole number'}), 400
|
||||
if not 60 <= timeout_s <= _PIXLET_EDITOR_MAX_TIMEOUT:
|
||||
return jsonify({'status': 'error',
|
||||
'message': f'timeout must be between 60 and '
|
||||
f'{_PIXLET_EDITOR_MAX_TIMEOUT} seconds'}), 400
|
||||
try:
|
||||
port = int(data.get('port') or _PIXLET_EDITOR_DEFAULT_PORT)
|
||||
except (TypeError, ValueError):
|
||||
return jsonify({'status': 'error', 'message': 'port must be a whole number'}), 400
|
||||
if not 1024 <= port <= 65535:
|
||||
return jsonify({'status': 'error', 'message': 'port must be between 1024 and 65535'}), 400
|
||||
|
||||
env = dict(os.environ)
|
||||
env['PIXLET_EDITOR_PORT'] = str(port)
|
||||
env['PIXLET_EDITOR_TIMEOUT'] = str(timeout_s)
|
||||
# A browser reaching this endpoint is remote by definition, so the
|
||||
# session needs to listen on more than loopback to be usable at all --
|
||||
# but only as a *default*. An operator who has already set
|
||||
# PIXLET_EDITOR_HOST (e.g. to keep it loopback-only even from the web
|
||||
# UI) must not have that overridden here.
|
||||
env.setdefault('PIXLET_EDITOR_HOST', '0.0.0.0')
|
||||
|
||||
log_path = Path(tempfile.gettempdir()) / 'ledmatrix_pixlet_editor.log'
|
||||
log_handle = open(log_path, 'w', encoding='utf-8') # noqa: SIM115 - owned by the child
|
||||
try:
|
||||
# start_new_session so the script leads its own process group: the
|
||||
# stop route signals the group, which is what lets the EXIT trap run
|
||||
# and hand the display back.
|
||||
process = subprocess.Popen( # nosec B603 - fixed script path, validated app_id
|
||||
['/bin/bash', str(_PIXLET_EDITOR_SCRIPT), app_dir.name],
|
||||
cwd=str(PROJECT_ROOT), env=env,
|
||||
stdout=log_handle, stderr=subprocess.STDOUT,
|
||||
start_new_session=True)
|
||||
finally:
|
||||
log_handle.close()
|
||||
|
||||
now = _pkg.time.time()
|
||||
state = {'pid': process.pid, 'app_id': app_dir.name, 'port': port,
|
||||
'timeout': timeout_s, 'started_at': now, 'deadline': now + timeout_s,
|
||||
'host': env['PIXLET_EDITOR_HOST'], 'log': str(log_path)}
|
||||
try:
|
||||
with open(_PIXLET_EDITOR_STATE, 'w', encoding='utf-8') as handle:
|
||||
json.dump(state, handle)
|
||||
except OSError as err:
|
||||
# The process is up but nothing records its PID: status and stop
|
||||
# would both report no session while the display stays down until
|
||||
# the timeout expires. Take the editor with us instead.
|
||||
# exc_info: this path answers 500, and the guard in
|
||||
# test_web_error_detail.py requires the traceback to reach the log
|
||||
# as well as the sanitized detail reaching the caller.
|
||||
logger.error('Started an editor session but could not record it: %s',
|
||||
err, exc_info=True)
|
||||
_terminate_editor_process(process.pid)
|
||||
return jsonify({'status': 'error',
|
||||
'message': 'Could not record the editor session; '
|
||||
'the editor was stopped.',
|
||||
'details': describe_exception(err)}), 500
|
||||
|
||||
logger.info('Pixlet editor started for %s on port %s (pid %s, %ss limit)',
|
||||
app_dir.name, port, process.pid, timeout_s)
|
||||
return jsonify({'status': 'success',
|
||||
'message': f"Editing '{app_dir.name}'. The display is stopped until "
|
||||
f"the session ends.",
|
||||
'data': _pixlet_editor_status()})
|
||||
except Exception as e:
|
||||
logger.exception('Error starting the pixlet editor')
|
||||
return jsonify({'status': 'error', 'message': 'Could not start the editor',
|
||||
'details': describe_exception(e)}), 500
|
||||
|
||||
@api_v3.route('/starlark/editor/stop', methods=['POST'])
|
||||
def stop_pixlet_editor():
|
||||
"""End the running session and give the display back."""
|
||||
try:
|
||||
state = _read_pixlet_editor_state()
|
||||
if not state or not _pixlet_editor_alive(state.get('pid')):
|
||||
_clear_pixlet_editor_state()
|
||||
return jsonify({'status': 'success', 'message': 'No editor session was running.',
|
||||
'data': {'running': False}})
|
||||
|
||||
pid = int(state['pid'])
|
||||
# SIGTERM the group, not the pid: bash forwards nothing to `timeout` and
|
||||
# its child on its own, and the trap needs to run to restart the display.
|
||||
try:
|
||||
os.killpg(os.getpgid(pid), signal.SIGTERM)
|
||||
except (ProcessLookupError, PermissionError, OSError) as err:
|
||||
logger.debug('Could not signal the editor process group: %s', err)
|
||||
with contextlib.suppress(ProcessLookupError, PermissionError):
|
||||
os.kill(pid, signal.SIGTERM)
|
||||
|
||||
# Give the trap a moment to stop pixlet and restart ledmatrix.
|
||||
deadline = _pkg.time.time() + 10
|
||||
while _pkg.time.time() < deadline and _pixlet_editor_alive(pid):
|
||||
_pkg.time.sleep(0.25)
|
||||
|
||||
escalated = False
|
||||
if _pixlet_editor_alive(pid):
|
||||
logger.warning('Editor session %s ignored SIGTERM; sending SIGKILL.', pid)
|
||||
with contextlib.suppress(ProcessLookupError, PermissionError, OSError):
|
||||
os.killpg(os.getpgid(pid), signal.SIGKILL)
|
||||
escalated = True
|
||||
|
||||
_clear_pixlet_editor_state()
|
||||
|
||||
if escalated:
|
||||
# SIGKILL gives the script's EXIT trap no chance to run, so nothing
|
||||
# has handed the display back. Reporting "the display is restarting"
|
||||
# here was simply untrue: restart it, and if that fails say so
|
||||
# rather than leave the panel dark behind a success response.
|
||||
result = _run_systemctl_command(
|
||||
['sudo', 'systemctl', 'start', 'ledmatrix.service'])
|
||||
if result.get('returncode') != 0:
|
||||
logger.error('Display restart after SIGKILL failed: %s',
|
||||
(result.get('stderr') or '').strip())
|
||||
return jsonify({
|
||||
'status': 'error',
|
||||
'message': 'Editor force-stopped, but the display could not be '
|
||||
'restarted automatically - start it manually.',
|
||||
'details': (result.get('stderr') or '').strip(),
|
||||
'data': {'running': False}}), 500
|
||||
return jsonify({'status': 'success',
|
||||
'message': 'Editor force-stopped; the display has been '
|
||||
'restarted.',
|
||||
'data': {'running': False}})
|
||||
|
||||
return jsonify({'status': 'success',
|
||||
'message': 'Editor stopped; the display is restarting.',
|
||||
'data': {'running': False}})
|
||||
except Exception as e:
|
||||
logger.exception('Error stopping the pixlet editor')
|
||||
return jsonify({'status': 'error', 'message': 'Could not stop the editor',
|
||||
'details': describe_exception(e)}), 500
|
||||
|
||||
Reference in New Issue
Block a user