mirror of
https://github.com/ChuckBuilds/LEDMatrix.git
synced 2026-08-21 02:19:07 +00:00
perf(health): stop rewriting a health record on every healthy cycle
Every successful plugin update called record_success(), which persisted the record unconditionally. In steady state the only fields that had changed were total_successes and last_success_time -- a counter and a timestamp that health_monitor surfaces for display and that nothing reads back after a restart. Nothing alerts on the age of last_successful_update; it is carried in the metrics dataclass and shown. Measured on a rig running 24 plugins, all steady-state (0 consecutive failures, circuit closed): a five-minute sample caught 22 health-file rewrites, about 4.4 a minute or 6,300 a day. Each write is ~400 bytes through cache_manager.set(), which writes a file per call, so each one costs a filesystem block plus an ext4 journal write. That lands on an SD card, where the unit of cost is an erase-block cycle rather than the bytes involved, and where wear is what eventually kills the card. Two cards have already failed on the other rig with the same signature -- unreadable block device, EIO on exec, sshd unable to read its host keys. The circuit breaker still has to survive a restart, so the write is kept for exactly the fields it is rebuilt from: consecutive_failures, circuit_state, circuit_opened_time, half_open_start_time. A failure, a circuit opening and a recovery are all still written the moment they happen. In-memory state is updated every time either way, so the health API and web UI show what they always did. Tested: 100 healthy cycles now perform zero writes after the first, the counters remain accurate in memory, and a failure, a recovery and a half-open-to-closed transition each still reach disk. One test kills and rebuilds the tracker from the cache to prove the breaker's state genuinely survives what is no longer written. Mutation-checked both ways: persisting unconditionally again fails the steady-state test, and widening _DURABLE_FIELDS to include last_success_time fails it too. The 46 existing health tests pass. (cherry picked from commit14abea2d24) (cherry picked from commit0f77bd2345)
This commit is contained in:
@@ -178,11 +178,21 @@ class PluginHealthTracker:
|
||||
)
|
||||
return self._health_state[plugin_id]
|
||||
|
||||
# Fields the circuit breaker is rebuilt from after a restart. Everything
|
||||
# else in a health record is reporting, read only for display.
|
||||
_DURABLE_FIELDS = ('consecutive_failures', 'circuit_state',
|
||||
'circuit_opened_time', 'half_open_start_time')
|
||||
|
||||
def _durable(self, state: Dict[str, Any]) -> tuple:
|
||||
"""The part of a health record whose loss would change behaviour."""
|
||||
return tuple(state.get(field) for field in self._DURABLE_FIELDS)
|
||||
|
||||
def record_success(self, plugin_id: str) -> None:
|
||||
"""Record a successful plugin execution."""
|
||||
state = self.get_health_state(plugin_id)
|
||||
current_time = time.time()
|
||||
|
||||
durable_before = self._durable(state)
|
||||
|
||||
# Reset consecutive failures
|
||||
state['consecutive_failures'] = 0
|
||||
state['total_successes'] = state.get('total_successes', 0) + 1
|
||||
@@ -198,9 +208,20 @@ class PluginHealthTracker:
|
||||
# Shouldn't happen, but handle it
|
||||
state['circuit_state'] = CircuitState.CLOSED.value
|
||||
state['circuit_opened_time'] = None
|
||||
|
||||
self._save_health_state(plugin_id, state)
|
||||
|
||||
|
||||
# A healthy plugin reports success every cycle, and in that steady state
|
||||
# the only fields changed above are a counter and a timestamp that
|
||||
# nothing reads back after a restart. Persisting them anyway rewrites a
|
||||
# small file per plugin per cycle: on a rig running 24 plugins, a
|
||||
# five-minute sample measured 22 rewrites, about 4.4 a minute or 6,300 a
|
||||
# day. Those land on an SD card, where the cost is an erase-block cycle
|
||||
# rather than the 400 bytes involved, and where wear is what eventually
|
||||
# kills the card.
|
||||
# In-memory state is still updated every time, so the health API and web
|
||||
# UI show exactly what they did before; only the write is skipped.
|
||||
if self._durable(state) != durable_before:
|
||||
self._save_health_state(plugin_id, state)
|
||||
|
||||
def record_failure(self, plugin_id: str, error: Optional[Exception] = None) -> None:
|
||||
"""Record a failed plugin execution."""
|
||||
state = self.get_health_state(plugin_id)
|
||||
|
||||
Reference in New Issue
Block a user