fix(recovery): prevent recursive handover amplification and poisoned session retries #375
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "fix/bounded-recovery-handover"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Reproduced production failure
Repeated interrupted recovery embeds the entire previous custom handover as JSON in the next handover. Production sessions grew 0.8 MB -> 496 MB, first causing repeated HTTP413, then Invalid string length during first persistence. The SDK creates its exclusive file before serializing all entries, leaving a metadata prefix and an unflushed in-memory manager; retries then fail EEXIST forever while the execution remains running. No provider config/user approval can repair this.
Generic fix (all Harness/PI installations)
Project evidence before JSON serialization; hard cap serialized UTF-8 JSON at 48KiB, preserve correlation IDs and current bounded excerpts, explicitly mark omissions/truncation. Prior handovers become source-session references, never recursive copies. Original transcript files/ledgers remain untouched. Preserve source file/session references and execution-bound checkpoint evidence, including when a nonempty transcript exists. Runtime rejects seeds >64KiB BEFORE SDK branch mutation or file creation. No replay grant, no invented absence of effects.
Tests
Fail-before/pass-after repeated recovery and large-evidence reproducers. Forty successive recoveries bounded. Real SDK new-file and existing metadata-only partial-file persistence/reopen tests; Unicode/JSON escape budget and idempotent seeding; oversized rejection before any mutation. Updated two test adapters to the actual ArchiveEntry/runtime status contracts (no weakening of recovery assertions). Full suite 2058 pass / 1 existing skip / zero fail, typecheck/build/diff check pass.
Deployment required: normal merge/release; graceful checkpoint restart clears the already-poisoned in-memory manager, preserving original files and durable tasks. Then verify original owner, actual model activity and fleet artifacts, not merely a scheduling receipt.
COMBINED LIVE AND FLEET PROOF COMPLETE for generic orchestration fixes #375/#379/#382/#383. Published release0.1.269 revision73418505750996a83bac175740551deb0c529331 (or verified descendant) activated on all THIRTEEN current active installations, including newly present Grace/Minerva. Each installed artifact passed 40 bounded synthetic recovery transfers, exact correlation preservation and guard checks, plus active monotonic decision revision>32, same-owner followUp tool/module and oversized readiness projection with unchanged authority/candidate. Serving API revision/authenticated identity checked where configured; configured API-disabled Matrix installations retained that policy and actual process-entry identity was verified. No new Slack/API/credentials/admin rights copied or enabled. Native graceful owner updates; Kubernetes normal GitOps PR merges and Recreate/checkpoint recovery.
Actual local original owner559dbf4d recovered from its poisoned in-memory session with old transcripts preserved. Parent used supported consolidation: 15 exact old records archived, revision16 accepted, followUp attached. Readiness probe proved former captured-evidence budget prevented all backend calls; new projection + first-safe-action policy selects exact candidate5 on configured actual reviewer, refuses missing-approval control. Production order became[0,1,3,4,2], original same owner ran18 real read/bash calls, independently completed source followUp using existing integrated commit7602e575a53cb1ed3599812e4e5e091003cc3e84 and exact-head green compatibility run141 (Core2084pass/1skip, Plugins139pass). No duplicate writer/push/merge performed by maintenance. Subsequent original-parent coordination completed another followUp; original live hold stays open, not falsely marked complete.
Final Core#383 local suite2095pass/1existing skip/zero fail; typecheck/build and exact-head full CI green before regular non-force merge. Fleet evidence ~/.cache/hayate-maintenance/orchestration-fleet-269/summary.json, per-instance installed JSON, original-writer-progress.json and test/provider logs. Image/control/config receipts in the same directory. Separate Randy volume startup mitigation agent-randy-waterhouse#176 passed normal required CI and merged, preserving UID/GID/permissions, to avoid re-chowning >740k files on every unchanged PVC mount.