fix: recover fsGroup-remounted execution partition startup without replay #466

Merged
karsten merged 3 commits from fix/partition-fsgroup-startup into main 2026-11-14 17:01:53 +00:00
Owner

URGENT observed production startup outage during normal owning promotion, separate from model admission: Minerva harness CrashLoopBackOff14, Error Missing/unreadable partition. Read-only sidecar audit proved all865 referenced blobs exist, all865 exact SHA256 correct, all865 owned UID/GID1000, all865 mode0660 after fsGroup remount. Manifest retains849 execution records,1 event chunk,15 turn chunks. No missing/corrupt data and no ledger mutation/replay/reset performed.

Root cause: new PartitionedJson reader requires exact0600, unlike existing execution dependency fsGroup-compatible reader. Fix permits ONLY same-owner regular final-no-symlink600 or640/660 with group in actual kernel primary/supplementary membership; rejects unrelated groups, executable/world/special modes, corruption, symlinks. Cold load/qualification remains read-only, normal durable writer reseals0600 through opened FD and fsync. No authority/recovery/task/model/budget change.

11 new offline regressions plus existing partition/compatibility tests33pass; independent review PASS after supplementary-group case addressed. Regression filename local-* discovered by existing trusted mandatory Core workflow; no workflow changes. Full exact-head CI required before merge. Activation through normal image publication/owning GitOps only, never production chmod or ledger edits.

Follow-up before merge: current committed taskBookRef also points to fsGroup2770 directory/660 pages, so the same read-only hydration/normal-writer resealing fix now covers TaskBookStorage. Missing committed task-book directories/pages are wrapped to prevent ExecutionStore ENOENT empty-ledger fallback. Normal hydration no longer mkdirs/fsyncs task-book directories. Positive read-only actual PRIVATE COPY acceptance retains all849 executions and unchanged manifest dd6dd68f083d9760b60868a6df07bdcd9d55efc54fc2b03c2a8e004e8e771b13 with all dependency and current task-book pages. Earlier incomplete fixture with task-book pages absent exposed the empty-store defect; it was local only, fixed before activation, and is NOT an acceptance receipt. Current exact head be15b23 CI pending; earlier 5a56a5b CI189 success is superseded, not evidence for this head.

URGENT observed production startup outage during normal owning promotion, separate from model admission: Minerva harness CrashLoopBackOff14, Error Missing/unreadable partition. Read-only sidecar audit proved all865 referenced blobs exist, all865 exact SHA256 correct, all865 owned UID/GID1000, all865 mode0660 after fsGroup remount. Manifest retains849 execution records,1 event chunk,15 turn chunks. No missing/corrupt data and no ledger mutation/replay/reset performed. Root cause: new PartitionedJson reader requires exact0600, unlike existing execution dependency fsGroup-compatible reader. Fix permits ONLY same-owner regular final-no-symlink600 or640/660 with group in actual kernel primary/supplementary membership; rejects unrelated groups, executable/world/special modes, corruption, symlinks. Cold load/qualification remains read-only, normal durable writer reseals0600 through opened FD and fsync. No authority/recovery/task/model/budget change. 11 new offline regressions plus existing partition/compatibility tests33pass; independent review PASS after supplementary-group case addressed. Regression filename local-* discovered by existing trusted mandatory Core workflow; no workflow changes. Full exact-head CI required before merge. Activation through normal image publication/owning GitOps only, never production chmod or ledger edits. Follow-up before merge: current committed taskBookRef also points to fsGroup2770 directory/660 pages, so the same read-only hydration/normal-writer resealing fix now covers TaskBookStorage. Missing committed task-book directories/pages are wrapped to prevent ExecutionStore ENOENT empty-ledger fallback. Normal hydration no longer mkdirs/fsyncs task-book directories. Positive read-only actual PRIVATE COPY acceptance retains all849 executions and unchanged manifest dd6dd68f083d9760b60868a6df07bdcd9d55efc54fc2b03c2a8e004e8e771b13 with all dependency and current task-book pages. Earlier incomplete fixture with task-book pages absent exposed the empty-store defect; it was local only, fixed before activation, and is NOT an acceptance receipt. Current exact head be15b23 CI pending; earlier 5a56a5b CI189 success is superseded, not evidence for this head.
fix: hydrate remounted fsGroup partitions without weakening authority checks
All checks were successful
decision-interface-ci / full-validation (pull_request) Successful in 3m36s
5a56a5b28c
Author
Owner

PI-REVIEW-RECEIPT

State: PASS
Head: 5a56a5b28cc6
Summary: PASS: No P0 or P1 findings. One non-blocking race condition merits attention.
Findings: 1

Top findings:

  • P2 Resealing does not prevent an already-open group writer from changing a verified partition (src/partitioned-json.ts:185)
<!-- PI-REVIEW-RECEIPT --> ## PI-REVIEW-RECEIPT State: **PASS** Head: `5a56a5b28cc6` Summary: PASS: No P0 or P1 findings. One non-blocking race condition merits attention. Findings: 1 Top findings: - **P2** Resealing does not prevent an already-open group writer from changing a verified partition (src/partitioned-json.ts:185)
fix: retain fsGroup task-book pages and fail closed on missing authority
All checks were successful
decision-interface-ci / full-validation (pull_request) Successful in 3m22s
be15b23eac
Author
Owner

PI-REVIEW-RECEIPT

State: PASS
Head: be15b23eac01
Summary: PASS: No P0 or P1 findings. The new permission checks introduce a non-blocking compatibility regression for previously accepted owner-read-only storage.
Findings: 1

Top findings:

  • P2 Preserve hydration of previously accepted owner-read-only pages and directories (src/task-book-storage.ts:20)
<!-- PI-REVIEW-RECEIPT --> ## PI-REVIEW-RECEIPT State: **PASS** Head: `be15b23eac01` Summary: PASS: No P0 or P1 findings. The new permission checks introduce a non-blocking compatibility regression for previously accepted owner-read-only storage. Findings: 1 Top findings: - **P2** Preserve hydration of previously accepted owner-read-only pages and directories (src/task-book-storage.ts:20)
fix: preserve read-only retained task-book qualification
All checks were successful
decision-interface-ci / full-validation (pull_request) Successful in 3m15s
1421d3d44b
Author
Owner

PI-REVIEW-RECEIPT

State: PASS
Head: 1421d3d44ba8
Summary: PASS — no P0 or P1 findings. One non-blocking edge case remains when staging an owner-read-only task-book page.
Findings: 1

Top findings:

  • P2 Staging cannot reuse an owner-read-only task-book page (src/task-book-storage.ts:41)
<!-- PI-REVIEW-RECEIPT --> ## PI-REVIEW-RECEIPT State: **PASS** Head: `1421d3d44ba8` Summary: PASS — no P0 or P1 findings. One non-blocking edge case remains when staging an owner-read-only task-book page. Findings: 1 Top findings: - **P2** Staging cannot reuse an owner-read-only task-book page (src/task-book-storage.ts:41)
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
harness/core!466
No description provided.