Skip to content

Non-record: XSA-all + mHC + Full QAT (val_bpb=1.1211)#928

Open
autocode-rayes wants to merge 1 commit intoopenai:mainfrom
autocode-rayes:mhc-xsa-all-fullqat
Open

Non-record: XSA-all + mHC + Full QAT (val_bpb=1.1211)#928
autocode-rayes wants to merge 1 commit intoopenai:mainfrom
autocode-rayes:mhc-xsa-all-fullqat

Conversation

@autocode-rayes
Copy link
Copy Markdown

Three changes on PR #549 stack:

  • XSA on all 11 layers (was last 4)
  • Manifold-constrained hyper-connections (22 extra params)
  • Full-training QAT (LATE_QAT_THRESHOLD=1.0)

Seed 1337: sliding_window=1.1229, legal_ttt=1.1211
Artifact: 15.95 MB, 8xH100 SXM, 600s train + 482s eval

Three changes on PR openai#549 stack:
- XSA on all 11 layers (was last 4)
- Manifold-constrained hyper-connections (22 extra params)
- Full-training QAT (LATE_QAT_THRESHOLD=1.0)

Seed 1337: sliding_window=1.1229, legal_ttt=1.1211
Artifact: 15.95 MB, 8xH100 SXM, 600s train + 482s eval

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
aryanbhosale added a commit to aryanbhosale/parameter-golf that referenced this pull request Mar 28, 2026
slope 0.75 + LR 0.027 + warmdown 3700 (PR openai#977)
No SWA with QAT (PR openai#989)
QAT from 50% + range fix [-31,31]
mHC 22-param residual mixing (PR openai#928)
VE128 + no gated_attn + no value_residual (PR openai#549)
LZMA preset 7 compression (PR openai#999)
Muon TTT with NS3 (PR openai#999)
Entropy-adaptive TTT epochs 2/3/4 (PR openai#999)
Per-layer TTT LR (PR openai#995)
TTT momentum 0.95 (PR openai#995)
@MatoTeziTanka
Copy link
Copy Markdown

Community Review — Non-record: XSA-all + mHC + Full QAT (val_bpb=1.1211)

BPB: 1.1211 | Compliance: LOOKS CLEAN — score-first-per-chunk TTT (legal #1416/#1423 pattern)

What I found in the code (head SHA 8294a75b78ca, file records/track_non_record_16mb/2026-03-27_XSA-all_mHC_FullQAT_ParallelMuon_1.1211/train_gpt.py):

The TTT path at line 1083 implements the score-first-per-chunk pattern: each chunk is scored under torch.no_grad() / inference_mode() before the base_model.train() + SGD adaptation runs on that same chunk, with an is_last_chunk guard so the final chunk gets no adaptation pass. This is the structural shape the legal frontier uses (PRs #1416 erichroepke, #1423 aryanbhosale).

Per Issue #402 and Issue #677, TTT is legal when each token is scored before the adapter updates on it, and that's what the code does here — chunk ci is scored under weights adapted only on chunks 0..ci-1. No prequant_ttt_adapt_adamw(val_tokens, ...) multi-epoch fine-tune, no scored-region SLOT, no target-in-key n-gram cache.

CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.06s, dim=512, layers=11, vocab=1024, code=90110 B, SMOKE_TEST_PASS

Verdict: LOOKS CLEAN.

Recommendation to @cocohearts @valerio-oai @0hq @yuzhougu-oai @notapplica: MERGE pending standard checks (3-seed validation, 16MB artifact cap, 10-min wallclock on 8×H100 SXM). The compliance picture matches the legal reference frontier and no flags were raised by the classification pass.

Auto-classification caveat: this review was drafted by the AST-based classifier against a template derived from manually-reviewed cluster PRs (#1420, #1450, #1487, #1541, #1529, #1533, #1518). If I've misread a subtlety in your eval path — e.g., multi-epoch TTT that I mistook for single-pass, or a target-in-key lookup I missed in a helper function — please flag it and I'll re-run the audit manually.


Reviewed by @MatoTeziTankaThe Agora. CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.06s, dim=512, layers=11, vocab=1024, code=90110 B, SMOKE_TEST_PASS. Classification via deterministic AST-based classify_prs.py (pattern bank derived from ~65 manually-reviewed PRs earlier in the 2026-04-11 sweep). This review was auto-drafted from a template and spot-checked before posting — if the template misread your code, please call it out so I can iterate the classifier.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants