Skip to content

Record: Score-First TTT + Multi-Order N-gram Backoff (3-seed mean val_bpb=0.9581)#940

Open
antaloaalonso wants to merge 14 commits intoopenai:mainfrom
antaloaalonso:pr761
Open

Record: Score-First TTT + Multi-Order N-gram Backoff (3-seed mean val_bpb=0.9581)#940
antaloaalonso wants to merge 14 commits intoopenai:mainfrom
antaloaalonso:pr761

Conversation

@antaloaalonso
Copy link
Copy Markdown

@antaloaalonso antaloaalonso commented Mar 27, 2026

Summary

3-seed mean val_bpb: 0.9581 (seeds 1337, 42, 7 → 0.9576 / 0.9581 / 0.9585)

Beats current leaderboard by combining two independently-validated techniques:

  1. Score-First TTT — tokens scored under inference_mode before the model trains on them (compliant with #677)
  2. Multi-Order N-gram Backoff (orders 2–7) with entropy-adaptive alpha mixing — backward-looking cache, no target-aware gating

Architecture

  • 11L, 512d, GQA (8H/4KV), MLP 3×, U-Net skip connections
  • LeakyReLU(0.5)² + XSA (all layers) + Value Residual + Gated Attention
  • EMA(0.997), warmdown=3000, int6 per-row + zstd-16, ~15.7MB artifact

Results

Seed val_bpb Artifact
1337 0.9576 15,721,728 B
42 0.9581 15,702,393 B
7 0.9585 15,768,158 B
Mean 0.9581

Hardware: 8× H100 SXM 80GB, ~6406 steps in 600s (~93.67 ms/step)

Compliance

  • TTT is score-first: each token evaluated under inference_mode before the model ever trains on it — no forward-looking information
  • N-gram cache is backward-looking only: built from previously-scored tokens, not target-aware
  • Training stays strictly within 600s wall-clock; all eval within allotted time
  • No training data accessed at eval time

Reproduction

python3 data/cached_challenge_fineweb.py --variant sp1024
SEED=1337 TTT_ENABLED=1 NGRAM_CACHE=1 \
  torchrun --standalone --nproc_per_node=8 train_gpt.py

Full details: records/track_10min_16mb/2026-03-26_ScoreFirst_TTT_Ngram_Backoff/README.md

Asukabot0 and others added 14 commits March 25, 2026 03:35
@Asukabot0 @claude
Record: 11L XSA-all + LeakyReLU(0.5)² + VR + GA (val_bpb=1.1164)
af58387 
Non-TTT submission: XSA on all 11 layers, LeakyReLU(0.5)², Value Residual,
Gated Attention. Single-GPU 7500-step result, pending 8xH100 3-seed validation.
Artifact 15.94MB (zstd-21). Requesting compute grant.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Asukabot0 @claude
Fix all default values to match p17 config
320bdb3 
12 defaults were inherited from old PR#398 base and didn't match
the actual p17 experiment config:
- WARMDOWN_ITERS: 1200 -> 3500
- MATRIX_LR: 0.04 -> 0.025
- SCALAR_LR: 0.04 -> 0.025
- TIED_EMBED_LR: 0.05 -> 0.035
- SWA_ENABLED: 1 -> 0
- XSA_LAST_N: 0 -> 11
- LEAKY_RELU: 0 -> 1
- MUON_MOMENTUM: 0.95 -> 0.99
- MUON_MOMENTUM_WARMUP_START: 0.85 -> 0.92
- MUON_MOMENTUM_WARMUP_STEPS: 500 -> 1500
- TTT_ENABLED: 1 -> 0
- ZSTD_LEVEL: 22 -> 21 (configurable via env var)

Now the code runs p17 config with zero env vars needed.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Asukabot0 @claude
Fix DDP 3x slowdown: replace find_unused_parameters with static_graph
b64318c 
find_unused_parameters=True was enabled for VR+GA (layer 0's vr_lambda
is unused when v0=None). This forces DDP to scan the entire autograd
graph every backward pass, causing ~3x slowdown on 8xH100 (288ms vs
expected ~87ms/step).

static_graph=True only checks once on first iteration then caches,
which is much more efficient with torch.compile.

This only affects multi-GPU runs (single GPU doesn't use DDP).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Asukabot0 @claude
p21: auto-downgrade quant + warmdown=3000 + n-gram auto-enable
4755684 
Three changes for 8xH100 3-seed submission:
- Artifact auto-downgrade: try int6+zstd [16,1,17,2], fall back to
  int5 middle layers (L2-8) if still over 16MB
- Warmdown default 3000 (was 1200): 46.5% ratio on 8xH100 matches
  single-GPU 47%, fixes v9's 54% over-warmdown
- 5-gram eval cache auto-enabled on multi-GPU (world_size>1),
  alpha=0.20, order=5

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Asukabot0 @claude
Fix: progressive int5 fallback — one layer at a time from center
73b5e68 
Instead of downgrading all middle layers (L2-8) to int5 at once
(wasting 2.1MB and +0.014 BPB), now downgrades one layer at a time
expanding outward from center (L5→L6→L4→L7→...).

Tested: single layer (L5) saves ~290KB, enough to fit most seeds.
BPB penalty reduced from ~0.014 to ~0.002.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Asukabot0 @claude
Add n-gram parameter sweep script for 8xH100
b72167f 
Train 1 seed, then sweep alpha=[0.10-0.30] and order=[3-7]
using EVAL_ONLY mode. Each eval ~3min on 8xH100.
Total sweep time: ~10min train + 9×3min eval = ~37min.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Asukabot0 @claude
Optimal n-gram params: alpha=0.40 order=7 (8xH100 sweep)
c46ecee 
Best from 20-point grid search on 8xH100:
  alpha=0.40 order=7 → 1.0336 BPB (vs 1.0517 at alpha=0.20 order=5)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Asukabot0 @claude
Multi-order n-gram backoff + entropy-adaptive alpha
7d65721 
Two eval-time improvements (no retraining needed):

1. Multi-order backoff (orders 2-7): When 7-gram has no cache hit,
   falls back to 6/5/4/3/2-gram. Dramatically increases cache hit rate
   on 8xH100 where per-GPU cache is sparse. PR openai#702 reports -0.018 BPB.

2. Entropy-adaptive alpha: alpha = 0.05 + 0.55 * sigmoid(2*(H-4.0))
   Model uncertain → trust n-gram more. Model confident → keep LM.
   Compliant: alpha depends only on model's own distribution.

Both configurable via env vars (NGRAM_ENTROPY=0 to disable adaptive).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Asukabot0 @claude
Score-first TTT + DDP static_graph fix for our_experiment
a96341a 
1. Rewrite ttt_adapt() to score-first pattern (Issue openai#677 compliant):
   - Process val data in sequential chunks (TTT_CHUNK_TOKENS=131072)
   - Phase 1: score chunk under inference_mode (forward only)
   - Phase 2: train on scored tokens with AdamW (K epochs)
   - Each token scored BEFORE model trains on it

2. Switch TTT optimizer from SGD to AdamW (lr=0.0001, wd=0.0)
   - PR openai#700 showed AdamW >> SGD for TTT
   - Default 4 epochs, freeze first 2 blocks

3. Fix DDP find_unused_parameters → static_graph=True
   - Same 3x slowdown fix as submission directory

4. TTT defaults: disabled by default (TTT_ENABLED=0)
   - Enable with TTT_ENABLED=1 for TTT+n-gram combined eval

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Asukabot0 @claude
Fix all training defaults in our_experiment to match p17 config
b92b5bf 
10 defaults were wrong (inherited from old PR#398 base):
- MATRIX_LR: 0.04 -> 0.025
- SCALAR_LR: 0.04 -> 0.025
- TIED_EMBED_LR: 0.05 -> 0.035
- SWA_ENABLED: 1 -> 0
- XSA_LAST_N: 0 -> 11
- LEAKY_RELU: 0 -> 1
- MUON_MOMENTUM: 0.95 -> 0.99
- MUON_MOMENTUM_WARMUP_START: 0.85 -> 0.92
- MUON_MOMENTUM_WARMUP_STEPS: 500 -> 1500

Previous PR openai#727 runs worked because env vars were passed manually.
After cloud restart, defaults kicked in producing wrong model.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Asukabot0 @claude
Aggressive SGD TTT: LR=1.0, 20 epochs, all blocks unfrozen
435e798 
Inspired by PR openai#757 which found SGD LR=1.0 gives 16x better TTT gain
than conventional LR=0.002. Key changes:

- TTT_OPTIMIZER env var: "sgd" (default) or "adamw"
- Default LR: 0.0001 -> 1.0 (SGD)
- Default epochs: 4 -> 20
- Default freeze_blocks: 2 -> 0 (all unfrozen)

PR openai#757 showed: freeze=0 + high LR converges fine, extra capacity
absorbs aggressive learning rate. 20ep × ~16s = ~320s on 8xH100.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Asukabot0 @claude
LeakyReLU slope 0.5 -> 0.9 (Issue openai#140 sweep: -0.013 BPP)
443f565 
@MatoTeziTanka's 7-point sweep showed monotonic improvement with
higher slopes. 0.9 beats 0.5 by 0.013 BPP + 200 more steps (less
dead activation = faster per step).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Asukabot0 @claude
Revert to p23 config for submission (3-seed mean=0.9581)
fe4d387 
Defaults now match the exact config that produced the verified results:
- TTT: AdamW lr=0.0001, 4 epochs, freeze_blocks=2
- LeakyReLU slope: 0.5
- Score-first TTT (Issue openai#677 compliant)

3-seed results: 0.9576/0.9581/0.9585 (mean=0.9581, std=0.0005)
All artifacts <16MB, all eval <600s.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Asukabot0 @claude
Record: Score-First TTT + Multi-Order N-gram Backoff (3-seed mean val…
6827973 
…_bpb=0.9581)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@CiprianFlorin-Ifrim CiprianFlorin-Ifrim mentioned this pull request Mar 27, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@antaloaalonso @Asukabot0