16L XSA-all + INT4 MLP QAT + GPTQ + EMA + Partial RoPE + 30ep Cosine TTT

[Bharath-970]

Submission: 16L XSA-all + INT4 MLP QAT + GPTQ-lite + EMA + Partial RoPE + 30ep Cosine TTT

val_bpb: pending 8×H100 run (expected ~1.02–1.05)

Techniques

Technique	Detail
XSA-all	All 16 layers share a single KV set from layer 0. Saves ~1.3MB vs XSA6, funding 2 extra layers
INT4 nibble MLP + QAT	MLP weights packed to 4-bit with STE quantization-aware training
INT6 attention	Attention weights quantized to INT6
GPTQ-lite clip search	Per-channel clip search for better quantization
EMA	Exponential moving average of weights, decay=0.999
Partial RoPE	RoPE applied to 25% of head dims only
Bigram hash	20480-entry bigram hash embedding (dim=128)
Trigram hash	10240-entry trigram hash embedding (dim=64)
LeakyReLU(0.5)²	MLP activation: leaky_relu(x, 0.5)²
SmearGate	Residual mix gating
Muon + WD	Muon optimizer with weight decay=0.04
U-Net skips	Skip connections between encoder/decoder layers
30ep Cosine TTT	Full-model AdamW on val tokens for 30 epochs, cosine LR decay from 5e-4, per-layer LR (3× MLP-out, 0.5× MLP-in)
Sliding window eval	stride=32 after TTT

Architecture

• 16 layers, 512 dim, 8 heads / 4 KV heads, MLP mult=3.5
• Warmdown 5000 iters, 20k total steps, train seq len=2048

Files

• records/track_10min_16mb/2026-03-25_16L_XSAall_GPTQ_EMA_PartialRoPE_TTT/train_gpt.py
• records/track_10min_16mb/2026-03-25_16L_XSAall_GPTQ_EMA_PartialRoPE_TTT/submission.json