From 13M Parameters to a Verified Small Code Executor

This is a publication-safe research preview. Full recipes, generated grammars, infrastructure identifiers, checkpoints, and several implementation details are intentionally withheld for possible future scientific publication.

Executive summary

The project started with a practical question: can a small Vietnamese-first code model be trained from random initialization, communicate at a basic level, ask for clarification or refuse when appropriate, and be evaluated by actual execution rather than loss alone?

The work progressed through data construction, a 13M-parameter trainer, empirical optimizer/objective tournaments, policy routing, execution-guided self-distillation, verified curricula, a 52M capacity study, deterministic identifier binding, depth curricula, and Blackwell saturation measurements.

The most important findings were:

• Capacity mattered: the 13M model hit a practical wall that a 52M probe crossed.
• Interface design mattered just as much: canonical signatures plus deterministic AST binding removed a major identifier bottleneck.
• Structural and depth coverage defined the useful generalization region.
• Self-distillation only became useful after the model could already produce successful trajectories.
• High GPU occupancy did not imply better quality.
• Promotion gates and checkpoint lineage prevented invalid runs from replacing working artifacts.

Research arc

The initial A-to-Z smoke pipeline built an AST-filtered Python corpus, tokenizer, verified Vietnamese code SFT data, and behavioral data for clarification, abstention, and refusal. A 12.66M model trained on 24.47M tokens reduced pretraining loss from 9.7476 to 5.8610 and SFT loss from 6.6090 to 2.2739. Its purpose was pipeline validation, not production capability.

V3.2 moved to one RTX PRO 6000 Blackwell and introduced deterministic resume, grouped splits, best-validation checkpointing, GQA, RoPE, RMSNorm, SwiGLU, Q/K normalization, and controlled auxiliary objectives. The 12.72M model reduced main validation loss from 5.8361 to 5.1485 across four corpus passes. The run was technically sound but severely data-constrained.

V4.1 compared AdamW, Muon, smaller token batches, masked-input regularization, and multi-token prediction from identical weights and multiple seeds. Plain next-token AdamW at roughly 32k tokens per update gave the best quality-per-credit tradeoff. Continued training improved validation from 5.0340 to 4.8261, yet executable pass@1 remained zero despite 100% syntax validity. This exposed the gap between language-model loss and contract-following code capability.

V5 added exact-signature completion, identifier augmentation, and a four-way policy router. The saved router reached macro-F1 0.8958, while code generalization remained zero. V6 then attempted incremental verified self-distillation without rerunning earlier phases. None of 8,064 sampled candidates passed verification, showing that self-distillation was premature.

V7 generated a family-split verified curriculum with 260 algorithm families and more than 14k training examples. The 13M model still produced zero executable pass@1. A separate 52M probe, however, reached 52% raw pass@1 on a 100-task synthetic sample.

The interface breakthrough

Many remaining failures were identifier-binding errors rather than algorithmic errors. The effective intervention was to canonicalize the signature, generate in a compact canonical namespace, deterministically bind identifiers back through an AST transformation, and then execute tests.

On the synthetic held-out grammar, the same 52M model reached approximately 83% test pass@1, 94% identifier robustness, and 100% syntax validity. These are strong in-scope results, not evidence of a general coding assistant.

Structure, depth, and self-training

A high score on “novel combinations” initially overstated compositional reasoning because the model was filling slots in one fixed template. Structural probes with reordered or repeated operations failed 0/6. After the curriculum included diverse program structures, the probe improved to 5/6 and a held-out synthetic set approached 99%.

Depth experiments showed a consistent coverage law. Training to depth 3 produced a sharp drop at depth 4; training to depth 6 made depth 6 strong and depth 7 partially usable; training to depth 8 yielded approximately 0.92 at depth 8, 0.88 at depth 9, and 0.74 at depth 10 before decaying further.

Verified self-training with base-data replay improved difficult-depth greedy accuracy only until pass@1 reached the sampling ceiling. At depth 7, pass@1, pass@8, and pass@32 converged near 0.633. Sampling more could not invent missing primitives or structures.

Inference-time chunking produced perfect results at much greater synthetic depths when the correct operation sequence was already available. This demonstrates a strong bounded executor under oracle decomposition, not autonomous planning.

Blackwell measurements

The hardware lab separated kernel ceilings from end-to-end training. Raw FFN GEMM reached 413.1 TFLOP/s in BF16 and 781.9 TFLOP/s in FP8. Peak sustained transformer training reached 224.7 TFLOP/s and MFU 0.547 on a roughly 1.45B model. In contrast, a naive end-to-end FP8 path was only 0.31x the BF16 token throughput on the tested configuration.

The practical conclusion was not “use more VRAM.” For 13M-52M models, the primary bottlenecks were data, interfaces, and evaluation. The Blackwell GPU was most valuable for fast ablations, batched generation, and verifier-heavy experiments.

Reliability lessons

One official 52M run became invalid because its data transformation silently dropped task descriptions. The validation gate correctly refused promotion. This failure reinforced three principles: validate schemas against real stage data, investigate metric discontinuities, and never let a candidate replace the active checkpoint solely because training completed.

The final system direction combined a small decoder, policy router, deterministic interface layer, execution verifier, validation-only promotion, persistent stage checkpoints, and lineage hashes.

Scope and next direction

What the project supports:

• A 13M capacity wall existed for this curriculum.
• A 52M model crossed that wall on verified synthetic tasks.
• Canonical interfaces and AST binding removed a major identifier bottleneck.
• Structural/depth coverage controlled the generalization boundary.
• Self-training helped only when verified solution headroom existed.

What it does not yet support:

• General-purpose code understanding.
• Autonomous decomposition at the reported 100% chunked-depth result.
• Transfer to real multi-file repositories or versioned APIs.
• State-of-the-art or paper-ready claims without independent benchmarks.

The next research target is autonomous Vietnamese specification-to-IR planning, followed by deterministic compilation and execution verification. This removes the oracle decomposition assumption and creates separately measurable planning and execution components.