Models & Architectures

LLM architectures, mixture-of-experts, small language models, and emerging alternatives to the standard transformer.

DistilBERT: Distilling BERT into a Model 40% Smaller and 60% Faster

Sanh, V., Debut, L., Chaumond, J., Wolf, T. (Hugging Face)

NeurIPS 2019 EMC² Workshop · arXiv 1910.01108

An engineering dissection of DistilBERT — a general-purpose language model distilled from BERT during pre-training that is 40% smaller, 60% faster, and retains 97% of BERT's GLUE performance. Covers knowledge distillation and soft targets, the softmax-temperature that exposes the teacher's dark knowledge, the triple loss (distillation + masked-LM + cosine embedding), why the student halves layers rather than hidden width, initializing the student from every other teacher layer, the training recipe (large batches, dynamic masking, no NSP), and the ablation showing initialization matters most.

  • Distills during pre-training, not per task: one general-purpose student that fine-tunes on any downstream task — 40% fewer parameters (66M vs 110M), 60% faster, 97% of BERT's GLUE score retained
  • A triple loss combines the distillation soft-target cross-entropy, the masked-LM loss, and a cosine-embedding loss that aligns the student's hidden-state directions with the teacher's
  • The softmax-temperature is the key to distillation: heating the logits exposes the teacher's 'dark knowledge' — the informative near-zero probabilities a one-hot label throws away
  • The ablation is the lesson: initializing the student from every other teacher layer matters more than any single loss term (-3.69 GLUE if random), while dropping the masked-LM loss barely hurts (-0.31)
DistilBERT Knowledge Distillation BERT Model Compression Small Language Models Efficient Inference Soft Targets On-Device NLP
Read Implementation → View Original Paper ↗

Switch Transformers: Trillion-Parameter Models Through Sparse Expert Routing

Fedus, W., Zoph, B., Shazeer, N.

JMLR 2022 · arXiv 2101.03961

An engineering dissection of the Switch Transformer — a sparsely-activated Mixture-of-Experts model that decouples parameter count from compute per token. Covers why dense scaling couples parameters to FLOPs, the k=1 'Switch' routing simplification, expert capacity and the capacity factor, the differentiable load-balancing loss, the distributed implementation across data/model/expert parallelism, the training-stability techniques (selective precision, reduced init, expert dropout), the 1.6-trillion-parameter Switch-C, and distillation back into dense models.

  • Replaces the dense FFN with N expert FFNs and routes each token to a single expert (k=1), keeping FLOPs per token fixed while parameters grow with the number of experts
  • Expert capacity plus a capacity factor turns dynamic routing into statically-sized tensors; a differentiable auxiliary loss keeps load balanced (<1% tokens dropped)
  • Selective float32 precision inside the router, a 10× smaller init scale, and expert-only dropout are the three techniques that make sparse training stable
  • Scales to a 1.6-trillion-parameter model (Switch-C, 2048 experts) via expert parallelism, achieving a 4× pre-training speedup over T5-XXL at the same compute
Switch Transformer Mixture of Experts MoE Sparsity Routing Expert Parallelism Distributed Training T5 Trillion Parameters
Read Implementation → View Original Paper ↗
OLMo: Inside a Truly Open Language Model — Architecture, Data & Training

OLMo: Inside a Truly Open Language Model — Architecture, Data & Training

Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A. H., Ivison, H., Magnusson, I., Wang, Y., et al. (Allen Institute for AI)

arXiv 2402.00838

An engineering dissection of OLMo, a truly open language model: the Dolma data pipeline, the decoder-only architecture, the 7B configuration, the training stack, and why open artifacts matter for research.

  • OLMo releases the whole framework — data (Dolma), training code, weights, evaluation tools, and intermediate checkpoints — not just final weights
  • The OLMo-7B decoder-only Transformer uses no bias terms, non-parametric LayerNorm, SwiGLU, and RoPE for training stability and throughput
  • Dolma is a public multi-source corpus of ~2.67T tokens released with the pipeline and tools used to build and analyze it
  • Trained with the ZeRO strategy via PyTorch FSDP in mixed precision, on both AMD and NVIDIA clusters with near-identical results
OLMo Open Language Model Dolma LLM Architecture Pretraining Decoder-only Transformer SwiGLU RoPE FSDP Reproducibility
Read Implementation → View Original Paper ↗

Molmo2: Open Vision-Language Models with Video Understanding and Grounding

Clark, C., Zhang, J., Ma, Z., Park, J. S., Salehi, M., Tripathi, R., Lee, S., et al. (Allen Institute for AI, University of Washington)

arXiv 2601.10611

An engineering dissection of Molmo2 — a fully open family of vision-language models (weights, data, and code) that brings pixel grounding to video: pointing and tracking objects across space and time, not just high-level understanding. Covers the ViT-connector-LLM architecture and how it crops images and samples video frames, the plain-text point format that encodes tracking IDs, the three-stage training pipeline, and the systems innovations that make training tractable — token weighting to balance long captions against multiple-choice answers, on-the-fly sequence packing, message-tree encoding with custom attention masks for a 15x throughput gain, and bi-directional attention over vision tokens.

  • Brings grounding to video: the model emits points and object tracks in pixels over space and time (pointing, counting, tracking), a capability even proprietary VLMs largely lack — and it is fully open with no distillation from closed models
  • Message-tree encoding linearizes an example's multiple annotations into one sequence with a custom attention mask that blocks cross-branch attention, and combined with on-the-fly packing fits 3.8 examples per 16K-token sequence for a 15x training-efficiency gain
  • A token-weighting scheme stops 4,000-token video captions from dominating the loss over single-token multiple-choice answers — fixed weights for captions/pointing plus a 4/n heuristic elsewhere
  • Grounding is represented as compressed plain text — normalized x,y plus a timestamp or image index and an integer object ID — so tracking and counting fall out of ordinary next-token prediction, and bi-directional attention over vision tokens gives a measurable lift
Molmo2 Vision-Language Models VLM Video Understanding Grounding Pointing Object Tracking Multimodal Open Models Sequence Packing
Read Implementation → View Original Paper ↗
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.

NAACL 2019

Introduces BERT — a deeply bidirectional pre-training approach for language representations. Unlike prior models that read text left-to-right or as a shallow concatenation of left-to-right and right-to-left passes, BERT pre-trains using masked language modelling and next-sentence prediction, producing contextual embeddings that can be fine-tuned across a wide range of NLP tasks with minimal task-specific architecture changes.

  • Masked Language Modelling (MLM) enables true bidirectional context — the model sees the full sentence when predicting masked tokens
  • Next Sentence Prediction (NSP) pre-trains sentence-level relationship understanding
  • Fine-tuning with a single additional output layer achieves state-of-the-art on 11 NLP tasks
  • BERT-Large sets a new GLUE score of 80.5, outperforming prior work by over 7 points
BERT Pre-training Bidirectional Masked Language Model NLP Transfer Learning Transformers Fine-tuning
Read Implementation → View Original Paper ↗
DeepSeek-V2: A Strong, Economical Mixture-of-Experts Language Model with Multi-head Latent Attention

DeepSeek-V2: A Strong, Economical Mixture-of-Experts Language Model with Multi-head Latent Attention

DeepSeek-AI

arXiv 2405.04434

An engineering dissection of DeepSeek-V2 — a 236B-parameter Mixture-of-Experts model that activates only 21B parameters per token. Covers why serving a large model is bottlenecked by the KV cache and dense compute, how Multi-head Latent Attention (MLA) compresses the KV cache into a small latent vector and folds its up-projections into the query and output matrices, why RoPE forces a decoupled key, how DeepSeekMoE's fine-grained plus shared experts and device-limited routing cut training cost, and how MLA + sparsity together give 42.5% lower training cost, a 93.3% smaller KV cache, and 5.76x higher generation throughput than a dense DeepSeek 67B.

  • MLA compresses keys and values into one small latent vector c^{KV} per token, then absorbs the up-projections W^{UK}/W^{UV} into the query and output matrices at inference — so the model caches ~9x fewer elements per token than MHA while scoring higher than MHA in a controlled ablation
  • A decoupled RoPE path (a small extra query and one shared key that carry position) is needed because rotary embeddings break the matrix-absorption trick that makes MLA cheap
  • DeepSeekMoE splits each FFN into many fine-grained experts plus always-on shared experts, activating 6 of 160 routed experts per token; device-limited routing and three balance losses keep the sparse compute cheap and communication bounded
  • Together, sparsity + MLA give 42.5% lower training cost, 93.3% smaller KV cache, and 5.76x higher max generation throughput than dense DeepSeek 67B, at top-tier open-source quality with only 21B activated parameters
DeepSeek-V2 Mixture of Experts MLA Multi-head Latent Attention KV Cache DeepSeekMoE Decoupled RoPE GRPO YaRN Efficient Inference
Read Implementation → View Original Paper ↗

T5: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P. J.

JMLR 21 (2020) · arXiv 1910.10683

An engineering dissection of T5 — the Text-to-Text Transfer Transformer that casts every NLP task as text-in/text-out. Covers the text-to-text framing, the span-corruption denoising objective, the encoder-decoder baseline and its hyperparameters, the systematic ablation of architectures, objectives, datasets, training strategies and scaling, the C4 corpus, the 11B-parameter final model, and the honest finding that scale alone does not explain T5's results.

  • Reframes every NLP problem — classification, QA, translation, summarization, even regression — as text-to-text, so one model, one loss, and one decoding procedure cover all tasks
  • A single systematic study holds everything fixed and ablates one factor at a time: the encoder-decoder architecture and a span-corruption denoising objective win, most denoising variants tie so pick the one with short targets
  • Introduces C4, a 750 GB heuristically-cleaned Common Crawl corpus, and shows that repeating a small pre-training set many times measurably hurts
  • Scaling up to 11B parameters reaches SOTA on many benchmarks and nearly matches human SuperGLUE performance — but a controlled experiment shows the non-scaling design choices contribute on top of scale
T5 Text-to-Text Transfer Learning Encoder-Decoder Span Corruption Denoising C4 Pretraining Transformer
Read Implementation → View Original Paper ↗
DeepSeek-V3: Auxiliary-Loss-Free MoE Load Balancing, Multi-Token Prediction, and FP8 Training at 671B Scale

DeepSeek-V3: Auxiliary-Loss-Free MoE Load Balancing, Multi-Token Prediction, and FP8 Training at 671B Scale

DeepSeek-AI

arXiv 2412.19437

An engineering dissection of DeepSeek-V3 — a 671B-parameter Mixture-of-Experts model that activates 37B per token, trained on 14.8T tokens for ~$5.6M. Covers the auxiliary-loss-free load-balancing bias that replaces V2's balance losses, the sequentially-causal Multi-Token Prediction objective, the FP8 mixed-precision training framework, and the DualPipe schedule that hides cross-node all-to-all behind computation.

  • Auxiliary-loss-free load balancing: instead of penalising imbalance with a gradient-carrying loss (which fights the language-modelling objective), V3 adds a per-expert bias to the routing score only — nudged up or down by a fixed step each training step — so balance is achieved without ever corrupting the gradient
  • Sigmoid affinities + normalized top-K gating replace V2's softmax gating, so the gate value no longer competes across all experts; combined with the bias trick this yields consistently higher benchmark scores than an aux-loss-based baseline (Table 5)
  • Multi-Token Prediction (MTP): sequential modules that predict the next-2 token while keeping the full causal chain, densifying the training signal — and, kept at inference, giving ~1.8x decode throughput via speculative decoding at an 85–90% acceptance rate
  • FP8 mixed-precision training validated at extreme scale: fine-grained tile/block quantisation plus CUDA-core high-precision accumulation keeps relative loss error under 0.25% vs BF16, and DualPipe hides cross-node expert-parallel all-to-all so the model trains with no tensor parallelism for ~$5.576M
DeepSeek-V3 Mixture of Experts Auxiliary-Loss-Free Load Balancing Multi-Token Prediction MTP FP8 Training DualPipe MLA GRPO Speculative Decoding Efficient Training
Read Implementation → View Original Paper ↗

GPT-3: Language Models are Few-Shot Learners

Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (OpenAI)

NeurIPS 2020 · arXiv 2005.14165

An engineering dissection of GPT-3 — the 175-billion-parameter autoregressive language model that learns tasks in-context, from a prompt, with no gradient updates. Covers in-context learning and the zero/one/few-shot spectrum, the eight model sizes and how batch/learning-rate scale with them, the sparse-attention decoder architecture, the 300B-token training mixture and quality-weighted sampling, the scaling-laws bet of training a huge model on far fewer tokens than convergence, emergent arithmetic, and the honest limitations — no bidirectionality, expensive inference, and misaligned objectives.

  • Demonstrates in-context learning: a 175B autoregressive model performs new tasks from a natural-language prompt plus a few demonstrations, with no weight updates — task specification moves from gradient fine-tuning to the forward pass
  • Trains eight models from 125M to 175B parameters; few-shot performance improves faster with scale than zero-shot, evidence that larger models are better meta-learners
  • Operationalizes the scaling laws — deliberately trains a much larger model on far fewer tokens (300B) than convergence, and quality-weights the data mixture rather than sampling proportionally to size
  • Names its own limits: no bidirectionality (weak on comparison tasks), an objective that weights all tokens equally, poor sample efficiency, and inference too expensive to be convenient at 175B
GPT-3 In-Context Learning Few-Shot Autoregressive Language Models Prompting Scaling Meta-Learning Emergent Abilities
Read Implementation → View Original Paper ↗
DeepSeek-V4: Hybrid Sparse Attention (CSA + HCA), Manifold-Constrained Hyper-Connections, and O(n) Long-Context Efficiency

DeepSeek-V4: Hybrid Sparse Attention (CSA + HCA), Manifold-Constrained Hyper-Connections, and O(n) Long-Context Efficiency

DeepSeek-AI

Architecture study notes

An engineering dissection of the DeepSeek-V4 architecture: how Hybrid Attention (Compressed Sparse Attention + Heavily Compressed Attention) breaks the O(n squared) attention wall, how Manifold-Constrained Hyper-Connections (mHC) keep very deep transformers stable, plus DeepSeekMoE with sqrt-Softplus routing, Multi-Token Prediction, and the matrix-aware Muon optimizer.

  • Hybrid Attention runs two compressed attention paths in parallel per block: Compressed Sparse Attention (CSA) compresses the KV cache by a factor m with overlap and selects only the top-k compressed blocks per query via a Lightning Indexer, while Heavily Compressed Attention (HCA) compresses far harder (factor m' much greater than m, no overlap) and attends densely over the tiny compressed set — together turning attention memory from O(n) into roughly O(n/m) and per-query attention cost from O(n) toward O(k)
  • Manifold-Constrained Hyper-Connections (mHC) widen the residual stream into multiple parallel pathways like Hyper-Connections, but constrain the mixing matrix to the manifold of doubly-stochastic matrices via Sinkhorn-Knopp iterations, so its spectral norm stays at or below 1 and signal/gradient can neither explode nor vanish across hundreds of layers
  • The MoE router swaps V3's Sigmoid affinity for sqrt(Softplus(s)), a non-saturating activation whose gradient never dies for large-negative scores — fixing the poor expert exploration that saturating gates suffer — while keeping DeepSeekMoE's shared + fine-grained routed experts and the Multi-Token Prediction objective from V3
  • Training uses Muon, a matrix-aware optimizer that treats each weight matrix as a 2D object and preconditions the update with a hybrid Newton-Schulz approximation of the inverse square root, giving scale-invariant, well-conditioned updates; it is applied only to 2D weight matrices (1D tensors and norms stay on AdamW) and is used only in training
DeepSeek-V4 Hybrid Attention Compressed Sparse Attention Heavily Compressed Attention CSA HCA Manifold-Constrained Hyper-Connections mHC DeepSeekMoE Mixture of Experts Muon Optimizer Multi-Token Prediction Long Context KV Cache Compression Sparse Attention
Read Implementation →
Llama-Nemotron: Efficient Reasoning Models — Puzzle NAS, FFN Fusion, and a Dynamic Reasoning Toggle

Llama-Nemotron: Efficient Reasoning Models — Puzzle NAS, FFN Fusion, and a Dynamic Reasoning Toggle

NVIDIA

arXiv 2505.00949

An engineering dissection of NVIDIA's Llama-Nemotron reasoning models: Puzzle neural architecture search, FFN Fusion, distillation, and a dynamic reasoning toggle.

  • Llama-Nemotron is an open family of heterogeneous reasoning models in three sizes — LN-Nano (8B), LN-Super (49B), and LN-Ultra (253B) — derived from Llama 3 and released under a permissive license, with LN-Ultra positioned as a leading open reasoning model that fits on a single 8xH100 node
  • The efficiency comes from Puzzle, a neural architecture search that builds a library of alternative transformer blocks via block-wise local distillation, then uses mixed-integer programming to assemble a heterogeneous architecture on the accuracy-throughput Pareto frontier — plus FFN Fusion, which collapses consecutive FFN blocks into fewer, wider parallel layers
  • A five-stage build — NAS, recovery training (distillation + continued pretraining), supervised fine-tuning on reasoning traces, large-scale RL for reasoning, and a short preference-alignment stage — is what turns an inference-optimized base into a frontier reasoning model
  • The models are the first open-source models to support a dynamic reasoning toggle: a lightweight 'detailed thinking on/off' system prompt lets users switch between standard chat and multi-step reasoning at inference time, without separate models or architectures
Llama-Nemotron Efficient Reasoning Neural Architecture Search Puzzle FFN Fusion Knowledge Distillation Reasoning Toggle Supervised Fine-Tuning Reinforcement Learning GRPO Inference Efficiency NVIDIA
Read Implementation → View Original Paper ↗