gemma-4-12B

conformance ✓ spec 1.0 gemma4 The Model Spec of google/gemma-4-12B in TheFormalEra.Spec.Registry. The constants below are transcribed from its published config.json, and the official transformers.Gemma4UnifiedForConditionalGeneration is checked against them.

At a glance


Model key	`google/gemma-4-12B`
Parameters (text model)	11,907,350,320
Layers	48 (40 sliding-window attention, 8 full attention)
Hidden size	3,840
MLP width	15,360 (gelu_pytorch_tanh)
Vocabulary	262,144
Norm	`w·x̂`, ε = 1e-06
Embedding	tied to the LM head, × √3840
Logits	soft-capped at 30
Hugging Face class	`Gemma4UnifiedForConditionalGeneration`
Shape domain	`T` ∈ [1, 262,144] tokens, `B` ∈ [1, 256] sequences, `S` ∈ [1, 262,144] cached positions, page size 16

What to watch for

Layer i is global iff i % 6 = 5 (gemma4_12b_pattern): 8 global layers, 40 sliding-window layers with a window of 1024.
Sliding layers: 16 query heads, 8 KV heads, head dim 256, RoPE θ = 10,000. Global layers: 16 query heads, 1 KV head, head dim 512, values = keys, RoPE θ = 1,000,000 with only the first 64 frequencies active.
Attention scale is 1 (the q/k norms take its place). Values get an unweighted RMSNorm (attn.v_norm).
Sandwich norms: post_mix_norm after attention and post_ffn_norm after the MLP; the MLP residual has a learned per-layer scalar.
GeGLU with the tanh approximation of GELU (geluMul). Embeddings are scaled by √3840. Logits are soft-capped: 30 · tanh(z / 30).
Scope: the text-only path. The vision and audio towers are out of scope.

Compute-graph nodes

Use these paths in Spec Keys: google/gemma-4-12B@1.0:layers[*].<node> for per-layer nodes, google/gemma-4-12B@1.0:<node> for the others.

Node	Op Spec	Layers	Formula	Constants
`embed`	`embed`	—	`y[t] = √3840 · w[tokens[t]]`	mul=√3840, norm=False
`input_norm`	`rmsnorm`	all 48	`y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · w[i]`	H=3840, eps=1e-06
`attn.qkv_proj`	`linear`	40: all but 5, 11, …, 47	`y[t,n] = Σₖ x[t,k] · w[n,k]`	K=3840, N=8192
`attn.qk_norm`	`rmsnorm`	40: all but 5, 11, …, 47	`per head: y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · w[i]`	D=256, eps=1e-06
`attn.v_norm`	`rmsnormUnit`	40: all but 5, 11, …, 47	`per head: y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε)`	D=256, eps=1e-06
`attn.rope`	`rope`	40: all but 5, 11, …, 47	`rotate pairs (i, i + 128) by pos · θ^(-2i/256), i < 128; coordinates ≥ 256 unchanged`	D=256, active=128, denom=256, rot=256, theta=10000
`attn.core`	`slidingDecodeGQA`	40: all but 5, 11, …, 47	`o[b,h] = Σ_{s < seq_lens[b]} softmax_s(1 · ⟨q[b,h], k[b,s,h / 2]⟩) · v[b,s,h / 2] over the last 1024 positions`	D=256, heads=16, kv_heads=8, scale=1, window=1024
`attn.o_proj`	`linear`	40: all but 5, 11, …, 47	`y[t,n] = Σₖ x[t,k] · w[n,k]`	K=4096, N=3840
`post_mix_norm`	`rmsnorm`	all 48	`y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · w[i]`	H=3840, eps=1e-06
`attn_residual`	`residual`	all 48	`y = x + h`	H=3840, layer_scalar=False, r=1
`post_attn_norm`	`rmsnorm`	all 48	`y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · w[i]`	H=3840, eps=1e-06
`mlp.gate_up`	`linear`	all 48	`y[t,n] = Σₖ x[t,k] · w[n,k]`	K=3840, N=30720
`mlp.act`	`geluMul`	all 48	`y[t,i] = gelu_tanh(gate_up[t,i]) · gate_up[t,15360+i]`	I=15360, act=gelu_pytorch_tanh
`mlp.down`	`linear`	all 48	`y[t,n] = Σₖ x[t,k] · w[n,k]`	K=15360, N=3840
`post_ffn_norm`	`rmsnorm`	all 48	`y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · w[i]`	H=3840, eps=1e-06
`mlp_residual`	`residual`	all 48	`y = s · (x + 1 · h)`	H=3840, layer_scalar=True, r=1
`attn.qkv_proj`	`linear`	8: 5, 11, …, 47	`y[t,n] = Σₖ x[t,k] · w[n,k]`	K=3840, N=8704
`attn.qk_norm`	`rmsnorm`	8: 5, 11, …, 47	`per head: y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · w[i]`	D=512, eps=1e-06
`attn.v_norm`	`rmsnormUnit`	8: 5, 11, …, 47	`per head: y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε)`	D=512, eps=1e-06
`attn.rope`	`rope`	8: 5, 11, …, 47	`rotate pairs (i, i + 256) by pos · θ^(-2i/512), i < 64; coordinates ≥ 512 unchanged`	D=512, active=64, denom=512, rot=512, theta=1000000
`attn.core`	`decodeGQA`	8: 5, 11, …, 47	`o[b,h] = Σ_{s < seq_lens[b]} softmax_s(1 · ⟨q[b,h], k[b,s,h / 16]⟩) · v[b,s,h / 16]`	D=512, heads=16, kv_heads=1, scale=1
`attn.o_proj`	`linear`	8: 5, 11, …, 47	`y[t,n] = Σₖ x[t,k] · w[n,k]`	K=8192, N=3840
`final_norm`	`rmsnorm`	—	`y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · w[i]`	H=3840, eps=1e-06
`lm_head`	`lmHead`	—	`z = x wᵀ, logits = 30 · tanh(z / 30)`	mul=1, softcap=30, tied=True

Nodes listed twice have different constants in different layers (for example, sliding-window and global attention). A kernel annotated with layers[*].<node> is verified against each variant.

Tensors of the main nodes

Node	Inputs and outputs
`embed`	`tokens` [T], `y` [T, 3840]
`input_norm`	`x` [T, 3840], `y` [T, 3840]
`attn.qkv_proj`	`x` [T, 3840], `y` [T, 8192]
`attn.qk_norm`	`q` [T, 16, 256], `k` [T, 8, 256], `q_out` [T, 16, 256], `k_out` [T, 8, 256]
`attn.v_norm`	`v` [T, 8, 256], `v_out` [T, 8, 256]
`attn.rope`	`q` [T, 16, 256], `k` [T, 8, 256], `pos` [T], `q_out` [T, 16, 256], `k_out` [T, 8, 256]
`attn.core`	`q` [B, 16, 256], `k` [B, S, 8, 256], `v` [B, S, 8, 256], `seq_lens` [B], `o` [B, 16, 256]
`attn.o_proj`	`x` [T, 4096], `y` [T, 3840]
`post_mix_norm`	`x` [T, 3840], `y` [T, 3840]
`attn_residual`	`x` [T, 3840], `h` [T, 3840], `y` [T, 3840]
`post_attn_norm`	`x` [T, 3840], `y` [T, 3840]
`mlp.gate_up`	`x` [T, 3840], `y` [T, 30720]
`mlp.act`	`gate_up` [T, 30720], `y` [T, 15360]
`mlp.down`	`x` [T, 15360], `y` [T, 3840]
`post_ffn_norm`	`x` [T, 3840], `y` [T, 3840]
`mlp_residual`	`x` [T, 3840], `h` [T, 3840], `y` [T, 3840]
`final_norm`	`x` [T, 3840], `y` [T, 3840]
`lm_head`	`x` [T, 3840], `logits` [T, 262144]

Conformance

The official transformers.Gemma4UnifiedForConditionalGeneration agrees with this spec, layer by layer in float64, to a maximum relative error of 6.7e-16 (reduced-width profile, 2 weight draws × 2 lengths).

theformalera-conformance verify google/gemma-4-12B                    # the official implementation
theformalera-conformance verify google/gemma-4-12B path/to/model.py   # your implementation
theformalera-conformance spec google/gemma-4-12B                      # the architecture, as Lean derives it

Published configuration

The hyperparameters as the spec renders them (config.json form)

{
  "tie_word_embeddings": true,
  "text_config": {
    "vocab_size": 262144,
    "use_double_wide_mlp": false,
    "tie_word_embeddings": true,
    "sliding_window": 1024,
    "rope_parameters": {
      "sliding_attention": {
        "rope_type": "default",
        "rope_theta": 10000
      },
      "full_attention": {
        "rope_type": "proportional",
        "rope_theta": 1000000,
        "partial_rotary_factor": 0.25
      }
    },
    "rms_norm_eps": 1e-06,
    "num_kv_shared_layers": 0,
    "num_key_value_heads": 8,
    "num_hidden_layers": 48,
    "num_global_key_value_heads": 1,
    "num_attention_heads": 16,
    "layer_types": [
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "full_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "full_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "full_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "full_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "full_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "full_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "full_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "full_attention"
    ],
    "intermediate_size": 15360,
    "hidden_size_per_layer_input": 0,
    "hidden_size": 3840,
    "hidden_activation": "gelu_pytorch_tanh",
    "head_dim": 256,
    "global_head_dim": 512,
    "final_logit_softcapping": 30,
    "enable_moe_block": false,
    "attention_k_eq_v": true,
    "attention_bias": false
  },
  "architectures": [
    "Gemma4UnifiedForConditionalGeneration"
  ]
}

Spec Hub

Qwen

Gemma

Granite

Muse

At a glance

What to watch for

Compute-graph nodes

Tensors of the main nodes

Conformance

Published configuration

​At a glance

​What to watch for

​Compute-graph nodes

​Tensors of the main nodes

​Conformance

​Published configuration

At a glance

What to watch for

Compute-graph nodes

Tensors of the main nodes

Conformance

Published configuration