gemma-4-31B

conformance ✓ spec 1.0 gemma4 The Model Spec of google/gemma-4-31B in TheFormalEra.Spec.Registry. The constants below are transcribed from its published config.json, and the official transformers.Gemma4ForConditionalGeneration is checked against them.

At a glance


Model key	`google/gemma-4-31B`
Parameters (text model)	30,697,345,340
Layers	60 (50 sliding-window attention, 10 full attention)
Hidden size	5,376
MLP width	21,504 (gelu_pytorch_tanh)
Vocabulary	262,144
Norm	`w·x̂`, ε = 1e-06
Embedding	tied to the LM head, × √5376
Logits	soft-capped at 30
Hugging Face class	`Gemma4ForConditionalGeneration`
Shape domain	`T` ∈ [1, 262,144] tokens, `B` ∈ [1, 256] sequences, `S` ∈ [1, 262,144] cached positions, page size 16

What to watch for

Layer i is global iff i % 6 = 5 (gemma4_31b_pattern): 10 global layers, 50 sliding-window layers with a window of 1024.
Sliding layers: 32 query heads, 16 KV heads, head dim 256, RoPE θ = 10,000. Global layers: 32 query heads, 4 KV heads, head dim 512, values = keys, RoPE θ = 1,000,000 with only the first 64 frequencies active.
Attention scale is 1. Values get an unweighted RMSNorm (attn.v_norm).
Sandwich norms (post_mix_norm, post_ffn_norm) and a learned per-layer scalar on the MLP residual.
GeGLU (geluMul, tanh GELU). Embeddings are scaled by √5376. Logits are soft-capped: 30 · tanh(z / 30).
Scope: the text-only path. The vision tower is out of scope.

Compute-graph nodes

Use these paths in Spec Keys: google/gemma-4-31B@1.0:layers[*].<node> for per-layer nodes, google/gemma-4-31B@1.0:<node> for the others.

Node	Op Spec	Layers	Formula	Constants
`embed`	`embed`	—	`y[t] = √5376 · w[tokens[t]]`	mul=√5376, norm=False
`input_norm`	`rmsnorm`	all 60	`y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · w[i]`	H=5376, eps=1e-06
`attn.qkv_proj`	`linear`	50: all but 5, 11, …, 59	`y[t,n] = Σₖ x[t,k] · w[n,k]`	K=5376, N=16384
`attn.qk_norm`	`rmsnorm`	50: all but 5, 11, …, 59	`per head: y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · w[i]`	D=256, eps=1e-06
`attn.v_norm`	`rmsnormUnit`	50: all but 5, 11, …, 59	`per head: y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε)`	D=256, eps=1e-06
`attn.rope`	`rope`	50: all but 5, 11, …, 59	`rotate pairs (i, i + 128) by pos · θ^(-2i/256), i < 128; coordinates ≥ 256 unchanged`	D=256, active=128, denom=256, rot=256, theta=10000
`attn.core`	`slidingDecodeGQA`	50: all but 5, 11, …, 59	`o[b,h] = Σ_{s < seq_lens[b]} softmax_s(1 · ⟨q[b,h], k[b,s,h / 2]⟩) · v[b,s,h / 2] over the last 1024 positions`	D=256, heads=32, kv_heads=16, scale=1, window=1024
`attn.o_proj`	`linear`	50: all but 5, 11, …, 59	`y[t,n] = Σₖ x[t,k] · w[n,k]`	K=8192, N=5376
`post_mix_norm`	`rmsnorm`	all 60	`y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · w[i]`	H=5376, eps=1e-06
`attn_residual`	`residual`	all 60	`y = x + h`	H=5376, layer_scalar=False, r=1
`post_attn_norm`	`rmsnorm`	all 60	`y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · w[i]`	H=5376, eps=1e-06
`mlp.gate_up`	`linear`	all 60	`y[t,n] = Σₖ x[t,k] · w[n,k]`	K=5376, N=43008
`mlp.act`	`geluMul`	all 60	`y[t,i] = gelu_tanh(gate_up[t,i]) · gate_up[t,21504+i]`	I=21504, act=gelu_pytorch_tanh
`mlp.down`	`linear`	all 60	`y[t,n] = Σₖ x[t,k] · w[n,k]`	K=21504, N=5376
`post_ffn_norm`	`rmsnorm`	all 60	`y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · w[i]`	H=5376, eps=1e-06
`mlp_residual`	`residual`	all 60	`y = s · (x + 1 · h)`	H=5376, layer_scalar=True, r=1
`attn.qkv_proj`	`linear`	10: 5, 11, …, 59	`y[t,n] = Σₖ x[t,k] · w[n,k]`	K=5376, N=18432
`attn.qk_norm`	`rmsnorm`	10: 5, 11, …, 59	`per head: y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · w[i]`	D=512, eps=1e-06
`attn.v_norm`	`rmsnormUnit`	10: 5, 11, …, 59	`per head: y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε)`	D=512, eps=1e-06
`attn.rope`	`rope`	10: 5, 11, …, 59	`rotate pairs (i, i + 256) by pos · θ^(-2i/512), i < 64; coordinates ≥ 512 unchanged`	D=512, active=64, denom=512, rot=512, theta=1000000
`attn.core`	`decodeGQA`	10: 5, 11, …, 59	`o[b,h] = Σ_{s < seq_lens[b]} softmax_s(1 · ⟨q[b,h], k[b,s,h / 8]⟩) · v[b,s,h / 8]`	D=512, heads=32, kv_heads=4, scale=1
`attn.o_proj`	`linear`	10: 5, 11, …, 59	`y[t,n] = Σₖ x[t,k] · w[n,k]`	K=16384, N=5376
`final_norm`	`rmsnorm`	—	`y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · w[i]`	H=5376, eps=1e-06
`lm_head`	`lmHead`	—	`z = x wᵀ, logits = 30 · tanh(z / 30)`	mul=1, softcap=30, tied=True

Nodes listed twice have different constants in different layers (for example, sliding-window and global attention). A kernel annotated with layers[*].<node> is verified against each variant.

Tensors of the main nodes

Node	Inputs and outputs
`embed`	`tokens` [T], `y` [T, 5376]
`input_norm`	`x` [T, 5376], `y` [T, 5376]
`attn.qkv_proj`	`x` [T, 5376], `y` [T, 16384]
`attn.qk_norm`	`q` [T, 32, 256], `k` [T, 16, 256], `q_out` [T, 32, 256], `k_out` [T, 16, 256]
`attn.v_norm`	`v` [T, 16, 256], `v_out` [T, 16, 256]
`attn.rope`	`q` [T, 32, 256], `k` [T, 16, 256], `pos` [T], `q_out` [T, 32, 256], `k_out` [T, 16, 256]
`attn.core`	`q` [B, 32, 256], `k` [B, S, 16, 256], `v` [B, S, 16, 256], `seq_lens` [B], `o` [B, 32, 256]
`attn.o_proj`	`x` [T, 8192], `y` [T, 5376]
`post_mix_norm`	`x` [T, 5376], `y` [T, 5376]
`attn_residual`	`x` [T, 5376], `h` [T, 5376], `y` [T, 5376]
`post_attn_norm`	`x` [T, 5376], `y` [T, 5376]
`mlp.gate_up`	`x` [T, 5376], `y` [T, 43008]
`mlp.act`	`gate_up` [T, 43008], `y` [T, 21504]
`mlp.down`	`x` [T, 21504], `y` [T, 5376]
`post_ffn_norm`	`x` [T, 5376], `y` [T, 5376]
`mlp_residual`	`x` [T, 5376], `h` [T, 5376], `y` [T, 5376]
`final_norm`	`x` [T, 5376], `y` [T, 5376]
`lm_head`	`x` [T, 5376], `logits` [T, 262144]

Conformance

The official transformers.Gemma4ForConditionalGeneration agrees with this spec, layer by layer in float64, to a maximum relative error of 8.6e-16 (reduced-width profile, 2 weight draws × 2 lengths).

theformalera-conformance verify google/gemma-4-31B                    # the official implementation
theformalera-conformance verify google/gemma-4-31B path/to/model.py   # your implementation
theformalera-conformance spec google/gemma-4-31B                      # the architecture, as Lean derives it

Published configuration

The hyperparameters as the spec renders them (config.json form)

{
  "tie_word_embeddings": true,
  "text_config": {
    "vocab_size": 262144,
    "use_double_wide_mlp": false,
    "tie_word_embeddings": true,
    "sliding_window": 1024,
    "rope_parameters": {
      "sliding_attention": {
        "rope_type": "default",
        "rope_theta": 10000
      },
      "full_attention": {
        "rope_type": "proportional",
        "rope_theta": 1000000,
        "partial_rotary_factor": 0.25
      }
    },
    "rms_norm_eps": 1e-06,
    "num_kv_shared_layers": 0,
    "num_key_value_heads": 16,
    "num_hidden_layers": 60,
    "num_global_key_value_heads": 4,
    "num_attention_heads": 32,
    "layer_types": [
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "full_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "full_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "full_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "full_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "full_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "full_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "full_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "full_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "full_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "full_attention"
    ],
    "intermediate_size": 21504,
    "hidden_size_per_layer_input": 0,
    "hidden_size": 5376,
    "hidden_activation": "gelu_pytorch_tanh",
    "head_dim": 256,
    "global_head_dim": 512,
    "final_logit_softcapping": 30,
    "enable_moe_block": false,
    "attention_k_eq_v": true,
    "attention_bias": false
  },
  "architectures": [
    "Gemma4ForConditionalGeneration"
  ]
}

Spec Hub

Qwen

Gemma

Granite

Muse

At a glance

What to watch for

Compute-graph nodes

Tensors of the main nodes

Conformance

Published configuration

​At a glance

​What to watch for

​Compute-graph nodes

​Tensors of the main nodes

​Conformance

​Published configuration

At a glance

What to watch for

Compute-graph nodes

Tensors of the main nodes

Conformance

Published configuration