> ## Documentation Index
> Fetch the complete documentation index at: https://docs.theformalera.com/llms.txt
> Use this file to discover all available pages before exploring further.

# gemma-4-31B

> Gemma 4: 5 sliding-window layers per global layer, sandwich norms, logit soft-capping.

conformance ✓spec 1.0gemma4

The Model Spec of [`google/gemma-4-31B`](https://huggingface.co/google/gemma-4-31B) in `TheFormalEra.Spec.Registry`. The constants below are transcribed from its published `config.json`, and the official `transformers.Gemma4ForConditionalGeneration` is checked against them.

## At a glance

| | |
| - | - |
| Model key | `google/gemma-4-31B` |
| Parameters (text model) | 30,697,345,340 |
| Layers | 60 (50 sliding-window attention, 10 full attention) |
| Hidden size | 5,376 |
| MLP width | 21,504 (gelu\_pytorch\_tanh) |
| Vocabulary | 262,144 |
| Norm | `w·x̂`, ε = 1e-06 |
| Embedding | tied to the LM head, × √5376 |
| Logits | soft-capped at 30 |
| Hugging Face class | `Gemma4ForConditionalGeneration` |
| Shape domain | `T` ∈ \[1, 262,144] tokens, `B` ∈ \[1, 256] sequences, `S` ∈ \[1, 262,144] cached positions, page size 16 |

## What to watch for

* Layer `i` is global iff `i % 6 = 5` (`gemma4_31b_pattern`): 10 global layers, 50 sliding-window layers with a window of 1024.
* Sliding layers: 32 query heads, 16 KV heads, head dim 256, RoPE θ = 10,000. Global layers: 32 query heads, 4 KV heads, head dim 512, values = keys, RoPE θ = 1,000,000 with only the first 64 frequencies active.
* Attention scale is 1. Values get an unweighted RMSNorm (`attn.v_norm`).
* Sandwich norms (`post_mix_norm`, `post_ffn_norm`) and a learned per-layer scalar on the MLP residual.
* GeGLU (`geluMul`, tanh GELU). Embeddings are scaled by √5376. Logits are soft-capped: `30 · tanh(z / 30)`.
* Scope: the text-only path. The vision tower is out of scope.

## Compute-graph nodes

Use these paths in Spec Keys: `google/gemma-4-31B@1.0:layers[*].<node>` for per-layer nodes, `google/gemma-4-31B@1.0:<node>` for the others.

| Node | Op Spec | Layers | Formula | Constants |
| - | - | - | - | - |
| `embed` | `embed` | — | `y[t] = √5376 · w[tokens[t]]` | mul=√5376, norm=False |
| `input_norm` | `rmsnorm` | all 60 | `y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · w[i]` | H=5376, eps=1e-06 |
| `attn.qkv_proj` | `linear` | 50: all but 5, 11, …, 59 | `y[t,n] = Σₖ x[t,k] · w[n,k]` | K=5376, N=16384 |
| `attn.qk_norm` | `rmsnorm` | 50: all but 5, 11, …, 59 | `per head: y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · w[i]` | D=256, eps=1e-06 |
| `attn.v_norm` | `rmsnormUnit` | 50: all but 5, 11, …, 59 | `per head: y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε)` | D=256, eps=1e-06 |
| `attn.rope` | `rope` | 50: all but 5, 11, …, 59 | `rotate pairs (i, i + 128) by pos · θ^(-2i/256), i < 128; coordinates ≥ 256 unchanged` | D=256, active=128, denom=256, rot=256, theta=10000 |
| `attn.core` | `slidingDecodeGQA` | 50: all but 5, 11, …, 59 | `o[b,h] = Σ_{s < seq_lens[b]} softmax_s(1 · ⟨q[b,h], k[b,s,h / 2]⟩) · v[b,s,h / 2] over the last 1024 positions` | D=256, heads=32, kv\_heads=16, scale=1, window=1024 |
| `attn.o_proj` | `linear` | 50: all but 5, 11, …, 59 | `y[t,n] = Σₖ x[t,k] · w[n,k]` | K=8192, N=5376 |
| `post_mix_norm` | `rmsnorm` | all 60 | `y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · w[i]` | H=5376, eps=1e-06 |
| `attn_residual` | `residual` | all 60 | `y = x + h` | H=5376, layer\_scalar=False, r=1 |
| `post_attn_norm` | `rmsnorm` | all 60 | `y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · w[i]` | H=5376, eps=1e-06 |
| `mlp.gate_up` | `linear` | all 60 | `y[t,n] = Σₖ x[t,k] · w[n,k]` | K=5376, N=43008 |
| `mlp.act` | `geluMul` | all 60 | `y[t,i] = gelu_tanh(gate_up[t,i]) · gate_up[t,21504+i]` | I=21504, act=gelu\_pytorch\_tanh |
| `mlp.down` | `linear` | all 60 | `y[t,n] = Σₖ x[t,k] · w[n,k]` | K=21504, N=5376 |
| `post_ffn_norm` | `rmsnorm` | all 60 | `y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · w[i]` | H=5376, eps=1e-06 |
| `mlp_residual` | `residual` | all 60 | `y = s · (x + 1 · h)` | H=5376, layer\_scalar=True, r=1 |
| `attn.qkv_proj` | `linear` | 10: 5, 11, …, 59 | `y[t,n] = Σₖ x[t,k] · w[n,k]` | K=5376, N=18432 |
| `attn.qk_norm` | `rmsnorm` | 10: 5, 11, …, 59 | `per head: y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · w[i]` | D=512, eps=1e-06 |
| `attn.v_norm` | `rmsnormUnit` | 10: 5, 11, …, 59 | `per head: y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε)` | D=512, eps=1e-06 |
| `attn.rope` | `rope` | 10: 5, 11, …, 59 | `rotate pairs (i, i + 256) by pos · θ^(-2i/512), i < 64; coordinates ≥ 512 unchanged` | D=512, active=64, denom=512, rot=512, theta=1000000 |
| `attn.core` | `decodeGQA` | 10: 5, 11, …, 59 | `o[b,h] = Σ_{s < seq_lens[b]} softmax_s(1 · ⟨q[b,h], k[b,s,h / 8]⟩) · v[b,s,h / 8]` | D=512, heads=32, kv\_heads=4, scale=1 |
| `attn.o_proj` | `linear` | 10: 5, 11, …, 59 | `y[t,n] = Σₖ x[t,k] · w[n,k]` | K=16384, N=5376 |
| `final_norm` | `rmsnorm` | — | `y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · w[i]` | H=5376, eps=1e-06 |
| `lm_head` | `lmHead` | — | `z = x wᵀ, logits = 30 · tanh(z / 30)` | mul=1, softcap=30, tied=True |

Nodes listed twice have different constants in different layers (for example, sliding-window and global attention). A kernel annotated with `layers[*].<node>` is verified against each variant.

## Tensors of the main nodes

| Node | Inputs and outputs |
| - | - |
| `embed` | `tokens` \[T], `y` \[T, 5376] |
| `input_norm` | `x` \[T, 5376], `y` \[T, 5376] |
| `attn.qkv_proj` | `x` \[T, 5376], `y` \[T, 16384] |
| `attn.qk_norm` | `q` \[T, 32, 256], `k` \[T, 16, 256], `q_out` \[T, 32, 256], `k_out` \[T, 16, 256] |
| `attn.v_norm` | `v` \[T, 16, 256], `v_out` \[T, 16, 256] |
| `attn.rope` | `q` \[T, 32, 256], `k` \[T, 16, 256], `pos` \[T], `q_out` \[T, 32, 256], `k_out` \[T, 16, 256] |
| `attn.core` | `q` \[B, 32, 256], `k` \[B, S, 16, 256], `v` \[B, S, 16, 256], `seq_lens` \[B], `o` \[B, 32, 256] |
| `attn.o_proj` | `x` \[T, 8192], `y` \[T, 5376] |
| `post_mix_norm` | `x` \[T, 5376], `y` \[T, 5376] |
| `attn_residual` | `x` \[T, 5376], `h` \[T, 5376], `y` \[T, 5376] |
| `post_attn_norm` | `x` \[T, 5376], `y` \[T, 5376] |
| `mlp.gate_up` | `x` \[T, 5376], `y` \[T, 43008] |
| `mlp.act` | `gate_up` \[T, 43008], `y` \[T, 21504] |
| `mlp.down` | `x` \[T, 21504], `y` \[T, 5376] |
| `post_ffn_norm` | `x` \[T, 5376], `y` \[T, 5376] |
| `mlp_residual` | `x` \[T, 5376], `h` \[T, 5376], `y` \[T, 5376] |
| `final_norm` | `x` \[T, 5376], `y` \[T, 5376] |
| `lm_head` | `x` \[T, 5376], `logits` \[T, 262144] |

## Conformance

The official `transformers.Gemma4ForConditionalGeneration` agrees with this spec, layer by layer in float64, to a maximum relative error of **8.6e-16** (reduced-width profile, 2 weight draws × 2 lengths).

```bash theme={"system"}
theformalera-conformance verify google/gemma-4-31B                    # the official implementation
theformalera-conformance verify google/gemma-4-31B path/to/model.py   # your implementation
theformalera-conformance spec google/gemma-4-31B                      # the architecture, as Lean derives it
```

## Published configuration

> [accordion] The hyperparameters as the spec renders them (config.json form)
>
> ```json theme={"system"}
> {
>   "tie_word_embeddings": true,
>   "text_config": {
>     "vocab_size": 262144,
>     "use_double_wide_mlp": false,
>     "tie_word_embeddings": true,
>     "sliding_window": 1024,
>     "rope_parameters": {
>       "sliding_attention": {
>         "rope_type": "default",
>         "rope_theta": 10000
>       },
>       "full_attention": {
>         "rope_type": "proportional",
>         "rope_theta": 1000000,
>         "partial_rotary_factor": 0.25
>       }
>     },
>     "rms_norm_eps": 1e-06,
>     "num_kv_shared_layers": 0,
>     "num_key_value_heads": 16,
>     "num_hidden_layers": 60,
>     "num_global_key_value_heads": 4,
>     "num_attention_heads": 32,
>     "layer_types": [
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "full_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "full_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "full_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "full_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "full_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "full_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "full_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "full_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "full_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "full_attention"
>     ],
>     "intermediate_size": 21504,
>     "hidden_size_per_layer_input": 0,
>     "hidden_size": 5376,
>     "hidden_activation": "gelu_pytorch_tanh",
>     "head_dim": 256,
>     "global_head_dim": 512,
>     "final_logit_softcapping": 30,
>     "enable_moe_block": false,
>     "attention_k_eq_v": true,
>     "attention_bias": false
>   },
>   "architectures": [
>     "Gemma4ForConditionalGeneration"
>   ]
> }
> ```


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.