> ## Documentation Index
> Fetch the complete documentation index at: https://docs.theformalera.com/llms.txt
> Use this file to discover all available pages before exploring further.

# granite-4.2-8b

> Dense Granite: GQA attention with a µP-style attention multiplier, SwiGLU.

conformance ✓spec 1.0granite

The Model Spec of [`ibm-granite/granite-4.2-8b`](https://huggingface.co/ibm-granite/granite-4.2-8b) in `TheFormalEra.Spec.Registry`. The constants below are transcribed from its published `config.json`, and the official `transformers.GraniteForCausalLM` is checked against them.

## At a glance

| | |
| - | - |
| Model key | `ibm-granite/granite-4.2-8b` |
| Parameters (text model) | 8,791,592,960 |
| Layers | 40 (40 full attention) |
| Hidden size | 4,096 |
| MLP width | 12,800 (silu) |
| Vocabulary | 100,352 |
| Norm | `w·x̂`, ε = 1e-05 |
| Embedding | separate from the LM head |
| Logits | no soft-capping |
| Hugging Face class | `GraniteForCausalLM` |
| Shape domain | `T` ∈ \[1, 131,072] tokens, `B` ∈ \[1, 256] sequences, `S` ∈ \[1, 131,072] cached positions, page size 16 |

## What to watch for

* Every layer is full causal attention: 32 query heads, 8 KV heads (4 query heads per KV head), head dim 128.
* The attention scale is the checkpoint's `attention_multiplier`, 1/128, not 1/√128.
* No q/k norms. RMSNorm weights are applied as `w`, with ε = 1e-5.
* RoPE over the full head, θ = 10,000,000. The language-model head is separate from the embedding.

## Compute-graph nodes

Use these paths in Spec Keys: `ibm-granite/granite-4.2-8b@1.0:layers[*].<node>` for per-layer nodes, `ibm-granite/granite-4.2-8b@1.0:<node>` for the others.

| Node | Op Spec | Layers | Formula | Constants |
| - | - | - | - | - |
| `embed` | `embed` | — | `y[t] = 1 · w[tokens[t]]` | mul=1, norm=False |
| `input_norm` | `rmsnorm` | all 40 | `y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · w[i]` | H=4096, eps=1e-05 |
| `attn.qkv_proj` | `linear` | all 40 | `y[t,n] = Σₖ x[t,k] · w[n,k]` | K=4096, N=6144 |
| `attn.rope` | `rope` | all 40 | `rotate pairs (i, i + 64) by pos · θ^(-2i/128), i < 64; coordinates ≥ 128 unchanged` | D=128, active=64, denom=128, rot=128, theta=10000000 |
| `attn.core` | `decodeGQA` | all 40 | `o[b,h] = Σ_{s < seq_lens[b]} softmax_s(1/128 · ⟨q[b,h], k[b,s,h / 4]⟩) · v[b,s,h / 4]` | D=128, heads=32, kv\_heads=8, scale=1/128 |
| `attn.o_proj` | `linear` | all 40 | `y[t,n] = Σₖ x[t,k] · w[n,k]` | K=4096, N=4096 |
| `attn_residual` | `residual` | all 40 | `y = x + h` | H=4096, layer\_scalar=False, r=1 |
| `post_attn_norm` | `rmsnorm` | all 40 | `y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · w[i]` | H=4096, eps=1e-05 |
| `mlp.gate_up` | `linear` | all 40 | `y[t,n] = Σₖ x[t,k] · w[n,k]` | K=4096, N=25600 |
| `mlp.act` | `siluMul` | all 40 | `y[t,i] = silu(gate_up[t,i]) · gate_up[t,12800+i]` | I=12800, act=silu |
| `mlp.down` | `linear` | all 40 | `y[t,n] = Σₖ x[t,k] · w[n,k]` | K=12800, N=4096 |
| `mlp_residual` | `residual` | all 40 | `y = x + h` | H=4096, layer\_scalar=False, r=1 |
| `final_norm` | `rmsnorm` | — | `y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · w[i]` | H=4096, eps=1e-05 |
| `lm_head` | `lmHead` | — | `z = x wᵀ` | mul=1, tied=False |

Nodes listed twice have different constants in different layers (for example, sliding-window and global attention). A kernel annotated with `layers[*].<node>` is verified against each variant.

## Tensors of the main nodes

| Node | Inputs and outputs |
| - | - |
| `embed` | `tokens` \[T], `y` \[T, 4096] |
| `input_norm` | `x` \[T, 4096], `y` \[T, 4096] |
| `attn.qkv_proj` | `x` \[T, 4096], `y` \[T, 6144] |
| `attn.rope` | `q` \[T, 32, 128], `k` \[T, 8, 128], `pos` \[T], `q_out` \[T, 32, 128], `k_out` \[T, 8, 128] |
| `attn.core` | `q` \[B, 32, 128], `k` \[B, S, 8, 128], `v` \[B, S, 8, 128], `seq_lens` \[B], `o` \[B, 32, 128] |
| `attn.o_proj` | `x` \[T, 4096], `y` \[T, 4096] |
| `attn_residual` | `x` \[T, 4096], `h` \[T, 4096], `y` \[T, 4096] |
| `post_attn_norm` | `x` \[T, 4096], `y` \[T, 4096] |
| `mlp.gate_up` | `x` \[T, 4096], `y` \[T, 25600] |
| `mlp.act` | `gate_up` \[T, 25600], `y` \[T, 12800] |
| `mlp.down` | `x` \[T, 12800], `y` \[T, 4096] |
| `mlp_residual` | `x` \[T, 4096], `h` \[T, 4096], `y` \[T, 4096] |
| `final_norm` | `x` \[T, 4096], `y` \[T, 4096] |
| `lm_head` | `x` \[T, 4096], `logits` \[T, 100352] |

## Conformance

The official `transformers.GraniteForCausalLM` agrees with this spec, layer by layer in float64, to a maximum relative error of **1.6e-15** (reduced-width profile, 2 weight draws × 2 lengths).

```bash theme={"system"}
theformalera-conformance verify ibm-granite/granite-4.2-8b                    # the official implementation
theformalera-conformance verify ibm-granite/granite-4.2-8b path/to/model.py   # your implementation
theformalera-conformance spec ibm-granite/granite-4.2-8b                      # the architecture, as Lean derives it
```

## Published configuration

> [accordion] The hyperparameters as the spec renders them (config.json form)
>
> ```json theme={"system"}
> {
>   "vocab_size": 100352,
>   "tie_word_embeddings": false,
>   "rope_theta": 10000000,
>   "rope_parameters": {
>     "rope_type": "default",
>     "rope_theta": 10000000
>   },
>   "rms_norm_eps": 1e-05,
>   "residual_multiplier": 1,
>   "num_key_value_heads": 8,
>   "num_hidden_layers": 40,
>   "num_attention_heads": 32,
>   "mlp_bias": false,
>   "logits_scaling": 1,
>   "intermediate_size": 12800,
>   "hidden_size": 4096,
>   "hidden_act": "silu",
>   "embedding_multiplier": 1,
>   "attention_multiplier": 0.0078125,
>   "attention_bias": false,
>   "architectures": [
>     "GraniteForCausalLM"
>   ]
> }
> ```


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.