> ## Documentation Index
> Fetch the complete documentation index at: https://docs.theformalera.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Qwen3.8-27B

> Hybrid Qwen3.5-family model: 48 Gated DeltaNet layers and 16 gated attention layers.

conformance ✓spec 1.0qwen3\_5

The Model Spec of [`Qwen/Qwen3.8-27B`](https://huggingface.co/Qwen/Qwen3.8-27B) in `TheFormalEra.Spec.Registry`. The constants below are transcribed from its published `config.json`, and the official `transformers.Qwen3_5ForConditionalGeneration` is checked against them.

## At a glance

| | |
| - | - |
| Model key | `Qwen/Qwen3.8-27B` |
| Parameters (text model) | 26,895,998,464 |
| Layers | 64 (48 Gated DeltaNet, 16 full attention) |
| Hidden size | 5,120 |
| MLP width | 17,408 (silu) |
| Vocabulary | 248,320 |
| Norm | `(1+w)·x̂`, ε = 1e-06 |
| Embedding | separate from the LM head |
| Logits | no soft-capping |
| Hugging Face class | `Qwen3_5ForConditionalGeneration` |
| Shape domain | `T` ∈ \[1, 262,144] tokens, `B` ∈ \[1, 256] sequences, `S` ∈ \[1, 262,144] cached positions, page size 16 |

## What to watch for

* Layer `i` is full attention iff `i % 4 = 3` (`qwen3_8_27b_pattern`): layers 3, 7, …, 63. The other 48 layers are Gated DeltaNet.
* **RMSNorm weights are applied as `(1 + w)`** (`rmsnormOffset`), including the per-head q/k norms and the final norm. A kernel that multiplies by `w` is a counterexample for every `T`.
* Gated Attention: 24 query heads, 4 KV heads (6 query heads per KV head), head dim 256. Each head's query and output gate come from the same projection, and the output is `o ⊙ σ(gate)` (`attn.gate`).
* Partial RoPE: only the first 64 of 256 dimensions rotate (`partial_rotary_factor = 1/4`), θ = 10,000,000. The mRoPE sections only matter for image and video input; for text the three position groups coincide.
* Gated DeltaNet: 16 key heads and 48 value heads of dimension 128 (value head `h` reads key head `h / 3`), a width-4 causal convolution with SiLU, L2-normalised q and k, and a gated RMSNorm on the output.
* Scope: the text-only language-model path. The vision encoder and the MTP head are out of scope.

## Compute-graph nodes

Use these paths in Spec Keys: `Qwen/Qwen3.8-27B@1.0:layers[*].<node>` for per-layer nodes, `Qwen/Qwen3.8-27B@1.0:<node>` for the others.

| Node | Op Spec | Layers | Formula | Constants |
| - | - | - | - | - |
| `embed` | `embed` | — | `y[t] = 1 · w[tokens[t]]` | mul=1, norm=False |
| `input_norm` | `rmsnormOffset` | all 64 | `y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · (1 + w[i])` | H=5120, eps=1e-06 |
| `gdn.in_proj` | `linear` | 48: all but 3, 7, …, 63 | `y[t,n] = Σₖ x[t,k] · w[n,k]` | K=5120, N=16480 |
| `gdn.conv` | `causalConvSilu` | 48: all but 3, 7, …, 63 | `y[t,c] = silu(Σ_{j<4} w[c,j] · x[t + j - 3,c])` | C=10240, K=4 |
| `gdn.core` | `gatedDeltaRule` | 48: all but 3, 7, …, 63 | `S_t = e^(g_t) · S_(t-1) · (I − β_t k̂_t k̂_tᵀ) + β_t v_t k̂_tᵀ,  o_t = S_t q̂_t,  q̂ = l2norm(q)/√128, k̂ = l2norm(k), value head h reads key head h / 3` | Dk=128, Dv=128, k\_heads=16, qk\_l2norm\_eps=1e-06, v\_heads=48 |
| `gdn.norm` | `rmsnormGated` | 48: all but 3, 7, …, 63 | `per head: y = w ⊙ x / √(mean(x²) + ε) ⊙ silu(z)` | Dv=128, eps=1e-06 |
| `gdn.out_proj` | `linear` | 48: all but 3, 7, …, 63 | `y[t,n] = Σₖ x[t,k] · w[n,k]` | K=6144, N=5120 |
| `attn_residual` | `residual` | all 64 | `y = x + h` | H=5120, layer\_scalar=False, r=1 |
| `post_attn_norm` | `rmsnormOffset` | all 64 | `y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · (1 + w[i])` | H=5120, eps=1e-06 |
| `mlp.gate_up` | `linear` | all 64 | `y[t,n] = Σₖ x[t,k] · w[n,k]` | K=5120, N=34816 |
| `mlp.act` | `siluMul` | all 64 | `y[t,i] = silu(gate_up[t,i]) · gate_up[t,17408+i]` | I=17408, act=silu |
| `mlp.down` | `linear` | all 64 | `y[t,n] = Σₖ x[t,k] · w[n,k]` | K=17408, N=5120 |
| `mlp_residual` | `residual` | all 64 | `y = x + h` | H=5120, layer\_scalar=False, r=1 |
| `attn.qkv_proj` | `linear` | 16: 3, 7, …, 63 | `y[t,n] = Σₖ x[t,k] · w[n,k]` | K=5120, N=14336 |
| `attn.qk_norm` | `rmsnormOffset` | 16: 3, 7, …, 63 | `per head: y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · (1 + w[i])` | D=256, eps=1e-06 |
| `attn.rope` | `rope` | 16: 3, 7, …, 63 | `rotate pairs (i, i + 32) by pos · θ^(-2i/64), i < 32; coordinates ≥ 64 unchanged` | D=256, active=32, denom=64, rot=64, theta=10000000 |
| `attn.core` | `decodeGQA` | 16: 3, 7, …, 63 | `o[b,h] = Σ_{s < seq_lens[b]} softmax_s(1/√256 · ⟨q[b,h], k[b,s,h / 6]⟩) · v[b,s,h / 6]` | D=256, heads=24, kv\_heads=4, scale=1/√256 |
| `attn.gate` | `sigmoidGate` | 16: 3, 7, …, 63 | `y = o ⊙ σ(gate)` | N=6144 |
| `attn.o_proj` | `linear` | 16: 3, 7, …, 63 | `y[t,n] = Σₖ x[t,k] · w[n,k]` | K=6144, N=5120 |
| `final_norm` | `rmsnormOffset` | — | `y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · (1 + w[i])` | H=5120, eps=1e-06 |
| `lm_head` | `lmHead` | — | `z = x wᵀ` | mul=1, tied=False |

Nodes listed twice have different constants in different layers (for example, sliding-window and global attention). A kernel annotated with `layers[*].<node>` is verified against each variant.

## Tensors of the main nodes

| Node | Inputs and outputs |
| - | - |
| `embed` | `tokens` \[T], `y` \[T, 5120] |
| `input_norm` | `x` \[T, 5120], `y` \[T, 5120] |
| `gdn.in_proj` | `x` \[T, 5120], `y` \[T, 16480] |
| `gdn.conv` | `x` \[T, 10240], `y` \[T, 10240] |
| `gdn.core` | `q` \[T, 16, 128], `k` \[T, 16, 128], `v` \[T, 48, 128], `beta` \[T, 48], `g` \[T, 48], `o` \[T, 48, 128] |
| `gdn.norm` | `x` \[T, 48, 128], `z` \[T, 48, 128], `y` \[T, 48, 128] |
| `gdn.out_proj` | `x` \[T, 6144], `y` \[T, 5120] |
| `attn_residual` | `x` \[T, 5120], `h` \[T, 5120], `y` \[T, 5120] |
| `post_attn_norm` | `x` \[T, 5120], `y` \[T, 5120] |
| `mlp.gate_up` | `x` \[T, 5120], `y` \[T, 34816] |
| `mlp.act` | `gate_up` \[T, 34816], `y` \[T, 17408] |
| `mlp.down` | `x` \[T, 17408], `y` \[T, 5120] |
| `mlp_residual` | `x` \[T, 5120], `h` \[T, 5120], `y` \[T, 5120] |
| `attn.qkv_proj` | `x` \[T, 5120], `y` \[T, 14336] |
| `attn.qk_norm` | `q` \[T, 24, 256], `k` \[T, 4, 256], `q_out` \[T, 24, 256], `k_out` \[T, 4, 256] |
| `attn.rope` | `q` \[T, 24, 256], `k` \[T, 4, 256], `pos` \[T], `q_out` \[T, 24, 256], `k_out` \[T, 4, 256] |
| `attn.core` | `q` \[B, 24, 256], `k` \[B, S, 4, 256], `v` \[B, S, 4, 256], `seq_lens` \[B], `o` \[B, 24, 256] |
| `attn.gate` | `o` \[T, 6144], `gate` \[T, 6144], `y` \[T, 6144] |
| `attn.o_proj` | `x` \[T, 6144], `y` \[T, 5120] |
| `final_norm` | `x` \[T, 5120], `y` \[T, 5120] |
| `lm_head` | `x` \[T, 5120], `logits` \[T, 248320] |

## Conformance

The official `transformers.Qwen3_5ForConditionalGeneration` agrees with this spec, layer by layer in float64, to a maximum relative error of **4.1e-14** (reduced-width profile, 2 weight draws × 2 lengths).

```bash theme={"system"}
theformalera-conformance verify Qwen/Qwen3.8-27B                    # the official implementation
theformalera-conformance verify Qwen/Qwen3.8-27B path/to/model.py   # your implementation
theformalera-conformance spec Qwen/Qwen3.8-27B                      # the architecture, as Lean derives it
```

## Published configuration

> [accordion] The hyperparameters as the spec renders them (config.json form)
>
> ```json theme={"system"}
> {
>   "tie_word_embeddings": false,
>   "text_config": {
>     "vocab_size": 248320,
>     "tie_word_embeddings": false,
>     "rope_parameters": {
>       "rope_type": "default",
>       "rope_theta": 10000000,
>       "partial_rotary_factor": 0.25
>     },
>     "rms_norm_eps": 1e-06,
>     "partial_rotary_factor": 0.25,
>     "num_key_value_heads": 4,
>     "num_hidden_layers": 64,
>     "num_attention_heads": 24,
>     "linear_value_head_dim": 128,
>     "linear_num_value_heads": 48,
>     "linear_num_key_heads": 16,
>     "linear_key_head_dim": 128,
>     "linear_conv_kernel_dim": 4,
>     "layer_types": [
>       "linear_attention",
>       "linear_attention",
>       "linear_attention",
>       "full_attention",
>       "linear_attention",
>       "linear_attention",
>       "linear_attention",
>       "full_attention",
>       "linear_attention",
>       "linear_attention",
>       "linear_attention",
>       "full_attention",
>       "linear_attention",
>       "linear_attention",
>       "linear_attention",
>       "full_attention",
>       "linear_attention",
>       "linear_attention",
>       "linear_attention",
>       "full_attention",
>       "linear_attention",
>       "linear_attention",
>       "linear_attention",
>       "full_attention",
>       "linear_attention",
>       "linear_attention",
>       "linear_attention",
>       "full_attention",
>       "linear_attention",
>       "linear_attention",
>       "linear_attention",
>       "full_attention",
>       "linear_attention",
>       "linear_attention",
>       "linear_attention",
>       "full_attention",
>       "linear_attention",
>       "linear_attention",
>       "linear_attention",
>       "full_attention",
>       "linear_attention",
>       "linear_attention",
>       "linear_attention",
>       "full_attention",
>       "linear_attention",
>       "linear_attention",
>       "linear_attention",
>       "full_attention",
>       "linear_attention",
>       "linear_attention",
>       "linear_attention",
>       "full_attention",
>       "linear_attention",
>       "linear_attention",
>       "linear_attention",
>       "full_attention",
>       "linear_attention",
>       "linear_attention",
>       "linear_attention",
>       "full_attention",
>       "linear_attention",
>       "linear_attention",
>       "linear_attention",
>       "full_attention"
>     ],
>     "intermediate_size": 17408,
>     "hidden_size": 5120,
>     "hidden_act": "silu",
>     "head_dim": 256,
>     "attention_bias": false
>   },
>   "architectures": [
>     "Qwen3_5ForConditionalGeneration"
>   ]
> }
> ```


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.