Qwen3.8-27B

conformance ✓ spec 1.0 qwen3_5 The Model Spec of Qwen/Qwen3.8-27B in TheFormalEra.Spec.Registry. The constants below are transcribed from its published config.json, and the official transformers.Qwen3_5ForConditionalGeneration is checked against them.

At a glance


Model key	`Qwen/Qwen3.8-27B`
Parameters (text model)	26,895,998,464
Layers	64 (48 Gated DeltaNet, 16 full attention)
Hidden size	5,120
MLP width	17,408 (silu)
Vocabulary	248,320
Norm	`(1+w)·x̂`, ε = 1e-06
Embedding	separate from the LM head
Logits	no soft-capping
Hugging Face class	`Qwen3_5ForConditionalGeneration`
Shape domain	`T` ∈ [1, 262,144] tokens, `B` ∈ [1, 256] sequences, `S` ∈ [1, 262,144] cached positions, page size 16

What to watch for

Layer i is full attention iff i % 4 = 3 (qwen3_8_27b_pattern): layers 3, 7, …, 63. The other 48 layers are Gated DeltaNet.
RMSNorm weights are applied as (1 + w) (rmsnormOffset), including the per-head q/k norms and the final norm. A kernel that multiplies by w is a counterexample for every T.
Gated Attention: 24 query heads, 4 KV heads (6 query heads per KV head), head dim 256. Each head’s query and output gate come from the same projection, and the output is o ⊙ σ(gate) (attn.gate).
Partial RoPE: only the first 64 of 256 dimensions rotate (partial_rotary_factor = 1/4), θ = 10,000,000. The mRoPE sections only matter for image and video input; for text the three position groups coincide.
Gated DeltaNet: 16 key heads and 48 value heads of dimension 128 (value head h reads key head h / 3), a width-4 causal convolution with SiLU, L2-normalised q and k, and a gated RMSNorm on the output.
Scope: the text-only language-model path. The vision encoder and the MTP head are out of scope.

Compute-graph nodes

Use these paths in Spec Keys: Qwen/Qwen3.8-27B@1.0:layers[*].<node> for per-layer nodes, Qwen/Qwen3.8-27B@1.0:<node> for the others.

Node	Op Spec	Layers	Formula	Constants
`embed`	`embed`	—	`y[t] = 1 · w[tokens[t]]`	mul=1, norm=False
`input_norm`	`rmsnormOffset`	all 64	`y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · (1 + w[i])`	H=5120, eps=1e-06
`gdn.in_proj`	`linear`	48: all but 3, 7, …, 63	`y[t,n] = Σₖ x[t,k] · w[n,k]`	K=5120, N=16480
`gdn.conv`	`causalConvSilu`	48: all but 3, 7, …, 63	`y[t,c] = silu(Σ_{j<4} w[c,j] · x[t + j - 3,c])`	C=10240, K=4
`gdn.core`	`gatedDeltaRule`	48: all but 3, 7, …, 63	`S_t = e^(g_t) · S_(t-1) · (I − β_t k̂_t k̂_tᵀ) + β_t v_t k̂_tᵀ, o_t = S_t q̂_t, q̂ = l2norm(q)/√128, k̂ = l2norm(k), value head h reads key head h / 3`	Dk=128, Dv=128, k_heads=16, qk_l2norm_eps=1e-06, v_heads=48
`gdn.norm`	`rmsnormGated`	48: all but 3, 7, …, 63	`per head: y = w ⊙ x / √(mean(x²) + ε) ⊙ silu(z)`	Dv=128, eps=1e-06
`gdn.out_proj`	`linear`	48: all but 3, 7, …, 63	`y[t,n] = Σₖ x[t,k] · w[n,k]`	K=6144, N=5120
`attn_residual`	`residual`	all 64	`y = x + h`	H=5120, layer_scalar=False, r=1
`post_attn_norm`	`rmsnormOffset`	all 64	`y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · (1 + w[i])`	H=5120, eps=1e-06
`mlp.gate_up`	`linear`	all 64	`y[t,n] = Σₖ x[t,k] · w[n,k]`	K=5120, N=34816
`mlp.act`	`siluMul`	all 64	`y[t,i] = silu(gate_up[t,i]) · gate_up[t,17408+i]`	I=17408, act=silu
`mlp.down`	`linear`	all 64	`y[t,n] = Σₖ x[t,k] · w[n,k]`	K=17408, N=5120
`mlp_residual`	`residual`	all 64	`y = x + h`	H=5120, layer_scalar=False, r=1
`attn.qkv_proj`	`linear`	16: 3, 7, …, 63	`y[t,n] = Σₖ x[t,k] · w[n,k]`	K=5120, N=14336
`attn.qk_norm`	`rmsnormOffset`	16: 3, 7, …, 63	`per head: y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · (1 + w[i])`	D=256, eps=1e-06
`attn.rope`	`rope`	16: 3, 7, …, 63	`rotate pairs (i, i + 32) by pos · θ^(-2i/64), i < 32; coordinates ≥ 64 unchanged`	D=256, active=32, denom=64, rot=64, theta=10000000
`attn.core`	`decodeGQA`	16: 3, 7, …, 63	`o[b,h] = Σ_{s < seq_lens[b]} softmax_s(1/√256 · ⟨q[b,h], k[b,s,h / 6]⟩) · v[b,s,h / 6]`	D=256, heads=24, kv_heads=4, scale=1/√256
`attn.gate`	`sigmoidGate`	16: 3, 7, …, 63	`y = o ⊙ σ(gate)`	N=6144
`attn.o_proj`	`linear`	16: 3, 7, …, 63	`y[t,n] = Σₖ x[t,k] · w[n,k]`	K=6144, N=5120
`final_norm`	`rmsnormOffset`	—	`y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · (1 + w[i])`	H=5120, eps=1e-06
`lm_head`	`lmHead`	—	`z = x wᵀ`	mul=1, tied=False

Nodes listed twice have different constants in different layers (for example, sliding-window and global attention). A kernel annotated with layers[*].<node> is verified against each variant.

Tensors of the main nodes

Node	Inputs and outputs
`embed`	`tokens` [T], `y` [T, 5120]
`input_norm`	`x` [T, 5120], `y` [T, 5120]
`gdn.in_proj`	`x` [T, 5120], `y` [T, 16480]
`gdn.conv`	`x` [T, 10240], `y` [T, 10240]
`gdn.core`	`q` [T, 16, 128], `k` [T, 16, 128], `v` [T, 48, 128], `beta` [T, 48], `g` [T, 48], `o` [T, 48, 128]
`gdn.norm`	`x` [T, 48, 128], `z` [T, 48, 128], `y` [T, 48, 128]
`gdn.out_proj`	`x` [T, 6144], `y` [T, 5120]
`attn_residual`	`x` [T, 5120], `h` [T, 5120], `y` [T, 5120]
`post_attn_norm`	`x` [T, 5120], `y` [T, 5120]
`mlp.gate_up`	`x` [T, 5120], `y` [T, 34816]
`mlp.act`	`gate_up` [T, 34816], `y` [T, 17408]
`mlp.down`	`x` [T, 17408], `y` [T, 5120]
`mlp_residual`	`x` [T, 5120], `h` [T, 5120], `y` [T, 5120]
`attn.qkv_proj`	`x` [T, 5120], `y` [T, 14336]
`attn.qk_norm`	`q` [T, 24, 256], `k` [T, 4, 256], `q_out` [T, 24, 256], `k_out` [T, 4, 256]
`attn.rope`	`q` [T, 24, 256], `k` [T, 4, 256], `pos` [T], `q_out` [T, 24, 256], `k_out` [T, 4, 256]
`attn.core`	`q` [B, 24, 256], `k` [B, S, 4, 256], `v` [B, S, 4, 256], `seq_lens` [B], `o` [B, 24, 256]
`attn.gate`	`o` [T, 6144], `gate` [T, 6144], `y` [T, 6144]
`attn.o_proj`	`x` [T, 6144], `y` [T, 5120]
`final_norm`	`x` [T, 5120], `y` [T, 5120]
`lm_head`	`x` [T, 5120], `logits` [T, 248320]

Conformance

The official transformers.Qwen3_5ForConditionalGeneration agrees with this spec, layer by layer in float64, to a maximum relative error of 4.1e-14 (reduced-width profile, 2 weight draws × 2 lengths).

theformalera-conformance verify Qwen/Qwen3.8-27B                    # the official implementation
theformalera-conformance verify Qwen/Qwen3.8-27B path/to/model.py   # your implementation
theformalera-conformance spec Qwen/Qwen3.8-27B                      # the architecture, as Lean derives it

Published configuration

The hyperparameters as the spec renders them (config.json form)

{
  "tie_word_embeddings": false,
  "text_config": {
    "vocab_size": 248320,
    "tie_word_embeddings": false,
    "rope_parameters": {
      "rope_type": "default",
      "rope_theta": 10000000,
      "partial_rotary_factor": 0.25
    },
    "rms_norm_eps": 1e-06,
    "partial_rotary_factor": 0.25,
    "num_key_value_heads": 4,
    "num_hidden_layers": 64,
    "num_attention_heads": 24,
    "linear_value_head_dim": 128,
    "linear_num_value_heads": 48,
    "linear_num_key_heads": 16,
    "linear_key_head_dim": 128,
    "linear_conv_kernel_dim": 4,
    "layer_types": [
      "linear_attention",
      "linear_attention",
      "linear_attention",
      "full_attention",
      "linear_attention",
      "linear_attention",
      "linear_attention",
      "full_attention",
      "linear_attention",
      "linear_attention",
      "linear_attention",
      "full_attention",
      "linear_attention",
      "linear_attention",
      "linear_attention",
      "full_attention",
      "linear_attention",
      "linear_attention",
      "linear_attention",
      "full_attention",
      "linear_attention",
      "linear_attention",
      "linear_attention",
      "full_attention",
      "linear_attention",
      "linear_attention",
      "linear_attention",
      "full_attention",
      "linear_attention",
      "linear_attention",
      "linear_attention",
      "full_attention",
      "linear_attention",
      "linear_attention",
      "linear_attention",
      "full_attention",
      "linear_attention",
      "linear_attention",
      "linear_attention",
      "full_attention",
      "linear_attention",
      "linear_attention",
      "linear_attention",
      "full_attention",
      "linear_attention",
      "linear_attention",
      "linear_attention",
      "full_attention",
      "linear_attention",
      "linear_attention",
      "linear_attention",
      "full_attention",
      "linear_attention",
      "linear_attention",
      "linear_attention",
      "full_attention",
      "linear_attention",
      "linear_attention",
      "linear_attention",
      "full_attention",
      "linear_attention",
      "linear_attention",
      "linear_attention",
      "full_attention"
    ],
    "intermediate_size": 17408,
    "hidden_size": 5120,
    "hidden_act": "silu",
    "head_dim": 256,
    "attention_bias": false
  },
  "architectures": [
    "Qwen3_5ForConditionalGeneration"
  ]
}

Spec Hub

Qwen

Gemma

Granite

Muse

At a glance

What to watch for

Compute-graph nodes

Tensors of the main nodes

Conformance

Published configuration

​At a glance

​What to watch for

​Compute-graph nodes

​Tensors of the main nodes

​Conformance

​Published configuration

At a glance

What to watch for

Compute-graph nodes

Tensors of the main nodes

Conformance

Published configuration