Muse-Glimmer-30B

conformance ✓ spec 1.0 muse_glimmer The Model Spec of meta-models/Muse-Glimmer-30B in TheFormalEra.Spec.Registry. The constants below are transcribed from its published config.json, and the official transformers.MuseGlimmerForConditionalGeneration is checked against them.

At a glance


Model key	`meta-models/Muse-Glimmer-30B`
Parameters (text model)	27,854,780,928
Layers	52 (39 sliding-window attention, 13 full attention)
Hidden size	6,656
MLP width	19,968 (silu)
Vocabulary	202,048
Norm	`(1+w)·x̂`, ε = 1e-05
Embedding	separate from the LM head, then RMSNorm
Logits	× 0.196116, soft-capped at 20
Hugging Face class	`MuseGlimmerForConditionalGeneration`
Shape domain	`T` ∈ [1, 131,072] tokens, `B` ∈ [1, 256] sequences, `S` ∈ [1, 131,072] cached positions, page size 16

What to watch for

Layer i is global iff i % 4 = 3: 13 global layers and 39 sliding-window layers with a window of 2048. The global layers are exactly the NoPE layers (museGlimmer_global_iff_nope): they use no position encoding.
32 query heads and 2 KV heads (16 query heads per KV head), head dim 128. Queries and keys get an unweighted RMSNorm; queries are then multiplied by 3.87.
Every layer has a sigmoid output gate from a separate projection (attn.gate).
RMSNorm weights are applied as (1 + w) (rmsnormOffset), except the final norm, which uses w. Sandwich norms use ε = 1e-8.
The embedding is followed by an unweighted RMSNorm. Logits are multiplied by 0.19611613513818404, then soft-capped: 20 · tanh(z / 20).
Scope: the text-only path. The vision tower is out of scope.

Compute-graph nodes

Use these paths in Spec Keys: meta-models/Muse-Glimmer-30B@1.0:layers[*].<node> for per-layer nodes, meta-models/Muse-Glimmer-30B@1.0:<node> for the others.

Node	Op Spec	Layers	Formula	Constants
`embed`	`embed`	—	`y[t] = 1 · w[tokens[t]], then RMSNorm`	mul=1, norm=True
`input_norm`	`rmsnormOffset`	all 52	`y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · (1 + w[i])`	H=6656, eps=1e-05
`attn.qkv_proj`	`linear`	all 52	`y[t,n] = Σₖ x[t,k] · w[n,k]`	K=6656, N=4608
`attn.qk_norm`	`rmsnormUnit`	all 52	`per head: y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε); q · 3.87`	D=128, eps=1e-05
`attn.rope`	`rope`	39: all but 3, 7, …, 51	`rotate pairs (i, i + 64) by pos · θ^(-2i/128), i < 64; coordinates ≥ 128 unchanged`	D=128, active=64, denom=128, rot=128, theta=500000
`attn.core`	`slidingDecodeGQA`	39: all but 3, 7, …, 51	`o[b,h] = Σ_{s < seq_lens[b]} softmax_s(1/√128 · ⟨q[b,h], k[b,s,h / 16]⟩) · v[b,s,h / 16] over the last 2048 positions`	D=128, heads=32, kv_heads=2, scale=1/√128, window=2048
`attn.gate`	`sigmoidGate`	all 52	`y = o ⊙ σ(gate)`	N=4096
`attn.o_proj`	`linear`	all 52	`y[t,n] = Σₖ x[t,k] · w[n,k]`	K=4096, N=6656
`post_mix_norm`	`rmsnormOffset`	all 52	`y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · (1 + w[i])`	H=6656, eps=1e-08
`attn_residual`	`residual`	all 52	`y = x + h`	H=6656, layer_scalar=False, r=1
`post_attn_norm`	`rmsnormOffset`	all 52	`y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · (1 + w[i])`	H=6656, eps=1e-05
`mlp.gate_up`	`linear`	all 52	`y[t,n] = Σₖ x[t,k] · w[n,k]`	K=6656, N=39936
`mlp.act`	`siluMul`	all 52	`y[t,i] = silu(gate_up[t,i]) · gate_up[t,19968+i]`	I=19968, act=silu
`mlp.down`	`linear`	all 52	`y[t,n] = Σₖ x[t,k] · w[n,k]`	K=19968, N=6656
`post_ffn_norm`	`rmsnormOffset`	all 52	`y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · (1 + w[i])`	H=6656, eps=1e-08
`mlp_residual`	`residual`	all 52	`y = x + h`	H=6656, layer_scalar=False, r=1
`attn.core`	`decodeGQA`	13: 3, 7, …, 51	`o[b,h] = Σ_{s < seq_lens[b]} softmax_s(1/√128 · ⟨q[b,h], k[b,s,h / 16]⟩) · v[b,s,h / 16]`	D=128, heads=32, kv_heads=2, scale=1/√128
`final_norm`	`rmsnorm`	—	`y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · w[i]`	H=6656, eps=1e-05
`lm_head`	`lmHead`	—	`z = 0.19611613513818404 · x wᵀ, logits = 20 · tanh(z / 20)`	mul=0.196116, softcap=20, tied=False

Nodes listed twice have different constants in different layers (for example, sliding-window and global attention). A kernel annotated with layers[*].<node> is verified against each variant.

Tensors of the main nodes

Node	Inputs and outputs
`embed`	`tokens` [T], `y` [T, 6656]
`input_norm`	`x` [T, 6656], `y` [T, 6656]
`attn.qkv_proj`	`x` [T, 6656], `y` [T, 4608]
`attn.qk_norm`	`q` [T, 32, 128], `k` [T, 2, 128], `q_out` [T, 32, 128], `k_out` [T, 2, 128]
`attn.rope`	`q` [T, 32, 128], `k` [T, 2, 128], `pos` [T], `q_out` [T, 32, 128], `k_out` [T, 2, 128]
`attn.core`	`q` [B, 32, 128], `k` [B, S, 2, 128], `v` [B, S, 2, 128], `seq_lens` [B], `o` [B, 32, 128]
`attn.gate`	`o` [T, 4096], `gate` [T, 4096], `y` [T, 4096]
`attn.o_proj`	`x` [T, 4096], `y` [T, 6656]
`post_mix_norm`	`x` [T, 6656], `y` [T, 6656]
`attn_residual`	`x` [T, 6656], `h` [T, 6656], `y` [T, 6656]
`post_attn_norm`	`x` [T, 6656], `y` [T, 6656]
`mlp.gate_up`	`x` [T, 6656], `y` [T, 39936]
`mlp.act`	`gate_up` [T, 39936], `y` [T, 19968]
`mlp.down`	`x` [T, 19968], `y` [T, 6656]
`post_ffn_norm`	`x` [T, 6656], `y` [T, 6656]
`mlp_residual`	`x` [T, 6656], `h` [T, 6656], `y` [T, 6656]
`final_norm`	`x` [T, 6656], `y` [T, 6656]
`lm_head`	`x` [T, 6656], `logits` [T, 202048]

Conformance

The official transformers.MuseGlimmerForConditionalGeneration agrees with this spec, layer by layer in float64, to a maximum relative error of 7.7e-15 (reduced-width profile, 2 weight draws × 2 lengths).

theformalera-conformance verify meta-models/Muse-Glimmer-30B                    # the official implementation
theformalera-conformance verify meta-models/Muse-Glimmer-30B path/to/model.py   # your implementation
theformalera-conformance spec meta-models/Muse-Glimmer-30B                      # the architecture, as Lean derives it

Published configuration

The hyperparameters as the spec renders them (config.json form)

{
  "tie_word_embeddings": false,
  "text_config": {
    "vocab_size": 202048,
    "tie_word_embeddings": false,
    "sliding_window": 2048,
    "rope_parameters": {
      "rope_type": "default",
      "rope_theta": 500000
    },
    "rms_norm_eps": 1e-05,
    "qk_scale_factor": 3.87,
    "post_norm_eps": 1e-08,
    "output_multiplier": 0.19611613513818404,
    "num_key_value_heads": 2,
    "num_hidden_layers": 52,
    "num_attention_heads": 32,
    "layer_types": [
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "full_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "full_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "full_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "full_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "full_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "full_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "full_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "full_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "full_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "full_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "full_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "full_attention",
      "sliding_attention",
      "sliding_attention",
      "sliding_attention",
      "full_attention"
    ],
    "layer_rope_theta": [
      500000,
      500000,
      500000,
      0,
      500000,
      500000,
      500000,
      0,
      500000,
      500000,
      500000,
      0,
      500000,
      500000,
      500000,
      0,
      500000,
      500000,
      500000,
      0,
      500000,
      500000,
      500000,
      0,
      500000,
      500000,
      500000,
      0,
      500000,
      500000,
      500000,
      0,
      500000,
      500000,
      500000,
      0,
      500000,
      500000,
      500000,
      0,
      500000,
      500000,
      500000,
      0,
      500000,
      500000,
      500000,
      0,
      500000,
      500000,
      500000,
      0
    ],
    "intermediate_size": 19968,
    "hidden_size": 6656,
    "hidden_activation": "silu",
    "head_dim": 128,
    "final_logit_softcapping": 20,
    "attention_bias": false
  },
  "architectures": [
    "MuseGlimmerForConditionalGeneration"
  ]
}

Spec Hub

Qwen

Gemma

Granite

Muse

At a glance

What to watch for

Compute-graph nodes

Tensors of the main nodes

Conformance

Published configuration

​At a glance

​What to watch for

​Compute-graph nodes

​Tensors of the main nodes

​Conformance

​Published configuration

At a glance

What to watch for

Compute-graph nodes

Tensors of the main nodes

Conformance

Published configuration