> ## Documentation Index
> Fetch the complete documentation index at: https://docs.theformalera.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Muse-Glimmer-30B

> Muse Glimmer: 3 sliding-window layers per NoPE global layer, gated attention, sandwich norms.

conformance ✓spec 1.0muse\_glimmer

The Model Spec of [`meta-models/Muse-Glimmer-30B`](https://huggingface.co/meta-models/Muse-Glimmer-30B) in `TheFormalEra.Spec.Registry`. The constants below are transcribed from its published `config.json`, and the official `transformers.MuseGlimmerForConditionalGeneration` is checked against them.

## At a glance

| | |
| - | - |
| Model key | `meta-models/Muse-Glimmer-30B` |
| Parameters (text model) | 27,854,780,928 |
| Layers | 52 (39 sliding-window attention, 13 full attention) |
| Hidden size | 6,656 |
| MLP width | 19,968 (silu) |
| Vocabulary | 202,048 |
| Norm | `(1+w)·x̂`, ε = 1e-05 |
| Embedding | separate from the LM head, then RMSNorm |
| Logits | × 0.196116, soft-capped at 20 |
| Hugging Face class | `MuseGlimmerForConditionalGeneration` |
| Shape domain | `T` ∈ \[1, 131,072] tokens, `B` ∈ \[1, 256] sequences, `S` ∈ \[1, 131,072] cached positions, page size 16 |

## What to watch for

* Layer `i` is global iff `i % 4 = 3`: 13 global layers and 39 sliding-window layers with a window of 2048. The global layers are exactly the NoPE layers (`museGlimmer_global_iff_nope`): they use no position encoding.
* 32 query heads and 2 KV heads (16 query heads per KV head), head dim 128. Queries and keys get an unweighted RMSNorm; queries are then multiplied by 3.87.
* Every layer has a sigmoid output gate from a separate projection (`attn.gate`).
* RMSNorm weights are applied as `(1 + w)` (`rmsnormOffset`), except the final norm, which uses `w`. Sandwich norms use ε = 1e-8.
* The embedding is followed by an unweighted RMSNorm. Logits are multiplied by 0.19611613513818404, then soft-capped: `20 · tanh(z / 20)`.
* Scope: the text-only path. The vision tower is out of scope.

## Compute-graph nodes

Use these paths in Spec Keys: `meta-models/Muse-Glimmer-30B@1.0:layers[*].<node>` for per-layer nodes, `meta-models/Muse-Glimmer-30B@1.0:<node>` for the others.

| Node | Op Spec | Layers | Formula | Constants |
| - | - | - | - | - |
| `embed` | `embed` | — | `y[t] = 1 · w[tokens[t]], then RMSNorm` | mul=1, norm=True |
| `input_norm` | `rmsnormOffset` | all 52 | `y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · (1 + w[i])` | H=6656, eps=1e-05 |
| `attn.qkv_proj` | `linear` | all 52 | `y[t,n] = Σₖ x[t,k] · w[n,k]` | K=6656, N=4608 |
| `attn.qk_norm` | `rmsnormUnit` | all 52 | `per head: y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε); q · 3.87` | D=128, eps=1e-05 |
| `attn.rope` | `rope` | 39: all but 3, 7, …, 51 | `rotate pairs (i, i + 64) by pos · θ^(-2i/128), i < 64; coordinates ≥ 128 unchanged` | D=128, active=64, denom=128, rot=128, theta=500000 |
| `attn.core` | `slidingDecodeGQA` | 39: all but 3, 7, …, 51 | `o[b,h] = Σ_{s < seq_lens[b]} softmax_s(1/√128 · ⟨q[b,h], k[b,s,h / 16]⟩) · v[b,s,h / 16] over the last 2048 positions` | D=128, heads=32, kv\_heads=2, scale=1/√128, window=2048 |
| `attn.gate` | `sigmoidGate` | all 52 | `y = o ⊙ σ(gate)` | N=4096 |
| `attn.o_proj` | `linear` | all 52 | `y[t,n] = Σₖ x[t,k] · w[n,k]` | K=4096, N=6656 |
| `post_mix_norm` | `rmsnormOffset` | all 52 | `y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · (1 + w[i])` | H=6656, eps=1e-08 |
| `attn_residual` | `residual` | all 52 | `y = x + h` | H=6656, layer\_scalar=False, r=1 |
| `post_attn_norm` | `rmsnormOffset` | all 52 | `y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · (1 + w[i])` | H=6656, eps=1e-05 |
| `mlp.gate_up` | `linear` | all 52 | `y[t,n] = Σₖ x[t,k] · w[n,k]` | K=6656, N=39936 |
| `mlp.act` | `siluMul` | all 52 | `y[t,i] = silu(gate_up[t,i]) · gate_up[t,19968+i]` | I=19968, act=silu |
| `mlp.down` | `linear` | all 52 | `y[t,n] = Σₖ x[t,k] · w[n,k]` | K=19968, N=6656 |
| `post_ffn_norm` | `rmsnormOffset` | all 52 | `y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · (1 + w[i])` | H=6656, eps=1e-08 |
| `mlp_residual` | `residual` | all 52 | `y = x + h` | H=6656, layer\_scalar=False, r=1 |
| `attn.core` | `decodeGQA` | 13: 3, 7, …, 51 | `o[b,h] = Σ_{s < seq_lens[b]} softmax_s(1/√128 · ⟨q[b,h], k[b,s,h / 16]⟩) · v[b,s,h / 16]` | D=128, heads=32, kv\_heads=2, scale=1/√128 |
| `final_norm` | `rmsnorm` | — | `y[t,i] = x[t,i] / √(Σⱼ x[t,j]² / H + ε) · w[i]` | H=6656, eps=1e-05 |
| `lm_head` | `lmHead` | — | `z = 0.19611613513818404 · x wᵀ, logits = 20 · tanh(z / 20)` | mul=0.196116, softcap=20, tied=False |

Nodes listed twice have different constants in different layers (for example, sliding-window and global attention). A kernel annotated with `layers[*].<node>` is verified against each variant.

## Tensors of the main nodes

| Node | Inputs and outputs |
| - | - |
| `embed` | `tokens` \[T], `y` \[T, 6656] |
| `input_norm` | `x` \[T, 6656], `y` \[T, 6656] |
| `attn.qkv_proj` | `x` \[T, 6656], `y` \[T, 4608] |
| `attn.qk_norm` | `q` \[T, 32, 128], `k` \[T, 2, 128], `q_out` \[T, 32, 128], `k_out` \[T, 2, 128] |
| `attn.rope` | `q` \[T, 32, 128], `k` \[T, 2, 128], `pos` \[T], `q_out` \[T, 32, 128], `k_out` \[T, 2, 128] |
| `attn.core` | `q` \[B, 32, 128], `k` \[B, S, 2, 128], `v` \[B, S, 2, 128], `seq_lens` \[B], `o` \[B, 32, 128] |
| `attn.gate` | `o` \[T, 4096], `gate` \[T, 4096], `y` \[T, 4096] |
| `attn.o_proj` | `x` \[T, 4096], `y` \[T, 6656] |
| `post_mix_norm` | `x` \[T, 6656], `y` \[T, 6656] |
| `attn_residual` | `x` \[T, 6656], `h` \[T, 6656], `y` \[T, 6656] |
| `post_attn_norm` | `x` \[T, 6656], `y` \[T, 6656] |
| `mlp.gate_up` | `x` \[T, 6656], `y` \[T, 39936] |
| `mlp.act` | `gate_up` \[T, 39936], `y` \[T, 19968] |
| `mlp.down` | `x` \[T, 19968], `y` \[T, 6656] |
| `post_ffn_norm` | `x` \[T, 6656], `y` \[T, 6656] |
| `mlp_residual` | `x` \[T, 6656], `h` \[T, 6656], `y` \[T, 6656] |
| `final_norm` | `x` \[T, 6656], `y` \[T, 6656] |
| `lm_head` | `x` \[T, 6656], `logits` \[T, 202048] |

## Conformance

The official `transformers.MuseGlimmerForConditionalGeneration` agrees with this spec, layer by layer in float64, to a maximum relative error of **7.7e-15** (reduced-width profile, 2 weight draws × 2 lengths).

```bash theme={"system"}
theformalera-conformance verify meta-models/Muse-Glimmer-30B                    # the official implementation
theformalera-conformance verify meta-models/Muse-Glimmer-30B path/to/model.py   # your implementation
theformalera-conformance spec meta-models/Muse-Glimmer-30B                      # the architecture, as Lean derives it
```

## Published configuration

> [accordion] The hyperparameters as the spec renders them (config.json form)
>
> ```json theme={"system"}
> {
>   "tie_word_embeddings": false,
>   "text_config": {
>     "vocab_size": 202048,
>     "tie_word_embeddings": false,
>     "sliding_window": 2048,
>     "rope_parameters": {
>       "rope_type": "default",
>       "rope_theta": 500000
>     },
>     "rms_norm_eps": 1e-05,
>     "qk_scale_factor": 3.87,
>     "post_norm_eps": 1e-08,
>     "output_multiplier": 0.19611613513818404,
>     "num_key_value_heads": 2,
>     "num_hidden_layers": 52,
>     "num_attention_heads": 32,
>     "layer_types": [
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "full_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "full_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "full_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "full_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "full_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "full_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "full_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "full_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "full_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "full_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "full_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "full_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "sliding_attention",
>       "full_attention"
>     ],
>     "layer_rope_theta": [
>       500000,
>       500000,
>       500000,
>       0,
>       500000,
>       500000,
>       500000,
>       0,
>       500000,
>       500000,
>       500000,
>       0,
>       500000,
>       500000,
>       500000,
>       0,
>       500000,
>       500000,
>       500000,
>       0,
>       500000,
>       500000,
>       500000,
>       0,
>       500000,
>       500000,
>       500000,
>       0,
>       500000,
>       500000,
>       500000,
>       0,
>       500000,
>       500000,
>       500000,
>       0,
>       500000,
>       500000,
>       500000,
>       0,
>       500000,
>       500000,
>       500000,
>       0,
>       500000,
>       500000,
>       500000,
>       0,
>       500000,
>       500000,
>       500000,
>       0
>     ],
>     "intermediate_size": 19968,
>     "hidden_size": 6656,
>     "hidden_activation": "silu",
>     "head_dim": 128,
>     "final_logit_softcapping": 20,
>     "attention_bias": false
>   },
>   "architectures": [
>     "MuseGlimmerForConditionalGeneration"
>   ]
> }
> ```


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.