Qwen3.5-4B-Q8_0-FlashNgram-NativeBridgeV3

Qwen3.5-4B-Q8_0-FlashNgram-NativeBridgeV3

Experimental research artifact. This is neither an official Qwen release nor a conversion of Qwen 3.8 into Qwen 3.5. It has known generation-stability limitations; see Limitations before use.

This GGUF retains a frozen Qwen 3.5 4B Q8_0 MTP backbone and embeds a frozen Qwen 3.8 Flash-Next n-gram lookup table. A 13.14M-parameter NativeBridge V3 adapter was trained to translate the retrieved Qwen 3.8 PLE features into an additive Qwen 3.5 residual update immediately before zero-based transformer block 2.

The published artifact is:

Qwen3.5-4B-Q8_0-FlashNgram-NativeBridgeV3-budget042-step2500-fixed.gguf

It is approximately 33.50 GB (31.20 GiB).

LiveCodeBench v6 — completed local run

Completed local evaluation on the LiveCodeBench v6 code_generation_lite set using the official dataset and checker. Generation uses one sample per problem (N=1). All 1,055 rows have a terminal outcome. This is not an official leaderboard result or a directly comparable full-benchmark score.

Difficulty breakdown — final results

Errors are incomplete generations or server-side failures rather than checker-verified incorrect solutions. The stricter 69.67% figure counts them as incorrect.

Earlier partial checkpoint

At an earlier 410-attempt checkpoint, the same in-progress run had 319 passes, 62 checker failures, and 29 generation/server errors: 83.73% excluding errors (319 / 381), or 77.80% when errors were counted as incorrect (319 / 410). This early result should not be treated as a stable estimate of the full-run result.

For context, the original Qwen3.5-4B model card reports 55.8 on LiveCodeBench v6. This NativeBridgeV3 evaluation is local, so it is not a direct leaderboard comparison.

Evaluation configuration

This completed N=1 LiveCodeBench v6 code_generation_lite run used the official dataset and checker.

• Model: Qwen3.5-4B-Q8_0-FlashNgram-NativeBridgeV3-budget042-step2500-fixed.gguf
• Runtime: llama.cpp, Vulkan backend; Qwen reasoning mode enabled
• NativeBridge configuration: ngram-gate=1
• Concurrency: 2 independent questions in parallel in the latest resumed segment (earlier segments used different concurrency; this does not change N=1 sampling)
• Sampling: temperature=0.6, top_p=0.95, top_k=20, min_p=0.0
• Penalties: presence_penalty=1.5, frequency_penalty=0.0, repeat_penalty=1.0
• Seed: not fixed; llama.cpp uses its random/default seed
• Checker timeout: 10 seconds per test case
• Output-token budget: initially 24,384; increased to 64,384 for the latest resumed portion of the run. The two portions should not be considered a strictly matched decoding configuration.

What is frozen and what was trained

Frozen Qwen 3.5 components:

• token embeddings, all 32 backbone blocks, MTP tensors, and LM head;
• the Q8_0 backbone quantization.

Frozen Qwen 3.8 Flash-Next PLE components:

• the embedded IQ4_NL n-gram table, per_layer_token_embd.weight, shape [160, 320001536];
• packed four-lane W_key, W_value, three RMSNorm scales, and the 10,240-channel depthwise causal-convolution kernel.

Trained NativeBridge V3 components:

• a Qwen-3.5-residual-to-PLE query projection, 2560 × 2560;
• four learned query-lane scales and biases;
• four-lane output-mixing logits;
• a PLE-to-Qwen-3.5 output projection, 2560 × 2560;
• one gate-bias scalar.

The learned checkpoint stores alpha=1, residual budget 0.42, and gate temperature 1.25.

Tokenizer compatibility and why a bridge is required

The Qwen 3.5 and Qwen 3.8 Flash-Next tokenizers were verified compatible for this experiment: they have the same 248,320-token vocabulary and matching BOS/EOS IDs. Consequently, identical text produces the same token-ID history and therefore the same n-gram hashes and PLE table-row lookups in both models.

That compatibility does not make the models' residual spaces compatible. An inspection of corresponding token embeddings found that same-token vectors had substantially different directions (low cosine similarity rather than an identity-like alignment). Directly adding a Qwen 3.8 PLE feature to the Qwen 3.5 residual stream would thus be an uncalibrated cross-space intervention. NativeBridge V3 is the learned translation layer: it creates Qwen-3.8-shaped query lanes from the Qwen 3.5 residual stream and maps the resulting PLE feature back into a Qwen-3.5-sized residual update.

Adapter computation

For each token, a bigram/trigram hash selects 16 table rows and produces a 2,560-wide PLE vector p_t. The bridge creates four Qwen-3.8-compatible query lanes from Qwen 3.5's single 2,560-wide residual stream:

key[t,l] = RMSNorm_key38(W_key38(p[t]))[l]

q_shared[t] = W_query35_to_PLE(h35[t])
query[t,l] = RMSNorm_query38(q_shared[t] * lane_scale[l] + lane_bias[l])

gate[t,l] = sigmoid((signed_sqrt(dot(key[t,l], query[t,l]) / sqrt(2560))
                     + gate_bias) / temperature)

z[t,l] = gate[t,l] * RMSNorm_conv38(W_value38(p[t]))
feature[t,l] = z[t,l] + SiLU(DepthwiseCausalConv38(z)[t,l])

feature35[t] = W_outputPLE_to_35(sum_l(softmax(lane_logits)[l] * feature[t,l]))
delta[t] = alpha * residual_budget * RMS(h35[t]) * feature35[t]
h35'[t] = h35[t] + delta[t]

The gate is per token × lane; it is not an attention distribution and does not sum to one. A low gate suppresses both the current PLE feature and its entry into the convolution's future state.

Four-lane bridge versus the original PLE routing

flowchart TB
    P["N-gram lookup<br/>pₜ ∈ R²⁵⁶⁰"]

    subgraph Q38["Original Qwen 3.8 Flash-Next PLE"]
        direction TB

        H38["Native HyperConnection residual<br/>h₃₈,t ∈ R¹⁰²⁴⁰"]

        H38 --> Q38L["Reshape into four native lanes"]
        Q38L --> Q380["query lane 0 · R²⁵⁶⁰"]
        Q38L --> Q381["query lane 1 · R²⁵⁶⁰"]
        Q38L --> Q382["query lane 2 · R²⁵⁶⁰"]
        Q38L --> Q383["query lane 3 · R²⁵⁶⁰"]

        P --> WK38["Frozen packed W_key38<br/>R²⁵⁶⁰ → R¹⁰²⁴⁰"]
        WK38 --> K38L["Four key lanes"]
        K38L --> K380["key lane 0 · R²⁵⁶⁰"]
        K38L --> K381["key lane 1 · R²⁵⁶⁰"]
        K38L --> K382["key lane 2 · R²⁵⁶⁰"]
        K38L --> K383["key lane 3 · R²⁵⁶⁰"]

        Q380 --> G38["four lane-wise<br/>key/query gates"]
        Q381 --> G38
        Q382 --> G38
        Q383 --> G38
        K380 --> G38
        K381 --> G38
        K382 --> G38
        K383 --> G38

        G38 --> F38["Native four-lane PLE feature"]
    end

    subgraph V3["Our Qwen 3.5 → Qwen 3.8 NativeBridge V3"]
        direction TB

        H35["Frozen Qwen 3.5 residual<br/>h₃₅,t ∈ R²⁵⁶⁰"]
        H35 --> WQ["Trained shared query projection<br/>W_query35→PLE · R²⁵⁶⁰ → R²⁵⁶⁰"]
        WQ --> QS["shared query · R²⁵⁶⁰"]

        QS --> A0["lane 0: q × scale₀ + bias₀"]
        QS --> A1["lane 1: q × scale₁ + bias₁"]
        QS --> A2["lane 2: q × scale₂ + bias₂"]
        QS --> A3["lane 3: q × scale₃ + bias₃"]

        A0 --> NQ["Frozen Qwen 3.8 query RMSNorm"]
        A1 --> NQ
        A2 --> NQ
        A3 --> NQ

        P --> WK["Frozen packed W_key38<br/>R²⁵⁶⁰ → R¹⁰²⁴⁰"]
        WK --> KL["four frozen key lanes"]

        NQ --> G["four gate values<br/>one per token × lane"]
        KL --> G

        P --> WV["Frozen W_value38<br/>R²⁵⁶⁰ → R²⁵⁶⁰"]
        WV --> RV["repeat value into four lanes"]

        G --> Z["gate × normalized value<br/>four lanes"]
        RV --> Z
        Z --> C["Frozen 4-lane causal<br/>depthwise convolution"]
        C --> F["z + SiLU(conv(z))<br/>four PLE feature lanes"]

        F --> MIX["Trained softmax lane mix<br/>R¹⁰²⁴⁰ → R²⁵⁶⁰"]
        MIX --> WO["Trained output bridge<br/>W_outputPLE→35"]
        WO --> D["deltaₜ = alpha × budget × RMS(h₃₅,t) × feature"]
        H35 --> ADD["Add before Qwen 3.5 block 2"]
        D --> ADD
    end

    style Q38 fill:#274860,color:#fff
    style V3 fill:#4b365b,color:#fff
    style WQ fill:#7c4d2f,color:#fff
    style MIX fill:#7c4d2f,color:#fff
    style WO fill:#7c4d2f,color:#fff
    style WK38 fill:#31594a,color:#fff
    style WK fill:#31594a,color:#fff
    style WV fill:#31594a,color:#fff
    style C fill:#31594a,color:#fff

The original model has a native four-lane 10,240-wide residual/query state. Qwen 3.5 has one 2,560-wide residual state, so V3 learns a shared query and four lightweight lane-specific affine views. The table, keys, values, normalization scales, and convolution remain frozen Qwen 3.8 components.

Training data

The training stream contained 150,000 packed 256-token sequences, sampled with fixed seed 1:

The mixture also has fixed-seed, non-overlapping eval and test splits of 4,096 sequences each. This checkpoint is a continuation-phase checkpoint, not a claim that all 150,000 available training positions were consumed in one run. Also training was done in around 26.500 (15.000 + 8.000 + 2500) steps (one sample per step due to 16GB VRAM) so not all training datapoints were touched. Gradually the graft was allowed to contribute more and more to the residuals.

Observed gate behavior

During the stronger-budget continuation that led into the selected 0.42 budget plateau, the mean calibrated gate decreased from roughly 0.58 early in the ramp to approximately 0.33–0.35. At the plateau, representative training batches showed:

gate mean ≈ 0.33–0.35
gate standard deviation ≈ 0.32–0.33
gate ≤ 0.1: ≈ 31–35% of token × lane values
gate ≥ 0.5: ≈ 30–34% of token × lane values
gate ≥ 0.9: ≈ 6–8% of token × lane values

Each token has four gate values, one for each PLE lane. Thus the mean is over all batch × tokens × 4 routing values in a log window. It means the bridge was selectively admitting PLE information rather than globally opening every lane; it does not mean that only 35% of n-gram table rows were looked up. The lookup still occurs, while low-gated features are suppressed before the causal convolution and residual update.

The delta telemetry is also logged as an RMS ratio, not as a signed-vector average:

delta / hidden = RMS(delta_t) / RMS(h35_t)

For the selected budget-0.42 checkpoint, the representative logged ratio was approximately 0.16–0.18 (0.162 at the saved step-2,500 checkpoint). In other words, the final additive update had an RMS magnitude of roughly 16–18% of the pre-block-2 Qwen 3.5 residual on the observed training batches. During the budget ramp it increased from about 0.06 to about 0.18 while the gate mean became more selective. This ratio describes the combined routed and projected PLE update; it is not the mean gate value and it does not say that individual residual dimensions are changed by a fixed 16–18%.

Full training charts, initiall 15.000 steps with 2000 steps ramp up, then another 8000 steps of ramp up then another 2500 steps that seemed to keep gates stable at 35% mean.

Perplexity evaluation

The following paired next-token PPL evaluations use identical deterministic 512-sequence samples for alpha 0 and 1 on each dataset. WikiText examples are packed to 256-token context. They were run in the PyTorch evaluation implementation with the Qwen 3.5 HF/BF16 backbone and the frozen IQ4_NL table. They are not GGUF-versus-GGUF runtime benchmarks.

The alpha-1 loss reductions are respectively 0.274442, 0.163656, 0.052000, and 0.319564 nats/token. These results show that the adapter improves next-token log-likelihood on these matched samples. They do not establish gains on math reasoning, coding tasks, factuality, or general user-facing quality.

Runtime requirement

This is not compatible with stock upstream llama.cpp. It requires the NativeBridge V3 runtime branch:

https://github.com/dburner/llama.cpp/tree/feature/qwen35-native-ple-bridge-v3

Build that branch with Vulkan support, then run target-model decoding:

.\llama-server.exe `
  -m .\Qwen3.5-4B-Q8_0-FlashNgram-NativeBridgeV3-budget042-step2500-fixed.gguf `
  --n-gpu-layers 99 `
  --ctx-size 48192 `
  --flash-attn on `
  --jinja `
  --reasoning on `
  --ngram-gate 1 `
  --spec-type none `
  --host 127.0.0.1 `
  --port 8080

Open http://127.0.0.1:8080/ for llama-server's built-in chat UI.

--ngram-gate is an inference-only multiplier on the trained V3 delta:

Do not use --spec-type draft-mtp / --spec-draft-n-max for this artifact. The MTP draft path is not V3-aware, so it does not provide a valid graft evaluation or serving path.

Limitations

• The PPL results were obtained in PyTorch. Exact numerical parity with the quantized custom llama.cpp runtime has not been established.
• PPL is a likelihood metric, not a reasoning, coding, factuality, safety, or instruction-following benchmark.
• The model has not been evaluated with a preregistered benchmark suite or a broad generation-stability evaluation.
• This artifact requires substantial local storage and a custom runtime; it is not a drop-in GGUF for standard llama.cpp frontends.

Reference training implementation: native_bridge.py

The following is the complete bridge module used for this release. The large n-gram table itself is supplied separately by the lookup wrapper; the module below contains the frozen Qwen 3.8 projections/norms/convolution and the trainable Qwen 3.5 bridge.

"""Frozen Qwen 3.8 PLE components and a trainable Qwen 3.5 bridge.

Only the compact PLE projections/norms/convolution are loaded here.  The huge
n-gram table remains in :mod:`ple_store` and is supplied as 2560-wide PLE
embeddings by the existing lookup wrapper.
"""
from __future__ import annotations

import math
from pathlib import Path

import torch
from safetensors.torch import load_file
from torch import nn
from torch.nn import functional as F


HIDDEN_SIZE = 2560
HC_COUNT = 4
HC_HIDDEN_SIZE = HIDDEN_SIZE * HC_COUNT


def _rms(x: torch.Tensor) -> torch.Tensor:
    return x.float().square().mean(dim=-1, keepdim=True).sqrt()


def _rms_norm(x: torch.Tensor, weight: torch.Tensor, eps: float = 1e-6) -> torch.Tensor:
    inv_rms = torch.rsqrt(x.float().square().mean(dim=-1, keepdim=True) + eps)
    return (x.float() * inv_rms) * weight.float()


class FrozenQwen38Ple(nn.Module):
    """Non-persistent frozen source PLE tensors extracted from Flash-Next."""

    REQUIRED = {
        "key_weight", "value_weight", "key_norm_weight", "query_norm_weight",
        "conv_norm_weight", "conv_weight",
    }

    def __init__(self, tensors: dict[str, torch.Tensor]) -> None:
        super().__init__()
        missing = self.REQUIRED - set(tensors)
        unexpected = set(tensors) - self.REQUIRED
        if missing or unexpected:
            raise ValueError(f"native PLE tensors mismatch; missing={sorted(missing)}, unexpected={sorted(unexpected)}")
        expected = {
            "key_weight": (HC_HIDDEN_SIZE, HIDDEN_SIZE),
            "value_weight": (HIDDEN_SIZE, HIDDEN_SIZE),
            "key_norm_weight": (HC_HIDDEN_SIZE,),
            "query_norm_weight": (HC_HIDDEN_SIZE,),
            "conv_norm_weight": (HC_HIDDEN_SIZE,),
            "conv_weight": (HC_HIDDEN_SIZE, 1, 4),
        }
        for name, shape in expected.items():
            value = tensors[name]
            if tuple(value.shape) != shape:
                raise ValueError(f"{name} shape {tuple(value.shape)} != {shape}")
            if not value.is_floating_point() or not torch.isfinite(value).all():
                raise ValueError(f"{name} must be finite floating point")
            # These buffers move with .to(), but are deliberately excluded
            # from checkpoints: each run names its immutable source file.
            self.register_buffer(name, value.contiguous(), persistent=False)

    @classmethod
    def from_safetensors(cls, path: Path) -> "FrozenQwen38Ple":
        if not path.is_file():
            raise FileNotFoundError(f"native Qwen 3.8 PLE source does not exist: {path}")
        return cls(load_file(path, device="cpu"))

    def key(self, ple: torch.Tensor) -> torch.Tensor:
        return _rms_norm(F.linear(ple.float(), self.key_weight.float()), self.key_norm_weight)

    def query_norm(self, query: torch.Tensor) -> torch.Tensor:
        return _rms_norm(query, self.query_norm_weight)

    def value(self, ple: torch.Tensor) -> torch.Tensor:
        return F.linear(ple.float(), self.value_weight.float())

    def conv_norm(self, value_hc: torch.Tensor) -> torch.Tensor:
        return _rms_norm(value_hc, self.conv_norm_weight)

    def causal_conv(self, x: torch.Tensor) -> torch.Tensor:
        """Apply the source depthwise kernel with explicit left-only padding."""
        if x.ndim != 3 or x.shape[-1] != HC_HIDDEN_SIZE:
            raise ValueError(f"native conv expects [batch, tokens, {HC_HIDDEN_SIZE}]")
        channels = x.transpose(1, 2)
        padded = F.pad(channels, (9, 0))  # dilation 3 * (kernel 4 - 1)
        return F.conv1d(padded, self.conv_weight.float(), dilation=3,
                        groups=HC_HIDDEN_SIZE).transpose(1, 2)


class Qwen35NativePleBridge(nn.Module):
    """Trainable Qwen 3.5 <-> frozen Qwen 3.8 four-lane PLE bridge (v3).

    ``gate_bias`` is deliberately a single trainable scalar, rather than a
    modification of a Qwen 3.8 tensor.  It calibrates how often the frozen
    source PLE is used for this new Qwen 3.5 residual stream.  Temperature is
    fixed per run so that the bridge cannot evade a gate-usage objective merely
    by making its logits arbitrarily sharp.
    """

    def __init__(self, native_ple: FrozenQwen38Ple, *, hidden_size: int = HIDDEN_SIZE,
                 ple_embed_dim: int = HIDDEN_SIZE, residual_budget: float = 0.05,
                 gate_bias_init: float = 0.0, gate_temperature: float = 1.0) -> None:
        super().__init__()
        if hidden_size != HIDDEN_SIZE or ple_embed_dim != HIDDEN_SIZE:
            raise ValueError("native bridge requires 2560-wide Qwen 3.5 residuals and PLE embeddings")
        if not math.isfinite(residual_budget) or residual_budget <= 0:
            raise ValueError("residual_budget must be finite and positive")
        if not math.isfinite(gate_bias_init):
            raise ValueError("gate_bias_init must be finite")
        if not math.isfinite(gate_temperature) or gate_temperature <= 0:
            raise ValueError("gate_temperature must be finite and positive")
        self.hidden_size = hidden_size
        self.ple_embed_dim = ple_embed_dim
        self.hc_count = HC_COUNT
        # Unlike a Python float, the budget is checkpointed.  That is required
        # for a continuation phase that ramps it: evaluation must reconstruct
        # the exact budget of the selected checkpoint, not just the run's
        # starting value.
        self.register_buffer("residual_budget", torch.tensor(residual_budget, dtype=torch.float32), persistent=True)
        self.native_ple = native_ple
        self.query_proj = nn.Linear(hidden_size, hidden_size, bias=False)
        self.query_lane_scale = nn.Parameter(torch.ones(HC_COUNT, hidden_size, dtype=torch.float32))
        self.query_lane_bias = nn.Parameter(torch.zeros(HC_COUNT, hidden_size, dtype=torch.float32))
        self.output_lane_logits = nn.Parameter(torch.zeros(HC_COUNT, hidden_size, dtype=torch.float32))
        self.output_proj = nn.Linear(hidden_size, hidden_size, bias=False)
        self.gate_bias = nn.Parameter(torch.tensor(gate_bias_init, dtype=torch.float32))
        # This is persistent so a checkpoint carries its exact calibration,
        # but is intentionally not trainable.
        self.register_buffer("gate_temperature", torch.tensor(gate_temperature, dtype=torch.float32), persistent=True)
        self.alpha = nn.Parameter(torch.zeros((), dtype=torch.float32))
        self.register_buffer("architecture_version", torch.tensor(3, dtype=torch.int32), persistent=True)
        self.last_activation_metrics: dict[str, torch.Tensor] = {}
        self.gate_mean_for_regularization: torch.Tensor | None = None

    def load_state_dict(self, state_dict: dict[str, torch.Tensor], strict: bool = True):
        """Load pre-calibration v3 checkpoints as an explicitly neutral gate.

        Early v3 runs did not contain calibration tensors.  Their behaviour is
        exactly represented by bias=0 and temperature=1, which also lets them
        remain valid alpha-ablation baselines after this extension.
        """
        compatible_state = dict(state_dict)
        compatible_state.setdefault("gate_bias", torch.zeros_like(self.gate_bias))
        compatible_state.setdefault("gate_temperature", torch.ones_like(self.gate_temperature))
        compatible_state.setdefault("residual_budget", self.residual_budget.detach().clone())
        return super().load_state_dict(compatible_state, strict=strict)

    def set_residual_budget(self, value: float) -> None:
        """Set the frozen global residual multiplier for the current step."""
        if not math.isfinite(value) or value <= 0:
            raise ValueError("residual budget must be finite and positive")
        with torch.no_grad():
            self.residual_budget.fill_(value)

    def forward(self, hidden_states: torch.Tensor, ple_embeddings: torch.Tensor) -> torch.Tensor:
        if hidden_states.ndim != 3 or ple_embeddings.ndim != 3:
            raise ValueError("hidden_states and ple_embeddings must have shape [batch, tokens, width]")
        if hidden_states.shape[:2] != ple_embeddings.shape[:2]:
            raise ValueError("hidden_states and ple_embeddings must have matching batch and token dimensions")
        if hidden_states.shape[-1] != self.hidden_size or ple_embeddings.shape[-1] != self.ple_embed_dim:
            raise ValueError("native bridge width mismatch")
        batch, tokens, _ = hidden_states.shape

        key = self.native_ple.key(ple_embeddings).reshape(batch, tokens, HC_COUNT, HIDDEN_SIZE)
        query_shared = self.query_proj(hidden_states.float())
        query_hc = query_shared.unsqueeze(-2) * self.query_lane_scale + self.query_lane_bias
        query = self.native_ple.query_norm(query_hc.reshape(batch, tokens, HC_HIDDEN_SIZE))
        query = query.reshape(batch, tokens, HC_COUNT, HIDDEN_SIZE)
        score = (key.float() * query.float()).sum(dim=-1, keepdim=True) / math.sqrt(HIDDEN_SIZE)
        signed_sqrt = score.sign() * score.abs().clamp_min(torch.finfo(torch.float32).tiny).sqrt()
        raw_gate = torch.sigmoid(signed_sqrt)
        gate_logits = (signed_sqrt + self.gate_bias) / self.gate_temperature
        gate = torch.sigmoid(gate_logits)
        # Retain the differentiable batch mean only until the trainer has
        # formed its optional usage regularizer for this forward pass.
        self.gate_mean_for_regularization = gate.mean()

        value = self.native_ple.value(ple_embeddings)
        value_hc = value.unsqueeze(-2).expand(-1, -1, HC_COUNT, -1)
        value_norm = self.native_ple.conv_norm(value_hc.reshape(batch, tokens, HC_HIDDEN_SIZE))
        z = gate.to(dtype=value_norm.dtype) * value_norm.reshape(batch, tokens, HC_COUNT, HIDDEN_SIZE)
        conv = F.silu(self.native_ple.causal_conv(z.reshape(batch, tokens, HC_HIDDEN_SIZE)))
        conv_hc = conv.reshape(batch, tokens, HC_COUNT, HIDDEN_SIZE)
        feature_hc = z + conv_hc
        lane_weights = torch.softmax(self.output_lane_logits, dim=0)
        mixed = (feature_hc.float() * lane_weights.unsqueeze(0).unsqueeze(0)).sum(dim=-2)
        feature35 = self.output_proj(mixed)
        # Gate is already inside feature35.  Do not divide by its RMS here.
        hidden_scale = _rms(hidden_states).detach()
        injected = (self.alpha.to(dtype=feature35.dtype) * self.residual_budget.to(dtype=feature35.dtype)
                    * hidden_scale.to(dtype=feature35.dtype) * feature35)

        with torch.no_grad():
            hidden_global_rms = hidden_states.detach().float().square().mean().sqrt()
            eps = torch.finfo(torch.float32).tiny
            def relative_rms(x: torch.Tensor) -> torch.Tensor:
                return x.detach().float().square().mean().sqrt() / hidden_global_rms.clamp_min(eps)
            gate_float = gate.detach().float()
            raw_gate_float = raw_gate.detach().float()
            injected_float = injected.detach().float()
            injected_rms = injected_float.square().mean().sqrt()
            self.last_activation_metrics = {
                "gate_mean": gate_float.mean(),
                "gate_std": gate_float.std(unbiased=False),
                "raw_gate_mean": raw_gate_float.mean(),
                "gate_bias": self.gate_bias.detach().float(),
                "gate_temperature": self.gate_temperature.detach().float(),
                "residual_budget": self.residual_budget.detach().float(),
                "gate_fraction_le_0_1": (gate_float <= 0.1).float().mean(),
                "gate_fraction_ge_0_5": (gate_float >= 0.5).float().mean(),
                "gate_fraction_ge_0_9": (gate_float >= 0.9).float().mean(),
                "value_to_hidden_rms": relative_rms(value),
                "z_to_hidden_rms": relative_rms(z),
                "conv_to_hidden_rms": relative_rms(conv_hc),
                "feature_hc_to_hidden_rms": relative_rms(feature_hc),
                "bridge_feature_rms": feature35.detach().float().square().mean().sqrt(),
                "injected_rms": injected_rms,
                "injected_to_hidden_rms": injected_rms / hidden_global_rms.clamp_min(eps),
            }
        return injected.to(dtype=hidden_states.dtype)

License and attribution

This artifact incorporates Qwen 3.5 model material and Qwen 3.8 Flash-Next PLE material. Redistribution requires compliance with all applicable Qwen licenses, acceptable-use terms, and the terms of the training datasets. This repository does not grant additional rights to those materials.

Describe this release as a Qwen 3.5 NativeBridge V3 experimental n-gram PLE graft, not as an official Qwen release or a converted Qwen 3.8 model.