gemma4-31B-1bit

1-Bit Quantisation for LLMs: DeepSeek-V4 Pro & Gemma 4 31B IT

Gemma 4 31B — Experimental 1-bit Quantization

Status (August 2026): Proof-of-concept.
True 1-bit packed weights (Gemma 4 31B IT) + Adaptive Mask approach (DeepSeek-V4 Pro).

What works right now

• Model packed into ~3.5 GB GGUF
• Structure successfully verified on Pixel 4 XL (6 GB RAM) via Termux + patched ik_llama.cpp
• Detailed per-layer R² analysis available

What does not work well yet

• Real text generation quality is significantly degraded (error accumulation across layers)
• Full correct inference on Android/iPhone is not stable
• Real-generation R² is much lower than weight-level / synthetic R²

This is research / experimental work, not a production-ready model.

GitHub (setup + patches): https://github.com/elliotlib/quant-1bit
Contact: [email protected] | X: @liberal17th

Looking for collaboration: compute, custom 1-bit kernels, or joint research.

---

Part 1: DeepSeek-V4 Pro — Adaptive Mask V2

1.1.1 Running on android

How run on android: Repository on GitHub

1.1 What We Are Approximating

DeepSeek-V4 Pro is a Mixture-of-Experts (MoE) model with the following architecture:

The model uses Multi-Head Latent Attention (MLA) — a compressed attention mechanism where Q, K, V projections are decomposed into low-rank matrices (q_a_proj, q_b_proj, kv_a_proj, kv_b_proj, q_down_proj, q_up_proj). This adds complexity because weight matrices are no longer simple [out_dim, in_dim] projections but involve compressed latent representations.

1.2 The Core Problem

We want to approximate every floating-point weight tensor in the model using a representation that uses effectively 1 bit per weight (or close to it), while preserving the model's output distribution during text generation. The challenge is that:

• A 1-bit weight has only 2 states: +scale or -scale
• LLM weights are normally distributed with mean ≈ 0 and small std
• Simply thresholding at zero destroys the fine-grained magnitude information that the model learned during training
• The approximation must preserve not just the weight values, but their behavior under matrix multiplication with real input activations

1.3 Technical Approach: Adaptive Mask V2

The method does NOT use SVD, quantization grids, or codebooks. Instead, it uses an iterative refinement process based on structured group scalars and adaptive block-wise masking.

Step-by-Step Algorithm

Step 1: Normalization

W_mean = mean(W)
W_std  = std(W) + 1e-8
W_norm = (W - W_mean) / W_std

This centers the weights and normalizes variance, making the subsequent masking scale-invariant.

Step 2: Block-wise Energy Analysis The weight matrix is divided into non-overlapping blocks (default 8×8 or 15×15). For a matrix of shape [rows, cols]:

pad_rows = (block_size - rows % block_size) % block_size
pad_cols = (block_size - cols % block_size) % block_size
W_padded = pad(W_norm, (0, pad_cols, 0, pad_rows))
num_row_blocks = padded_rows // block_size
num_col_blocks = padded_cols // block_size
W_blocks = W_padded.reshape(num_row_blocks, block_size, num_col_blocks, block_size)
block_energy = sum(abs(W_blocks), dim=(1, 3))  # Sum of absolute values per block

Step 3: Adaptive Mask Selection Based on the layer type, a coverage ratio is selected:

The top-k blocks by energy are selected, where k = total_blocks × coverage_ratio.

Step 4: Multi-Pass Scalar Refinement (8-15 passes) This is the core innovation. Instead of storing individual weights, we store group scalars that reconstruct the weights:

For each pass:
    For each group of 64 rows:
        W_group = current_residual[start:end]
        mask_group = adaptive_mask[start:end]

        # Positive scalar: mean of positive masked elements
        pos_mask = (W_group > 0) * mask_group
        s_plus = sum(W_group * pos_mask) / (sum(pos_mask) + 1e-8)

        # Negative scalar: mean of negative masked elements
        neg_mask = (W_group < 0) * mask_group
        s_minus = sum(W_group * neg_mask) / (sum(neg_mask) + 1e-8)

        # Reconstruct this group
        reconstruction += pos_mask * s_plus + neg_mask * s_minus

    # Compute residual for next pass
    residual = W_norm - reconstruction
    current_residual = residual * adaptive_mask

    # Adaptively shrink coverage for next pass (focus on hardest elements)
    if not last_pass:
        current_ratio = max(initial_ratio * (0.95^pass_idx), 0.50)
        recompute mask on |residual| with new ratio

Step 5: Denormalization and Residual Bias

W_recon = reconstruction * W_std + W_mean
# For elements NOT covered by mask, add mean residual bias
uncovered_mask = (mask < 0.5)
residual_mean = sum((W_orig - W_recon) * uncovered_mask) / sum(uncovered_mask)
W_recon += residual_mean * uncovered_mask

Step 6: Real Generation R² Check This is critical: we don't just compare W_orig vs W_recon elementwise. Instead:

1. Download real token embeddings from the model's embedding layer
2. Generate random token sequences
3. Run forward passes through the actual layer operation (linear, rmsnorm, embedding)
4. Compare output activations: R² = 1 - SS_res / SS_tot

1.4 What Gets Stored (The "Compressed" Representation)

For each tensor, we store:

1. Global norm params: W_mean, W_std (2 floats)
2. Scalar layers: For each of 8 passes, for each of N groups: (s_plus, s_minus) — stored as flat list
3. Mask metadata: The actual mask is NOT stored; it's recomputed from the stored scalars and norm params during reconstruction
4. Coverage ratio: The actual ratio used

This is NOT true 1-bit storage — it's a structured scalar approximation. The theoretical compression is extreme because we store ~2×num_groups×num_passes scalars instead of millions of weights.

1.5 Results on Layer 0 (14 Tensors)

Aggregate Statistics:

1.6 Key Observations from DeepSeek Results

1. RMSNorm layers are trivially compressible: R² > 0.999 because they have only ~5K-7K elements, small variance, and the operation (elementwise multiply with reciprocal sqrt of variance) is linear in the weight.

2. Small 1D tensors (hc_base, attn_sink) also compress well: R² > 0.99 because there are only 24-128 elements — the scalar model has enough degrees of freedom.

3. Large matrices (FFN gate, compressor) are harder: The FFN gate at [384, 7,168] = 22M elements achieves only R² = 0.8479. This is the hardest layer type because:

   • Gate projections use SwiGLU: output = gate(x) * up(x), where gate = sigmoid(linear(x)) — the nonlinearity amplifies weight errors
   • The gate matrix has a "bottleneck" structure (384 → 7,168) that is sensitive to sign flips

4. MLA projections show consistent R² ~0.9065: The compressed attention mechanism (q_a, q_b, kv_a) has structured low-rank properties that make it more amenable to scalar approximation than standard attention.

---

Part 2: Gemma 4 31B IT — Packed 1-Bit V11

2.1 What We Are Approximating

Gemma 4 31B IT (Instruction-Tuned) has a dense transformer architecture (no MoE):

Vision encoder (for multimodal understanding):

2.2 The Core Problem (Same, Different Approach)

Unlike DeepSeek's scalar-only approach, Gemma uses true 1-bit packing:

• Each weight is represented by a single bit: 1 if ≥ 0, 0 if < 0
• Magnitude is recovered via per-row (or per-group) floating-point scales
• This achieves genuine ~16x compression vs FP16

The challenge is that sign-only representation loses all magnitude information within each sign class. The row-wise scales recover only the mean magnitude per row, not per-element magnitudes.

2.3 Technical Approach: Packed 1-Bit V11

Phase 1: Weight Quantization (Same as DeepSeek's masking but with true bit-packing)

Step 1-5: Same as DeepSeek — normalization, block-wise energy masking, multi-pass scalar refinement, denormalization.

Step 6: Sign Extraction and Bit Packing

# After reconstruction, extract signs
signs = (W_recon >= 0).flatten().cpu().numpy()[:numel]
# Pack 8 booleans into 1 byte
packed_bits = np.packbits(signs)  # torch.uint8 tensor
# Store: packed_bits (1 bit per element), row_scales (float32 per row × 2)

Step 7: Row-wise Scale Extraction

num_rows = shape[0]
row_scales = zeros(num_rows, 2)  # [pos_mean, neg_mean] per row
for r in range(num_rows):
    row = W_recon[r]
    pos = row[row > 0]
    neg = row[row <= 0]
    row_scales[r, 0] = mean(pos) if len(pos) > 0 else 0.0
    row_scales[r, 1] = mean(neg) if len(neg) > 0 else 0.0

Phase 2: Recursive Scale Quantization (The V11 Innovation)

The row_scales themselves are float32 tensors. For a [8,192, 2] scales tensor (from Q_Proj), that's 16,384 floats = 64 KB. While small, at scale across all layers this adds up. The V11 method quantizes scales themselves using the same 1-bit approach:

Algorithm for scale quantization:

Input: row_scales [num_rows, 2], target_r2=0.90, max_attempts=4

Configs (from coarse to fine):
  Attempt 1: block_size=4, passes=4, ratio=0.70
  Attempt 2: block_size=4, passes=6, ratio=0.80
  Attempt 3: block_size=2, passes=6, ratio=0.90
  Attempt 4: block_size=2, passes=8, ratio=0.97

For each config:
    packed_s, coverage = PackedOneBitTensor.from_float(S, config)
    S_recon = packed_s.to_dense()
    r2_s = 1 - sum((S - S_recon)^2) / sum((S - mean(S))^2)
    if r2_s >= target_r2: break
    track best attempt

Store: scales_packed (1-bit packed) + scales_of_scales (float32, tiny)

Phase 3: ShardStats — Cross-Shard Adaptive Ratio

This is the most important innovation for practical deployment. The problem: static ratios (e.g., attention=0.88) don't account for weight distribution variance across layers. Layer 0's Q_Proj may compress well at 88%, but Layer 30's may need 95%.

ShardStats Algorithm:

class ShardStats:
    TARGET_R2 = 0.90
    EMA_ALPHA = 0.3
    MIN_SAMPLES = 2
    ADAPT_STEP = 0.03
    CAP = 0.12

    def suggest_ratio(layer_type, numel):
        base = RATIOS[layer_type]  # static base
        if count[layer_type] < MIN_SAMPLES:
            return base  # not enough data yet
        adjusted = clip(base + offset[layer_type], 0.50, 0.99)
        return adjusted

    def update(layer_type, measured_r2, ratio_used):
        # Update EMA of R² for this layer type
        ema_r2[layer_type] = (ALPHA * measured_r2 + (1-ALPHA) * ema_r2[layer_type])

        # Adjust offset based on whether we're hitting target
        if ema_r2 < TARGET_R2 - 0.02:
            offset[layer_type] = min(CAP, offset + ADAPT_STEP)  # need more coverage
        elif ema_r2 > TARGET_R2 + 0.05:
            offset[layer_type] = max(-CAP, offset - ADAPT_STEP)  # can afford less

Why this matters:

• Early layers (0-10) may have "cleaner" weight distributions (closer to Gaussian)
• Later layers (40-60) may have "spikier" distributions (specialized features)
• Without adaptation, you'd use the same ratio for all, causing some layers to fail
• ShardStats learns this online as it processes shards, improving consistency

2.4 What Gets Stored (The "Compressed" Representation)

For each tensor:

1. packed_bits: torch.uint8 tensor, ceil(numel / 8) bytes (true 1-bit packing)
2. row_scales: [num_rows, 2] float32 tensor (or recursively quantized)
3. shape: Original shape for reconstruction
4. numel: Number of elements
5. W_mean, W_std: Global normalization params
6. coverage: Actual mask coverage achieved

If scales are quantized: 7. scales_packed_bits: 1-bit packed version of row_scales 8. scales_packed_scales: Scalars for the scales (meta-scales) 9. scales_packed_numel: Element count for scale tensor

2.5 Results: Real Generation Check on Layer 0 & Layer 1

These results use real prompt-based activation checks — not just weight R², but R² of the actual layer outputs when fed real token embeddings from the model.

Layer 0: Post-Feedforward Layernorm

Analysis: The weight R² is terrible (0.0076) because the 1-bit approximation of a 5K-element vector is extremely coarse. However, in RMSNorm the weight acts as a gain factor after variance normalization. The activation R² (0.6594) is much higher because:

• The input activations x have their own variance structure
• The error in the weight is multiplied by x, but x is normalized: output = x / sqrt(var(x) + eps) * weight
• The normalization step absorbs some of the weight error

Layer 0: Pre-Feedforward Layernorm

Analysis: Same pattern — weight R² is near-zero, but activation R² is 0.94. The pre-FFN norm weight has smaller relative error (0.34 vs 0.78), leading to better activation preservation. This shows that weight R² is not predictive of activation R² for normalization layers.

Layer 0: K_Norm (Attention Key Norm)

Analysis: Perfect reconstruction. Only 256 elements — the scalar model has more parameters than data points. This is trivial.

Layer 0: K_Proj (Key Projection)

Scale Quantization Attempts:

Analysis:

• Weight R² (0.4614) and activation R² (0.4812) are close — for linear layers, weight error directly propagates to activation error
• The quantized scales achieve R² = 0.9481, meaning the scale quantization itself is high-quality
• However, even perfect scales can't fix the fundamental limitation: 1-bit signs lose intra-row magnitude variation
• Coverage of 72.8% means 27.2% of elements are reconstructed from residual bias only

Layer 0: O_Proj (Output Projection)

Analysis: O_Proj is the hardest attention matrix because it projects from concatenated heads (8,192) back to hidden size (5,376). The high output dimension means each row has 8K elements — row-wise scales average over too many values, losing fine structure.

Layer 0: Q_Norm (Query Norm)

Same as K_Norm — trivially perfect.

Layer 0: Q_Proj (Query Projection)

Scale Quantization:

Analysis: Q_Proj shows the same pattern as K_Proj. The 1-bit weight R² (~0.45) is the fundamental bottleneck. Even with near-perfect scale quantization (R²=0.9371), the activation R² caps at ~0.49.

Layer 0: V_Proj (Value Projection)

Scale Quantization:

Analysis: V_Proj achieves the highest weight R² (0.5122) among attention projections. This is because value projections typically have smoother, more Gaussian weight distributions than query/key projections. The activation R² (0.5398) is correspondingly the highest.

Layer 1: Input Layernorm

Analysis: Layer 1's input norm achieves activation R² = 0.9086 — better than Layer 0's post-FFN norm (0.6594). This suggests that deeper layer norms may be more robust to weight approximation, possibly because their inputs have been "pre-conditioned" by previous layers.

2.6 Compression Metrics

Forward Pass Timing:

Key Insight: The packed forward pass is actually slightly faster than original in some cases (Q_Proj: 4.90ms vs 5.42ms). This is because:

• The packed weights are smaller, improving cache locality
• The reconstruction (unpack) happens once and the dense tensor is cached
• However, the unpack step (43-49ms) is expensive — it must happen before the first forward pass

---

Part 3: Critical Analysis — Why Real Generation R² is Lower

3.1 The Compounding Error Problem

The experiments above measure R² per layer with the original upstream activations. In a real generation scenario:

Layer 0: x0 (real) → approx_layer_0 → x1 (approximated)
Layer 1: x1 (approximated, not real!) → approx_layer_1 → x2 (doubly approximated)
...
Layer 60: x60 (accumulated error) → approx_layer_60 → output

Error propagation in deep networks is multiplicative, not additive.

If each layer has activation R² = 0.5 (generous estimate for attention layers), after 60 layers:

• The effective R² degrades exponentially
• Even with perfect norm layers (R²=1.0), the attention and FFN errors compound

3.2 Resource Limitations

The experiments were conducted on a single GPU with limited VRAM (evidenced by frequent torch.cuda.empty_cache() calls and 0.28 GB VRAM usage). This constrains:

1. Batch size: Only batch=2, seq=1000 could be tested — real generation uses larger contexts
2. Layer coverage: Only Layer 0 and scattered layers were validated, not all 60 layers end-to-end
3. Embedding quality: The input embeddings are loaded from FP16 weights, but in a fully quantized model, even embeddings would be 1-bit
4. No end-to-end generation: Perplexity (PPL) and BLEU/ROUGE scores were not computed due to memory constraints

3.3 Why RMSNorm is Perfect but Attention is Not

The fundamental issue: 1-bit quantization of 44M-element matrices with only row-wise scales is massively underparameterized.

For Q_Proj [8,192, 5,376]:

• Original parameters: 44,040,192 floats
• 1-bit packed: 44,040,192 bits = 5.5 MB
• Row scales: 8,192 × 2 = 16,384 floats = 64 KB
• Information loss: We're representing 44M continuous values with 44M bits + 16K floats — a compression ratio of ~256:1 in information-theoretic terms

3.4 Theoretical Limits

From information theory, the minimum bits needed to represent a weight matrix with distortion D is given by rate-distortion theory. For Gaussian weights with variance σ²:

R(D) = (1/2) * log2(σ² / D)  bits per dimension

For D/σ² ≈ 0.5 (R² ≈ 0.5), R(D) ≈ 0.5 bits per weight. True 1-bit quantization is at the theoretical limit. Achieving R² > 0.9 would require:

• Non-uniform quantization (learned codebooks)
• Vector quantization (grouping weights into vectors)
• Mixed-precision (1-bit for some layers, 2-4 bit for critical layers)
• Outlier-aware quantization (keep top-k weights in full precision)

---

Part 4: What Would Be Needed to Succeed

4.1 Immediate Technical Improvements

1. Mixed-Precision Strategy: Use 1-bit for norms and small tensors, 2-bit for attention, 4-bit for FFN gates. This preserves the 10-15x compression while keeping R² > 0.9.

2. Outlier Preservation: Keep the top 1% of weights (by magnitude) in FP16. Experiments show 1% outliers contain ~20% of the "information" in the matrix.

3. Learned Codebooks: Instead of row-wise scales, use k-means on weight clusters to learn 16-32 centroids per layer. This is 4-5 bit but with much better R².

4. Activation-Aware Quantization (AWQ-style): Weight the quantization error by activation magnitude. Rarely-activated weights can tolerate more error.

5. Layer-wise Fine-Tuning: After quantization, run 100-1000 steps of distillation on the quantized model to recover accuracy. This requires the full model loaded in memory.

4.2 Resource Requirements

4.3 Collaboration Opportunities

This work demonstrates a proof-of-concept with the following validated components:

• ✅ Streaming safetensors parsing without full model download
• ✅ Block-wise energy analysis for structured masking
• ✅ Multi-pass scalar refinement with convergence
• ✅ True 1-bit packing with np.packbits
• ✅ Recursive scale quantization (scales of scales)
• ✅ Cross-shard adaptive ratio via EMA (ShardStats)
• ✅ Real activation R² measurement (not just weight R²)

What is needed from partners:

• Access to multi-GPU cluster (8× A100 minimum)
• Existing quantization infrastructure (vLLM, TensorRT-LLM, or custom kernels)
• Dataset for calibration (C4, WikiText, or domain-specific corpora)
• Evaluation framework (LM Evaluation Harness, HELM, etc.)

Potential collaborators:

• Companies building edge AI chips (need ultra-small models)
• Cloud providers offering quantized model serving
• Research labs working on sub-1-bit quantization
• Open-source projects (llama.cpp, ollama, etc.)

---

Part 5: Detailed Results Tables

5.1 DeepSeek-V4 Pro: Complete Layer 0 Results

5.2 Gemma 4 31B IT: Real Generation Results (Layers 0-1)

5.3 Gemma 4 31B IT: Scale Quantization Attempts Detail

K_Proj Scales (Layer 0)

Q_Proj Scales (Layer 0)

V_Proj Scales (Layer 0)

5.4 Compression Summary: Gemma 4 31B IT

---

Part 6: Conclusions and Call for Collaboration

What Works

1. RMSNorm layers: Perfectly approximated (R² > 0.99) with trivial compression
2. Small 1D tensors: Perfect reconstruction due to overparameterization
3. Scale quantization: Recursive 1-bit on scales achieves R² > 0.92 in 3 attempts
4. ShardStats: Cross-shard adaptation improves consistency
5. Streaming processing: Can process 64-shard models without loading full weights

What Doesn't Work (Yet)

1. Attention projections: R² ~0.45-0.54 on real activations — too low for usable generation
2. Error compounding: Per-layer R² doesn't reflect end-to-end degradation
3. Resource constraints: Single GPU limits validation to small batches and few layers
4. No fine-tuning: Quantization-aware training would recover 10-20% R²

The Path Forward

To make 1-bit LLMs practical, we need:

Call for Investment / Collaboration

This research demonstrates a novel approach to extreme LLM quantization with the following intellectual property:

• Multi-pass block-wise scalar refinement with adaptive masking
• Recursive scale quantization (1-bit on scales of scales)
• Cross-shard EMA-based ratio adaptation (ShardStats)
• Streaming safetensors processing for models of any size

I am seeking:

• Compute partnership: Access to 8+ A100/H100 GPUs for 2-4 weeks
• Research collaboration: Joint paper on sub-1-bit quantization
• Investment: Seed funding to build custom inference kernels and validate at scale
• Industry partnership: Integration with existing serving stacks (vLLM, TensorRT-LLM)

Contact: [email protected] or X: @liberal17th