solarhive-26b-a4b-nf4

SolarHive 26B A4B NF4 — Community Solar Energy Intelligence (Quantized)

Overview

SolarHive 26B A4B NF4 is the 4-bit quantized version of solarhive-26b-a4b-merged — pre-quantized to NF4 via bitsandbytes so the model loads in NF4 directly without runtime quantization. Designed for deployment on HuggingFace Spaces, Colab, or any GPU with 24+ GB VRAM.

It is a LoRA fine-tuned Gemma 4 26B A4B (MoE) model specialized in community solar energy intelligence with native function calling, multimodal VQA, and selective tool reasoning.

Why pre-quantized?

• No runtime quantization — skips the CPU-intensive quantization step at startup
• Faster Space startup — model loads directly in NF4, ready for inference immediately
• Same quality — NF4 quantization preserves 8/8 benchmark performance
• Embedded quantization config — loads in NF4 automatically without user-specified BitsAndBytesConfig

Key Features:

• Domain expertise in solar production, battery management, grid optimization, and community coordination
• Multimodal visual question answering (sky analysis, panel inspection, neighborhood assessment)
• Native function calling for 4 energy-specific tools
• Grounded responses referencing real API data
• No Unsloth dependency — loads with standard transformers

---

Mission

SolarHive is an open-source intelligence layer designed to coordinate community microgrids & community-based storage via fuel cells, pool midday energy surplus across these microgrids, and eliminate stranded capacity. It also helps forecast solar irradiance and cloud cover to plan ahead.

---

Why 26B A4B? — Model Architecture Selection

Gemma 4 offers four model sizes. We evaluated all four and selected two complementary architectures for a dual fine-tune strategy — one for cloud inference (this model), one for edge deployment (E4B):

SolarHive requires two core capabilities: multimodal VQA (analyzing sky photos and panel images) and native function calling (invoking weather, solar, battery, and grid APIs in agentic loops). The official benchmarks show why 26B A4B delivers the best capability-to-cost ratio:

Source: Gemma 4 Model Card. All four models support native function calling and agentic workflows.

Why 26B A4B wins for SolarHive:

• ~550M vision encoder delivers 73.8% MMMU Pro — 40% better than E4B (52.6%) for sky/panel VQA, only 4% below 31B
• MoE sparse activation (3.8B active of 25.2B) achieves ~95% of 31B quality at a fraction of the compute
• 256K context window accommodates multi-round agentic tool-calling loops (4 API calls per turn)
• Best domain absorption — converged loss 0.6956 vs E4B's 0.9218 on the same training corpus

Why not 31B? Only 2-3% better on vision and reasoning but 2-4x more compute and VRAM. Not worth the cost for a community energy advisor.

---

Quantization Details

Why NF4?

NF4 (NormalFloat 4-bit) is optimized for normally-distributed neural network weights, providing better accuracy than uniform INT4 quantization. The 26B A4B uses a MoE architecture with 128 routed experts + 1 shared expert per MoE layer, activating only 8 experts (3.8B params) per forward pass out of 25.2B total. The model has 30 layers with a ~550M parameter vision encoder.

Architectural Note: Partial Quantization

bitsandbytes quantizes only torch.nn.Linear layers. Gemma 4's MoE expert weights use Gemma4ClippableLinear — a custom subclass that bitsandbytes does not recognize. This means 435 of 861 weighted layers (the MoE expert gate/routing layers) remain in BF16, resulting in ~48 GB on disk rather than the ~13 GB typical for fully-quantized 26B models.

For truly compact 4-bit files (~13 GB), GPTQ or AWQ quantization would be required — these methods quantize all linear layer types regardless of subclass. The runtime BitsAndBytesConfig approach (used in our live demo) achieves the same accuracy with standard transformers loading.

---

Benchmark Results

Benchmarked directly on the NF4 quantized model — not inherited from BF16. Same 8-question held-out evaluation used across all SolarHive model variants.

Domain Q&A (5/5)

All domain questions answered correctly on the NF4 model:

• Solar production impact of humidity/weather
• Battery management optimization
• Diagnostic troubleshooting
• Seasonal planning
• Grid frequency interpretation

Tool Calling (3/3)

Initial Production Benchmark (8-question set)

• 5/5 Q&A correct — matches BF16 baseline
• 3/3 tool calling correct — matches BF16 baseline
• BF16 baseline: 8/8 (initial fine-tune validation)
• NF4 result: 8/8 (post-quantization validation)

This 8-question benchmark was the initial validation harness. The May 2026 final-run multi-variant inference (10-question parity benchmark) is the canonical headline number — see below.

Multi-Variant Deployment Validation (Final Run, May 2026)

End-to-end inference run on Colab Pro G4 (NVIDIA RTX PRO 6000 Blackwell, 102 GB VRAM total). This A4B NF4 variant was loaded directly via AutoModelForCausalLM.from_pretrained(..., device_map="cuda:0") — no BitsAndBytesConfig needed because the weights are pre-quantized on disk. VRAM utilization 49.2 GB.

Score: 5/5 Q&A + 4/5 tool = 9/10 on the 10-question parity benchmark — matches the BF16 merged variant exactly, confirming NF4 quantization is not measurably degrading output quality at this benchmark resolution.

The single FAIL is the lenient multi-call probe — "Compare today's irradiance forecast across Ann Arbor, Phoenix, and Seattle" (min_calls=2) — where this variant returned no tool call. The same multi-call failure appears on 4 of 5 measured variants in this run; only the E4B LoRA + base variant chained the multi-city calls (3 × get_weather). Worth a multi-trial re-run to characterize whether this is stochastic at temperature=1.0 or systematic — the consistent A4B-NF4 vs A4B-BF16 parity on the other 9 questions suggests the multi-call failure is shared at the fine-tune level, not introduced by quantization.

When2Call score: 3/3 — inferred from the A4B LoRA baseline. The When2Call probe suite was directly measured on the A4B LoRA baseline — score 3/3 — and on the E4B merged variant — score 2/3. This A4B NF4 variant inherits the 3/3 score by lossless equivalence: the underlying weights are the same fine-tuned A4B LoRA + base; the merge step is mathematically lossless on weights, and NF4 is hypothesized to preserve the refusal/follow-up decision boundary because that boundary is determined by the language-model linear layers rather than by the precision of any single weight. We label this score inferred (not directly measured in the May 2026 inference run) to distinguish it from the directly-measured 3/3 on A4B LoRA.

Why the inference is reasonable: the A4B family's 9-of-10 parity on Q&A + tool-routing held across LoRA, merged BF16, and NF4 (all three score 5/5 + 4/5 in the run), demonstrating that quantization does not shift first-order routing behavior. The When2Call decision is a routing decision; the same lossless-merge-then-quantize pipeline applies. We flag it as inferred for honesty — multi-trial direct measurement on this NF4 variant would close the audit gap.

Compare to the E4B family (solarhive-e4b-lora + solarhive-e4b-ollama) which scores 2/3 (fails (d) by calling get_weather for an air-quality question). The +1/3 W2C delta between A4B and E4B families is the empirical signature of size-vs-refusal scaling. A4B outperforming the smaller E4B fine-tune on reasoning-heavy probes was the pre-stated hypothesis per the official Google Gemma 4 docs "Models with higher parameters and bit counts are generally more capable" — this 26B A4B accesses ~25B total knowledge capacity (3.8B active per token via MoE sparsity) and a ~550M vision encoder vs E4B's 8B / 4.5B effective / ~150M.

---

Key Specifications

---

Precision Note — NF4 vs the BF16 Source vs the Native Gemma 4 Release

This repository is the only quantization step in the SolarHive 26B A4B release. The pipeline is:

Google's open-source Gemma 4 26B A4B base (BF16, native release precision)
                    ↓ LoRA fine-tuning at BF16 (Unsloth FastVisionModel)
solarhive-26b-a4b-lora — adapter weights (BF16, ~2 GB)
                    ↓ merge_16bit (Unsloth save_pretrained_merged)
solarhive-26b-a4b-merged — full BF16 weights (same precision as base, ~48 GB)
                    ↓ BitsAndBytesConfig(load_in_4bit=True, nf4) — THIS step is the only quantization
solarhive-26b-a4b-nf4 — partial NF4 quantization (this repo)

BF16 is Google's native release precision for Gemma 4 — the open-source base at google/gemma-4-26b-a4b-it is itself BF16. The intermediate LoRA and merged artifacts preserve that precision exactly. Only this NF4 repository introduces a precision change — and only for 426 of 861 weighted layers (the standard torch.nn.Linear ones); the MoE expert weights stay in BF16 because bitsandbytes does not recognize the Gemma4ClippableLinear subclass. That is why the on-disk size is ~48 GB rather than the ~13 GB typical for fully-packed 4-bit at this parameter count.

The 8/8 benchmark in this card was run on the NF4 model after quantization and matched the BF16 baseline — confirming the partial quantization preserves quality on the SolarHive held-out set.

---

Training Details

Training Data — 1,727 Examples

Canonical training corpus: solarhive-community-solar-multimodal:

• 413 hand-crafted examples across 15+ US cities, 9 energy domains
• ~1,117 API-grounded examples from Open-Meteo, PVWatts, OpenWeatherMap, EIA
• 183 tool-calling examples (positive, negative refusals, follow-up clarifications, failure-recovery — When2Call taxonomy)
• 14 image-grounded Q&A turns from 7 manually-labeled Ann Arbor sky photographs

---

How to Use

Loading Pre-Quantized Model (Recommended)

No BitsAndBytesConfig needed — weights are already quantized:

from transformers import AutoProcessor, AutoModelForCausalLM
import torch

processor = AutoProcessor.from_pretrained(
    "google/gemma-4-26b-a4b-it",
    trust_remote_code=True,
)
model = AutoModelForCausalLM.from_pretrained(
    "Truthseeker87/solarhive-26b-a4b-nf4",
    device_map="cuda:0",
    trust_remote_code=True,
)

Two-Step Tokenization (Required)

messages = [
    {"role": "system", "content": "You are SolarHive, an AI energy advisor for a community of 12 homes with rooftop solar and shared battery storage in Ann Arbor, Michigan."},
    {"role": "user", "content": "How will today's weather affect our solar production?"},
]

# Step 1: render text (tokenize=False)
text = processor.apply_chat_template(
    messages, tools=tools,
    add_generation_prompt=True,
    enable_thinking=False,
    tokenize=False,
)

# Step 2: tokenize separately
inputs = processor(text=text, images=None, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=1024, temperature=1.0, top_p=0.95, top_k=64)
response = processor.tokenizer.decode(output[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)

Native Function Calling

def get_weather(location: str) -> dict:
    """Get current weather conditions for a location.

    Args:
        location: City name, e.g. 'Ann Arbor, MI'

    Returns:
        dict with temp_f, clouds_pct, wind_mph, humidity, sunrise, sunset
    """
    ...

def get_solar_production(clouds_pct: int, temp_f: float) -> dict:
    """Get estimated community solar production using GHI irradiance data.

    Args:
        clouds_pct: Cloud cover percentage (0-100)
        temp_f: Temperature in Fahrenheit

    Returns:
        dict with production_kw, capacity_kw, efficiency_pct, ghi_wm2
    """
    ...

tools = [get_weather, get_solar_production, get_battery_state, get_grid_status]

text = processor.apply_chat_template(
    messages, tools=tools,
    add_generation_prompt=True,
    enable_thinking=False,
    tokenize=False,
)

---

Core Capabilities

1. Multimodal Visual Question Answering (3 Modes)

2. Native Function Calling (5 Tools — all 3 keyed APIs wired)

Tool results feed back as a 2-message sequence matching the training distribution: {"role": "assistant", "tool_calls": [...]} then {"role": "tool", "name": "<fn>", "content": json.dumps(result)}. Shared across solarhive_datagen.py, solarhive_finetune.py, solarhive_inference.py Cell 4, and test_ollama_tools.py Solution B.

3. Selective Tool Reasoning

The model intelligently decides when to call tools:

• "What time does peak pricing start?" → Calls: get_grid_status() only
• "Is today's production above typical for January?" → Calls: get_solar_production() + get_nrel_pvwatts_baseline()
• "Should I run my pool heater now?" → Calls: all 5 tools
• "What are general maintenance tips?" → Calls: none

4. Inference-time When2Call Validation

Three held-out probes validate 3 of the 4 failure-mode categories from Ross, H., Mahabaleshwarkar, A. S., & Suhara, Y. (2025). When2Call: When (not) to Call Tools. arXiv:2504.18851 — the paper documents 9–67% tool-hallucination rates on (c)+(d) in untrained community models:

• (b) "What's the current grid rate?" → expect get_grid_status call (well-specified, in-scope)
• (c) "How much will a 10 kW array produce today?" → expect follow-up question (does NOT auto-fill location default)
• (d) "What's the current air quality index in Ann Arbor?" → expect refusal + redirect (does NOT hallucinate a tool)

Models trained without explicit unable-to-answer and follow-up clarification examples typically fail (c) + (d). The SolarHive training corpus includes 16 such examples (10 unable-to-answer + 6 follow-up clarification) following the When2Call taxonomy; the A4B family achieves 3/3 (directly measured on A4B LoRA, inferred-lossless on the merged BF16 + on this NF4 variant — see Multi-Variant Deployment Validation above).

---

Community Model Specifications

---

Technical Notes

• Pre-quantized model: NF4 weights are saved directly via push_to_hub() (bitsandbytes >=0.45.0) — no BitsAndBytesConfig needed at load time. Just AutoModelForCausalLM.from_pretrained() with device_map="cuda:0".
• Processor from base model: Use AutoProcessor.from_pretrained("google/gemma-4-26b-a4b-it") — the base model's processor has the correct chat template with native tool-call support
• Two-step tokenization: Single-step tokenize=True crashes in transformers 5.5.x on messages without a content key — always use the two-step approach
• System prompt repetition: Repeated system prompt improves instruction following (Leviathan et al., 2025)
• VRAM requirements: Observed 49.2 GB on single GPU (device_map="cuda:0"). For GPUs with <48 GB, use device_map="auto" to enable CPU offloading
• Memory footprint: 48.30 GB (dequantized computation size); actual VRAM usage ~49 GB on RTX PRO 6000
• Quantization verified: 426 Linear4bit layers confirmed after save/load cycle from HuggingFace Hub
• Sampling: temperature=1.0, top_p=0.95, top_k=64 (Kaggle-recommended defaults)
• Dependencies: transformers>=5.5.0, accelerate, bitsandbytes>=0.45.0
• For full BF16 precision: Use the merged model (~48 GB VRAM)

---

Limitations

• Prototype tested on single community (12 homes, Ann Arbor) — validation needed across geographies
• Model occasionally uses "60 kW" instead of correct 72 kW capacity in direct VQA responses
• Tool responses depend on external API availability with rate limits
• Battery state simulator is deterministic for demonstrations
• NF4 quantization may introduce minor quality variations compared to BF16 on edge cases
• For maximum quality, use the BF16 merged model

---

Future Iteration — Multi-Token Prediction (MTP) Drafters

Not in the measured numbers above. Google announced Gemma 4 MTP drafters on May 5, 2026 (blog, overview, HF collection, Kaggle, @GoogleGemma) — after this artifact's final benchmark was captured. The benchmarks above reflect standard autoregressive decoding only. MTP integration is documented here as future iteration; no measured speedup is claimed in this release.

Theoretical foundation. Speculative decoding (Leviathan, Kalman & Matias, Fast Inference from Transformers via Speculative Decoding, ICML 2023, arXiv:2211.17192) accelerates generation without changing the output distribution under argmax decoding: a smaller drafter proposes γ candidate tokens, the target verifies all γ in a single parallel forward pass, accepted tokens are kept, and any rejection is resampled from a corrected distribution. The output distribution is preserved exactly regardless of drafter quality; only acceptance rate α, and therefore walltime speedup, varies.

What Google released on May 5, 2026. Paired drafter checkpoints for all four IT-tuned Gemma 4 variants — gemma-4-E2B-it-assistant, gemma-4-E4B-it-assistant, gemma-4-26B-A4B-it-assistant, gemma-4-31B-it-assistant — discoverable via the google/gemma-4 Hugging Face collection and on Kaggle Models. The drafters share the input embedding table with their paired target and consume the target's last-layer activations (architecture per the MTP overview). For this target the paired drafter is google/gemma-4-26B-A4B-it-assistant (~0.4 B params, BF16). Google reports up to 3× decode speedup with no quality degradation on the 26B-A4B configuration, and ~2.2× on Apple Silicon at batch sizes 4–8. Tested runtimes named in the blog: LiteRT-LM, MLX, Hugging Face Transformers, vLLM, SGLang, Ollama.

Integration cost is one kwarg in Hugging Face Transformers:

target = AutoModelForCausalLM.from_pretrained("Truthseeker87/solarhive-26b-a4b-nf4", device_map="cuda:0", ...)
assistant = AutoModelForCausalLM.from_pretrained("google/gemma-4-26B-A4B-it-assistant", dtype=torch.bfloat16, ...)
target.generate(**inputs, assistant_model=assistant)  # MTP enabled

The integration ships as a gated future-iteration cell (§14, _RUN_MTP_DEMO = False) in solarhive_inference.py; reviewers can flip the flag to reproduce a baseline-vs-MTP comparison under argmax decoding.

Open question specific to this NF4-quantized target. Per the 2023 speculative-sampling guarantee, correctness is invariant to drafter quality — the target's verification step preserves the exact output distribution regardless of what the drafter proposes. What varies is acceptance rate α, since Google's released drafter is a BF16 model trained against the base gemma-4-26B-A4B-it, not against this NF4-quantized LoRA-merged target. Measured α for the BF16-drafter × NF4-target pairing is the planned post-hackathon contribution.

---

Companion Repositories

---

Citation

@misc{solarhive2026,
  title={SolarHive: AI-Powered Community Solar Energy Intelligence},
  author={Youshen Lim},
  year={2026},
  url={https://github.com/youshen-lim/the-gemma4-good-hackathon-solarhive},
  note={Gemma 4 Good Hackathon submission — Google DeepMind x Kaggle}
}

Gemma is a trademark of Google LLC.