t5gemma-9b-2b-ul2-encoder-only

Encoder-only extraction for google/t5gemma-9b-2b-ul2

This repo contains the extracted T5 encoder-only weights for google/t5gemma-9b-2b-ul2. Gemma is provided under and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms.

Reference table

This extraction is part of a complete T5Gemma suite (270m, 1b, 2b, 4b, 9b) created to provide a unified environment for LLM-SDXL adapter research.

Why?

Standard T5-Gemma checkpoints are designed for seq2seq tasks, carrying heavy decoder overhead. This extraction isolates the V1 text-only encoder (Gemma-2 based), stripping the decoder layers to reduce VRAM consumption by ~25%.

How to load the model

from transformers import T5GemmaEncoderModel

model_path = "PhatcatDK/t5gemma-9b-2b-ul2-encoder-only"

# Load the weights
model = T5GemmaEncoderModel.from_pretrained(
    model_path,
    torch_dtype="auto", # Recommended: bfloat16
    is_encoder_decoder=False, # CRITICAL: Tells Transformers there is no decoder
    trust_remote_code=True
)

Acknowledgements & Inspiration

This work was directly inspired by the pioneering research of Minthy.

Minthy was the first to demonstrate the power of the t5gemma-2b-2b-ul2 encoder within this ecosystem. This specific repository was created to round out a standardized suite (270m through 9b) with a "clean-config" setup, ensuring consistent architectural parity for cross-attention testing.

I am incredibly grateful for the groundwork Minthy laid; this suite exists to support and verify the technical infrastructure he built.