qwen3.6-27B-cq3

<center><h1 style="font-size:42px;">Qwen3.5-27B-cq3</h1> 🦋️ <a href="https://bsky.app/profile/crumb.bsky.social">@crumb.bsky.social</a> 🐦️ <a href="https://x.com/cephaloform">@cephaloform</a> 🌐️ <a href="https://cephaloform.neocities.com">cephaloform.neocities.com</a><br> <em>(hqq)</em> <strong>Quantized 🤗 <code>transformers</code> model</strong> with lora integration that works <strong>with zero extra installs!</strong> Just `trust_remote_code=True`. <em>You trust me!</em> 🩷️ <br>- 🌷️ _ - \. 🪨️☘️</center> <hr> Load the quantized model!

from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"

model_id = "crumb/qwen3.6-27B-cq3"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model     = AutoModelForCausalLM.from_pretrained(
    model_id, dtype=torch.bfloat16, 
    device_map="auto",        # <- however you please
    trust_remote_code=True    # <- it's that easy!.. just takes a while LOL
)

Add Loras!

model.add_adapter("my_adapter", r=32, alpha=32**0.5, trainable=True)
model.add_adapter("my_other_adapter", r=45, alpha=1.772453851, trainable=True)

Generate!

inputs = tokenizer("Once upon a time,", return_tensors="pt")
inputs = {k:v.to(device) for k,v in inputs.items()}
outputs = model.generate(
    **inputs, do_sample=True,
    max_new_tokens=16,
    temperature=1.0,
    top_k=0,
)
print(tokenizer.decode(outputs[0]))
# [transformers] Setting `pad_token_id` to `eos_token_id`:248044 for open-end generation.
# Once upon a time, a single drive DJ grew out of a drug‑ware would run it and eventually

Reinforcement learning!

advantage = 10.0 # <- what it generated was awesome lol

optimizer = torch.optim.AdamW(
    [p for n,p in model.named_parameters() if "my_adapter" in n],
    3e-5, weight_decay=0.1, betas=(0.9, 0.95)
)
logits = model(outputs.clone().detach()).logits
logits = logits[:, inputs['input_ids'].shape[-1]:, :]
labels = outputs[:, inputs['input_ids'].shape[-1]:].clone()
loss = torch.nn.functional.cross_entropy(
    logits.reshape(-1, logits.size(-1)),
    labels.reshape(-1),
    reduction="none",
).view(labels.shape)
policy_loss = (-loss * advantage).mean()
policy_loss.backward()

grad_norm = torch.nn.utils.clip_grad_norm_(model.parameters(), 0.5)
optimizer.step()

Disable your adapters!

with model.disable_adapters(["my_other_adapter"]):
    # ...whatever code here...

Save and load adapters to and from the 🤗 hub!

model.push_to_hub("my_adapter", "username/my_cq_adapter")
model.push_to_hub("my_other_adapter", "username/my_other_cq_adapter")

model.load_adapter(
    adapter_name="another_adapter", 
    load_directory="username/yet_another_cq_adapter"
)

tested w/ transformers-5.7.0