Downloads · 30 days
118
11% of all-time downloads
RichardErkhov/han1997_-_mamba-2.8b-zephyr-hf-gguf
han1997_-_mamba-2.8b-zephyr-hf-gguf is a machine learning model from RichardErkhov. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Downloads · 30 days
118
11% of all-time downloads
All-time downloads
1.1K
Public
Repo size
27.9 GB
Likes
0
Public
Click a slice to open those files.
.gguf32.7 GB · 100%
From the Hugging Face model README
Quantization made by Richard Erkhov.
mamba-2.8b-zephyr-hf - GGUF
| Name | Quant method | Size |
|---|---|---|
| mamba-2.8b-zephyr-hf.Q2_K.gguf | Q2_K | 0.94GB |
| mamba-2.8b-zephyr-hf.Q3_K_S.gguf | Q3_K_S | 1.19GB |
| mamba-2.8b-zephyr-hf.Q3_K.gguf | Q3_K | 1.19GB |
| mamba-2.8b-zephyr-hf.Q3_K_M.gguf | Q3_K_M | 1.19GB |
| mamba-2.8b-zephyr-hf.Q3_K_L.gguf | Q3_K_L | 1.19GB |
| mamba-2.8b-zephyr-hf.IQ4_XS.gguf | IQ4_XS | 1.43GB |
| mamba-2.8b-zephyr-hf.Q4_0.gguf | Q4_0 | 1.51GB |
| mamba-2.8b-zephyr-hf.IQ4_NL.gguf | IQ4_NL | 1.51GB |
| mamba-2.8b-zephyr-hf.Q4_K_S.gguf | Q4_K_S | 1.51GB |
| mamba-2.8b-zephyr-hf.Q4_K.gguf | Q4_K | 1.51GB |
| mamba-2.8b-zephyr-hf.Q4_K_M.gguf | Q4_K_M | 1.51GB |
| mamba-2.8b-zephyr-hf.Q4_1.gguf | Q4_1 | 1.66GB |
| mamba-2.8b-zephyr-hf.Q5_0.gguf | Q5_0 | 1.81GB |
| mamba-2.8b-zephyr-hf.Q5_K_S.gguf | Q5_K_S | 1.82GB |
| mamba-2.8b-zephyr-hf.Q5_K.gguf | Q5_K | 1.82GB |
| mamba-2.8b-zephyr-hf.Q5_K_M.gguf | Q5_K_M | 1.82GB |
| mamba-2.8b-zephyr-hf.Q5_1.gguf | Q5_1 | 1.97GB |
| mamba-2.8b-zephyr-hf.Q6_K.gguf | Q6_K | 2.15GB |
| mamba-2.8b-zephyr-hf.Q8_0.gguf | Q8_0 | 2.76GB |
The following content is mostly from https://huggingface.co/state-spaces/mamba-2.8b-hf
This repository contains the transfromers compatible mamba-2.8b-zephyr. The checkpoints are untouched, but the full config.json and tokenizer are pushed to this repo.
For details of the original model before conversion, see https://huggingface.co/xiuyul/mamba-2.8b-zephyr.
You need to install transformers from main until transformers=4.39.0 is released.
pip install git+https://github.com/huggingface/transformers@main
We also recommend you to install both causal_conv_1d and mamba-ssm using:
pip install causal-conv1d>=1.2.0
pip install mamba-ssm
If any of these two is not installed, the "eager" implementation will be used. Otherwise the more optimised cuda kernels will be used.
You can use the classic generate API:
>>> from transformers import MambaConfig, MambaForCausalLM, AutoTokenizer
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("han1997/mamba-2.8b-zephyr-hf")
>>> model = MambaForCausalLM.from_pretrained("han1997/mamba-2.8b-zephyr-hf")
>>> input_ids = tokenizer("Hey how are you doing?", return_tensors="pt")["input_ids"]
>>> out = model.generate(input_ids, max_new_tokens=10)
>>> print(tokenizer.batch_decode(out))
["Hey how are you doing?\n\nI'm doing great.\n\nI"]
In order to finetune using the peft library, we recommend keeping the model in float32!
from datasets import load_dataset
from trl import SFTTrainer
from peft import LoraConfig
from transformers import AutoTokenizer, AutoModelForCausalLM, TrainingArguments
tokenizer = AutoTokenizer.from_pretrained("han1997/mamba-2.8b-zephyr-hf")
model = AutoModelForCausalLM.from_pretrained("han1997/mamba-2.8b-zephyr-hf")
dataset = load_dataset("Abirate/english_quotes", split="train")
training_args = TrainingArguments(
output_dir="./results",
num_train_epochs=3,
per_device_train_batch_size=4,
logging_dir='./logs',
logging_steps=10,
learning_rate=2e-3
)
lora_config = LoraConfig(
r=8,
target_modules=["x_proj", "embeddings", "in_proj", "out_proj"],
task_type="CAUSAL_LM",
bias="none"
)
trainer = SFTTrainer(
model=model,
tokenizer=tokenizer,
args=training_args,
peft_config=lora_config,
train_dataset=dataset,
dataset_text_field="quote",
)
trainer.train()