Zephyr-7B-quantized

---

Model Card for zephyr-7b-beta-quantized

Model Details

• Model Name: dhanasekharB/Zephyr-7B-quantized (Quantized)
• Repository: dhanasekharB/Zephyr-7B-quantized
• Quantization: 4-bit quantized using bitsandbytes
• Intended Use: General-purpose causal language model with efficient inference on CPUs due to quantization
• Model Type: Auto-regressive Transformer for text generation

Model Description

This quantized version of HuggingFaceH4/zephyr-7b-beta is optimized for low-memory usage and fast inference on CPUs. By reducing the model size through 4-bit quantization, it’s suitable for applications that require text generation on resource-constrained devices.

Quantization was performed using the bitsandbytes library, specifically with 4-bit NormalFloat quantization (nf4). This model supports tasks like text generation and interactive dialogue, and is tailored for efficient, on-device deployment.

Usage

This model is loaded with quantization configuration, making it efficient for CPU-based inference. Below is an example to load and run it for inference.

from transformers import AutoTokenizer, AutoModelForCausalLM

# Load model and tokenizer
model_ckpoint = "dhanasekharB/Zephyr-7B-quantized"
tokenizer = AutoTokenizer.from_pretrained(model_ckpoint)
model = AutoModelForCausalLM.from_pretrained(model_ckpoint)

# Sample inference
my_text = "Hi, how are you?"
inputs = tokenizer(my_text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=10)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Quantization Details

• Quantization Method: 4-bit quantization with NormalFloat (NF4)
• Quantization Library: bitsandbytes
• Configuration: bnb_4bit_use_double_quant=True, bnb_4bit_quant_type="nf4"

Intended Use and Limitations

Use Cases

• Text generation for chatbots
• Summarization and paraphrasing (limited to short contexts)
• Creative writing and ideation

Limitations

• This model may struggle with long or complex prompts due to reduced precision from quantization.
• Fine-tuning or other modifications may be required for domain-specific applications.
• Not intended for: Tasks requiring high-accuracy arithmetic or logical consistency, since quantization may reduce precision.

Performance

• Memory Efficiency: This model uses approximately 4x less memory compared to the standard 32-bit float model.
• Inference Speed: Optimized for faster inference on CPUs, but exact performance depends on the CPU’s specifications.

Training and Evaluation Data

The base model, zephyr-7b-beta, was trained on diverse text data, but this quantized version was not further fine-tuned. Quantization was applied to adapt the model for CPU inference.

Ethical Considerations

This model should not be used for generating harmful, biased, or misleading information. Users should monitor outputs, especially for public-facing applications, to ensure content aligns with ethical guidelines.

Acknowledgments

The base model was developed by [Model Original Authors/Organization]. Special thanks to the bitsandbytes library for making efficient quantization possible.

Citation

If you use this model, please cite it as follows:

@misc{your_model_name,
  title={Quantized zephyr-7b-beta},
  author={Your Name/Organization},
  year={2024},
  url={your Hugging Face Hub URL}
}

---