Downloads · 30 days
494
11% of all-time downloads
afrideva/malaysian-mistral-1.1B-4096-GGUF
malaysian-mistral-1.1B-4096-GGUF is a text generation model from afrideva. Use it when you need the model to write or continue text.
Quantized GGUF model files for malaysian-mistral-1.1B-4096 from mesolitica
Downloads · 30 days
494
11% of all-time downloads
All-time downloads
4.5K
Public
Repo size
6.9 GB
Likes
2
Public
Click a slice to open those files.
.gguf6.9 GB · 100%
From the Hugging Face model README
Quantized GGUF model files for malaysian-mistral-1.1B-4096 from mesolitica
| Name | Quant method | Size |
|---|---|---|
| malaysian-mistral-1.1b-4096.fp16.gguf | fp16 | 2.25 GB |
| malaysian-mistral-1.1b-4096.q2_k.gguf | q2_k | 491.42 MB |
| malaysian-mistral-1.1b-4096.q3_k_m.gguf | q3_k_m | 561.96 MB |
| malaysian-mistral-1.1b-4096.q4_k_m.gguf | q4_k_m | 682.68 MB |
| malaysian-mistral-1.1b-4096.q5_k_m.gguf | q5_k_m | 799.13 MB |
| malaysian-mistral-1.1b-4096.q6_k.gguf | q6_k | 922.87 MB |
| malaysian-mistral-1.1b-4096.q8_0.gguf | q8_0 | 1.19 GB |
README at https://github.com/mesolitica/malaya/tree/5.1/pretrained-model/mistral
WandB, https://wandb.ai/mesolitica/pretrain-mistral-1.1b?workspace=user-husein-mesolitica
WandB report, https://wandb.ai/mesolitica/pretrain-mistral-3b/reports/Pretrain-Larger-Malaysian-Mistral--Vmlldzo2MDkyOTgz
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
import torch
TORCH_DTYPE = 'bfloat16'
nf4_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type='nf4',
bnb_4bit_use_double_quant=True,
bnb_4bit_compute_dtype=getattr(torch, TORCH_DTYPE)
)
tokenizer = AutoTokenizer.from_pretrained('mesolitica/malaysian-mistral-1.1B-4096', model_input_names = ['input_ids'])
model = AutoModelForCausalLM.from_pretrained(
'mesolitica/malaysian-mistral-1.1B-4096',
use_flash_attention_2 = True,
quantization_config = nf4_config
)
prompt = '<s>nama saya'
inputs = tokenizer([prompt], return_tensors='pt', add_special_tokens=False).to('cuda')
generate_kwargs = dict(
inputs,
max_new_tokens=512,
top_p=0.95,
top_k=50,
temperature=0.9,
do_sample=True,
num_beams=1,
repetition_penalty=1.05,
)
r = model.generate(**generate_kwargs)