mamba-1.4b-OA

this is a test, not a useful SOTA bot

• state-spaces/mamba-1.4b finetuned on Open Assistant conversations, 3 epochs
• talks ChatML (w/o system message)
• training code: https://github.com/geronimi73/mamba

inference:

import torch
from transformers import AutoTokenizer
from mamba_ssm.models.mixer_seq_simple import MambaLMHeadModel

modelpath="g-ronimo/mamba-1.4b-OA"

model = MambaLMHeadModel.from_pretrained(
    modelpath,    
    dtype=torch.bfloat16,
    device="cuda"
)

tokenizer = AutoTokenizer.from_pretrained(modelpath) 

question="Why am I so tired?"

template="<|im_start|>user\n{q}\n<|im_end|>\n<|im_start|>assistant"
prompt=template.format(q=question)    
prompt_tokenized=tokenizer(prompt, return_tensors="pt").to("cuda")["input_ids"]
output_tokenized = model.generate(
    input_ids=prompt_tokenized, 
    max_length=len(prompt_tokenized[0])+100,
    cg=True,
    output_scores=True,
    enable_timing=False,
    temperature=0.7,
    top_k=40,
    top_p=0.1,
    )
answer=tokenizer.decode(output_tokenized[0])

print(answer)

example conversation:

<|im_start|>user
hello
<|im_end|>
<|im_start|>assistant
Hello, how can I help you today?
<|im_end|>
<|im_start|>user
who are you?
<|im_end|>
<|im_start|>assistant
I am Open Assistant, a chatbot designed to assist you with various tasks. Is there anything in particular that I can help you with today?
<|im_end|>
<|im_start|>user
tell me about quantum mechanics
<|im_end|>
<|im_start|>assistant
Quantum mechanics is a branch of physics that describes the behavior of matter and energy at the atomic and subatomic level. It is based on the principles of quantum theory, which provides a framework for understanding the behavior of particles at the smallest scales.

In quantum mechanics, particles are described by wave functions that represent the probability of finding a particle at a specific location. These wave functions can be used to predict the behavior of particles, and they can be modified using concepts such as superposition and entanglement.

Quantum mechanics also addresses the nature of space and time, as it suggests that these concepts are merely human constructs that arise from our perception of the world. According to quantum mechanics, space and time are not objective entities, but rather subjective constructs that arise from the way we perceive particles.

Quantum mechanics has many applications in fields such as physics, chemistry, and materials science, and it has led to the development of new technologies, such as quantum computing and quantum communication.
<|im_end|>