Quark-0.5M

🔬 Quark-0.5M

Quark-0.5M is an ultra-lightweight Llama-based model with only 465,504 parameters. It was trained from scratch to demonstrate the power of high-quality data (FineWeb-Edu) on extremely small architectures.

Model Details

• Architecture: Llama-based
• Parameters: 465,504
• Vocabulary Size: 500 (Custom Byte-Level BPE)
• Hidden Size: 96
• Intermediate Size: 192
• Layers: 4
• Heads: 4
• Context Length: 256 tokens

Training

• Dataset: 400 Million Tokens of HuggingFaceFW/fineweb-edu (Sample-10BT)
• Training Time: ~42 minutes on a single Kaggle T4 GPU
• Final Loss: 2.46
• Optimizer: AdamW with Cosine Learning Rate Decay

Intended Use

Quark is a research project to explore the limits of "Micro-LLMs". It is surprisingly capable of forming grammatically correct English sentences and structured lists, despite fitting into less than 2MB of disk space.

Performance Example

Prompt: "Artificial intelligence is "<br> Output: "Artificial intelligence is very important. These are more likely to be adapted with the people of the following: - Subjects, evidence and social treatment for reduces costs..."

Prompt: "The future of science is "<br> Output:: "The future of science is just a sense of someone. There are many people today, according to these radioactive systems. The most popular capitalists are classified in the country. They are also accessible. These are not rarely available. In the middle of the direction, there are some of the body to raise the physical results."

Prompt: "Albert Einstein was "<br> Output:: "Albert Einstein was "for the first "the same "shap"." The "forms" of the British "measurement" and the "subject" and then the "content" of the "specific" and then the "contents". The period of the people of the movement of the manner. It is not actually accepted by the Christian culture of the Christianity"

How to use

from transformers import LlamaForCausalLM, PreTrainedTokenizerFast

model = LlamaForCausalLM.from_pretrained("LH-Tech-AI/Quark-0.5M")
tokenizer = PreTrainedTokenizerFast.from_pretrained("LH-Tech-AI/Quark-0.5M")

prompt = "The scientific method is"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50, do_sample=True, temperature=0.4)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))