Qwen2-36m-hf

Qwen2 HF model (36M Parameters)

This is a Qwen2 architecture model trained completely from scratch with 36 million parameters. It uses a custom tokenizer and vocabulary, and is designed for experimentation with compact, task-specific language models.

Training Details

• Architecture: Qwen2ForCausalLM
• Parameters: 36M
• Training from scratch: Yes
• Pretrained base: None
• Tokenizer: ByteLevelBPETokenizer
• Language: English
• Dataset: wikimedia/wikipedia
• Subset: 20231101.en
• Subset rows: 35,000
• Max position embeddings: 512
• Number of steps: 100K
• Hidden size: 384
• Number of attention heads: 12
• Number of transformer layers: 12
• Dropout rate: 0.2
• Vocabulary size: 10K
• Minimum token frequency: 5
• GPU Used: NVIDIA T4 (16GB)

Purpose

This is a quick experiment to see how well Qwen2 handles a small amount of data. It seems to be working reasonably well so far. This is a raw model not fine-tuned and it's currently only trained on 80K rows from the Wikipedia-20231101.en dataset, and just 100K training steps have been completed. The training process resumed from a previously saved checkpoint to continue model optimization.

For now, I’m leaving the model in its current state. However, if anyone wants to resume training, you can access the checkpoint here.

Intended Use

• Small-scale research
• Testing text generation on limited data
• Fine-grained experimentation with custom language models
• Educational purposes

Limitations

• Not general-purpose
• Limited vocabulary and context length
• Struggles outside its trained domain
• English-only
• Not production-ready

Inference Example

from transformers import Qwen2ForCausalLM, Qwen2Tokenizer

model = Qwen2ForCausalLM.from_pretrained("green19d25y/Qwen2-36m-hf")
tokenizer = Qwen2Tokenizer.from_pretrained("green19d25y/Qwen2-36m-hf")

prompt = "Once upon a time,"

input_ids = tokenizer.encode(prompt, return_tensors="pt")

output = model.generate(
    input_ids,
    max_length=100,
    num_return_sequences=1,
    do_sample=True,
    temperature=0.7
)

generated_text = tokenizer.decode(output[0], skip_special_tokens=True)
print(generated_text)

Model Test

Here’s a sample prompt and the output. I know it’s not very good… :(

prompt = "Once upon a time,"

Output: Once upon a time, he was aware of the problem with the British and was unable to do the way to the island. Late in the 1930s, the army participated in the reconsecution of the Dutch Republic in the 1950s, and was able to secure the military dictatorship system in Europe in 1960. His chief military commander was the commander-in-chief of the Royal Army Reserve, 

prompt = "At midnight"

Output: At midnight, the Cedar Stones were in their base at the top of the bay. A week after the suspension, the Clydefoot Stones, a 363-foot, were disassembled and dragged into the lakes, inflicted and, "The Cairn Drive" became the largest in the world. During the first three months of its existence, the Cairn Mountain Sutton Creek

prompt = "At the first stage"

Output: At the first stage. The club was disbanded in 1934, and the club again became an NCAA Division I men's basketball team in 1955, having the best hopes of becoming an amateur player. After winning the 1955–56 season, the Iowa Sixs Division club won the Junior championship.

prompt = "When I was a"

Output: When I was a child, Clyde, who was born in London. He was educated in the United Kingdom and, for the second time in a number of other countries, and was sentenced to six years in prison. After a seven-year stint in Scotland, Clyde returned to Scotland when he was sixteen. In 1904, he was appointed Chief Executive of Hastings to the British Government, and at the same time he was appointed

❤️ Big Thanks to the Qwen Team