qwen3-0.6b-horoscope

Qwen3-0.6B fine-tuned on horoscopes

Qwen/Qwen3-0.6B continued-pretrained on karthiksagarn/astro_horoscope (21,959 horoscopes; 19,763 train / 2,196 eval, seed=42).

Trained on raw text with no chat template, so it is a style continuation model, not an assistant. Prompt it with a fragment and it continues in horoscope register.

Results

Perplexity fell 27% from epoch 1 to epoch 3. Eval loss decreased monotonically - no overfitting observed.

Before / after

Same prompt, same seed (42), same sampling (temperature=0.8, top_p=0.9); only the weights differ.

Prompt: Leo, the stars suggest

• Base: "...that the galaxy is not just a flat plane, but a sphere, and that the universe is expanding..."
• Fine-tuned: "...that you are going to find someone who is very attractive. They may be a friend or a romantic partner..."

The base model read "stars" astronomically; the fine-tuned model reads it astrologically.

Hyperparameters

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("VizRohan/qwen3-0.6b-horoscope")
model = AutoModelForCausalLM.from_pretrained("VizRohan/qwen3-0.6b-horoscope", dtype="auto")

enc = tok("Aries, today", return_tensors="pt")
out = model.generate(**enc, max_new_tokens=60, do_sample=True,
                     temperature=0.8, top_p=0.9)
print(tok.decode(out[0], skip_special_tokens=True))

Limitations

Entertainment only. Trained on ~20k short texts, so output is formulaic and reproduces the genre's cliches. Low perplexity partly reflects that horoscope text is deliberately vague and therefore easy to predict - it is not a measure of quality.