spark-qwen3.5-0.8b

Spark

Spark is a small, low-latency conversational model — a LoRA fine-tune of Qwen/Qwen3.5-0.8B (0.8B params, Apache 2.0), developed by a team from Helios as part of the Arche model family. It is meant to serve as the general chat / quick-response layer inside ELYSIUM, a local AI agent.

This repository contains the merged model weights (LoRA already merged into the base model), ready to load directly with transformers.

LoRA configuration

Training data

• A subset of mlabonne/FineTome-100k (general instruction / chat tuning).
• A synthetic identity dataset (generated programmatically) teaching Spark its identity — name "Spark", developed by "a team from Helios", part of the "Arche" family. Upsampled ~6× so identity behavior is reliably learned.

Training ran on a single Colab T4 free-tier GPU in non-thinking mode (enable_thinking=False) for low latency. Training code and the identity generator live in the GitHub repo: https://github.com/Helios-4U

Intended use

General-purpose chat and fast responses, designed to sit as a quick-response layer inside ELYSIUM. Not fine-tuned for tool use, coding, or long-form reasoning.

Context length limitation

⚠️ Spark was trained at a sequence length of 512 tokens. Although the base model Qwen/Qwen3.5-0.8B natively supports up to 262,144 tokens of context, Spark itself was not trained or validated on long sequences. Do not assume Spark inherits the base model's long-context behavior — keep inputs near the 512-token training length for expected behavior.

Known Issues

Real bugs encountered during training are documented in the GitHub repo: https://github.com/Helios-4U/Spark/blob/main/docs/KNOWN_ISSUES.md (transformers git-install requirement, trl max_length rename, loss_type="nll" requirement, ~248k-vocab OOM, Colab session-time cap).

No formal benchmark evaluation has been run on Spark. Performance claims are unverified; the base model's published scores do not transfer by assumption.

Example usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Helios4U/spark-qwen3.5-0.8b"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")

messages = [
    {"role": "system", "content": "You are Spark, a helpful, honest, and concise AI assistant."},
    {"role": "user", "content": "Who are you?"},
]

# Non-thinking mode (Spark was trained this way for low latency)
prompt = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, enable_thinking=False
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

License

Spark is a derivative work of Qwen/Qwen3.5-0.8B, released under Apache 2.0. Spark is distributed under the same Apache 2.0 license.