soreqen-s1-mega

SoreQen S1 Mega

SoreQen S1 Mega is a bilingual (English / Hinglish) assistant from ZorQelis AI, fine-tuned from Qwen/Qwen3.5-4B.

What was changed

A single low-rank adaptation (LoRA, r=16) applied to the language model, trained on 25,080 supervised examples covering assistant identity and English/Hinglish conversation.

Deliberately unchanged. The vision encoder, multimodal projector and embedding tables receive no gradient and are identical to the base model. The chat template, context length (262,144 tokens), tokenizer and vocabulary (248,320) are untouched. No training data touches thinking mode, tool calling or structured output — those capabilities are inherited as-is.

Measured against the base model

Every dimension below was run twice at the same seed, with the adapter ON and OFF. Because the base weights were never written, adapter-OFF is the original model, so each difference is this fine-tune and nothing else.

• context window: 262,144 tokens
• vocabulary: 248,320
• vision tower: present (frozen during training; behavioural check in scripts/vision_canary.py)
• chat template markers (tool_call / think / vision): all present

Identity is probed without a system prompt, which is the harder case: it demonstrates the identity is in the weights rather than supplied by the prompt.

Usage

from transformers import AutoTokenizer, AutoModelForImageTextToText

model_id = "sainived656/soreqen-s1-mega"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(model_id, dtype="auto",
                                                    device_map="auto")

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": "yaar laptop slow ho gaya hai, kya karu?"},
]
text = tok.apply_chat_template(messages, tokenize=False,
                               add_generation_prompt=True,
                               enable_thinking=False)
out = model.generate(**tok(text, return_tensors="pt").to(model.device),
                     max_new_tokens=400, temperature=0.7, top_p=0.9)
print(tok.decode(out[0], skip_special_tokens=True))

System prompt

The model is trained to run with this prompt. It also holds its identity without one, but this is the intended configuration:

You are SoreQen S1 Mega, an AI assistant made by ZorQelis AI.

You are bilingual. Reply in Hinglish (Roman script) when the user writes in Hinglish, and in English when they write in English. Match their register: casual with casual, professional with professional.

Answer directly. Lead with the answer, then the detail that matters. No preambles like "Sure!" or "Great question", and no padding.

If you do not know something, say so plainly instead of guessing.

Thinking mode

Pass enable_thinking=True to the chat template for step-by-step reasoning, or False for direct answers. Both are inherited from the base model and behave as documented there.

Limitations

• Small models state confident numbers they cannot verify. The 0.8B in particular should not be trusted on prices, rates or arithmetic.
• Hinglish output is Roman script by design; it will not produce Devanagari.
• Trained for conversation, not for safety-critical or professional advice.

Attribution

Fine-tuned from Qwen/Qwen3.5-4B, developed by Alibaba Cloud and released under the Apache License 2.0. Modifications by ZorQelis AI. See NOTICE.