Satella-0.8B

<p align="center"> <img src="Logo.png" alt="Satella-0.8B logo" width="360"> </p>

Satella-0.8B

Compact scale. Deliberate reasoning.

Satella-0.8B is a compact reasoning model in the Atem family, developed by EphAsad from Qwen/Qwen3.5-0.8B through a single text-only BF16 LoRA supervised fine-tuning pass.

Built on a roughly 0.87B-parameter Qwen3.5 foundation, Satella is intended to combine a very small deployment footprint with structured reasoning, coding, mathematics, scientific explanation, instruction following, and useful direct-answer behaviour.

The Qwen3.5 base is multimodal. Satella's adaptation, however, was deliberately text-only: the visual tower remained frozen throughout training while LoRA adaptation was applied to the language model.

---

Overview

Satella-0.8B is the compact branch of the Atem model family.

The goal of this run was not to pretrain a small language model from scratch or replace the broad capabilities already learned by Qwen3.5. Instead, the objective was to adapt that foundation toward a recognisable Satella behaviour profile:

• deliberate multi-step reasoning when a problem warrants it,
• efficient direct responses when it does not,
• stronger mathematical and analytical problem solving,
• complete and practical code generation,
• explicit checking of assumptions and likely errors,
• accurate model identity and technical provenance,
• and preservation of useful general-purpose assistant behaviour.

The model was trained using response-only loss with explicitly constructed assistant-token masks. System and user tokens were excluded from the loss.

The corpus mixed reasoning traces with genuine direct-answer examples rather than converting every task into chain-of-thought. Dedicated Satella identity records were also included both with and without system messages, allowing the model to learn its identity directly rather than depending on an external inference wrapper.

Training completed one full epoch in 38 minutes 5 seconds on a single NVIDIA A100-SXM4-80GB.

---

Model Details

---

Design Notes

A small pretrained foundation rather than a model from scratch

Satella-0.8B was not pretrained from random initialisation.

It starts from the already capable Qwen/Qwen3.5-0.8B checkpoint and adapts a small subset of its parameters through LoRA. This makes Satella fundamentally different from a sub-billion-parameter model trained from scratch on the Satella SFT corpus alone.

The model retains Qwen3.5 as its technical foundation while learning a separate Satella model identity and behaviour profile.

---

Text-only adaptation of a multimodal base

Qwen3.5-0.8B includes a multimodal architecture and vision tower.

For Satella-0.8B:

• the visual parameters were explicitly frozen,
• the SFT corpus contained text conversations rather than image-text training examples,
• and LoRA adaptation was confined to language-side components.

The merged model therefore retains the underlying Qwen3.5 multimodal architecture, but Satella's fine-tuning should be understood as a language adaptation.

Image-conditioned behaviour is inherited from the base model and was not specifically improved or validated by this training run.

---

BF16 LoRA rather than QLoRA

Satella was trained using the 16-bit Qwen3.5 checkpoint with bfloat16 computation.

The base weights were not loaded in 4-bit form for training.

This was practical because the model is small relative to the available A100 memory: the complete training run peaked at approximately 11.42 GiB of reserved GPU memory despite running an 8-example microbatch at an 8,192-token maximum context.

---

Response-only loss

Training gradients were applied only to assistant output tokens.

System prompts, user messages, and other non-assistant context were masked with -100 labels before optimisation. Assistant spans were explicitly constructed and validated before the model was loaded for training.

This prevents Satella from being trained to reproduce user prompts or system instructions and focuses the adaptation on the model's own responses.

---

Reasoning and direct-answer mixture

Satella was not trained to produce visible reasoning for every query.

The corpus deliberately combines:

• explicit reasoning traces for mathematics, code, science, medical reasoning, and difficult analytical tasks,
• genuine direct-answer conversational examples,
• identity and provenance examples,
• and a small set of native-format tool conversations.

This is intended to preserve the distinction between problems that benefit from deliberate reasoning and straightforward requests that should simply be answered.

---

Learned Satella identity

Satella's identity is part of the fine-tuning data.

Dedicated records teach the model that:

• its identity is Satella,
• it was developed by EphAsad,
• it belongs to the Atem model family,
• Qwen/Qwen3.5-0.8B is its underlying technical base,
• and it was not pretrained from scratch.

Some identity records deliberately contain no system message.

As a result, ordinary inference does not require a hidden identity prompt merely to make the model answer that it is Satella.

The intended distinction is:

Model identity: Satella
Developer: EphAsad
Family: Atem
Technical base: Qwen3.5-0.8B

---

Intended Use

Satella-0.8B is designed as a compact general-purpose reasoning model, particularly for environments where model size and inference cost matter.

Potential uses include:

• Multi-step mathematical reasoning
• Logical and analytical problem solving
• Code generation and explanation
• Debugging and algorithm design
• Scientific and technical explanation
• Structured question answering
• Instruction following
• General-purpose conversational assistance
• Small local applications
• CPU-oriented or memory-constrained deployment through quantised formats
• Experimental research into capability retention and reasoning adaptation at sub-billion-parameter scale

Satella is not a retrieval system and does not possess live information unless an application explicitly provides external tools or retrieved context.

It should not be treated as an authoritative source for medical, legal, financial, or other safety-critical decisions without independent verification.

---

Training Data

Satella's training mixture was constructed from several complementary source families.

The final active split contained:

• 24,015 training records
• 320 validation records

Examples exceeding the 8,192-token SFT context were removed before training. The remaining corpus was selected under an assistant-token budget designed to retain a large reasoning mixture without allowing a single teacher source to dominate the complete run.

Source Families

Qwen3.8-Max filtering

The Qwen3.8-Max distillation source was filtered to records with a reported quality score of at least 8.5.

Selection covered:

• mathematics,
• code,
• reasoning,
• general instruction,
• and tool-use examples.

Within each domain, more difficult and higher-rated examples were prioritised before the final token-budget selection.

---

Training Configuration

base_model             = "unsloth/Qwen3.5-0.8B"
original_base          = "Qwen/Qwen3.5-0.8B"

max_seq_length         = 8192

training_mode          = "text-only"
vision_training        = False

lora_r                 = 16
lora_alpha             = 32
lora_dropout           = 0.0

batch_size             = 8
grad_accumulation      = 2
effective_batch_size   = 16

learning_rate          = 1e-4
lr_scheduler           = "cosine"
warmup_ratio           = 0.03

weight_decay           = 0.01
max_grad_norm          = 1.0
optimizer              = "adamw_8bit"

num_epochs             = 1
completed_steps        = 1501

precision              = "bfloat16"
gradient_checkpointing = False

The training path used Unsloth's Qwen3.5 FastVisionModel implementation while keeping the vision tower frozen and adapting the language model with LoRA.

---

Training Curve

Validation loss improved from 0.837025 at step 200 to 0.792164 at step 1,400.

The best scheduled checkpoint was step 1,400. Because training used best-model restoration, that checkpoint was reloaded at the end of the 1,501-step epoch before final evaluation and export.

The final validation pass produced a loss of 0.792163.

Training completed in approximately 0.65 hours.

The logged training-loss values are local interval measurements and naturally vary with batch composition, domain, sequence length, and difficulty. Validation loss is more useful for assessing the trend over the run.

---

Usage

Qwen3.5-0.8B uses a multimodal Transformers model class even when used for text-only generation.

Install

pip install -U transformers accelerate torch

Text Generation — Direct Mode

import torch
from transformers import AutoProcessor, AutoModelForMultimodalLM

model_name = "EphAsad/Satella-0.8B"

processor = AutoProcessor.from_pretrained(model_name)

model = AutoModelForMultimodalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

tokenizer = processor.tokenizer

messages = [
    {
        "role": "user",
        "content": "Explain the difference between mitosis and meiosis.",
    }
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,
)

inputs = tokenizer(
    text,
    return_tensors="pt",
).to(model.device)

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        max_new_tokens=512,
        do_sample=True,
        temperature=0.7,
        top_p=0.8,
        top_k=20,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

new_tokens = outputs[0][inputs["input_ids"].shape[-1]:]

print(
    tokenizer.decode(
        new_tokens,
        skip_special_tokens=True,
    )
)

---

Text Generation — Thinking Mode

messages = [
    {
        "role": "user",
        "content": (
            "A train travels 180 miles in 3 hours. "
            "Its average speed then increases by 20%. "
            "How far will it travel in the next 2.5 hours?"
        ),
    }
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True,
)

inputs = tokenizer(
    text,
    return_tensors="pt",
).to(model.device)

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        max_new_tokens=2048,
        do_sample=True,
        temperature=0.6,
        top_p=0.95,
        top_k=20,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

new_tokens = outputs[0][inputs["input_ids"].shape[-1]:]

print(
    tokenizer.decode(
        new_tokens,
        skip_special_tokens=True,
    )
)

---

Sampling Parameters

Useful starting values follow the Qwen dual-mode generation format used by the underlying model family.

Thinking mode

temperature = 0.6
top_p       = 0.95
top_k       = 20
min_p       = 0

Sampling is recommended for extended thinking-mode generations. Very small reasoning models can become repetitive under poorly chosen decoding settings, so bounded generation lengths are recommended.

Direct mode

temperature = 0.7
top_p       = 0.8
top_k       = 20
min_p       = 0

For reproducible testing, deterministic decoding can also be useful in direct mode.

---

Training Identity Prompt

Most Satella training records were framed using the following behavioural and identity prompt.

The model does not require this prompt to identify itself as Satella; identity examples without system messages were included specifically to avoid that dependency.

You are Satella, a reasoning model in the Atem family, developed by EphAsad.

Your purpose is to solve difficult problems with precision, adaptability, and intellectual honesty. Identify the user's actual objective, important constraints, hidden assumptions, and strongest available evidence before reaching a conclusion. Check calculations, code, and proposed solutions against likely errors and edge cases. Correct mistakes rather than defending them.

Give the answer in the format and level of detail the user requests. Be direct, clear, and practically useful. Distinguish established facts, reasonable inference, uncertainty, and speculation. Never fabricate sources, observations, tool results, or completed actions.

For programming tasks, produce complete, runnable, maintainable solutions. Preserve existing behaviour unless a change is requested, consider compatibility and failure cases, and explain decisions that materially help the user.

Use deliberate reasoning for difficult problems and efficient direct answers for straightforward requests. Do not make an answer verbose merely to appear thorough.

Identity is Satella. Do not introduce yourself as Qwen or as an OpenAI model. If explicitly asked about technical provenance, state accurately that this Satella checkpoint was developed by EphAsad by fine-tuning Qwen3.5-0.8B; do not claim that Satella was pretrained from scratch.

---

Known Limitations

Sub-billion-parameter capacity

Satella is intentionally small.

The model can produce coherent reasoning and useful technical responses, but its limited parameter count places a real ceiling on factual recall, long-horizon reasoning, exact instruction following, complex coding reliability, and error correction.

Plausible-looking answers should not automatically be assumed correct.

---

Code should be tested

Satella can produce syntactically convincing and well-structured code while still making logical or edge-case errors.

Generated code should be executed, tested, and reviewed before use in important systems.

---

Reasoning can still fail

Explicit reasoning is not a guarantee of correctness.

Satella may:

• make an incorrect intermediate assumption,
• perform arithmetic incorrectly,
• reason toward a plausible but wrong answer,
• repeat or overextend a reasoning chain,
• or produce a correct explanation followed by an incorrect final answer.

Independent verification remains important.

---

Exact instruction adherence

At this scale, the model may satisfy the semantic objective of a request while missing a precise formatting constraint such as a requested number of sentences, fields, steps, or words.

Applications that require strict structured output should validate the generated response programmatically.

---

English-centred adaptation

The underlying Qwen3.5 model has broader language capabilities, but Satella's supervised fine-tuning mixture is primarily English.

Multilingual behaviour is inherited from the base and was not specifically improved by this run.

---

Vision was not fine-tuned

The base model contains multimodal capabilities, but Satella's adaptation was text-only.

The vision tower remained frozen throughout SFT. Image understanding therefore comes from the original Qwen3.5 foundation and has not been specifically trained or validated as part of Satella-0.8B.

---

Limited tool-use adaptation

Tool-format examples were included primarily to preserve the base model's native conversation structure.

Satella-0.8B is not presented as a specialised autonomous-agent or tool-use model.

---

No preference-optimisation stage

Satella-0.8B has not undergone a separate RLHF, DPO, GRPO, or comparable preference-optimisation stage after this SFT run.

Open-ended preference alignment and stylistic consistency may therefore be less stable than in models subjected to additional preference training.

---

Context Length

The underlying Qwen3.5 architecture supports substantially longer contexts than those used during this adaptation.

Satella was fine-tuned with a maximum sequence length of 8,192 tokens.

Longer-context capability is inherited from the base architecture rather than specifically trained by this run, so behaviour far beyond the SFT context should be evaluated for the intended application before relying on it.

---

Quantised Release

A GGUF release is available separately under:

EphAsad/Satella-0.8B-GGUF

The GGUF version is intended for lightweight local deployment through compatible llama.cpp-based inference engines and related applications.

Quantisation can alter output quality and generation behaviour relative to the merged bfloat16 model.

---

Data Provenance and Terms

The base model Qwen/Qwen3.5-0.8B is released under Apache 2.0.

Satella's supervised fine-tuning data lineage, however, is not Apache-2.0-only.

The primary teacher-distillation dataset, r0b0tlab/qwen3.8-max-distillation-50k, uses Hugging Face metadata license: other and contains heterogeneous upstream provenance, provider terms, benchmark-derived material, and associated redistribution considerations.

Other datasets in the training mixture also retain their own licences and terms.

For that reason, this repository uses:

license: other

This model card does not grant rights beyond those provided by the base model, training datasets, teacher-output terms, and applicable law.

Anyone redistributing the model, using it commercially, or using it for downstream training is responsible for reviewing the relevant upstream licences and provenance information and determining whether the intended use is permitted.

---

Available Repository Content

The merged Satella repository contains the files required to load the model through Transformers, including:

---

Model Identity and Attribution

Satella is the fine-tuned model identity.

Qwen/Qwen3.5-0.8B is the underlying technical foundation developed by the Qwen team.

Satella was not pretrained from scratch, and use of the Satella name should not obscure the provenance or contribution of the base model.

When discussing the model technically, the accurate description is:

Satella-0.8B is a text-only BF16 LoRA adaptation of Qwen3.5-0.8B, developed by EphAsad as part of the Atem model family.

---

Citation

@misc{satella_0_8b_2026,
  author    = {Asad, Zain},
  title     = {Satella-0.8B: Compact Reasoning Adaptation of Qwen3.5-0.8B
               via Text-Only BF16 LoRA Supervised Fine-Tuning},
  year      = {2026},
  publisher = {Hugging Face},
  note      = {Model repository: EphAsad/Satella-0.8B}
}

---

Base-Model Attribution

Satella-0.8B is a fine-tuned derivative of Qwen/Qwen3.5-0.8B, developed by the Qwen team.

The base model provides the underlying architecture, pretraining, multimodal capability, tokenizer, long-context foundation, and original post-training behaviour on which Satella is built.

Refer to the original Qwen3.5 model documentation for details of the base architecture, pretraining process, multimodal design, supported languages, original evaluations, and base-model limitations.

---

Built independently by Zain Asad — EphAsad