Teleste-Learner-12B

Teleste Learner 12B

Teleste Learner 12B is a 4-bit QLoRA fine-tune of Gemma 4 12B Instruct trained to adapt to the current request instead of assuming a fixed job.

Give it a new rule, a few examples, or a mid-conversation rule change. It is supposed to infer the contract from this conversation, apply it, and drop the old rule if you change it. That is in-context task induction, not a new form of AGI and not online weight updates while you chat.

v2 vs the 4B Qwen run: Gemma 4 12B Unified base, gold hand-written traces only, assistant-token SFT, native Gemma 4 thought channels, no Super-NaturalInstructions filler.

What it is good at

• Invented mappings shown with a few labeled examples
• Following a procedure you just defined (format, cipher, filter, schema)
• Switching behavior when a later message replaces the rule
• Staying quiet on extra commentary when the contract is strict
• Short replies vs stock Gemma 4 12B, which often spends a long thought channel first (faster wall time on T4; not a formal speed bench)

What it is not

• Not a general agent with memory across sessions
• Not trained as a specialist in one domain (medicine, law, a single company's docs)
• Not guaranteed to invent the correct rule when the examples are ambiguous

How to use

This repo is a LoRA adapter. Load it with Unsloth FastModel on the adapter id (do not 4-bit-load the base and then PeftModel.from_pretrained on a second copy — that path produced garbage on T4):

from unsloth import FastModel
from unsloth.chat_templates import get_chat_template

model, tok = FastModel.from_pretrained(
    "Davitotty1/Teleste-Learner-12B",
    max_seq_length=1024,
    load_in_4bit=True,
)
tok = get_chat_template(tok, chat_template="gemma-4-thinking")
FastModel.for_inference(model)

system = (
    "You adapt to the current request. Infer the user's goal, the hidden rules, "
    "and the output contract from this conversation only. If examples are present, "
    "the mapping in those examples is the law. If a later message changes the rules, "
    "the new rules replace the old ones. If this message is a new question, a correction, "
    "or a topic change, drop the last task. Do not reuse the last number or list. "
    "Check the answer against the inferred contract before you finish. Do not keep a default job."
)
messages = [
    {"role": "system", "content": system},
    {"role": "user", "content": "show then you finish.\noak -> O1K\nmaple -> M3E\nfig -> F1G\npine ->"},
]
inputs = tok.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    enable_thinking=True,
    return_tensors="pt",
    return_dict=True,
)
inputs = inputs.to(model.device) if hasattr(inputs, "to") else {k: v.to(model.device) for k, v in inputs.items()}
out = model.generate(**{k: inputs[k] for k in ("input_ids", "attention_mask") if k in inputs}, max_new_tokens=256, temperature=1.0, top_p=0.95, top_k=64)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

If this repo also has merged weights at the root, load that folder directly with AutoModelForCausalLM.from_pretrained.

Sampling that works well for Gemma 4: temperature=1.0, top_p=0.95, top_k=64. Allow enough max_new_tokens for a short thought channel plus the answer.

Thinking is enabled with enable_thinking=True (or <|think|> at the start of the system prompt). The model may emit:

<|channel>thought
...
<channel|>
answer

Training

Data: ~1000 hand-written adaptation traces (few-shot invented tasks, rule shifts, self-checks, stacked constraints, messy specs, topic interrupts, corrections). No public many-task filler in v2.

Evaluation

Held-out Kaggle suite vs stock Gemma 4 12B Instruct (unsloth/gemma-4-12b-it), same system prompt, 22 items (15 adaptation / 3 rule-switch / 4 control). Not MMLU. Exact / last-line / normalized match.

Teleste is not a general-benchmark win on this board. Control stayed at 4/4 (the LoRA did not collapse ordinary Q&A). Item-level preds were not copies of the base run. Notable swaps:

• Teleste right, base wrong: vowel_count (3), consonants_only (slvr), hyphen_swap_reverse (aet-neerg), switch_transform (MAPLE; base kept the old reverse cipher ELPAM)
• Base right, Teleste wrong: first_last_upper (PR vs PE), drop_last (penc vs penci), minus_four (12 vs 11), switch_scoring (numbers only vs CSV)

Several base cells were empty because the first Gemma pass truncated long thoughts. Treat the adaptation 40/40 as noisy. The MAPLE vs ELPAM item is the cleanest rule-switch check.

On the same Kaggle T4, Teleste felt much faster than stock Gemma 4 12B. That is generation length, not a timed latency bench: the base model often writes a long <|channel>thought block, while Teleste often answers in a few tokens (e.g. Paris with no thought). After Unsloth kernels are warm, the 22-item Teleste pass finished quickly. We did not log seconds per item.

License

Apache 2.0, inherited from Gemma 4 12B Instruct.