tekyerema-1-native-v2-tool-pilot

Tɛkyerɛma-1 tool adapter, v2 pilot (arm ②, native text)

Author: Prince Nasamu Alhassan

Overview

Qwen3-4B + LoRA, trained on the v2 corpus utterance column — the raw target-language text — so no translator runs at inference. Arm ②, the native arm.

Loaded as STUDENT2 in LLM/brain_server.py alongside arm ①, which beat it 81.7 to 76.7 on the 2026-09-06 leak-free head-to-head. It is kept because removing the translator removes a whole failure mode, and because the two arms fail differently.

Use it

A LoRA adapter: load the base, then apply it. Take the tokenizer from the base, not from this repo — the adapter's saved tokenizer carries a chat template that silently ignores enable_thinking, and without that flag Qwen3 opens a reasoning block and never reaches the JSON.

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE = "Qwen/Qwen3-4B-Instruct-2507"
tok  = AutoTokenizer.from_pretrained(BASE)         # the BASE, deliberately
model = PeftModel.from_pretrained(
    AutoModelForCausalLM.from_pretrained(
        BASE, dtype=torch.bfloat16, device_map="auto"),
    "PrinceAlhassanNasamu/tekyerema-1-native-v2-tool-pilot").eval()

prompt = tok.apply_chat_template(
    [{"role": "user", "content": TOOL_PROMPT}],    # schema + user command
    add_generation_prompt=True, tokenize=False,
    enable_thinking=False)                         # not optional
enc = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**enc, max_new_tokens=96, do_sample=False)
print(tok.decode(out[0][enc["input_ids"].shape[1]:],
                 skip_special_tokens=True))

TOOL_PROMPT must carry the full tool schema exactly as training did — see LLM/student_prompt_frozen.py. Asked without it, the model has no tool names to choose from and scores zero.

Measured, naked, on the full bench

This adapter ran the complete bench on a Kaggle T4 through its byte-identical training prompt, with no safety stack around it:

The strongest and the weakest arm at once.

1. Best English / Ghanaian-English router on this bench. eng 0.88 and gh_eng 1.00 beat every other arm measured — e5 at 0.68/0.62, rules at 0.52/0.50, gpt-oss-20b at roughly 0.66 overall. The distillation worked for the command distribution it saw.
2. Catastrophic false-action rate as a naked decision-maker (0.358). The corpus contains only command→tool rows, so it routes everything, including non-actionable questions and the PIN/OTP adversarials — guard-hold 0.333 against gpt-oss-20b's 11/12. It was never taught to abstain.
3. Native-language transfer is poor on FRESH phrasings: twi 0.30, ewe 0.00, hau 0.33 — despite this arm scoring 93.3% on held-out corpus rows. That gap is the clearest evidence in the project that the corpus eval measured template-distribution fit rather than generalisation. The bench is the honest test.

Verdict: it earns a place as a fast English / Ghanaian-English routing rung behind the safety stack, not as the sole brain.

Training data

The v2 distillation corpus, utterance column — target-language text directly, no MT round trip.

A warning from how the v1 pair was built. The first attempt produced reply adapters for the two arms that were identical to 17 decimal places, because the training script applied its TEXT_COL switch only inside the tool branch. Two runs, two repos, one model. If you are comparing arm ① and arm ②, verify they differ before believing any difference you measure.

Licence

This repo previously declared no licence, rendering as "unknown" on the Hub. Corrected to CC BY-NC 4.0.

Limitations, stated plainly

• A validation loss is not a benchmark. Several cards on this account used to print the training run's own eval loss under a heading that read like a result. Where this card gives a WER or CER it names the slice and the scorer; where no such number exists it says the model is unscored.
• Scripture-derived audio does not generalise to speech. The Kusaal and much of the Ewe material is read scripture with a small number of narrators. Held out by CLIP rather than by SPEAKER it produces scores that look excellent and mean nothing -- the public bible_Kusaal figure of 11.69 is invalid for exactly this reason and must never be quoted.
• Evaluation is on read and machine-translated text. No corpus of people speaking natural phone commands in these languages exists. Numbers measured this way are optimistic about phrasing and pessimistic about code-switching, and should not be read as field performance.
• Research work from a hackathon entry, not a supported product.

The rest of the family

Twi — tekyerema-asr-ctc (v1, w2v-BERT CTC) · tekyerema-asr-ctc-twi-v2 (continuation)

Kusaal — tekyerema-asr-mms-kus (MMS adapter, the one to use) · tekyerema-asr-ctc-kus-v2 (new w2v-BERT line) · kasa42-asr (third-party baseline)

Ewe — tekyerema-asr-mms-ewe (the one to use) · tekyerema-asr-ctc-ewe · tekyerema-asr-ctc-ewe-v3

Other recognisers — tekyerema-asr-mms-dag · tekyerema-asr-mms-hau · whisper-large-v3-turbo-tekyerema-eng

Voices — tekyerema-tts-twi · tekyerema-tts-kus · tekyerema-tts-ewe · tekyerema-tts-hau · tekyerema-tts-eng

Agent models — tekyerema-1-v2-tool-pilot · tekyerema-1-native-v2-tool-pilot · tekyerema-audio-native-4k

Translation & routing — tekyerema-nllb600m-v1 · kusaal-nllb-600M · tekyerema-intent-afroxlmr

Acknowledgements

Compute resources provided by AI Skills and Compute Africa (AISCA). Trained on the Ghana NLP H200 GPU and on Kaggle T4s. Please keep derivatives non-commercial and share improvements back with the Ghana NLP community (ghananlpcommunity).