gpt-oss-20b-toolcall-two-phase-v3-general-lora

gpt-oss-20b-toolcall-two-phase-v3-general-lora

Two-phase LoRA fine-tune for Wish Engine implementor tool-calling quality, using generalized tool-name training data.

Training Plan

• Phase 1 (strict): sahilmob/wish-engine-toolcall-next-v3-strict-general
  • train 732, validation 91, test 90
• Phase 2 (curriculum): sahilmob/wish-engine-toolcall-next-v3-general
  • train 1770, validation 245, test 222

Job Lineage

• v3 two-phase training:
  • 699b6ba31aad19adb8aacee3
• post-train eval (base, same datasets):
  • 699b735552d1c53b7df7d347
• post-train eval (tuned, this adapter):
  • 699b73601aad19adb8aacf4d
• prior baseline eval on same datasets (v2 adapter):
  • 699b4a1a1aad19adb8aacd13

Training Summary (v3)

From 699b6ba31aad19adb8aacee3:

• Phase 1 final:
  • train_loss: 2.904
  • mean_token_accuracy: 0.6004
  • last eval checkpoint:
    • eval_loss: 2.486
    • eval_mean_token_accuracy: 0.5683
• Phase 2 final:
  • train_loss: 1.924
  • mean_token_accuracy: 0.6225
  • last eval checkpoint:
    • eval_loss: 1.801
    • eval_mean_token_accuracy: 0.6285

Eval Summary (Post-Train, v3 Datasets)

Base vs tuned (tuned - base):

• mean_masked_nll: 4.3395 -> 0.5443 (-3.7952, lower is better)
• masked_perplexity: 76.6717 -> 1.7235 (-74.9483, lower is better)
• tool_name_exact_accuracy: 0.0619 -> 0.0000 (-0.0619)
• tool_name_canonical_exact_accuracy: 0.0667 -> 0.3762 (+0.3095)
• tool_name_contains_expected_accuracy: 0.0810 -> 0.3762 (+0.2952)
• tool_name_canonical_contains_expected_accuracy: 0.0810 -> 0.3762 (+0.2952)
• tool_name_parsed_ratio: 0.2619 -> 0.9810 (+0.7190)

Change vs v2 adapter on same eval set:

• tool_name_canonical_exact_accuracy: +0.0048
• tool_name_canonical_contains_expected_accuracy: +0.0048
• tool_name_parsed_ratio: -0.0190
• mean_masked_nll: +0.0031

Notes

• Canonical tool-name metrics normalize wrapper prefixes like functions.* and mcp__*__.
• This adapter is optimized for implementor tool-calling behavior, not general-purpose coding.

Related

• Central runbook: sahilmob/wish-engine-toolcall-runbook
• Post-train eval artifact in project repo:
  • training/hf-jobs/eval-results/v3-general-eval-post-train-2026-02-22.json