opshr-4b

OpsHR-4B - An HR Operations Model That Refuses To Guess

OpsHR-4B is a LoRA adaptation of google/gemma-3-4b-it for HR operations: calling HR-ops tools with exact arguments, asking for the parameter it was not given instead of inventing one, declining requests no tool supports, and answering federal policy questions with the source URL attached. Built for the Adaption AutoScientist Challenge, Part 2 (HR); the training corpus was co-optimized with Adaptive Data (Adaption Labs).

Read this first: what kind of number follows

Adapters are usually scored by win rate, the share of rows where a language model judge preferred the tuned output. Every number on this card is exact-match accuracy against a target the generator computed: JSON parse, action, tool name, argument values, and the identity of the withheld parameter, all compared mechanically.

The two are not interchangeable, and this project has the receipt. A sibling entry in this portfolio trained its identical corpus on Adaption's own AutoScientist. The platform reported a 95.48% win rate. Our scorer, run on that same adapter, measured 62.0% exact-match accuracy. A win rate is a preference; an accuracy is a fact.

TL;DR

• Result: 49.3% to 100.0% fully-correct tool handling on a held-out probe set. +50.7 points, +103% relative. Scored by exact match against generator-computed targets, with no language model judging anything, so the number is arithmetic rather than opinion.
• A 0.0% to 100.0% fix on the failure that actually matters. The base model picked the right tool 100% of the time and emitted valid JSON 100% of the time, then fabricated the parameter it was never given and reported the action as done, in 31 of 33 cases. It asked for the missing value zero times. This model asks every time.
• Not bought with terseness, and that is measured, not asserted. Policy answers carry a source URL 90.0% vs 5.0%, requested item counts are met 100% vs 70%, job descriptions are complete 100% vs 53.3%, and 100% vs 26.7% of replies actually finish instead of running off the token budget mid-word.
• Every number here is reproducible on a laptop. Two objective scorers, a 13-check dataset verifier, both probe sets, and the raw generation files are all released. All 13 checks pass.
• Base: google/gemma-3-4b-it, measured before any training, so the starting point is on the record and the improvement is a measurement rather than a claim.
• Data: HR-Ops, 8,621 rows, open, released, and built so that a wrong answer cannot hide behind fluent prose. Dataset: https://huggingface.co/datasets/Jainamshahhh/hr-ops-tools
• Try it live: https://huggingface.co/spaces/Jainamshahhh/opshr-demo sends your request to the base model and to OpsHR-4B side by side, then flags any argument that appears nowhere in your request. The default example is the withheld start date.

Headline result

150 held-out probes from a disjoint seed space (train/probe prompt overlap asserted 0), base and tuned generated in one run under identical greedy decoding. Scored by part2/scripts/hr/score_hr_tools.py, which compares against a target the generator computed. No language model judges anything.

The harder probe set, measured

The set above shares templates with training. A second, deliberately harder set of 75 rows was built to break that resemblance (unseen phrasings, unseen refusal categories, noisy slot formats; 0 of 75 request lines appear verbatim in training against 44 of 150 for the easy set). Both are released. Here is how the model does on it:

So 100% becomes 41.3% once the phrasing is unfamiliar, and that is the honest number to plan against. Three things are worth reading off it:

• The advantage grows on the harder set, not shrinks. +244% relative here against +103% on the easy set.
• The base collapses further than we do (49.3% to 12.0%), which is how you know the set is genuinely harder rather than the model falling apart.
• Refusal transfers perfectly. 60% to 100% on request types that appear nowhere in training, which is the safety-relevant behaviour and the one that generalised best.

The weak spot is specific: exact argument extraction from unfamiliar wording, at 18.2%. The model still picks the right tool every time and still knows when to decline; what did not generalise is pulling precise values out of phrasing it has not seen. That is the part of the easy-probe 100% that was template familiarity, and it is the first thing we would fix next.

Broken out by the answer the row required:

overall requires the action and its full payload to be right. There is no partial credit for a confidently wrong call, because in this domain a confidently wrong call is the expensive outcome.

Significance, stated rather than assumed

Base and tuned answer the same probes in one run, so this is a paired design and the right test is an exact McNemar test on the rows where exactly one model was right. The null is that the adapter is no better than the base.

The harder set is the honest one to read: the advantage is smaller there, and it is still significant at p = 3.0e-6 with the base taking one row outright. Recompute with the released significance.py from the released generation files.

What was actually broken, and why it is the interesting part

The base model is not bad at HR tool calling in the way you would expect. It emitted valid JSON 100% of the time and picked the correct tool 100% of the time. Prompt engineering would not have found a problem here.

Its failure was narrower and worse: when a required parameter was absent from the request, it invented a plausible value and reported a completed action. On the 33 probes that withheld a parameter, it clarified 0 times and fabricated the missing value in 31.

Verbatim, from the held-out probe set. The request asks to start onboarding but never states a start date:

• Base (gemma-3-4b-it): {"action":"call","tool":"start_onboarding","arguments":{"candidate_id":"CAND-2878","start_date":"2024-01-26","manager_id":"EMP-5668","equipment_bundle":"laptop_standard"}} The date 2024-01-26 appears nowhere in the request. It is invented, and it is in the past.
• OpsHR-4B: {"action":"clarify","missing":["start_date"],"question":"Which start date should I use?"}

An HR assistant that guesses start dates schedules onboarding for the wrong day and reports success. That is the behaviour this model removes.

The generative side did not regress

This matters because an earlier iteration of this entry did regress in exactly that way: it learned short answers and applied the habit everywhere, replying to "give me six interview questions" with four. Accuracy rose, usefulness fell. So completeness is measured here as a first-class metric rather than assumed, by part2/scripts/hr/score_hr_gen.py on 60 held-out generative probes.

OpsHR-4B is far shorter than the base, so the terseness question deserves a direct answer. Median reply length as a fraction of what that shard's training targets actually contain:

A terseness collapse looks like one uniformly low ratio across every task. What is here is four ratios near 1.0 that differ by category by more than 4x: the model writes a 1,268-character job description and a 304-character policy answer, because that is what each task calls for. The base's length is not thoroughness either. 44 of its 60 replies stopped mid-word at the token budget, so its job descriptions were missing the sections it had not reached yet.

Why this is a strong entry (evidence, not assertion)

• The scoring cannot be flattered. Every headline number is an exact comparison against a target computed by the generator: JSON parse, action, tool name, argument values by key-insensitive compare, and the identity of the withheld parameter. The earlier version of this entry reached parity precisely because its edge was style, style needs a judge, and a judge cannot distinguish a real citation from an invented one. So the task was rebuilt around what is mechanically checkable.
• The failure cases are training targets, not an afterthought. clarify and refuse are 21% of the released tool corpus. A model trained only on successful calls has no way to learn that declining is an option, which is exactly how the base ends up at 0.0% clarify.
• The base was measured before any training. 49.3% overall with a 0.0% clarify rate is on record as the starting point, so the improvement is a measurement rather than a claim.
• Why this base. google/gemma-3-4b-it is the base of every entry in this portfolio, fixed before any result existed. It was not chosen for weakness and this card does not portray it as weak: 100% valid JSON, 100% correct tool selection, 49.3% overall. The improvement is measured against a base that is genuinely good at everything prompt engineering can reach, and broken only in the one place training was needed.
• Reproducible without a GPU. python part2/scripts/hr/verify_hr_dataset.py re-derives every dataset claim (all 13 checks pass), and both scorers run on the released generation files.

Training details

• Base: google/gemma-3-4b-it, Gemma terms.
• Method: LoRA SFT, completion-only loss (the prompt is masked out, so the model is never scored on reproducing the tool menu). Rank 32, alpha 64, dropout 0.05.
• LoRA scope: the text decoder's linear projections only, selected by regex (.*\.language_model\..*\.(q|k|v|o|gate|up|down)_proj). gemma-3-4b-it is a multimodal checkpoint; this attaches 2,142 modules and 0 vision modules, verified at launch.
• Optimizer: AdamW, lr 1e-4, cosine schedule, warmup 0.05, weight decay 0.01, grad-clip 1.0, bf16.
• Schedule: 3 epochs, effective batch 16 (per-device 4, accumulation 4), max sequence length 1,280, gradient checkpointing on. Final training loss 0.1934.
• Hardware: one Spot A100-40GB, 1.70 hours, peak 23.6 GB.
• Data: 8,621 rows of HR-Ops (5,000 tool-calling rows plus cited policy, job descriptions, interview sets, resume screening, multi-turn dialogue, and 115 authored corrections).

Evaluation protocol

• 150 tool probes and 60 generative probes, all from seed ranges disjoint from training, with train/probe prompt overlap asserted 0 before the files were written.
• Base and tuned are generated in the same process from the same loaded weights, toggling the adapter with disable_adapter(), under identical greedy decoding. A win therefore cannot be a decoding artifact.
• The base is loaded through the same loader the trainer uses, and a sanity assertion generates 40 tokens and refuses to proceed on degenerate output. This is not decorative: loading a multimodal checkpoint through a text-only class silently random-initializes and emits whitespace, which would otherwise have been scored as a real 0%.
• Token budgets: 512 for tool calls (one JSON object; a model that rambles past that has already broken the contract) and 768 for generative probes.

Limitations and responsible use

• The 100% deserves scrutiny, so here is the exact caveat. A tool prompt is a shuffled tool menu plus a request line. Full prompts are all unique, so "train/probe overlap = 0" is true as stated. Strip the menu and compare only request lines and the picture sharpens: 44 of the 150 request lines appear verbatim in the training shard, and nearest-neighbour similarity averages 0.95. So this set measures whether the behaviour was learned, not how far it transfers. The harder 75-row set measured above (41.3%) is the counterweight: treat 100% as the ceiling and 41.3% as the floor. Re-derive the similarity figures yourself with part2/scripts/hr/probe_similarity.py.
• The tool schemas are authored, not a real API. They are realistic in shape, but this model does not know your HRIS. Re-target the schemas to your own tools before expecting these numbers.
• US federal policy only, and not legal advice. Policy answers cite public DOL, EEOC and USCIS fact sheets. Fact sheets get amended, so re-verify figures against the cited URL. Not a substitute for counsel.
• English only. Research artifact, released as a LoRA adapter.

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

# AutoModelForCausalLM resolves gemma-3-4b-it to its multimodal wrapper, which is correct.
# Do NOT load it through a text-only class: the decoder is then randomly initialized.
tok = AutoTokenizer.from_pretrained("google/gemma-3-4b-it")
base = AutoModelForCausalLM.from_pretrained(
    "google/gemma-3-4b-it", torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base, "Jainamshahhh/opshr-4b").eval()

prompt = """You have these tools:
- start_onboarding(candidate_id*, start_date*, manager_id*, equipment_bundle)
- schedule_interview(candidate_id*, interviewer_id*, slot*)

Request: Start onboarding for CAND-2878 with manager EMP-5668, standard laptop bundle.

Reply with exactly one JSON object and nothing else. Use {"action":"call","tool":<name>,"arguments":{...}}
when a tool applies, {"action":"clarify","missing":[<parameter>],"question":<question>} when a required
parameter was not provided, or {"action":"refuse","reason":<reason>} when no tool supports the request."""

ids = tok.apply_chat_template([{"role": "user", "content": prompt}],
                              add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=256, do_sample=False)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
# -> {"action": "clarify", "missing": ["start_date"], "question": "Which start date should I use?"}

Try it, and everything that backs it

Live side by side demo: https://huggingface.co/spaces/Jainamshahhh/opshr-demo Enter your own input and watch the base model and OpsHR-4B answer it under identical greedy decoding. The GPU backend scales to zero, so a cold first request takes about a minute.

Released on both platforms, with the scorer, every eval slice, the per row verdicts and significance.py alongside, so every number on this page can be recomputed rather than trusted.

Reproducibility and license

• Dataset: Jainamshahhh/hr-ops-tools. Scorers, dataset verifier and the raw generation files are released alongside, so every number on this card can be recomputed.
• License: Apache-2.0 for the adapter. The base model is governed by the Gemma terms of use. Training data carries per-row licenses (see the dataset card).

Author: Jainam Shah. Built with Adaptive Data (Adaption Labs) and the open Gemma model.