joby-it-servicedesk

Joby IT Service Desk — Gemma 4 31B (Fine-Tuned, Tool-Aware)

A LoRA fine-tune of Google's Gemma 4 31B-it (dense, multimodal) specialized for Joby Aviation's internal IT service desk workflows, with native tool-calling preserved through a mixed Joby + general-purpose function-calling training corpus.

Private, internal model. Weights contain Joby-specific references (Jira project keys, Confluence KB content, internal hostnames, infrastructure conventions). Distributed only to Joby engineering staff via the PRE installer. Not for public release.

• Repo (this card): sunkencity/joby-it-servicedesk (GGUF, private)
• Adapter repo: sunkencity/joby-it-servicedesk-lora (private)
• Data pipeline: https://github.com/sunkencity999/joby-datasets (private)
• Released: 2026-05-30 (v0.4)
• Maintainer: Christopher Bradford, Systems Administration / AI Engineering, Joby Aviation

v0.4 — tool-protocol fix

v0.3 fabricated tool responses inline: it would emit a valid tool call and then immediately roleplay the tool's reply and a final answer in one generation (response:NAME{value:<|"|>{...}<|"|>}<tool_response|>The status is...). Root cause was a training-format defect — Gemma 4's stock chat_template.jinja packs the tool call, tool response, and assistant follow-up into a single <|turn>model block, and the LoRA learned to imitate that one-shot shape.

v0.4 corrects it at the source:

• New renderer (joby_chat_template.py) puts the tool call, tool response, and assistant follow-up in separate <|turn> blocks (<|tool_call>...<tool_call|><turn|> → <|turn>tool ... <turn|> → fresh <|turn>model). Same Gemma 4 native special tokens, corrected boundaries.
• Pre-rendered training data (build_mixed_dataset.py emits {"text": ...} records) bypasses MLX-LM's default chat-template application.
• Wider LoRA: attention + FFN rank-32 (q/v/gate/up/down) instead of attention-only rank-16. The "preserve FFN tool-call pathway" hypothesis was wrong — that pathway was already broken.
• Best-val checkpoint shipped: iter 6,750 of the retrain (val 0.638).

Inference: Ollama's stock gemma4 renderer doesn't reproduce the corrected turn structure, so PRETeams routes this model through /api/generate raw mode with a JS port of joby_chat_template (preteams:src/joby-template.js). Base PRE keeps using /api/chat for other models.

---

1. At a glance

---

2. What it's good at

• Native tool-calling preserved. Smoke-tests at 3 / 3 on PRE-style probes (date, bash, memory_search) — same activation rate as the unmodified base model. Designed to operate as a tool-augmented agent, not a closed-book oracle.
• Joby IT vocabulary and patterns. Internal terminology, ticket structure, and resolution conventions for license provisioning, network ergonomics, hardware lifecycle, account ops, and the day-to-day Joby IT helpdesk shape. Strong familiarity with the systems landscape: Jira (IT / DHLP / JSD projects), Confluence (ITKB space), D365, Smartsheet, Active Directory / Entra ID, Intune / Jamf, VPN / SSO / SAML, M365.
• Structured technical responses. Clean step-by-step procedures, properly fenced shell snippets, headers when length warrants. Inherits Gemma 4's strong instruction-following on top of the Joby-specific stylistic prior learned from agent replies.
• Multi-turn agent loops. Mixed in 241 real PRE tool-use traces during training, so the model has seen the full ChatML shape (system → user → assistant{tool_calls} → tool{result} → assistant) and handles multi-step plans without falling out of tool format.
• Long-context aware. Inherits Gemma 4's 256 K (262,144) token position embeddings. In practice PRE deploys at 8K–128K depending on RAM headroom (see PRE's context sizing table); the model itself is not the bottleneck.

3. What it isn't

• Not a substitute for live tool calls. Joby-specific facts — current ticket IDs, URLs, account numbers, AWS account IDs, network ranges, on-call rotations — must come from confluence.search, jira.get_issue, smartsheet.*, etc., at inference time, not from the model's baked-in knowledge. The mixed-data recipe intentionally weakened memorization in favor of tool-use behavior. Expect hallucinated URLs, ticket IDs, and account references if the model is run without tools.
• Not a general-purpose chatbot. Capability outside Joby's operational footprint is no better than base Gemma 4 31B-it, and may be slightly worse in stylistic register (responses skew toward ticket-resolution prose).
• Not for incident response without human review. A senior IT staff member must validate any output that triggers operational change (account provisioning, group-membership changes, MFA resets, MDM commands).
• Not multimodal at inference. Although the base model is multimodal (image + audio tokens in its vocabulary), the fine-tune is text-only. Images submitted at inference will be tokenized but the model has no learned grounding for them in the Joby domain.
• Not bilingual. Training data is English-only. Spanish/Portuguese fallback is base-Gemma-quality at best.

---

4. Architecture (base model)

Only the language-model trunk is touched by the LoRA. Vision and audio towers are frozen and effectively unused in this deployment.

---

5. Training data

A deliberately mixed corpus — 10,990 ChatML conversations. The mix is the central design choice of v0.3 and is what made tool-calling survive (see §9 Version History).

27.6% of examples contain structured tool_calls fields. 72.4% are plain assistant-text completions.

5.1 Joby tickets + KB (7,949)

Built by the joby-datasets pipeline (~/joby-datasets/). Each row is one Jira ticket or one Confluence section, rewritten by a local LLM (pre-gemma4 via Ollama) into a clean instruction → agent-style answer pair:

Extract  →  Transform (synthesized)  →  Filter  →  Format (ChatML)
 Jira/Confluence    LLM-rewrite             dedup/trim     embed system prompt

• Jira JQL. project = IT AND statusCategory = Done AND resolved >= "-730d" — last two years of resolved IT tickets, paginated 100 at a time with retry on 429/5xx. Fields requested: summary, description, status, resolution, priority, issuetype, labels, components, created/updated/resolved dates, reporter, assignee, full comment thread.
• Confluence space. ITKB (IT Knowledge Base), 69 pages, recursive descent. Min page length 200 chars.
• Transform (synthesized mode). Each Jira ticket's full comment thread is rewritten by pre-gemma4 into a clean agent reply. Confluence pages are chunked at H1/H2 boundaries; each chunk is paired with LLM-generated questions to form instruction/answer pairs.
• Filter. Length bounds, one-liner rejection (fixed, duplicate, see IT-123), email-signature stripping, attachment-marker removal, fuzzy dedup with rapidfuzz, mojibake/non-ASCII bloat removal.
• System prompt baked in (ChatML): identifies the model as a Joby IT service-desk assistant.

5.2 Glaive function-calling v2 (2,800)

glaiveai/glaive-function-calling-v2 — the open-source canonical function-call training set. Streamed via ijson from the local HF cache; only conversations containing at least one <functioncall>{...}</functioncall> block are kept (non-tool turns are discarded). Glaive's idiosyncratic markup (USER: / A: / ASSISTANT: / FUNCTION RESPONSE: role tags, single-quote-inside-double-quote argument strings) is parsed by build_mixed_dataset.py::parse_glaive_chat into proper ChatML messages with structured tool_calls fields. 2,800 records were sampled with seed=42.

5.3 PRE session tool-use traces (241)

Real conversation histories from ~/.pre/sessions/ exported via web/src/training.js's exportTrainingData({format:'chatml', minToolCalls:1}). PRE emits tool calls as inline <tool_call>{...}</tool_call> markup in assistant content; the build script (_convert_pre_record) promotes those to structured tool_calls fields so Gemma 4's chat template renders them with native function-call tokens.

These are the most "in-domain" tool-use examples in the mix — they contain real PRE tool names (bash, memory_search, confluence_search, jira_get_issue, calendar_list_events, apple_mail_compose, rag_search, …) and the real ChatML shape the deployed model will see.

5.4 Quality controls and what was not done

• No PII redaction. Joby IT tickets are confirmed PII-free at design time. Internal account IDs, hostnames, and infrastructure conventions are present in the weights — treat outputs as internal-classification material.
• No license/copyright scrubbing of Confluence content beyond the standard filter pipeline.
• No safety RLHF. This is a supervised LoRA only. The base model's RLHF/safety tuning is the only behavioral guardrail.
• No multimodal data. Text-only fine-tune; vision/audio pathways untouched.

---

6. Training recipe

Full reproducible config in lora_config.yaml. Mixing script in build_mixed_dataset.py.

6.1 Why attention-only LoRA

The v0.2 fine-tune (Joby-only, full LoRA targeting q/k/v/o + gate/up/down) collapsed tool-calling to 0 / 3. The Joby corpus has zero tool-call traces, and full LoRA strongly overfit the FFN toward plain-text replies, overriding the base model's learned tool-call routing.

v0.3 fixes this two ways: (a) mix in 27.6% structured tool-call examples so the model actually sees tool calls during training, and (b) restrict LoRA to attention q_proj / v_proj only, leaving the FFN — where tool routing lives — completely untouched. The two changes are complementary; ablating either reproduces the regression.

6.2 Hardware & throughput

• M4 Max, 128 GB unified memory, macOS, MLX-LM trainer (Apple Silicon native, Metal-backed).
• Sustained throughput: ~120 tokens/sec, ~0.4 iter/sec (batch 1, seq 4096).
• Peak memory: 102.96 GB unified (≈80% of 128 GB; macOS swap headroom kept the system responsive).
• Wall time: ~5 hours for 10,000 iters, ~80 sec/eval for 50 val batches.

6.3 Training dynamics

Validation loss every 250 iters. Best checkpoint is iter 9,250 (val 0.678). The final iter's spike to 1.077 is consistent with the noisy single-batch SGD on a mixed corpus and is not what we released — the published GGUF uses iter 9,250.

Loss is noisy because batch size = 1 is the dominant variance source. The trend is clearly downward through ~iter 9,250 with the floor stepping down through 0.840 → 0.808 → 0.762 → 0.740 → 0.678.

---

7. Evaluation

Run with evaluate.py after ollama create joby-it-servicedesk -f Modelfile.joby:

python evaluate.py --model joby-it-servicedesk:q8_0 --base pre-gemma4

7.1 Tool-calling smoke test

Three prompts that have no overlap with the training corpus, designed to probe whether native tool activation survived the LoRA. Each prompt is paired with the minimum tool schema (bash, date, memory_search) the model needs to respond correctly.

The fine-tune matches the base model's tool-activation rate. This was the binary success criterion that v0.1 and v0.2 failed.

7.2 Held-out validation loss

7.3 Domain knowledge probes

Free-form, no-tool generation. Used as a qualitative sanity check (correctness must still be verified against live Confluence/Jira):

• How do I request a Fusion 360 license at Joby?
• Where is the IT Knowledge Base in Confluence?
• What's the Jira project key for the IT service desk?
• How do I connect to the Joby VPN?

These produce on-format, Joby-styled answers. They are not authoritative — the model may invent URLs, ticket numbers, or KB titles. Always couple with a real confluence.search / jira.get_issue tool call before acting.

7.4 What is not evaluated here

• Exact-match accuracy on closed-book Joby facts. Deliberately not measured, because we want the model to defer to tools.
• Toxicity / safety. Inherits Gemma 4's RLHF; no separate red-teaming for the adapter.
• Long-context comprehension >32K. Inherits Gemma 4's 256K positional embeddings but was trained at max_seq_length=4096. Behavior at 32K–128K context is governed by the base; expect graceful degradation, not measured here.

---

8. Files in this repo

Companion adapter repo (sunkencity/joby-it-servicedesk-lora) ships:

• adapter_model.safetensors — 22.27 MB LoRA weights (rank-16, attention-only).
• lora_config.yaml — exact MLX-LM config used to train.

---

9. Version history

---

10. Usage

10.1 Ollama (recommended)

# Private repo — needs an HF token
export HF_TOKEN=<your-hf-token>
ollama pull hf.co/sunkencity/joby-it-servicedesk:q8_0
ollama run hf.co/sunkencity/joby-it-servicedesk:q8_0

10.2 Via PRE installer

cd ~/pre && ./install.sh
# Installer detects Joby access and offers to pull this model,
# aliasing it locally as `pre-gemma4-itsd`.

PRE then routes requests to it via the standard pre-gemma4 channel — same tool wiring, same ~/.pre/ data dir, same sessions.

10.3 Sampling defaults (Modelfile.joby)

For Q&A-style use with strict factual answers, override at runtime: temperature 0.2, drop top_p/top_k/min_p to defaults.

10.4 Chat template

The merged model carries Gemma 4's chat_template.jinja unchanged. Tool calls in the OpenAI/Glaive tool_calls shape are rendered by the template into native Gemma function-call tokens; client code does not need a custom wrapper.

---

11. Limitations & responsible use

• Internal references baked in. AWS account IDs, hostnames, internal URL patterns, and Joby-specific infrastructure conventions are present in the weights. Treat all outputs as internal-only material — equivalent to forwarding a Jira ticket excerpt.
• No PII redaction pass. The pipeline does not redact, because the source corpus is confirmed PII-free. If future Joby ticket data contains PII, add a redaction stage between transform and filter in ~/joby-datasets/.
• Hallucination of specifics is expected without tools. The training mix deliberately downweights memorized factual recall (URLs, ticket IDs, account numbers) in favor of tool-using behavior. Always run with confluence.search, jira.get_issue, smartsheet.*, rag.search available.
• No incident-response autonomy. Human review required for any action-taking output (account changes, MDM commands, permission grants).
• Knowledge cutoff = data cutoff. Jira pull is bounded to resolved >= "-730d" from the data extraction date (2026-05-15). Anything resolved before mid-2024 or after that snapshot is not represented.
• Gemma 4 license applies. Use of the weights is governed by the Gemma Terms of Use. Derivative weights inherit the license.

---

12. Reproducibility

The full pipeline lives in ~/joby-datasets/:

joby-datasets/
├── src/                          # Jira/Confluence extraction + transform + filter + format
├── config.yaml                   # JQL queries, Confluence spaces, filter thresholds
├── training/
│   ├── build_mixed_dataset.py    # Joby + Glaive + PRE mixer (this fine-tune's secret sauce)
│   ├── prepare_data.py           # Stage ChatML into MLX-LM-expected layout
│   ├── lora_config.yaml          # MLX-LM hyperparameters
│   ├── train.sh                  # mlx_lm.lora -c lora_config.yaml
│   ├── fuse.sh                   # mlx_lm.fuse adapter → base
│   ├── convert_to_gguf.sh        # llama.cpp GGUF + quant
│   ├── evaluate.py               # val loss + tool-call probe
│   └── publish_hf.sh             # push to HF (private)

To rebuild end-to-end:

# 1. Extract
python -m src.extract_jira
python -m src.extract_confluence

# 2. Transform + filter + format
python -m src.transform --mode synthesized
python -m src.filter
python -m src.format --format chatml --split 0.1

# 3. Mix + stage
cd training && python build_mixed_dataset.py --glaive-n 2800 --seed 42
python prepare_data.py

# 4. Train (≈5 hours on M4 Max 128GB)
./train.sh

# 5. Fuse → convert → register → evaluate → publish
./fuse.sh
./convert_to_gguf.sh
ollama create joby-it-servicedesk -f Modelfile.joby
python evaluate.py --model joby-it-servicedesk:q8_0 --base pre-gemma4
./publish_hf.sh

---

13. Citation

@misc{joby_it_servicedesk_2026,
  author       = {Bradford, Christopher},
  title        = {Joby IT Service Desk — Gemma 4 31B (Fine-Tuned, Tool-Aware)},
  year         = {2026},
  version      = {0.3},
  howpublished = {HuggingFace private repo: sunkencity/joby-it-servicedesk},
  note         = {Private internal Joby Aviation model. Derived from google/gemma-4-31B-it.}
}

14. Acknowledgements

• Google DeepMind for Gemma 4.
• MLX team (Apple) for MLX-LM — the only LoRA trainer that comfortably handles a dense 31B base in 128 GB unified memory.
• Glaive AI for glaive-function-calling-v2, the function-call corpus that made v0.3 possible.
• llama.cpp maintainers for keeping the Gemma 4 GGUF converter current.
• Joby IT for two years of clean, well-resolved service-desk tickets.