nemotron-testing

AdaptKey/AdaptKey-Nemotron-30b

Overview

AdaptKey-Nemotron-30b is a LoRA fine-tuned version of NVIDIA's Nemotron-3-Nano-30B model, specialized for telecommunications and network engineering applications. The model was trained on 1.3M+ telecom domain examples covering 3GPP standards, IETF protocols, network traces, anomaly detection, and network function configuration.

This model achieved a composite benchmark score of 596 — a +58 point improvement (+10.8%) over the NVIDIA Nemotron-3-Nano-30B-A3B baseline of 538 — while using conservative anti-forgetting training strategies to preserve general capabilities.

Benchmark Results

Evaluated via the TeleFlow evaluation system on 2/9/2026. See Evaluation Methodology below for full details on scoring.

Strongest Gains

• TeleYaml +16.8 pts (+26.9%) — structured YAML generation for network configs
• TeLogs +12.8 pts (+26.2%) — network log analysis and fault diagnosis
• TeleTables +11.8 pts (+19.3%) — tabular reasoning over network parameters

---

Evaluation Methodology

Overview

Adaptkey uses a two-tier scoring system designed to minimize judge cost while maximizing evaluation accuracy:

1. Deterministic scoring — applied first whenever the answer is objectively verifiable (exact-match multiple choice, numeric answers). Scores are 10 (correct) or 0 (incorrect). The LLM judge is skipped entirely for these cases, eliminating variance and cost.
2. LLM-as-a-Judge — invoked for all remaining responses where deterministic checking cannot conclusively score quality.

Judge Model

Scoring Rubrics

Two rubrics are applied depending on benchmark type:

Rubric A — Free-Text Technical Answers

Applied to: TeleQnA, TeleMath, TeleLogs, TSG-3GPP

The judge evaluates three criteria simultaneously:

• Factual Accuracy — Are the key technical facts correct?
• Completeness — Does the response cover the main points from the reference answer?
• Correctness — Are there any incorrect statements that would mislead an engineer?

Rubric B — Structured Configuration Answers

Applied to: TeleYaml, TeleTables

The judge evaluates two weighted axes:

• Structural Validity (40%) — Is the output a valid configuration with correct syntax?
• Content Accuracy (60%) — Do field names and values match the expected configuration? Partial credit awarded proportionally based on ratio of correct fields to total fields.

Judge Prompt Structure

Each judge invocation consists of two messages:

System message:

You are a strict telecom evaluation judge. Score accurately based on the rubric.
Output ONLY the JSON object.

User message:

Question: {question}

Reference Answer: {reference_answer}

Model Response: {model_response}

Scoring Rubric:
{applicable_rubric}

Output JSON: {"score": <0-10>, "reasoning": "<brief explanation>"}

Retry Policy

If the judge scores a response below a configurable threshold, the model is re-prompted up to 5 times. The best score across all attempts is recorded. This measures the model's capability ceiling rather than single-shot performance, and is applied consistently across all models evaluated including the baseline.

Benchmark-to-Rubric Mapping

What We Did

• Goal: Create a specialized telecom AI assistant with expert-level knowledge of 3GPP, IETF, ITU, and TM Forum standards
• Approach: LoRA fine-tuning with conservative hyperparameters to prevent catastrophic forgetting
• Dataset: 1.3M+ telecom Q&A examples with augmented network slicing and network function configuration data
• Base model: NVIDIA Nemotron-3-Nano-30B-A3B (Megatron format)

Training Data

Dataset Composition (~1.31M examples)

Domain Coverage

• Network Traces & Anomaly Detection: 5G trace analysis, KPI statistics, anomaly classification
• Network Slicing: S-NSSAI configuration, slice types (eMBB, URLLC, mMTC), resource allocation
• Network Function Configuration: Open5GS YAML generation, AMF/SMF/UPF configuration
• 3GPP Standards Q&A: Core network procedures, RAN protocols, signaling
• Network Forecasting: Trend analysis, traffic prediction
• Troubleshooting: Root cause analysis, diagnostic procedures

Data Format

{
  "input": "System: You are an expert telecommunications engineer...\nUser: [question with context]",
  "output": "[detailed answer with reasoning]"
}

Training Details

LoRA Hyperparameters

Training Configuration

Infrastructure

Parallelism

Training Progress

Version History

Key Improvements in This Version

• Augmented network slicing examples to address weak benchmark performance
• Enhanced network function configuration coverage
• Improved system prompts (removed misleading "telco expert" framing for non-telco questions)
• +10.8% absolute improvement on composite benchmark over NVIDIA baseline

Post-Training Pipeline

# Merge LoRA weights
torchrun --nproc-per-node=4 \
  /opt/Megatron-Bridge/examples/peft/merge_lora.py \
  --lora-checkpoint /models/AdaptKey-Nemotron-30b-lora/iter_0010500 \
  --hf-model-path /models/nemotron-30b \
  --output /models/AdaptKey-Nemotron-30b-merged

# Export to HuggingFace format
python /opt/Megatron-Bridge/examples/conversion/convert_checkpoints.py export \
  --hf-model /models/nemotron-30b \
  --megatron-path /models/AdaptKey-Nemotron-30b-merged \
  --hf-path /models/AdaptKey-Nemotron-30b-hf-export

Usage

With Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "AdaptKey/AdaptKey-Nemotron-30b",
    trust_remote_code=True,
    torch_dtype="bfloat16",
)
tokenizer = AutoTokenizer.from_pretrained(
    "AdaptKey/AdaptKey-Nemotron-30b",
    trust_remote_code=True,
)

prompt = """System: You are an expert telecommunications engineer. Answer questions accurately based on your knowledge of telecom standards (3GPP, IETF, ITU, TM Forum).

User: Explain the difference between eMBB, URLLC, and mMTC slice types in 5G network slicing."""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

With vLLM

from vllm import LLM, SamplingParams

llm = LLM(
    model="AdaptKey/AdaptKey-Nemotron-30b",
    trust_remote_code=True,
    tensor_parallel_size=1,
    gpu_memory_utilization=0.90,
)

sampling_params = SamplingParams(temperature=0.7, max_tokens=512)
outputs = llm.generate([prompt], sampling_params)

Docker Compose (vLLM Server)

services:
  vllm-adaptkey:
    image: vllm/vllm-openai:latest
    container_name: vllm-adaptkey-nemotron-30b
    runtime: nvidia
    environment:
      - NVIDIA_VISIBLE_DEVICES=0
    ports:
      - "8090:8000"
    volumes:
      - /opt/models:/models:ro
    command: >
      --model /models/AdaptKey-Nemotron-30b
      --trust-remote-code
      --max-model-len 8196
      --gpu-memory-utilization 0.90
      --tensor-parallel-size 1
    restart: unless-stopped

Lessons Learned

1. Anti-forgetting strategy works: Conservative LoRA params (64/128/0.1) with 5e-5 LR preserved general capabilities
2. Data quality matters more than quantity: Improving weak-area examples had more impact than adding more data
3. System prompt alignment: Mismatched system prompts (e.g., "telco expert" for ethics questions) hurt performance
4. Mixed datasets: Combining diverse telecom subcategories prevents narrow specialization

License

This model is derived from NVIDIA's Nemotron-3-Nano-30B and is subject to the NVIDIA Open Model License Agreement. Please review the license terms before use in commercial applications.

Citation

@misc{adaptkey_nemotron_30b_2026,
  title={AdaptKey-Nemotron-30b: A Telecom-Specialized Language Model},
  author={AdaptKey},
  year={2026},
  publisher={HuggingFace},
  url={https://huggingface.co/AdaptKey/AdaptKey-Nemotron-30b}
}