Stockmark-Nemotron-3-Nano-Omni-JapanDocReader

Stockmark-Nemotron-3-Nano-Omni-JapanDocReader

Introduction

Stockmark-Nemotron-3-Nano-Omni-JapanDocReader is a Japanese document-reading multimodal model built on top of NVIDIA's Nemotron-3-Nano-Omni-30B-A3B-Reasoning. This model injects Japanese structured document parsing capability while preserving as much of the original VQA ability as possible.

It is trained with a two-stage recipe:

1. Mixed SFT — supervised fine-tuning on a mixture of structured parsing data and VQA data. Mixing VQA into the SFT set mitigates catastrophic forgetting of the base model's reasoning/VQA ability that pure structured-parsing-only SFT would cause. SFT was performed with NeMo Megatron-Bridge.
2. DAPO — reinforcement learning applied to the structured document parsing capability on top of the mixed-SFT base. This pushes structured document parsing quality above the SFT base while keeping the VQA regression small. DAPO was performed with NeMo-RL.

All training keeps thinking mode on and uses English reasoning traces, which is central to preserving multi-step-reasoning VQA performance (switching to no-think or changing the reasoning language degrades the base model's reasoning distribution).

Training data was synthesized with Stockmark's Japanese document synthesis pipeline using Nemotron-Personas-Japan and Qwen3.6-27B as the generator.

For a detailed write-up (in Japanese), see our our blog.

Quickstart

The model serves as a standard OpenAI-compatible endpoint via vLLM (≥ 0.20). It is a reasoning model — keep thinking mode on and inject a reasoning budget so it closes </think> and answers.

1. Launch the server

vllm serve stockmark/Stockmark-Nemotron-3-Nano-Omni-JapanDocReader \
  --served-model-name japandocreader \
  --host 0.0.0.0 --port 8000 \
  --dtype bfloat16 \
  --max-model-len 210000 \
  --tensor-parallel-size 1 \
  --trust-remote-code \
  --reasoning-parser nemotron_v3 \
  --allowed-local-media-path / \
  --media-io-kwargs '{"video": {"fps": 2, "num_frames": 256}}' \
  --video-pruning-rate 0.5

2. Recommended inference parameters

max_tokens > reasoning_budget is required: the budget caps the think block, and the remaining max_tokens − reasoning_budget tokens hold the answer. Setting them equal starves the answer.

3. Structured document parsing

⚠️ Use this exact prompt. The model was trained with the fixed Japanese structured document parsing prompt below. The prompt defines the task, the JSON schema, the allowed class values, and the coordinate convention — the model's output format is conditioned on it. Do not paraphrase, translate, or reorder it; changing the prompt degrades layout accuracy and JSON validity. Keep it verbatim, including the trailing Return ONLY the JSON object instruction.

import base64, json, re, urllib.request

PROMPT = """画像に含まれるドキュメントの構造をJSON形式で抽出してください。
出力フォーマット:
{
  "document_structure": [
    {
      "class": "title" | "heading" | "text" | "table" | "list" | "picture" | "formula",
      "bbox": [x1, y1, x2, y2],
      "contents": "内容（pictureの場合は画像内容の説明、formulaの場合は数式のlatex表記）",
      "caption": "pictureのキャプション文字（オプション）"
    }
  ]
}
classの種類: title（タイトル）、heading（見出し）、text（本文）、table（表）、list（リスト）、picture（画像）、formula（数式）
bboxは左上(x1,y1)と右下(x2,y2)の座標です。bboxの座標系は0-1000の相対座標です。
Return ONLY the JSON object. No markdown, no extra commentary."""

def parse_document(image_path, base_url="http://127.0.0.1:8000"):
    with open(image_path, "rb") as f:
        b64 = base64.b64encode(f.read()).decode()
    payload = {
        "model": "japandocreader",
        "messages": [{"role": "user", "content": [
            {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}},
            {"type": "text", "text": PROMPT},
        ]}],
        "temperature": 0.6, "top_p": 0.95,
        "repetition_penalty": 1.0,
        "max_tokens": 20480,
        "chat_template_kwargs": {"enable_thinking": True, "reasoning_budget": 16384},
        "thinking_token_budget": 17408,   # reasoning_budget + grace
    }
    req = urllib.request.Request(
        base_url.rstrip("/") + "/v1/chat/completions",
        data=json.dumps(payload).encode(),
        headers={"Content-Type": "application/json", "Authorization": "Bearer EMPTY"},
        method="POST")
    with urllib.request.urlopen(req, timeout=1800) as r:
        msg = r.read(); msg = json.loads(msg)["choices"][0]["message"]
    # thinking is split into reasoning_content; the answer is the JSON in `content`
    answer = msg.get("content") or ""
    m = re.search(r"\{.*\}", answer, re.DOTALL)
    return json.loads(m.group(0) if m else answer)   # -> {"document_structure": [...]}

result = parse_document("document.png")
print(json.dumps(result, ensure_ascii=False, indent=2))

4. Document VQA

For VQA, send the image plus the question (no docparse prompt); the model reasons in <think> and answers in natural language:

payload["messages"] = [{"role": "user", "content": [
    {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}},
    {"type": "text", "text": "この文書について: <your question in Japanese>"},
]}]
# same sampling params as above; the answer is in message.content, the reasoning in reasoning_content

License

Released under the NVIDIA Open Model License, inherited from the base model. Please also review the base model's terms.

Acknowledgements

In this experiment, we used 8×B300 Blackwell Ultra compute resources provided by NVIDIA, and conducted training on NVIDIA Brev using NVIDIA NeMo. We sincerely thank everyone at NVIDIA for supporting the large-scale SFT and RL experiments.

Developed by

Stockmark Inc.

Citation

@misc{stockmark_japandocreader_2026,
  title={Stockmark-Nemotron-3-Nano-Omni-JapanDocReader: Structured Document Parsing via Capability Injection and Forgetting Control}, 
  author={Stockmark Inc.},
  year={2026},
  eprint={2608.06758},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2608.06758}, 
}