Downloads · 30 days
31
26% of all-time downloads
ZhangYuchi/modelbest-Qwen-2.5-Omni-7B-SFT-only
modelbest-Qwen-2.5-Omni-7B-SFT-only is a any-to-any model from ZhangYuchi. Use it for the any-to-any task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as apache-2.0.
This is a fully fine-tuned version of Qwen/Qwen2.5-Omni-7B on the OmniGAIA training data, trained as a tool-integrated omni-modal agent that natively reasons over video + audio + image and calls external tools (websea…
Downloads · 30 days
31
26% of all-time downloads
All-time downloads
119
Public
Parameters
8.9B
17.9 GB on disk
Likes
2
Public
Click a slice to open those files.
.safetensors17.9 GB · 100%
From the Hugging Face model README
This is a fully fine-tuned version of Qwen/Qwen2.5-Omni-7B on the OmniGAIA training data, trained as a tool-integrated omni-modal agent that natively reasons over video + audio + image and calls external tools (web_search, page_browser, code_executor, read_image/audio/video).
The key differences from the recipe in the OmniGAIA paper are:
with_obs trajectories from the OmniGAIA train split as-is, rather than re-synthesizing trajectories via hindsight-guided tree exploration (the paper's OmniAtlas-SFT recipe).role=tool (observation) messages are normalized to role=user to fit the Qwen2.5-Omni chat template. Combined with ms-swift's default --loss_scale default (which only computes loss on assistant turns), this means only assistant tokens contribute to the loss; tool / observation content is fully masked, functionally the same as the paper's trajectory-level masked SFT.The training objective is functionally equivalent to OmniAtlas-SFT (only assistant tokens are supervised; tool / observation tokens are masked). The improvement over the paper's 7B numbers is therefore attributable to the training-data recipe (raw
with_obstrajectories + sliding-window long-context handling), not to a different loss target.
This is checkpoint-150 (epoch ≈ 1.28), which gave the most stable validation performance during training.
Evaluated using the official OmniGAIA eval script (run_base_agent_original_paper.py) with DeepSeek-V3.2 as the LLM judge, same tools/active-perception setup as published baselines.
| Seed | Pass@1 (LLM_Equal) | EM | Avg tool calls | Non-empty rate |
|---|---|---|---|---|
| 42 | 16.94 | 8.61 | 12.16 | 100% |
| 1337 | 15.28 | 5.28 | 12.27 | 100% |
| 2024 | 14.44 | 6.11 | 11.91 | 100% |
| 7 | 16.39 | 4.17 | 11.86 | 100% |
| Mean ± std (4 seeds) | 15.76 ± 1.12 | 6.04 ± 1.89 | 12.05 | 100% |
| Method | Pass@1 |
|---|---|
| Qwen2.5-Omni-7B (zero-shot) | 3.6 |
| + OmniAtlas-SFT (paper) | 11.4 |
| + OmniDPO (paper) | 13.3 |
| + with_obs SFT (this model) | 15.76 |
The cross-seed variance comes mainly from (i) real-time
web_searchresults drifting between runs, and (ii) DeepSeek-V3.2 judge non-determinism, not from the model itself (vLLM is run in greedy mode).
| Level | Count | Pass@1 |
|---|---|---|
| Easy | 122 | 25.41 |
| Medium | 160 | 12.97 |
| Hard | 78 | 7.05 |
| Item | Value |
|---|---|
| Base model | Qwen/Qwen2.5-Omni-7B |
| Trainer | ms-swift (full-param SFT, bf16) |
| Train data | OmniGAIA train split, kept as with_obs cached dataset (~2,150 trajectories), split via sliding window (width=8192, overlap=4096) → 4,682 windows |
| Epochs | 2 |
| Effective batch | 1 × 8 GPUs × gradient_accumulation_steps=5 |
| Learning rate | 1e-5, warmup ratio 0.03 |
| Max length | 8,192 |
| Precision | bf16 |
| Attention | flash_attention_2 |
| Other | padding_free, use_liger_kernel, gradient & vit_gradient checkpointing, DeepSpeed ZeRO-2 |
This checkpoint corresponds to global step 150 / 234 (epoch ≈ 1.28). Later checkpoints (200, 234) showed slightly higher train loss reduction but no monotonic Pass@1 gain in our evaluation, so we use step 150 for the leaderboard submission.
We verified that no test-set question text from test_metadata.json appears verbatim in any training sample (0 / 2,151 overlap). The training set shares media with the test set (~23%) because the OmniGAIA dataset is constructed from a shared media pool — this is the same condition under which all OmniAtlas / Qwen baselines in the paper were evaluated, so the comparison is apples-to-apples.
from transformers import Qwen2_5OmniForConditionalGeneration, AutoProcessor
model_path = "ZhangYuchi/modelbest-Qwen-2.5-Omni-7B-SFT-only"
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)
model = Qwen2_5OmniForConditionalGeneration.from_pretrained(
model_path,
torch_dtype="bfloat16",
device_map="auto",
trust_remote_code=True,
)
For best results with OmniGAIA-style multi-turn tool-use evaluation, serve it with vLLM:
vllm serve <model_path> \
--trust-remote-code \
--tensor-parallel-size 1 \
--max-model-len 32768 \
--max-num-seqs 1 \
--max-num-batched-tokens 32768
and drive it via the OmniGAIA agent loop (run_base_agent_original_paper.py --enable-active-perception ...) with web_search, page_browser, code_executor, and read_image/audio/video tools.
Qwen2.5-Omni-7B (audio perception errors on long clips, vision recognition errors on rare entities, etc.).Functionally equivalent loss masking, different role wrapper.
tool / observation messages are remapped to role=user (to satisfy the Qwen2.5-Omni chat template), and ms-swift then masks all non-assistant tokens via --loss_scale default. The end effect — supervising only assistant tokens (reasoning + tool_call) — matches the paper's trajectory-level masked SFT, but the on-disk role schema differs. Downstream tooling that special-cases role=tool may not recognize the observation segments in this dataset / format.
Trajectories are not re-synthesized.
The paper's OmniAtlas-SFT uses hindsight-guided tree exploration with Gemini-3-Flash to synthesize new tool-integrated trajectories. This model trains directly on the raw with_obs trajectories shipped in the OmniGAIA train split, without any re-synthesis or trajectory cleanup.
No DPO / RLHF stage.
The paper reports its strongest 7B number with OmniAtlas-SFT + OmniDPO. This model is SFT-only.
Shared media pool between train and test. The OmniGAIA train and test sets are constructed from a shared media pool (~23% of train trajectories share at least one media file with a test item, even though no test-set question text appears verbatim in any train sample — see "Data contamination self-check" above). This is a property of the benchmark itself, the same as all baselines in the paper.
What this implies for the reported numbers:
+OmniAtlas-SFT (11.4) and +OmniDPO (13.3) therefore reflects differences in training data recipe (raw with_obs + sliding window) rather than a different loss mask.with_obs, holding everything else fixed) has not been run in this codebase, so the per-component contribution of "raw with_obs" vs "sliding window" vs "different trajectory pool" is not yet decomposed.Apache-2.0, inherited from the base Qwen2.5-Omni-7B.
If you use this model, please also cite the underlying base model and the OmniGAIA benchmark:
@misc{qwen25omni,
title = {Qwen2.5-Omni Technical Report},
author = {Qwen Team},
year = {2025},
}
@misc{omnigaia,
title = {OmniGAIA: Towards Native Omni-Modal AI Agents},
author = {RUC-NLPIR et al.},
year = {2026},
url = {https://huggingface.co/spaces/RUC-NLPIR/OmniGAIA-Leaderboard},
}