Qwen3-8B-MegaScience

MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoning

This repository hosts the Qwen3-8B-MegaScience model, a key component of the research presented in the paper "MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoning".

Abstract

Scientific reasoning is critical for developing AI scientists and supporting human researchers in advancing the frontiers of natural science discovery. However, the open-source community has primarily focused on mathematics and coding while neglecting the scientific domain, largely due to the absence of open, large-scale, high-quality, verifiable scientific reasoning datasets. To bridge this gap, we first present TextbookReasoning, an open dataset featuring truthful reference answers extracted from 12k university-level scientific textbooks, comprising 650k reasoning questions spanning 7 scientific disciplines. We further introduce MegaScience, a large-scale mixture of high-quality open-source datasets totaling 1.25 million instances, developed through systematic ablation studies that evaluate various data selection methodologies to identify the optimal subset for each publicly available scientific dataset. Meanwhile, we build a comprehensive evaluation system covering diverse subjects and question types across 15 benchmarks, incorporating comprehensive answer extraction strategies to ensure accurate evaluation metrics. Our experiments demonstrate that our datasets achieve superior performance and training efficiency with more concise response lengths compared to existing open-source scientific datasets. Furthermore, we train Llama3.1, Qwen2.5, and Qwen3 series base models on MegaScience, which significantly outperform the corresponding official instruct models in average performance. In addition, MegaScience exhibits greater effectiveness for larger and stronger models, suggesting a scaling benefit for scientific tuning. We release our data curation pipeline, evaluation system, datasets, and seven trained models to the community to advance scientific reasoning research.

Code

The official implementation and full codebase are available on the MegaScience GitHub repository.

Qwen3-8B-MegaScience

Training Recipe

• LR: 5e-6
• LR Schedule: Cosine
• Batch Size: 512
• Max Length: 4,096
• Warm Up Ratio: 0.05
• Epochs: 3

Usage

You can load the model and tokenizer using the transformers library and perform text generation. For Qwen models, it is often recommended to use the apply_chat_template method for instruction-following tasks.

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "MegaScience/Qwen3-8B-MegaScience"

# Load model with bfloat16 for better performance on compatible GPUs
# or use torch.float16 if bfloat16 is not supported
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto" # Automatically maps model to available devices (e.g., GPU)
)
tokenizer = AutoTokenizer.from_pretrained(model_id)

# Example for chat completion
messages = [
    {"role": "user", "content": "Explain Newton's first law of motion in simple terms."},
]

input_ids = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True, # Important for instruction-tuned models
    return_tensors="pt"
).to(model.device)

outputs = model.generate(input_ids, max_new_tokens=256, do_sample=True, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Evaluation Results

<div style="display: flex; justify-content: left; gap: 20px;"> <img src="https://cdn-uploads.huggingface.co/production/uploads/616bfc2b40e2f69baa1c7add/abIVZ2XB9D-o-TCyvOkDE.png" alt="Data Pipeline" style="width:80%;"> </div> <div style="display: flex; justify-content: left; gap: 20px;"> <img src="https://cdn-uploads.huggingface.co/production/uploads/616bfc2b40e2f69baa1c7add/xFTJ7nevc3S4UYJxUS7ue.png" alt="Data Pipeline" style="width:80%;"> </div>

More about MegaScience

<div style="display: flex; justify-content: left; gap: 20px;"> <img src="https://cdn-uploads.huggingface.co/production/uploads/616bfc2b40e2f69baa1c7add/VogIpBbjfNxXFP9DfVMms.png" alt="Data Pipeline" style="width:100%;"> </div>

Citation

Check out our paper for more details. If you use our dataset or find our work useful, please cite

@article{fan2025megascience,
  title={MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoning},
  author={Fan, Run-Ze and Wang, Zengzhi and Liu, Pengfei},
  year={2025},
  journal={arXiv preprint arXiv:2507.16812},
  url={https://arxiv.org/abs/2507.16812}
}