JustRL-Qwen3-4B

JustRL-Qwen3-4B

JustRL-Qwen3-4B is a research checkpoint from the Direct-OPD collection. It is released for reproducible research on post-training and reinforcement-learning-style optimization for reasoning-oriented language models.

Model Details

• Repository: BytedTsinghua-SIA/JustRL-Qwen3-4B
• Collection: Direct-OPD
• Source checkpoint: c7wc7w/JustRL_4B_KL2_LEN2k_ckpt220
• Model family: Qwen3
• Parameter scale: 4B
• Training recipe: JustRL
• Checkpoint: ckpt220
• License: Apache-2.0

Training and Provenance

A JustRL Direct-OPD checkpoint for the Qwen3 4B family, with KL coefficient 2 as indicated by the source checkpoint name.

The checkpoint was mirrored from ModelScope to Hugging Face for easier access and inclusion in the Direct-OPD collection. The source checkpoint name records the available release metadata, including method, model family, KL/adaptive-KL setting when present, sequence length setting, and checkpoint step. Full training data, hyperparameters, and evaluation protocol should be taken from the associated Direct-OPD release materials when available.

Intended Use

This model is intended for research use, including:

• studying post-training and RL-style optimization behavior;
• comparing Direct-OPD checkpoints across model families and parameter scales;
• running offline evaluation on reasoning, instruction-following, and alignment benchmarks;
• serving as an initialization or comparison point for further research.

It is not intended for direct deployment in safety-critical, medical, legal, financial, or other high-stakes settings without independent evaluation and safeguards.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "BytedTsinghua-SIA/JustRL-Qwen3-4B"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

messages = [
    {"role": "user", "content": "Explain the main idea of Direct-OPD in one paragraph."},
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
inputs = tokenizer([text], return_tensors="pt").to(model.device)

output_ids = model.generate(
    **inputs,
    max_new_tokens=512,
    temperature=0.7,
    top_p=0.95,
)
print(tokenizer.decode(output_ids[0], skip_special_tokens=True))

Evaluation

No benchmark results are included in this model card yet. Users should evaluate the checkpoint on the tasks and safety criteria relevant to their use case before drawing conclusions or deploying derived systems.

Recommended reporting for downstream evaluations:

• benchmark name and version;
• decoding configuration;
• prompt template and chat template;
• number of samples and random seeds;
• comparison baselines from the same model family.

Limitations

• The model may generate incorrect, misleading, biased, or unsafe content.
• The model card does not currently document the full training dataset or complete hyperparameter schedule.
• Performance can be sensitive to prompt format, decoding settings, and evaluation harness.
• Research checkpoints can regress on general instruction-following or safety behavior while improving on targeted optimization objectives.

Citation

Citation information for the associated Direct-OPD research release will be added when available. If you use this checkpoint, please cite the Direct-OPD project or paper once published, and include the model repository URL in your reproducibility artifacts.

Acknowledgements

This checkpoint is part of the Direct-OPD model collection maintained by BytedTsinghua-SIA and mirrored from the original ModelScope release.