Qwen2.5-Coder-3B-PT

Qwen2.5-Coder for Pharo Code Completion

This repository provides a Pharo-specialized variant of Qwen2.5-Coder, fine-tuned for fill-in-the-middle (FIM) code completion in Pharo, a Smalltalk-inspired, low-resource programming language.

The model is designed to complete partially written Pharo methods by predicting the missing code between a given prefix and suffix, making it suitable for IDE-assisted code completion.

Model Details

• Base model: Qwen2.5-Coder
• Model type: Decoder-only causal language model
• Programming language: Pharo / Smalltalk
• Primary task: Fill-in-the-middle (FIM) code completion
• Training pipeline: Continued pre-training followed by supervised fine-tuning
• Maximum context length: 32,768 tokens

Motivation

Large language models perform remarkably well on widely used programming languages such as Python, Java, and JavaScript, but their performance remains limited for low-resource languages such as Pharo.

Pharo inherits several unique characteristics from Smalltalk, including keyword-based message syntax, message-oriented control flow, and concise methods. These models are specifically adapted to these characteristics to provide more accurate and practical code completion while remaining compact enough for efficient local inference and IDE integration.

Training Overview

The model is trained using a two-stage specialization pipeline.

Continued Pre-Training

The base Qwen2.5-Coder model is first adapted to Pharo source code through continued pre-training on a curated corpus of Pharo methods. This stage teaches the model Pharo syntax, APIs, and coding conventions.

Supervised Fine-Tuning

The adapted checkpoint is then fine-tuned on realistic fill-in-the-middle completion examples generated from Pharo methods. Completion targets begin at arbitrary cursor positions, better reflecting real-world code editing scenarios.

Usage

Install the required libraries:

pip install -U "transformers>=4.37.0" accelerate torch

Load the model:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_name = "pharo-llm/Qwen2.5-Coder-7B-SFT"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto",
).eval()

Example

The model is trained for fill-in-the-middle (FIM) code completion using the Qwen FIM prompt format.

prefix = """Collection >> selectEven

    ^ self select: [ :each |
"""

suffix = """
]
"""

prompt = (
    "<|fim_prefix|>"
    + prefix +
    "<|fim_suffix|>"
    + suffix +
    "<|fim_middle|>"
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=16,
    do_sample=False,
)

completion = tokenizer.decode(
    outputs[0][inputs.input_ids.shape[1]:],
    skip_special_tokens=True,
)

print(completion)

Expected completion:

each even

Paper

For additional details about the training methodology, datasets, and evaluation, see:

Teaching LLMs a Low-Resource Language: Enhancing Code Completion in Pharo
https://huggingface.co/papers/2607.04939

Citation

If you use this model in your research or applications, please cite our paper:

@article{teachingllmslowresourcelanguage,
  title={Teaching LLMs a Low-Resource Language: Enhancing Code Completion in Pharo},
  author={Kilian Kier and Alessandro Giagnorio and Omar AbedelKader and Oleksandr Zaitsev and Robert Peharz and Romain Robbes and Gabriele Bavota and Stéphane Ducasse},
  journal={arXiv},
  year={2026},
  doi={10.48550/arXiv.2607.04939}
}