Llama-3.1-Future-Code-Ja-8B

Llama 3.1 Future Code Ja

Llama 3.1 Future Code Ja is a large language model with 8B parameters built on top of the Meta Llama 3.1 model. The model was first experienced continual pre-trained on the mixture of code and mostly-Japanese natural language data. The training data is mainly from The Stack V2 dataset and the subset of LLM-jp Corpus v3, which comprises 204.9B code and 85.7B natural language tokens after carefully designed data cleaning. The model was then merged with the instruct variant of the Meta Llama 3.1 model to acquire abilities to follow general task instructions, followed by supervised fine-tuning (SFT) and direct preference optimization (DPO) on our own magpie-generated code instruction data.

The model officially supports Japanese and English for natural languages and more than 40 programming languages ranging from popular Python, Java etc. to some legacy languages such as COBOL. In addition to causal (left-to-right) inference, the model supports Fill-in-the-Middle (FIM) capability, where the model fills in the blank attending to bidirectional context, a common use case in IDEs.

The model outperforms the original Llama 3.1 model in both Japanese, and English-instructed code completion tasks in various programming languages, and outperforms Qwen families in Japanese generation tasks, attaining a good balance between specialty in code-related tasks and general ability in Japanese.

Usage

Here are the sample inference scripts with transformers. We recommend using vLLM for faster inference.

pip install torch transformers accelerate

Chat

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "future-architect/Llama-3.1-Future-Code-Ja-8B"

model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_name)

# we recommend using the following system prompt:
# for Japanese completion : "あなたは様々なソフトウェア開発タスクをサポートするAIアシスタントです。"
# for English completion : "You are an AI assistant who support various software development tasks."

message = [
    {
        "role": "system",
        "content": "あなたは様々なソフトウェア開発タスクをサポートするAIアシスタントです。"
    },
    {
        "role": "user",
        "content": "PythonでFizzBuzzを書いてください。",
    },
]

input_ids = tokenizer.apply_chat_template(
    message, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)

output = model.generate(**input_ids, max_new_tokens=1024)

print(tokenizer.decode(output[0, input_ids["input_ids"].shape[1]:]))

Fill-in-the-Middle

With the idea that the users may not want line breaks just after their cursor positions, we did not create any middle splits that start with newline symbols (\n), but included them at the end of the prefix instead. This also holds true for the boundaries of suffix and middle splits, causing great sensitivity against which split to include newline symbols. Please remove one new line symbol (if exists) from the beginning of the suffix for improved performance.

You may set a larger repetition penalty to avoid nonsense generations with too many signs.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

FIM_PREFIX = "<|fim_prefix|>"
FIM_MIDDLE = "<|fim_middle|>"
FIM_SUFFIX = "<|fim_suffix|>"

model_name = "future-architect/Llama-3.1-Future-Code-Ja-8B"

model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_name)

# prepend <|begin_of_text|> to inform that this is the beginning of "the content" (not whole sequence with special tokens)
prefix = "<|begin_of_text|>def fizzbuzz(n"
suffix = "return n"

# PSM mode (infilling)
input_txt = FIM_PREFIX + prefix + FIM_SUFFIX + suffix + FIM_MIDDLE
# SPM mode (reverse infilling)
# input_txt = FIM_PREFIX + FIM_SUFFIX + suffix + FIM_MIDDLE + prefix

# set add_special_tokens to False, so that the tokenizer does NOT add <|begin_of_text|> before special tokens
input_ids = tokenizer(input_txt, add_special_tokens=False, return_tensors="pt").to(model.device)

output = model.generate(**input_ids, max_new_tokens=1024, temperature=0.2, top_p=0.95)

print(tokenizer.decode(output[0, input_ids["input_ids"].shape[1]:]))

Model Performance

Code completion (Japanese)

• JHumanEval (Sato et al., 2024)
• JMultiPL-E (Taneguchi et al., 2025)

Note: We do not report scores for two programming languages (Julia and Racket), which we did not include in the training data. All the scores below are pass@1 with 10 trials.

Code completion (English)

• HumanEval (Chen et al., 2021)
• MultiPL-E (Cassano et al., 2022)

Note: We do not report scores for two programming languages (Julia and Racket), which we did not include in the training data. All the scores below are pass@1 with 10 trials.

Fill-in-the-Middle

• SantaCoder-FIM (Allal et al., 2023)

Note: The models with asterisk (*) do not support FIM. We used the SPM prompt in Gong et al., 2024 and truncated the generated output just before the point that matched the beginning of the provided suffix. The scores of Llama models on PSM mode are not reported here since we got almost 0 scores for all those settings. All the scores below are exact match (EM) with 1 trial.

Japanese tasks

• JCommonSenseQA (Kurihara et al., 2022, Exact Match)
• JEMHopQA (Ishii et al., 2024, chr-F1)
• NIILC (Sekine, 2003, chr-F1)
• JSQuAD (Kurihara et al., 2022, chr-F1)
• XL-Sum (Hasan et al., 2021, ROUGE-2)
• MGSM (Shi et al., 2023, Exact Match)
• WMT20 en-ja (Barrault et al., 2020, BLEU)
• WMT20 ja-en (Barrault et al., 2020, BLEU)

English tasks

• TriviaQA (Joshi et al., 2017, Exact Match)
• SQuAD2 (Rajpurkar et al., 2018, Exact Match)
• GSM8K (Cobbe et al., 2021, Exact Match)

Evaluation Details

We used the Code Generation LM Evaluation Harness toolkit to evaluate code completion and FIM capabilities.

We adopted the settings below for decoding. We mostly followed the recommendations however, we set max_new_tokens instead of max_tokens to avoid truncation while handling long input sequences.

• Temperature: 0.2
• Top-p: 0.95
• Number of completions to generate: 10 (for completion tasks), 1 (for FIM tasks)
• Maximum number of new tokens: 512

We followed the evaluation strategy adopted in the Swallow project for Japanese and English tasks. More specifically, we used the llm-jp-eval toolkit for Japanese tasks and the Language Model Evaluation Harness toolkit for English (and some Japanese) tasks.

We adopted the default decoding strategy for all the tasks.

Risks and Limitations

The model is trained on general tasks related to software development, not on organization-specific, and/or non-standardized tasks. We recommend further fine-tuning the model to make it work better with those tasks. The model may produce incorrect output and all the suggestions from the model must be carefully examined before adopting in real-world applications.

Acknowledgements

The model is developed as part of the Generative AI Accelerator Challenge (GENIAC) project. We thank great support from the New Energy and Industrial Technology Development Organization (NEDO) and the Ministry of Economy, Trade and Industry (METI) for financial support.

Contact

• pj-geniac at future.co.jp

License

META LLAMA 3.1 COMMUNITY LICENSE

Copyright © 2025 by Future Corporation