game-comment-vlm

Game Comment VLM

A vision-language model fine-tuned to act as a live game commentator — it looks at a gameplay screenshot and generates a short, natural, spoken-style comment reacting to the moment, similar to how a live streamer or esports caster would react in real time.

• Developed by: TK17250
• License: apache-2.0
• Finetuned from model: unsloth/Llama-3.2-11B-Vision-Instruct
• Model type: Vision-Language Model (mllama architecture, 11B parameters)
• Language: English

This model was trained 2x faster with Unsloth and Hugging Face's TRL library.

<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>

Supported Games

The model is trained specifically on gameplay screenshots from three titles:

Commentary style and vocabulary (e.g. weapon names, map callouts, in-game terminology) are tailored to these three games. Performance on other games or genres outside this scope is not guaranteed, since the training data does not cover them.

Training Data

• Fine-tuned on a synthetic dataset of ~1,600+ gameplay screenshots paired with commentator-style captions.
• Captions were generated using Minimax-M3 as a teacher model, then quality-filtered using a 0–5 relevance/quality score before training.
• Each training example consists of: a gameplay screenshot, a system persona prompt ("passionate live game commentator"), and a target caption written in a natural, spoken commentary style.

<!-- - Dataset: [`TK17250/game-comment-synthetic-dataset`](https://huggingface.co/datasets/TK17250/game-comment-synthetic-dataset) -->

How to Use

from transformers import AutoProcessor, AutoModelForVision2Seq
from PIL import Image
import torch

processor = AutoProcessor.from_pretrained("TK17250/game-comment-vlm")
model = AutoModelForVision2Seq.from_pretrained(
    "TK17250/game-comment-vlm",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

image = Image.open("your_gameplay_screenshot.jpg").convert("RGB")

system_prompt = (
    "You are a passionate live game commentator. You watch a screenshot from a "
    "video game and give a short, natural spoken-style comment on the gameplay "
    "moment, reacting to how good or bad the play looks."
)
user_prompt = "Comment on this gameplay moment."

messages = [
    {"role": "user", "content": [
        {"type": "image"},
        {"type": "text", "text": f"{system_prompt}\n\n{user_prompt}"}
    ]}
]

input_text = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(image, input_text, add_special_tokens=False, return_tensors="pt").to(model.device)

output = model.generate(**inputs, max_new_tokens=128, temperature=0.7, min_p=0.1)
print(processor.decode(output[0], skip_special_tokens=True))

Note: This model's chat template does not support a separate system role alongside image inputs. Combine the persona/system instruction directly into the user message text, as shown above.

Using Unsloth (faster inference)

from unsloth import FastVisionModel
from transformers import TextStreamer

model, tokenizer = FastVisionModel.from_pretrained("TK17250/game-comment-vlm", load_in_4bit=False)
FastVisionModel.for_inference(model)

# ... build messages/inputs as above using `tokenizer` instead of `processor`
_ = model.generate(**inputs, streamer=TextStreamer(tokenizer, skip_prompt=True), max_new_tokens=128)

Intended Use

• Generating live-style commentary/captions for gameplay clips or screenshots (Fortnite, Rocket League, VALORANT).
• Powering auto-commentary features for streaming tools, highlight reels, or gameplay analysis apps.

Limitations

• Trained only on the three games listed above; output quality and terminology accuracy drop significantly for other games.
• As a fine-tune of an 11B vision-language model on a relatively small synthetic dataset (~1,600 examples), the model may occasionally produce generic or repetitive commentary, or misread fine-grained HUD/UI details (e.g. exact kill counts, health values).
• Captions are stylistic/entertainment-oriented and should not be relied on for precise, factual game-state reporting.