Qwen3-4B-Game-semantic-IDs

Qwen3-4B QLoRA -- Generative Recommender over Semantic IDs

QLoRA fine-tune of Qwen/Qwen3-4B to reason over semantic IDs -- short discrete codes produced by an RQ-VAE trained on game-catalog item embeddings -- instead of ever seeing raw item IDs or embeddings directly. Trained on the paired dataset pblrvo/steam-games-semanticIds-instructions.

Pipeline

Game catalog items -> embedded (Qwen3-0.6B) -> compressed into 4-level semantic IDs via RQ-VAE -> those ID tokens become new vocabulary the LLM is fine-tuned to reason over. Two-stage fine-tuning:

1. Embedding warmup: only embed_tokens/lm_head trained (codebook-grounded initialization + short high-LR run) so the new ID tokens carry meaningful structure before task-specific training begins.
2. QLoRA (this checkpoint): a real LoRA adapter (rank 8) across all attention/MLP projections, trained on top of stage 1's warmed-up embeddings, via Unsloth.

Both stages run through 4-bit quantization -- full-parameter fine-tuning of a ~4B model doesn't fit a single 12GB consumer GPU alongside gradients/optimizer state.

Tasks

• Sequential recommendation: predict the next item's semantic ID from a user's play history
• Grounding: map a semantic ID <-> item name/genres, both directions
• Similar item: given an item, suggest another one real users also engaged with
• ASY (asymmetric item prediction, LC-Rec, arXiv 2311.09049): same history/target pairs as sequential, rendered as the target's name instead of its semantic ID

Evaluation

Recall@K / NDCG@K via constrained beam search (candidates restricted to real catalog items through a trie over valid semantic IDs / descriptions), the same methodology TIGER (Rajput et al. 2023) and LC-Rec use for semantic-ID generative recommenders -- a single greedy decode is a top-1 prediction, not a ranking, so beam search stands in for the ranking step a classic recommender gets for free from a dot-product over all items.

(n=100 per task, temperature=0.8, beam=10; catalog size ~8,563 items, so random-chance Recall@10 is ~0.12% -- the recommendation-shaped tasks land well above chance, the two grounding directions remain a clear, unresolved weak point.)

Known limitations

• Grounding (exact ID<->name lookup) is weak. grounding_id2name produces plausible, valid catalog descriptions (~85-90% valid-format) but essentially never the specific correct one, even given a 10-candidate retry budget. Unlike the relational tasks (sequential/similar item), the ID<->name mapping has no exploitable structure to generalize from -- it's closer to an arbitrary lookup table than a learnable pattern, and a rank-8 LoRA adapter may simply not have enough capacity to memorize it precisely for ~8,500 items. Some of this may also be a property of the semantic ID space itself: items that collide on their coarser RQ-VAE codes are only disambiguated by a tiebreaker digit that carries no learnable signal connecting it to the item's name.
• Compute-constrained training. QLoRA rank 8, ~0.74 epochs of the fine-tuning stage (wall-clock capped on a single 12GB GPU, not run to convergence). Results should be read as "what's achievable under this specific hardware budget," not a ceiling on the approach.
• No classical-recommender baseline (e.g. SASRec) has been run against the same data -- these numbers show the model beats random chance substantially, not that the semantic-ID + LLM approach outperforms a much simpler sequential recommender.
• Single training run, single seed, throughout.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

ADAPTER = "pblrvo/Qwen3-4B-Game-semantic-IDs"
BASE_MODEL = "Qwen/Qwen3-4B"

tokenizer = AutoTokenizer.from_pretrained(ADAPTER)
quantization_config = BitsAndBytesConfig(
    load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True, llm_int8_skip_modules=["lm_head"],
)
base_model = AutoModelForCausalLM.from_pretrained(BASE_MODEL, dtype=torch.bfloat16, quantization_config=quantization_config)
base_model.resize_token_embeddings(len(tokenizer))
model = PeftModel.from_pretrained(base_model, ADAPTER)

The semantic-ID vocabulary (<|sid_start|>, <|sid_L{level}_{code}|>, <|sid_end|>) is only meaningful relative to the specific RQ-VAE codebook trained in the source project -- this model isn't usable standalone without that catalog/codebook context.