Downloads · 30 days
0
ypolatog/24679-hw2-movie-genre-prompting
24679-hw2-movie-genre-prompting is a text classification model from ypolatog. Use it when you need a label for a piece of text. The card lists the license as other.
Zero-, one- and five-shot prompting of Qwen/Qwen2.5-3B-Instruct on ArinRoths/movie-data, compared against the DistilBERT model fine-tuned in Part 3 on the same 15 test descriptions.
Downloads · 30 days
0
Access
Public
Updated Sep 21, 2026
Repo size
—
Likes
0
Public
Click a slice to open those files.
.md3.1 KB · 41%
From the Hugging Face model README
Zero-, one- and five-shot prompting of Qwen/Qwen2.5-3B-Instruct on
ArinRoths/movie-data, compared against the DistilBERT
model fine-tuned in Part 3 on the same 15 test descriptions.
No weights are published here — prompting trains nothing. This card records the method.
| Method | K | Selection | Accuracy | Macro F1 | Parse failures | Prompt tokens | s/example |
|---|---|---|---|---|---|---|---|
| DistilBERT (Part 3) | - | - | 0.667 | 0.659 | 0 | 45 | 0.05 |
| 0-shot | 0 | - | 0.667 | 0.591 | 1 | 145 | 0.08 |
| 1-shot adaptive | 1 | adaptive | 0.600 | 0.600 | 1 | 193 | 0.09 |
| 5-shot adaptive | 5 | adaptive | 0.533 | 0.543 | 1 | 383 | 0.15 |
| 5-shot random | 5 | random | 0.600 | 0.590 | 1 | 388 | 0.15 |
Chance is 0.200. Test size is 15 descriptions, so one error moves accuracy by 6.7 points — these differences are indicative, not conclusive.
Model: Qwen/Qwen2.5-3B-Instruct, fp16, greedy decoding (do_sample=False,
max_new_tokens=8), left-padded for batched generation. Cost: $0.00 — the model
runs locally on a Colab GPU, so cost is reported as
7 GPU-seconds for all test pipelines combined.
Prompt. A system message instructing a single-word answer with no explanation, followed by an instruction naming the allowed genres and forbidding punctuation or reasoning. Worked examples, when present, precede the query in ascending order of similarity so the most relevant sits nearest the question.
Exemplar selection. adaptive embeds the 70-description training pool with
sentence-transformers/all-MiniLM-L6-v2 and takes the k nearest by cosine similarity; random samples k
uniformly from the same pool. Only original (non-augmented) training descriptions are eligible, and
no validation or test description is ever used as an exemplar.
Output parsing. The first genre named anywhere in the output is taken. If none is named, each genre is scored by its average token log-probability and the best is chosen, so every description receives a prediction. Parse failures are reported per pipeline above.
K was chosen on the validation split; the test split was scored once per pipeline.
Coursework for CMU 24-679 Design AI; not for any real decision. The test split is 15 descriptions from one student's survey — far too few to rank these methods conclusively. The LLM may well have seen these films during pretraining, which flatters the zero-shot row in a way a genuinely novel description would not.
The source dataset declares no license. Reuse should be cleared with its author, ArinRoths.
Seed 24679, transformers 5.17.0. Splits taken from ypolatog/24679-hw2-movie-genre-distilbert/split_manifest.json.
results.json holds every number above including the full K sweep.