Downloads · 30 days
14
3% of all-time downloads
andresnowak/Qwen3-0.6B-instruction-finetuned-MCQA
Qwen3-0.6B-instruction-finetuned-MCQA is a text generation model from andresnowak. Use it when you need the model to write or continue text. It is set up for transformers.
should probably proofread and complete it, then remove this comment. --
Downloads · 30 days
14
3% of all-time downloads
All-time downloads
415
Public
Parameters
596M
11.9 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors1.2 GB · 99%
From the Hugging Face model README
This model is a fine-tuned version of andresnowak/Qwen3-0.6B-instruction-finetuned on an unknown dataset.
More information needed
More information needed
More information needed
This model was trained with the same methodology as https://huggingface.co/andresnowak/MNLP_M2_mcqa_model, where we only do a feedforward on the prompt we get the last logit token and we do cross entropy loss on that token and the 4 options of the question (so the idea is that we want to maximize the likelihood of the model of printing the correct letter to the question)
The following hyperparameters were used during training:
The model was evaluated on a suite of Multiple Choice Question Answering (MCQA) benchmarks (on its validation and test sets repsectively for each one), and NLP4education is only the approximated 1000 question and answers given to use.
The performance on the MCQA benchmarks is:
| Benchmark | Accuracy (Acc) | Normalized Accuracy (Acc Norm) |
|---|---|---|
| ARC Challenge | 61.39% | 59.96% |
| ARC Easy | 79.43% | 76.51% |
| GPQA | 32.59% | 28.57% |
| Math QA | 24.69% | 24.80% |
| MCQA Evals | 41.82% | 39.22% |
| MMLU | 52.11% | 52.11% |
| MMLU Pro | 15.41% | 14.31% |
| MuSR | 51.06% | 48.41% |
| NLP4Education | 44.14% | 42.73% |
| Overall | 44.74% | 42.96% |
The tests where done with this prompt (And only MusR used a different one where you add the Question: and Narrative: )
This question assesses challenging STEM problems as found on graduate standardized tests. Carefully evaluate the options and select the correct answer.
---
[Insert Question Here]
---
[Insert Choices Here, e.g.:
A. Option 1
B. Option 2
C. Option 3
D. Option 4]
---
Your response should include the letter and the exact text of the correct choice.
Example: B. Entropy increases.
Answer:
And the teseting was done on [Letter]. [Text answer]