Downloads · 30 days
0
MohammadKhosravi/llama3.1-8b-cefr-prefix-tuning-6k-286M-Variant
llama3.1-8b-cefr-prefix-tuning-6k-286M-Variant is a machine learning model from MohammadKhosravi. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for pytorch. The card lists the license as apache-2.0.
This repository contains the ~286M CEFR Multi-Prefix Tuning model used in the master's thesis:
Downloads · 30 days
0
Access
Public
Updated Sep 13, 2026
Repo size
589 MB
Likes
0
Public
Click a slice to open those files.
.pt572 MB · 97%
From the Hugging Face model README
This repository contains the ~286M CEFR Multi-Prefix Tuning model used in the master's thesis:
Beyond Prompting: Resource-Efficient Explicit Control for CEFR-Aligned Language Generation
The model uses meta-llama/Llama-3.1-8B-Instruct as a frozen backbone and introduces six CEFR-specific prefix banks, corresponding to the six Common European Framework of Reference for Languages (CEFR) proficiency levels:
A1, A2, B1, B2, C1, and C2.
Each CEFR level is associated with 30 dedicated continuous virtual tokens. The requested target level selects the corresponding prefix bank, while a shared projection network maps those prefix representations into the layer-wise key/value states consumed by the frozen Llama backbone.
The resulting controller contains:
286,019,584 trainable parameters (~286.02M)
This experiment investigates the effect of controller capacity within the CEFR Multi-Prefix Tuning architecture.
The architecture retains the same explicit CEFR-conditioning mechanism used by the smaller ~68M variant while substantially increasing the dimensionality and capacity of the shared prefix controller.
The experiment therefore forms the intermediate point in the following capacity-scaling study:
CEFR Multi-Prefix — ~68M
│
▼
CEFR Multi-Prefix — ~286M
│
▼
CEFR Multi-Prefix — ~537M
The central research question is:
How does increasing controller capacity affect CEFR controllability when the underlying CEFR-specific prefix-selection mechanism remains unchanged?
Only the CEFR prefix controller is trained. The parameters of the underlying Llama-3.1-8B-Instruct backbone remain frozen.
The high-level architecture is:
Target CEFR
│
▼
┌──────────────────────────┐
│ CEFR-Specific Prefix Bank│
│ A1 A2 B1 B2 C1 C2 │
└────────────┬─────────────┘
│
▼
30 Virtual Tokens
│
▼
┌──────────────────────────┐
│ Shared Projection MLP │
│ 4096 → 4096 → 65536 │
│ Tanh │
└────────────┬─────────────┘
│
▼
Per-Layer Prefix
Key/Value States
│
▼
┌──────────────────────────┐
│ Frozen Llama-3.1-8B │
│ 32 Layers │
└──────────────────────────┘
The model contains six independent CEFR-specific prefix embedding sets:
A1
A2
B1
B2
C1
C2
Each level contains 30 virtual tokens:
6 CEFR levels × 30 tokens
=
180 CEFR-specific virtual-token embeddings
Each virtual-token representation has dimension 4096, matching the hidden dimension of Llama-3.1-8B-Instruct.
The shared prefix projection network is:
4096 → 4096 → 65536
with a Tanh activation between the two linear transformations.
More explicitly:
CEFR-Specific Embedding
R^4096
│
▼
Linear(4096, 4096)
│
▼
Tanh
│
▼
Linear(4096, 65536)
│
▼
Per-Layer Key/Value
Prefix Representation
│
▼
Frozen Llama-3.1-8B
| Component | Value |
|---|---|
| Frozen backbone | meta-llama/Llama-3.1-8B-Instruct |
| CEFR levels | 6 |
| Virtual tokens per level | 30 |
| Total CEFR prefix embeddings | 180 |
| Prefix embedding dimension | 4096 |
| Projection architecture | 4096 → 4096 → 65536 |
| Trainable controller parameters | 286,019,584 |
| Trainable controller size | ~286.02M |
The trainable parameter count is reported directly by the executed training notebook.
The Llama backbone remains completely frozen throughout training.
The specific target CEFR label is not inserted directly into the textual prompt.
Instead, the requested proficiency level is supplied to the controller as a CEFR class ID:
Target CEFR
│
▼
cefr_id
│
▼
Select One of Six
CEFR Prefix Banks
│
▼
30 Level-Specific
Virtual Tokens
│
▼
Shared Projection
│
▼
Per-Layer Prefix
Key/Value States
│
▼
Frozen Llama-3.1-8B
Thus, the specific target class such as A1, B2, or C2 is communicated through architectural prefix selection rather than direct textual label insertion.
The textual instruction still contains general proficiency-oriented guidance, but the requested CEFR class itself is supplied through the prefix controller.
Training uses the Balanced CEFR Steering Subset derived from the EFCAMDAT Steering Training Dataset.
The dataset contains exactly 5,568 examples, balanced equally across all six CEFR classes:
| CEFR Level | Samples |
|---|---|
| A1 | 928 |
| A2 | 928 |
| B1 | 928 |
| B2 | 928 |
| C1 | 928 |
| C2 | 928 |
| Total | 5,568 |
A stratified 90/10 train-validation split is applied using random_state=42.
| Partition | Samples |
|---|---|
| Training | 5,011 |
| Validation | 557 |
| Total | 5,568 |
Each training sample contains:
The specific target CEFR class is not written directly into the textual prompt.
The formatted prompt is concatenated with the reference response for autoregressive training.
Prompt tokens and padding positions are masked using the standard -100 ignore index, so the causal language-modeling loss is computed only over target response tokens.
The training objective is:
-\sum_{t \in \mathcal{T}{\mathrm{target}}} \log p{\theta} \left( y_t \mid y_{<t},x,c \right) $$
where:
Only the CEFR Multi-Prefix controller is optimized.
| Setting | Value |
|---|---|
| Backbone | meta-llama/Llama-3.1-8B-Instruct |
| Backbone state | Frozen |
| Epochs | 3 |
| Optimizer | AdamW |
| Learning rate | 2e-4 |
| Weight decay | 0.01 |
| Learning-rate schedule | Cosine |
| Warmup ratio | 5% |
| Training batch size | 16 |
| Gradient accumulation | 2 |
| Effective training batch size | 32 |
| Validation batch size | 32 |
| Maximum sequence length | 512 |
| Precision | BF16 |
| Random seed | 42 |
| Checkpoint criterion | Minimum validation loss |
The recorded training trajectory is:
| Epoch | Train Loss | Validation Loss | Validation Perplexity |
|---|---|---|---|
| 1 | 2.6608 | 2.5036 | 12.23 |
| 2 | 2.3967 | 2.4328 | 11.39 |
| 3 | 2.2449 | 2.4366 | 11.43 |
The best validation checkpoint is obtained at Epoch 2.
Validation loss follows:
2.5036 → 2.4328 → 2.4366
while validation perplexity follows:
12.23 → 11.39 → 11.43
Although training loss continues to decrease during Epoch 3, validation loss increases slightly from 2.4328 to 2.4366.
Therefore, the Epoch 2 checkpoint is retained.
The complete three-epoch training run produced the following hardware measurements:
| Metric | Value |
|---|---|
| Total training time | 1497.77 s |
| Total training time | 0.42 h |
| Peak GPU memory allocation | 69.43 GB |
| Average GPU utilization | 97.4% |
| Trainable controller parameters | 286,019,584 |
These statistics correspond to the complete three-epoch CEFR Multi-Prefix training run.
The trained controller is evaluated on the final In-Domain Evaluation Prompt Matrix used throughout the thesis.
The benchmark contains:
117 topics × 6 CEFR levels = 702 generation conditions
The evaluation pipeline is:
Input Topic
│
▼
CEFR-Blind Textual Prompt
│
├────────────────────────────┐
│ │
▼ ▼
Text Tokens Target CEFR
│
▼
cefr_id
│
▼
CEFR-Specific Prefix
│
┌────────────────────────────┘
▼
Frozen Llama-3.1-8B
│
▼
Generated Response
│
▼
CEFR Evaluator
For every condition:
| Setting | Value |
|---|---|
| Evaluation batch size | 32 |
| Maximum new tokens | 200 |
| Temperature | 0.6 |
| Sampling | Enabled |
| Backbone | meta-llama/Llama-3.1-8B-Instruct |
Generated texts are evaluated using the primary thesis CEFR evaluator:
MohammadKhosravi/roberta-large-cefr-classifier-JointLoss
The evaluator is independent of the CEFR prefix controller and is not involved in training the generation model.
The ~286M CEFR Multi-Prefix model obtains the following results across all 702 in-domain generation conditions:
| Metric | Result |
|---|---|
| Total generations | 702 |
| Strict Accuracy | 52.42% |
| Adjacent Accuracy | 76.35% |
| Mean Absolute Error (MAE) | 0.8533 |
| Mean Dependency Distance (MDD) | 1.82 |
| Mean Flesch Reading Ease | 79.49 |
| Mean Sentence-Level CEFR Drift | 2.34 levels |
Strict Accuracy requires the CEFR evaluator's prediction to exactly match the requested target level.
For example:
Target B1:
B1 → Correct
A2 / B2 → Incorrect
A1/C1/C2 → Incorrect
Adjacent Accuracy additionally accepts predictions one CEFR level away from the requested target.
For example:
Target B1:
B1 → Strict + Adjacent correct
A2 / B2 → Adjacent correct
A1/C1/C2 → Incorrect
CEFR levels are mapped to ordered class indices:
A1 = 0
A2 = 1
B1 = 2
B2 = 3
C1 = 4
C2 = 5
MAE measures the average ordinal distance between the requested and predicted CEFR classes:
\frac{1}{N} \sum_{i=1}^{N} \left| \hat{y}_i-y_i \right| $$
Lower values indicate smaller proficiency-control errors.
The evaluation additionally records:
These diagnostics complement the primary CEFR-control metrics.
This model represents the intermediate-capacity configuration of the CEFR Multi-Prefix family.
The in-domain results across the three configurations are:
| Variant | Trainable Parameters | Strict Accuracy |
|---|---|---|
| CEFR Multi-Prefix — ~68M | 68.41M | 45.44% |
| CEFR Multi-Prefix — ~286M | 286.02M | 52.42% |
| CEFR Multi-Prefix — ~537M | ~537M | 64.10% |
The progression is therefore:
~68M
45.44%
│
▼
~286M
52.42%
│
▼
~537M
64.10%
From the ~68M model to the ~286M model:
45.44% → 52.42%
corresponding to an absolute improvement of:
+6.98 percentage points
and a relative improvement of approximately:
+15.4%
From the ~286M model to the ~537M model:
52.42% → 64.10%
corresponding to an absolute improvement of:
+11.68 percentage points
The observed progression provides evidence that controller capacity is an important factor in CEFR controllability within the Multi-Prefix architecture.
For additional experimental context:
| Method | Trainable Parameters | Strict Accuracy |
|---|---|---|
| LoRA | 41.94M | 26.21% |
| Standard Prefix-Tuning | 68.25M | 32.34% |
| Vanilla PMT | 536.87M | 15.67% |
| CEFR Multi-Prefix — 68M | 68.41M | 45.44% |
| CEFR Multi-Prefix — 286M | 286.02M | 52.42% |
| CEFR Multi-Prefix — 537M | ~537M | 64.10% |
These comparisons should be interpreted within the shared experimental setup used in the thesis.
In particular, the comparison between the ~68M and ~286M CEFR Multi-Prefix variants isolates the effect of increasing controller capacity while retaining the same core CEFR-specific prefix-selection mechanism.
This experiment addresses the relationship between architectural conditioning and controller capacity.
The experimental progression can be summarized as:
CEFR-Controlled Generation
│
▼
Explicit Multi-Prefix Control
│
┌─────────────┼─────────────┐
│ │ │
▼ ▼ ▼
~68M ~286M ~537M
45.44% 52.42% 64.10%
│ │ │
└─────────────┴─────────────┘
│
▼
Controller-Capacity Analysis
The architecture maintains the same core CEFR-conditioning mechanism while varying controller capacity.
This allows the thesis to distinguish between performance gains attributable to:
The ~286M configuration provides the intermediate experimental point required to characterize this scaling behavior.
The complete training and evaluation notebook is available in the thesis GitHub repository at:
notebooks/05_cefr_prefix_tuning/cefr_prefix_tuning_286m_train_and_eval.ipynb
The balanced training subset is constructed in:
notebooks/01_data_preparation/balanced_cefr_steering_subset.ipynb
The in-domain evaluation matrix is constructed in:
notebooks/01_data_preparation/efcamdat_preprocessing_and_partitioning.ipynb
The notebook retains the original experimental outputs, including:
This model is intended for research on:
Important: This model is an experimental research artifact and is not intended to serve as a standalone CEFR assessment model.
If you use this model or the accompanying experimental resources, please cite the associated master's thesis and repository.
@mastersthesis{khosravi2026beyond,
author = {Mohammad Khosravi},
title = {Beyond Prompting: Resource-Efficient Explicit Control for CEFR-Aligned Language Generation},
school = {University of Padova},
year = {2026}
}