Downloads · 30 days
14
25% of all-time downloads
MohammadKhosravi/llama3.1-8b-standard-prefix-tuning-6k
llama3.1-8b-standard-prefix-tuning-6k is a text generation model from MohammadKhosravi. Use it when you need the model to write or continue text. It is set up for peft. The card lists the license as apache-2.0.
This repository contains the Standard Prefix-Tuning baseline used in the master's thesis:
Downloads · 30 days
14
25% of all-time downloads
All-time downloads
56
Public
Repo size
25.1 MB
Likes
0
Public
Click a slice to open those files.
.json17.2 MB · 69%
From the Hugging Face model README
This repository contains the Standard Prefix-Tuning baseline used in the master's thesis:
Beyond Prompting: Resource-Efficient Explicit Control for CEFR-Aligned Language Generation
The model applies the Hugging Face PEFT implementation of Prefix-Tuning to meta-llama/Llama-3.1-8B-Instruct.
The Llama-3.1-8B-Instruct backbone remains frozen while a continuous learnable prefix is optimized for CEFR-conditioned text generation across the six proficiency levels:
A1, A2, B1, B2, C1, and C2.
This model serves as a generic PEFT baseline for comparison with the CEFR-specific control architectures investigated in the thesis.
Standard Prefix-Tuning learns continuous virtual prefix representations while keeping the underlying language-model parameters frozen.
The prefix representations are injected into the transformer's attention key/value states through the PEFT Prefix-Tuning implementation.
The configuration used in this experiment is:
Frozen Llama-3.1-8B-Instruct
+
30 Virtual Prefix Tokens
+
Prefix Projection
| Setting | Value |
|---|---|
| PEFT method | Prefix-Tuning |
| Backbone | meta-llama/Llama-3.1-8B-Instruct |
| Number of virtual tokens | 30 |
| Prefix projection | True |
| Task type | Causal Language Modeling |
| Trainable parameters | 68,254,720 |
| Total parameters | 8,098,515,968 |
| Trainable fraction | 0.8428% |
Only the Prefix-Tuning parameters are optimized during training. The Llama-3.1-8B-Instruct backbone remains frozen.
Training uses the Balanced CEFR Steering Subset derived from the EFCAMDAT Steering Training Dataset.
The dataset contains exactly 5,568 examples, balanced equally across the six CEFR levels:
| CEFR Level | Samples |
|---|---|
| A1 | 928 |
| A2 | 928 |
| B1 | 928 |
| B2 | 928 |
| C1 | 928 |
| C2 | 928 |
| Total | 5,568 |
A stratified 90/10 train-validation split is used with random_state=42:
| Partition | Samples |
|---|---|
| Training | 5,011 |
| Validation | 557 |
| Total | 5,568 |
Each training example contains:
The textual instruction explicitly specifies the requested CEFR level.
The training sequence consists of the formatted instruction followed by the reference response.
Prompt and padding tokens are excluded from the causal language-modeling loss using the standard -100 ignore index, so optimization is performed only on the target response tokens.
Training uses the standard autoregressive causal language-modeling cross-entropy objective.
The backbone parameters remain frozen and gradients update only the learnable Prefix-Tuning parameters.
Conceptually, the target-only objective is:
-\sum_{t \in \mathcal{T}{\text{target}}} \log p{\theta}(y_t \mid y_{<t},x) $$
where $\mathcal{T}_{\text{target}}$ contains only the response-token positions included in the loss.
| Setting | Value |
|---|---|
| Backbone | meta-llama/Llama-3.1-8B-Instruct |
| Epochs | 3 |
| Optimizer | AdamW |
| Learning rate | 2e-4 |
| Weight decay | 0.01 |
| Learning-rate schedule | Cosine |
| Warmup ratio | 5% |
| Training batch size | 16 |
| Gradient accumulation steps | 2 |
| Effective training batch size | 32 |
| Validation batch size | 32 |
| Maximum sequence length | 512 |
| Precision | bfloat16 |
| Gradient clipping | 1.0 |
| Checkpoint criterion | Minimum validation loss |
The recorded training trajectory is:
| Epoch | Train Loss | Validation Loss | Validation Perplexity |
|---|---|---|---|
| 1 | 2.6629 | 2.5390 | 12.67 |
| 2 | 2.4541 | 2.4701 | 11.82 |
| 3 | 2.3293 | 2.4610 | 11.72 |
Validation performance improves across all three epochs.
Validation loss decreases from:
2.5390 → 2.4701 → 2.4610
while validation perplexity decreases from:
12.67 → 11.82 → 11.72
The lowest validation loss and perplexity are obtained at Epoch 3. The Epoch 3 adapter is therefore retained as the final checkpoint.
The trained Prefix-Tuning adapter is evaluated on the final In-Domain Evaluation Prompt Matrix used throughout the thesis.
The benchmark contains:
117 topics × 6 CEFR levels = 702 generation conditions
For evaluation, the trained Prefix-Tuning adapter is attached to the frozen Llama-3.1-8B-Instruct backbone using PEFT.
| Setting | Value |
|---|---|
| Evaluation batch size | 32 |
| Maximum new tokens | 200 |
| Temperature | 0.6 |
| Sampling | Enabled |
| Backbone | meta-llama/Llama-3.1-8B-Instruct |
Generated texts are evaluated using the primary CEFR evaluator:
MohammadKhosravi/roberta-large-cefr-classifier-JointLoss
The Standard Prefix-Tuning baseline obtains the following results over the complete 702-condition in-domain evaluation matrix:
| Metric | Result |
|---|---|
| Total generations | 702 |
| Strict Accuracy | 32.34% |
| Adjacent Accuracy | 64.67% |
| Mean Absolute Error (MAE) | 1.2422 |
| Mean Dependency Distance (MDD) | 1.80 |
| Mean Flesch Reading Ease | 81.73 |
| Mean Sentence-Level CEFR Drift | 2.17 levels |
A generation is counted as correct only when the predicted CEFR level exactly matches the requested target level.
Adjacent Accuracy additionally treats predictions one CEFR level away from the requested target as correct.
For example:
Target B1:
B1 → Strict + Adjacent correct
A2 / B2 → Adjacent correct
A1 / C1 / C2 → Incorrect
CEFR levels are mapped to ordered class indices:
A1 = 0
A2 = 1
B1 = 2
B2 = 3
C1 = 4
C2 = 5
MAE measures the average ordinal distance between the requested and predicted CEFR levels:
\frac{1}{N} \sum_{i=1}^{N} \left| \hat{y}_i-y_i \right| $$
Lower values indicate smaller CEFR-control errors.
The evaluation additionally reports:
These diagnostics complement the primary CEFR-control metrics.
Standard Prefix-Tuning is included as a generic parameter-efficient baseline against which the explicitly CEFR-conditioned methods are compared.
Unlike the proposed CEFR-specific controllers, this baseline does not contain a dedicated architectural CEFR-conditioning mechanism.
The target proficiency level is supplied through the textual instruction, while the same shared learnable prefix mechanism is used across CEFR levels.
This provides a comparison between:
Generic Prefix-Tuning
vs.
Explicit CEFR-Conditioned Prefix Control
The experiment is therefore useful for separating improvements due to explicit CEFR conditioning from improvements that can be obtained through standard parameter-efficient adaptation alone.
This repository implements the Standard Prefix-Tuning baseline using the PEFT Prefix-Tuning mechanism.
It should be distinguished from the PrefixMemory-Tuning and CEFR-conditioned controller variants evaluated elsewhere in the thesis.
In this baseline:
Conceptually:
Standard Prefix-Tuning
│
┌─────────────┴─────────────┐
│ │
CEFR Information Prefix Parameters
│ │
Textual Instruction Shared Prefix
│ │
└─────────────┬─────────────┘
│
▼
Frozen Llama-3.1-8B
This differs from the explicitly conditioned architectures, where CEFR information directly controls dedicated trainable architectural components.
The complete training and evaluation notebook is available in the thesis GitHub repository at:
notebooks/04_generic_peft_baselines/standard_prefix_tuning_train_and_eval.ipynb
The balanced training subset is constructed in:
notebooks/01_data_preparation/balanced_cefr_steering_subset.ipynb
The in-domain evaluation benchmark is constructed in:
notebooks/01_data_preparation/efcamdat_preprocessing_and_partitioning.ipynb
The experimental notebook retains the original outputs, including:
This adapter is intended for research on:
Important: This repository contains an experimental research artifact and is not intended to serve as a standalone CEFR assessment model.
If you use this model or the accompanying experimental resources, please cite the associated master's thesis and repository.
@mastersthesis{khosravi2026beyond,
author = {Mohammad Khosravi},
title = {Beyond Prompting: Resource-Efficient Explicit Control for CEFR-Aligned Language Generation},
school = {University of Padova},
year = {2026}
}