Downloads · 30 days
0
MohammadKhosravi/llama3.1-8b-cefr-steering-1layer-head-ordinal-universal
llama3.1-8b-cefr-steering-1layer-head-ordinal-universal is a text classification model from MohammadKhosravi. Use it when you need a label for a piece of text. It is set up for pytorch.
This repository contains the CEFR latent attribute classifier used by the PPLM baseline in the master's thesis:
Downloads · 30 days
0
Access
Public
Updated Sep 13, 2026
Repo size
100 KB
Likes
0
Public
Click a slice to open those files.
.pt100 KB · 90%
From the Hugging Face model README
This repository contains the CEFR latent attribute classifier used by the PPLM baseline in the master's thesis:
Beyond Prompting: Resource-Efficient Explicit Control for CEFR-Aligned Language Generation
The classifier operates on hidden-state representations extracted from meta-llama/Llama-3.1-8B-Instruct and predicts one of the six Common European Framework of Reference for Languages (CEFR) proficiency levels:
A1, A2, B1, B2, C1, and C2.
Its role is not standalone text generation. Instead, it provides the differentiable CEFR objective used by the Plug and Play Language Model (PPLM) baseline during inference-time steering.
The PPLM pipeline uses this classifier as an attribute model over internal Llama-3.1-8B representations.
Llama-3.1-8B hidden representation
↓
Dropout (p = 0.35)
↓
Linear(4096 → 6)
↓
CEFR class logits
A1 A2 B1 B2 C1 C2
During PPLM generation, the CEFR classification objective provides a differentiable signal that can be propagated with respect to the model's latent representation to steer generation toward a requested CEFR level.
The classifier is intentionally lightweight:
The model can be summarized as:
Dropout(0.35) → Linear(4096, 6)
No intermediate hidden layer, nonlinear activation, or LayerNorm module is used.
The shallow architecture was selected to keep the attribute model lightweight and fully differentiable during PPLM decoding.
The classifier is trained on the latent CEFR dataset:
MohammadKhosravi/cefr-llama3.1-8b-hidden-states-efcamdat-universal
The dataset contains 66,494 latent representations paired with CEFR labels.
Each sample consists of:
The latent dataset includes representations constructed from the EFCAMDAT Steering Training Dataset together with the pre-existing CEFR latent dataset used in the PPLM experiment.
The construction pipeline is available in the thesis GitHub repository:
notebooks/01_data_preparation/pplm_latent_dataset_construction.ipynb
The full latent dataset is divided using a stratified split:
The resulting training partition contains 56,519 latent vectors, with the remaining samples used for validation.
Stratification preserves the CEFR class distribution across the two partitions.
The CEFR latent dataset is strongly imbalanced, particularly for the advanced C1 and C2 levels.
Inverse-frequency class weights are computed from the training partition using sklearn.utils.class_weight.compute_class_weight.
The resulting weights used during training were:
| CEFR | Class Weight |
|---|---|
| A1 | 0.7386 |
| A2 | 0.7372 |
| B1 | 0.7341 |
| B2 | 0.7367 |
| C1 | 2.2107 |
| C2 | 8.5017 |
The substantially larger C1 and C2 weights compensate for the much lower representation of advanced-level samples in the latent training data.
Training uses a custom Ordinal Weighted Loss combining weighted cross-entropy with an ordinal distance penalty.
The total objective is:
\mathcal{L}{\mathrm{WCE}} + \lambda{\mathrm{ord}}\mathcal{L}_{\mathrm{ordinal}} $$
where:
This encourages the classifier to account for the ordered structure of the CEFR scale in addition to exact class prediction.
| Setting | Value |
|---|---|
| Optimizer | AdamW |
| Initial learning rate | 3e-4 |
| Weight decay | 0.01 |
| Batch size | 128 |
| Maximum epochs | 50 |
| Input dropout | 0.35 |
| Input dimension | 4096 |
| Number of CEFR classes | 6 |
| Train / validation split | 85% / 15% |
| Split seed | 42 |
| Checkpoint criterion | Validation Strict Accuracy |
The learning rate is reduced when validation Strict Accuracy plateaus.
The training run produced the following recorded checkpoints:
| Epoch | Learning Rate | Training Loss | Validation Strict Accuracy | Validation Adjacent Accuracy |
|---|---|---|---|---|
| 1 | 0.000300 | 1.3687 | 77.19% | 94.93% |
| 5 | 0.000300 | 0.7541 | 82.68% | 96.25% |
| 10 | 0.000300 | 0.6760 | 83.79% | 96.87% |
| 15 | 0.000150 | 0.6453 | 84.37% | 97.03% |
| 20 | 0.000150 | 0.6360 | 84.21% | 96.83% |
| 25 | 0.000150 | 0.6311 | 85.08% | 97.24% |
| 30 | 0.000075 | 0.6125 | 85.05% | 97.36% |
| 35 | 0.000019 | 0.6150 | 84.94% | 97.18% |
| 40 | 0.000009 | 0.6064 | 85.01% | 97.18% |
| 45 | 0.000002 | 0.6143 | 85.03% | 97.21% |
| 50 | 0.000001 | 0.6149 | 85.00% | 97.21% |
The training run used 56,519 training vectors.
The trajectory shows rapid improvement during the first 10–15 epochs, followed by a plateau around 84–85% Strict Accuracy, while Adjacent Accuracy remains around 97%.
The model checkpoint is selected according to the highest observed validation Strict Accuracy.
The best result obtained during training was:
| Metric | Result |
|---|---|
| Best Validation Strict Accuracy | 85.16% |
| Corresponding Validation Adjacent Accuracy | 97.32% |
The corresponding Adjacent Accuracy is the value recorded at the checkpoint selected by the highest Strict Accuracy; it is not independently selected as the maximum Adjacent Accuracy across epochs.
The training procedure saves a new checkpoint whenever validation Strict Accuracy exceeds the previous best value.
Strict Accuracy requires the predicted CEFR level to exactly match the target CEFR class.
Adjacent Accuracy counts a prediction as correct when it is either the exact CEFR level or one immediately neighboring level on the ordinal CEFR scale.
For example:
Target B1 → B1 = Strict + Adjacent correct
Target B1 → A2 / B2 = Adjacent correct
Target B1 → A1 / C1 / C2 = Incorrect
The adjacent metric is useful because CEFR is ordinal and classification errors between neighboring levels are less severe than errors spanning several proficiency levels.
This classifier is used exclusively as the attribute classifier for the PPLM baseline.
The PPLM experiments use gradients derived from the classifier's CEFR objective during inference-time latent perturbation while the parameters of Llama-3.1-8B-Instruct remain frozen.
The classifier itself is trained separately from the generation experiments.
Evaluation of the resulting PPLM-controlled generations is performed in a separate notebook in the thesis repository.
The classifier-training notebook is available at:
notebooks/03_pplm/pplm_latent_classifier_train.ipynb
The latent-dataset construction notebook is available at:
notebooks/01_data_preparation/pplm_latent_dataset_construction.ipynb
The training notebook retains the original experimental outputs, including:
This model is intended for research on:
Important: This model is not intended to serve as a standalone general-purpose CEFR assessment model for arbitrary learner texts.
It was trained specifically on Llama-3.1-8B latent representations for use within the PPLM experimental pipeline.
If you use this model or the accompanying experimental resources, please cite the associated master's thesis and repository.
@mastersthesis{khosravi_cefr_control,
title = {Beyond Prompting: Resource-Efficient Explicit Control for CEFR-Aligned Language Generation},
author = {Mohammad Khosravi},
school = {University of Padova},
year = {2026}
}