plutchik-emotions-polish-poc

plutchik-emotions-polish-poc

This model is a fine-tuned version of sdadas/polish-gpt2-small on a synthetically annotated WiktorS/polish-news dataset. It achieves the following results on the evaluation set: Every list contains results for threshold [0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]

• Loss: 0.4958
• Hamming Accuracy: [0.8251, 0.8363, 0.8424, 0.8461, 0.8471, 0.848, 0.8464]
• F1 Macro: [0.5061, 0.4948, 0.4802, 0.4641, 0.4376, 0.4023, 0.3423]
• Precision Macro: [0.4873, 0.5201, 0.5486, 0.5752, 0.5986, 0.6279, 0.6586]
• Recall Macro: [0.5270, 0.4738, 0.4304, 0.3929, 0.3494, 0.3008, 0.2362]

Model description

Trained from sdadas/polish-gpt2-small as a Proof of Concept.

Intended uses & limitations

Detecting emotions described in Robert_Plutchik#Theory_of_emotion

Labels:

0: joy
1: trust
2: anticipation
3: surprise
4: fear
5: sadness
6: disgust
7: anger

How to use

<details><summary>Transformers AutoModel</summary>

from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"
checkpoint = "nie3e/plutchik-emotions-polish-poc"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForSequenceClassification.from_pretrained(
    checkpoint, problem_type="multi_label_classification"
).to(device)

text = "To jest model wykrywający super emocje w tekście! :D"

input_ids = tokenizer(text, return_tensors="pt").to(device)
logits = model(**input_ids)["logits"].to("cpu")

threshold = 0.3
predicted_class_ids = torch.arange(
    0, logits.shape[-1]
)[torch.sigmoid(logits).squeeze(dim=0) > threshold]

percent = torch.sigmoid(logits).squeeze(dim=0)

id2class = model.config.id2label
print([id2class[c] for c in predicted_class_ids.tolist()])
print({id2class[i]: f"{(p*100):.2f}%" for i, p in enumerate(percent.tolist())})

['joy', 'anticipation']
{'joy': '91.15%', 'trust': '2.06%', 'anticipation': '61.14%', 'surprise': '23.00%', 'fear': '1.93%', 'sadness': '0.40%', 'disgust': '3.02%', 'anger': '2.54%'}

</details> <details><summary>Transformers Pipeline</summary>

from transformers import pipeline
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"

pipe = pipeline(
    "text-classification",
    "nie3e/plutchik-emotions-polish-poc",
    top_k=-1,
    device=device
)
text = "To jest model wykrywający super emocje w tekście! :D"

result = pipe(text)
print(result)

[[{'label': 'joy', 'score': 0.9111796617507935}, {'label': 'anticipation', 'score': 0.6109178066253662}, {'label': 'surprise', 'score': 0.23231014609336853}, {'label': 'disgust', 'score': 0.030675798654556274}, {'label': 'anger', 'score': 0.025565214455127716}, {'label': 'trust', 'score': 0.02096424251794815}, {'label': 'fear', 'score': 0.019419347867369652}]]

</details> <details><summary>vLLM OpenAI serving (recommended)</summary>

docker run --gpus 1 --ipc=host -p 8000:8000 vllm/vllm-openai:v0.9.2 --model nie3e/plutchik-emotions-polish-poc

using curl:

curl -X 'POST' \
  'http://127.0.0.1:8000/classify' \
  -H 'Content-Type: application/json' \
  -d '{
  "model": "nie3e/plutchik-emotions-polish-poc",
  "input": ["To jest model wykrywający super emocje w tekście! :D"]
}'

result:

{
	"id": "classify-2cea66aa49b84b239277ad8cdb8ad662",
	"object": "list",
	"created": 1756143116,
	"model": "nie3e/plutchik-emotions-polish-poc",
	"data": [
		{
			"index": 0,
			"label": "joy",
			"probs": [
				0.8406059741973877,
				0.0017006286652758718,
				0.12716063857078553,
				0.024033185094594957,
				0.0015728245489299298,
				0.00032968190498650074,
				0.0025133665185421707,
				0.002083654049783945
			],
			"num_classes": 8
		}
	],
	"usage": {
		"prompt_tokens": 12,
		"total_tokens": 12,
		"completion_tokens": 0,
		"prompt_tokens_details": null
	}
}

using python:

import requests

response = requests.post(
    f"http://127.0.0.1:8000/classify",
    headers={"Content-Type": "application/json"},
    json={
        "model": "nie3e/plutchik-emotions-polish-poc",
        "input": ["To jest model wykrywający super emocje w tekście! :D"]
    }
)

print(response.json())

{'id': 'classify-003f1db547504a5398ee76907f6f5e30',
 'object': 'list',
 'created': 1756143205,
 'model': 'nie3e/plutchik-emotions-polish-poc',
 'data': [{'index': 0,
   'label': 'joy',
   'probs': [0.8406059741973877,
    0.0017006286652758718,
    0.12716063857078552,
    0.024033185094594955,
    0.0015728245489299297,
    0.00032968190498650074,
    0.0025133665185421705,
    0.002083654049783945],
   'num_classes': 8}],
 'usage': {'prompt_tokens': 12,
  'total_tokens': 12,
  'completion_tokens': 0,
  'prompt_tokens_details': None}}

</details>

Training and evaluation data

Dataset: WiktorS/polish-news

LLM used for annotation: bartowski/mistralai_Mistral-Small-3.2-24B-Instruct-2506-GGUF Q8

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 2
• eval_batch_size: 4
• seed: 42
• distributed_type: multi-GPU
• num_devices: 2
• gradient_accumulation_steps: 16
• total_train_batch_size: 64
• total_eval_batch_size: 8
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 10

Training results

Framework versions

• Transformers 4.54.1
• Pytorch 2.7.1+cu128
• Datasets 4.0.0
• Tokenizers 0.21.4