MemoryBERT

🧠 About NOETIV

This project is part of the NOETIV initiative — a modular AI platform for healthcare proffesionals.
🔗 Visit us at noetiv.com

🧠 MemoryBERT

A RoBERTa-based transformer model for Cognitive Memory Recognition (CMR) – classifying natural language into six memory categories inspired by cognitive science.

---

🧭 Overview

MemoryBERT is fine-tuned to classify user-generated text into:

• Episodic memory
• Semantic memory
• Spatial memory
• Emotional memory
• Associative memory
• Non-memory

This model supports research into memory-type classification, schema formation, and personalized AI interaction systems.

🧪 Model Details

• Base model: roberta-base
• Task: Multi-class sequence classification
• Classes: 6
• Max sequence length: 128 tokens
• Training epochs: 1.5
• Label smoothing: 0.1
• Loss function: CrossEntropyLoss
• Optimizer: AdamW
• Batch size: 8

---

📊 Evaluation Results

On a synthetic 400-example test set balanced across classes:

• Macro F1: 1.00
• Eval loss: 0.423
• Epochs: 1.5
• Accuracy: 100%

⚠️ Note: These results are from a synthetic dataset — further real-world validation is ongoing and expansion of baseline dataset used for version 1 of memoryBERT

---

🧠 Dataset

MemoryBERT was trained on a synthetic dataset of 4,000 curated examples (2,000 memory and 2,000 non-memory)

Each entry is labeled with one of six memory types and tagged by domain and span group.

---

🚀 Usage

from transformers import RobertaTokenizer, RobertaForSequenceClassification

model = RobertaForSequenceClassification.from_pretrained("DimitriosPanagoulias/MemoryBERT")
tokenizer = RobertaTokenizer.from_pretrained("DimitriosPanagoulias/MemoryBERT")

def predict_memory_type(text):
    inputs = tokenizer(text, return_tensors="pt", truncation=True, padding=True, max_length=128)
    outputs = model(**inputs)
    predicted_id = outputs.logits.argmax(dim=-1).item()
    return model.config.id2label[predicted_id]

predict_memory_type("Without a map, I navigated the winding back roads to reach my childhood home.")

or via huggingface pipeline

# Use a pipeline as a high-level helper
from transformers import pipeline
import torch
device = 0 if torch.cuda.is_available() else -1  # 0 = GPU, -1 = CPU
pipe = pipeline("text-classification", model="DimitriosPanagoulias/MemoryBERT", device=device)
pipe("I remember the long walk to my childhood school.")

outputs:

[{'label': 'episodic', 'score': 0.9272529482841492}]

Authors

• Dimitrios P. Panagoulias, Department of Informatics, University of Piraeus
• Persephone Papatheodosiou, Sleep Research Unit, Department of Psychiatry, National and Kapodistrian University of Athens
• Anastasios Bonakis, Second Department of Neurology, National and Kapodistrian University of Athens
• Dimitris Dikeos, Sleep Research Unit, Department of Psychiatry, National and Kapodistrian University of Athens
• Maria Virvou, Lab of Software Engineering, Department of Informatics, University of Piraeus
• George A. Tsihrintzis, Lab of Pattern Recognition and Machine Learning – Multimedia Systems, Department of Informatics, University of Piraeus

Citation

You can cite either one or both of the following previous related work:

• Panagoulias, D.P. et al. “Memory and Schema in Human–Generative Artificial Intelligence Interactions.” 2024 IEEE ICTAI Conference (in press)

Available at: https://ieeexplore.ieee.org/document/10849404

• Panagoulias, D.P. et al. Mathematical representation of memory and schema for improving human-generative AI interactions.” 2024 IEEE IISA Conference (in press)

Available at: https://ieeexplore.ieee.org/document/10786703