CogStateIR-Supra50M-v0.0.3-medium_dataset

Supra 50 M Instruct (third time, medium dataset)

The model is available here.

Training setup

• Base model : https://huggingface.co/SupraLabs/Supra-50M-Instruct
• Parameters : 51.8M
• Dataset : 315 examples
• Epochs : 100
• Learning rate : 0.0001
• Hardware : Apple M2 Pro on GPU
• Duration : ~1h (thanks to Metal)

Results

Loss after training : 0.393566. Validation loss after training: 0.832417.

Prediction - training example

Model's raw output :

"emotion|anger|increases_a_lot\nemotion|hurt|increases_a_little\nrelationship|closeness|increases_a_little\nbelief|user_is_frustrated|increases_a_little"

Model's prediction :

state_changes:
  emotion:
    anger: increases_a_lot
    hurt: increases_a_little
  relationship:
    closeness: increases_a_little
  belief:
    user_is_frustrated: increases_a_little
  memory: null
  reflection: null

Output of the dataset :

state_changes:
  relationship:
    trust: increases_a_little
  emotion:
    anger: decreases
  belief:
    people_can_change: increases_a_little
  memory: reinforce_previous_conflict
  reflection: required

The model didn't really integrate the difference between relationship and emotion, didn't really integrate the subtleties about the character personnality, but didn't totally hallucinate : it integrate the output format, didn't repeat himself, and proposed a pretty relevant IR (not totally absurd). The result is quite the same as for the first model, but a little better (more content).

Prediction - held-out example

Model's raw output :

RAW OUTPUT:
"emotion|anger|increases\nemotion|hope|increases\nemotion|hope|increases\nrelationship|trust|increases\nrelationship|trust_is_frustration|increases\nrelationship|trust_is_frustration|increases\nrelationship|trust_is_frustration|increases\nrelationship|trust_is_frustration|increases\nrelationship|trust_is_frustration|increases\nrelationship|trust_is_frustration|increases\nrelationship|trust_is_frustration|increases\nrelationship|trust_is_frustration|increases\nrelationship|trust_is_frustration|increases\nrelationship|trust_is_frustration|increases\nrelationship|trust_is_frustration|increases\nrelations"

Model's output :

state_changes:
  emotion:
    anger: increases
    hope: increases
  relationship:
    trust: increases
    trust_is_frustration: increases
  belief: null
  memory: null
  reflection: null

Output of the test dataset :

state_changes:
  relationship:
    respect: increases_a_little
    disappointment: decreases
  emotion:
    irritation: decreases
    pride: increases_a_little
  belief:
    effort_matters: increases
  memory: reinforce_previous_conflict
  reflection: required

The problem is that here, the model seems worst than the one with 135 examples... so I decided to run other tests :

Prediction - held-out example 2

RAW OUTPUT: "no_changes"

Model's output :

state_changes: no_changes

Output of the dataset :

state_changes:
  emotion:
    contentment: increases_a_little

This time, the model didn't hallucinate. It only decided that no changes were required, which is not absurd.

![NOTE] This examples and the following ones are from the dataset, but the model wasn't trained on these.

Prediction - held-out example 3

RAW OUTPUT: "emotion|contentment|increases_a_little\nemotion|warmth|increases_a_little\nbelief|increases_a_little\nbelief|increases_a_little\nbelief|increases_a_little\nbelief|decreases_a_little\nbelief|decreases_a_little\nbelief|decreases_a_little\nbelief|decreases_a_little\nbelief|decreases_a_little\nbelief|decreases_a_little\nbelief|decreases_a_little\nbelief|decreases_a_little\nbelief|decreases_a_little\nbelief|decreases_a_little\nbelief|decreases_a_little\nbelief|decreases_a_little\nbelief|decreases_a_little\nbelief|decreases_a_little\nbelief|decreases_a"

Model's output :

state_changes:
  emotion:
    contentment: increases_a_little
    warmth: increases_a_little
  relationship: null
  belief: null
  memory: null
  reflection: null

Output from the dataset :

state_changes:
  emotion:
    contentment: increases_a_little

This time, the result is very good. While the model wasn't trained over this examples (he didn't know this one), its prediction is adequate : he predicted contentment, but also added warmth, which is not absurd. We still have a problem : the hallucination. The model repeated himself. Anyway, thanks to the internal parser, this doesn't affect the final output.

Prediction - held-out example 4

RAW OUTPUT: "no_changes"

Output of the model :

state_changes: no_changes

Output of the dataset :

state_changes: no_changes

While the model wasn't trained over this example, it prediction is correct. He 'understands' that there are no changes or minor changes in basic conversations.

Prediction - held-out example 5

RAW OUTPUT: "emotion|determination|increases\nemotion|determination|increases\nemotion|determination|increases\nemotion|determination|increases\nrelationship|determination|increases\nbelief|exhaustion|increases\nbelief|exhaustion|increases\nbelief|exhaustion|increases\nbelief|exhaustion|increases\nbelief|exhaustion|increases\nbelief|exhaustion|increases\nbelief|exhaustion|increases\nbelief|exhaustion|increases\nreflection|required"

Model's output :

state_changes:
  emotion:
    determination: increases
  relationship:
    determination: increases
  belief:
    exhaustion: increases
  memory: null
  reflection: required

Output of the dataset :

state_changes:
  emotion:
    exhaustion: increases_a_little
    relief: increases
    warmth: increases_a_little
  relationship:
    closeness: increases_a_little

This examples is more complex. It manages to predict something coherent (exhaustion, determination, null memory is right...), but not totally right : exhaustion is NOT a belief, reflection wasn't really required, and closeness was required.

Current model's behaviour :

Next steps

I think the real bottleneck is the model size : a 360M parameters could be more appropriate. Also, a larger dataset would make the model understand more situations and more complex ones.

Next things to do are :

• train a larger model (~360M parameters) with a larger dataset / the same one (~500 examples) and ~100 epochs (to avoid overtraining and memorization)
• train a LoRA instead of finetuning all the model.
• train with an even datasets : 500, 1500, 2500 examples.
• evaluate the model's predictions over many tests
• integrate the IR in the engine