vntl-7b-v0.3-lora

This is an experimental llama2 7B lora created using the VNTL-v2-1k dataset. There have been some minor changes in the dataset since version 0.2, and I have made the following adjustments to the training arguments:

• Model loaded in 8 bits.
• Sequence length limited to 1024 tokens to speed up experiments.
• Effective batch size changed to 30 (6 batch size + 5 grad acc).
• 2 epochs.

Eval Loss: 0.78

This lora was trained alongside a 4-bit lora (qlora), the goal being to see if training a lora would be better than training a qlora. In the end, I don't think that there was much of a difference. At most I saw a consistent 0.01 drop in loss, but the loss graph looked the same, which meant both fine-tunes converged the same way.

This is an prompt example:

<<START>>
Name: Uryuu Shingo (瓜生 新吾) | Gender: Male | Aliases: Onii-chan (お兄ちゃん)
Name: Uryuu Sakuno (瓜生 桜乃) | Gender: Female
<<JAPANESE>>
[桜乃]: 『……ごめん』
<<ENGLISH>> (fidelity = absolute)
[Sakuno]: 『... Sorry.』
<<JAPANESE>>
[新吾]: 「ううん、こう言っちゃなんだけど、迷子でよかったよ。桜乃は可愛いから、いろいろ心配しちゃってたんだぞ俺」
<<ENGLISH>> (fidelity = high)

The generated translation for that prompt, with temperature 0, is:

[Shingo]: 「No, don't apologize. I'm just glad you're safe. You're so cute, Sakuno, I was worried sick.」