plamo-2-translate-MP-GGUF

license: plamo community license. TLDR: free for use unless inside a big commercial corporation.

Mixed Precision GGUF layer quantization of plamo 2 translate by pfnet

Original model: https://huggingface.co/pfnet/plamo-2-translate

The hybrid quant employs different quantization levels on a per layer basis to enable both high performance and small file size at the same time. This particular quant achieves a ~7.3G gguf for the 9.53B parameter translation model . The quants employed are all K to avoid slow CPU or older GPU processing of IQ quants. For this file the layer quants are as follows:

Q5_K_L : attn_v = q8_0 attn_o = q6_k ffn_d = q6_k
Q6_K_S : Q6_K
Q6_K_M : attn_v = q8_0 ffn_d = q8_0
Q6_K_L : attn_v = q8_0 attn_o = q8_0 ffn_d = q8_0

   LAYER_TYPES='[
   [0 ,"Q6_K_S"],[1 ,"Q5_K_L"],[2 ,"Q5_K_M"],[3 ,"Q5_K_M"],[4 ,"Q5_K_M"],[5 ,"Q5_K_M"],[6 ,"Q5_K_M"],[7 ,"Q5_K_M"],
   [8 ,"Q5_K_M"],[9 ,"Q5_K_M"],[10,"Q5_K_M"],[11,"Q5_K_M"],[12,"Q5_K_M"],[13,"Q5_K_M"],[14,"Q5_K_M"],[15,"Q5_K_M"],
   [16,"Q5_K_L"],[17,"Q5_K_L"],[18,"Q5_K_L"],[19,"Q5_K_L"],[20,"Q5_K_L"],[21,"Q5_K_L"],[22,"Q5_K_L"],[23,"Q5_K_L"],
   [24,"Q6_K_S"],[25,"Q6_K_S"],[26,"Q6_K_S"],[27,"Q6_K_S"],[28,"Q6_K_S"],[29,"Q6_K_S"],[30,"Q6_K_M"],[31,"Q6_K_L"]
   ]'
   FLAGS="--token-embedding-type Q6_K --output-tensor-type Q6_K --layer-types-high"

The layer quants were optimized to enable the model to fit into 8G VRAM cards with room left over for some context while using a minimum layer quant of Q5_K_M across all layers for high performance.

Comparison:

Usage:

This model is a machine language translation (MT) model. Its prompt format is as follows:

<|plamo:op|>dataset\ntranslation\n<|plamo:op|>input lang=SOURCE\nHello, how are you?\n<|plamo:op|>output lang=TARGET\n

where SOURCE and TARGET are language codes from the set {de, en, es, fr, ja, ru, zh} At time of writing it is not clear if the model supports more than these languages or not.

Example use:

This example shows use using a downstream llama.cpp server accessed through a shell based CLI running in a bash shell which handles unicode correctly. The example will cycle a starting phrase through all the languages:

lm '<|plamo:op|>dataset\ntranslation\n<|plamo:op|>input lang=en\nThe secret code to open the safe is "dead mens chest"\n<|plamo:op|>output lang=de\n'
Der Geheimcode zum Öffnen des Safes lautet „dead mens chest“

lm '<|plamo:op|>dataset\ntranslation\n<|plamo:op|>input lang=de\nDer Geheimcode zum Öffnen des Safes lautet „dead mens chest“\n<|plamo:op|>output lang=es\n'
La clave secreta para abrir la caja fuerte es "dead mens chest"

lm '<|plamo:op|>dataset\ntranslation\n<|plamo:op|>input lang=es\nLa clave secreta para abrir la caja fuerte es "dead mens chest"\n<|plamo:op|>output lang=fr\n'
Le mot de passe secret pour ouvrir la coffre-fort est « dead mens chest »

lm '<|plamo:op|>dataset\ntranslation\n<|plamo:op|>input lang=fr\nLe mot de passe secret pour ouvrir la coffre-fort est « dead mens chest »\n<|plamo:op|>output lang=ru\n'
Ключевой пароль для открытия сейфа — «dead mens chest»

lm '<|plamo:op|>dataset\ntranslation\n<|plamo:op|>input lang=ru\nКлючевой пароль для открытия сейфа — «dead mens chest»\n<|plamo:op|>output lang=zh\n'
钥匙密码用于打开保险柜是“dead mens chest”

lm '<|plamo:op|>dataset\ntranslation\n<|plamo:op|>input lang=zh\n钥匙 密码用于打开保险柜是“dead mens chest”\n<|plamo:op|>output lang=jp\n'
金庫を開けるための鍵のパスワードは「dead mens chest」です

lm '<|plamo:op|>dataset\ntranslation\n<|plamo:op|>input lang=jp\n金庫 を開けるための鍵のパスワードは「dead mens chest」です\n<|plamo:op|>output lang=en\n'
The password for the key that unlocks the safe is "dead mens chest".

This example shows semantics held up fairly well through the indo european language and oriental language sets but syntax broke down a small amount in this simple test (somewhere along the line secret code got translated to password and open got translated to unlock)

Note that "<|plamo:op|>" is a special token in the model vocab, i.e. correct tokenization of the starting prompt will be:

tokenize '<|plamo:op|>dataset\ntranslation\n<|plamo:op|>input lang=en\nThe secret code to open the safe is "dead mens chest" \n<|plamo:op|>output lang=de\n'
[ 4, 45474, 10, 18053, 10, 4, 1760, 98700, 61, 1227, 10, 1097, 5160, 1461, 35112, 2473, 45119, 290, 21236, 39021, 9786, 34, 32, 10, 4, 3045, 98700, 61, 1453, 10 ]

The translator can also be easily scripted in a bash shell environment as follows:

translate()

{
   phrase=$1
   SOURCE=$2
   TARGET=$3
   echo -n $(lm "<|plamo:op|>dataset\ntranslation\n<|plamo:op|>input lang=${SOURCE}\n${phrase}\n<|plamo:op|>output lang=${TARGET}")
}

langs=(de es fr ru ja zh en)

phrase='The secret code to open the safe is "dead mens chest"'
SOURCE=en
echo "${SOURCE}: $phrase"
for TARGET in ${langs[@]}; do
   phrase=$(translate "$phrase" $SOURCE $TARGET)
   # Remove lead and trail \n delimiters
   phrase=${phrase#\\n}
   phrase="${phrase%\\n}"
   echo "${TARGET}: $phrase"
   SOURCE=$TARGET
done

Running this script produces:

en: The secret code to open the safe is "dead mens chest"
de: Der Geheimcode zum Öffnen des Safes lautet „dead mens chest“
es: La clave secreta para abrir la caja fuerte es "dead mens chest"
fr: Le mot de passe secret pour ouvrir la coffre-fort est « dead mens chest »
ru: Ключевой пароль для открытия сейфа — «dead mens chest»
ja: 金庫を開けるための鍵のパスワードは「dead mens chest」です
zh: 钥匙的密码是“dead mens chest”，用于打开保险柜。
en: The password for the key is "dead mens chest," used to unlock the safe.

MT benchmarks for the model are given here: https://huggingface.co/spaces/steampunque/benchlm

Download the file from below:

A discussion thread about the hybrid layer quant approach can be found here on the llama.cpp git repository:

https://github.com/ggml-org/llama.cpp/discussions/13040