DElefant

DElefant:

<img src="https://huggingface.co/SebastianBodza/DElefant/resolve/main/badge_gerlefant.png" style="max-width:200px"> DElefant is a LLM developed for instruction tuned German interactions. This version is built on top of the adapted BLOOM version from <a href="https://huggingface.co/malteos/bloom-6b4-clp-german">Malte Ostendorff</a> with a opus-mt translated and afterwards filtered <a href="https://huggingface.co/datasets/SebastianBodza/Ger_WizardLM_evol_instruct_70k_V0">WizardLM</a> dataset. The evolved dataset led to SOTA english LLMs and we hope by incoperating the dataset to a german base model we can leverage the capabilities for various tasks including Code generation. Due to limitation in translation, the comments inside of the code blocks remained english, however the Coding was kept in working condition.

Model Description:

Full-Finetuning of the German-BLOOM model on an RTX 3090 with the translated WizardLM Dataset.

Roadmap:

If there is sufficient demand, additional adjustments can be made:

• Native German generated dataset
• Full Fine-Tuning of larger LLMs e.g. Falcon, Starcoderplus, ...

How to use:

Prompt-Template:

{instruction}\n\n### Response:

Code example for inference:

from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("SebastianBodza/DElefant")
model = AutoModelForCausalLM.from_pretrained("SebastianBodza/DElefant", device_map="auto")
frage = "Wie heißt der Bundeskanzler?"
prompt = f"{frage}\n\n### Response:"

txt = tokenizer(prompt, return_tensors="pt").to("cuda")
txt = model.generate(**txt, 
                          max_new_tokens=256, 
                          eos_token_id=tokenizer.eos_token_id)
tokenizer.decode(txt[0], skip_special_tokens=True)

Training:

Training was based on Llama-X with the adaptions of WizardLMs training script.

deepspeed Llama-X/src/train_freeform.py \
                --model_name_or_path malteos/bloom-6b4-clp-german \
                --data_path ger_alpaca_evol_instruct_70k_e.json \
                --output_dir ./full_finetune \
                --num_train_epochs 2 \
                --model_max_length 2048 \
                --per_device_train_batch_size 2 \
                --per_device_eval_batch_size 1 \
                --gradient_accumulation_steps 8 \
                --evaluation_strategy "no" \
                --save_strategy "steps" \
                --save_steps 400 \
                --save_total_limit 3 \
                --learning_rate 2e-5 \
                --warmup_steps 2 \
                --logging_steps 2 \
                --lr_scheduler_type "cosine" \
                --report_to "tensorboard" \
                --gradient_checkpointing True \
                --deepspeed deepspeed.json \
                --bf16 True

<img src="https://huggingface.co/SebastianBodza/DElefant/resolve/main/train_loss_DElefant.svg" style="max-width:350px">