Saxo_-_yunsung-llama-2-koen-13b-linkbricks-sft-basic-v1-gguf

Quantization made by Richard Erkhov.

Github

Discord

Request more models

yunsung-llama-2-koen-13b-linkbricks-sft-basic-v1 - GGUF

• Model creator: https://huggingface.co/Saxo/
• Original model: https://huggingface.co/Saxo/yunsung-llama-2-koen-13b-linkbricks-sft-basic-v1/

Original model description:

library_name: transformers license: apache-2.0 basemodel: beomi/llama-2-koen-13b datasets:

• Saxo/total_ko_train_set_small_basic
• beomi/KoAlpaca-v1.1a
• kyujinpy/KOR-OpenOrca-Platypus-v2
• nlpai-lab/databricks-dolly-15k-ko language:
• ko
• en pipeline_tag: text-generation

---

Model Card for Model ID

<div align="center"> <img src="https://www.linkbricks.com/wp-content/uploads/2022/03/%E1%84%85%E1%85%B5%E1%86%BC%E1%84%8F%E1%85%B3%E1%84%87%E1%85%B3%E1%84%85%E1%85%B5%E1%86%A8%E1%84%89%E1%85%B3%E1%84%85%E1%85%A9%E1%84%80%E1%85%A9-2-1024x804.png" /> </div>

AI 와 빅데이터 분석 전문 기업인 Linkbricks의 데이터사이언티스트인 지윤성 박사(Saxo)가 beomi/llama-2-koen-13b 베이스모델을 GCP상의 A100-40G 4개를 통해 SFT 훈련을 한(2048 Tokens) 인스트럭션 모델. Accelerate, Deepspeed Zero-3 라이브러리를 사용했으며 Flash Attention 은 Disable 로 설정

Dr. Yunsung Ji (Saxo), a data scientist at Linkbricks, a company specializing in AI and big data analytics, trained the beomi/llama-2-koen-13b base model on 4 A100-40Gs on GCP for 4 hours of instructional training (2048 Tokens). Accelerate, Deepspeed Zero-3 libraries were used.

www.linkbricks.com, www.linkbricks.vc

Configuration including BitsandBytes

---

learning_rate = 2e-4 num_epochs = 5 batch_size = 4 block_size = 2048 trainer = "sft" warmup_ratio = 0.1 weight_decay = 0.01 gradient_accumulation = 4 mixed_precision = "fp16" peft = True quantization = "int4" lora_r = 64 lora_alpha = 16 lora_dropout = 0.1 model_max_length = 2048

Dataset Format

Alpaca Format Prompt Text