komt-Llama-2-13b-hf-ggml

komt-Llama-2-13b-hf-ggml

https://github.com/davidkim205/komt

This model quantized the korean Llama 2 13B using llama.cpp to 4-bit quantization.

Our model, being in the same format as TheBloke's ggml, supports the following libraries or UI.

The following content references TheBloke/Llama-2-13B-chat-GGML.

GGML files are for CPU + GPU inference using llama.cpp and libraries and UIs which support this format, such as:

• KoboldCpp, a powerful GGML web UI with full GPU acceleration out of the box. Especially good for story telling.
• LoLLMS Web UI, a great web UI with GPU acceleration via the c_transformers backend.
• LM Studio, a fully featured local GUI. Supports full GPU accel on macOS. Also supports Windows, without GPU accel.
• text-generation-webui, the most popular web UI. Requires extra steps to enable GPU accel via llama.cpp backend.
• ctransformers, a Python library with LangChain support and OpenAI-compatible AI server.
• llama-cpp-python, a Python library with OpenAI-compatible API server.

Model Details

• Model Developers : davidkim(changyeon kim)
• Repository : https://github.com/davidkim205/komt
• quant methods : q4_0, q4_1, q5_0, q5_1, q2_k, q3_k, q3_k_m, q3_k_l, q4_k, q4_k_s, q4_k_m, q5_k, q5_k_s, q5_k_m, q8_0, q4_0

Prompt Template

### instruction: {prompt}

### Response: 

Examples:

### instruction: 자동차 종합(정기)검사 의무기간은 얼마인가요?

### Response:

response:

### instruction: 자동차 종합(정기)검사 의무기간은 얼마인가요?

### Response:자동차 종합(정기)검사는 2년
1991년 7월 1일에 고시된 '자동차 보험료 조정기준'에서 취리로부터 제정된 기준 상 경량 살수차를 제외한 자동차 모든 승용자동차는 2년마다 필요하다. 이 법은 차량에 관계없이 2년마다 정기검사를 해야한다고 규제했다.

Usage

When using the original llama.cpp

make -j && ./main -m models/komt-Llama-2-13b-hf-ggml/ggml-model-q8_0.bin -p "### instruction: 영화 해리포터 시리즈 배급사가 어디야\n\n### Response:"

When using the modified llama.cpp for korean multi-task (recommended): Refer https://github.com/davidkim205/komt/tree/main/llama.cpp

make -j && ./main -m models/komt-Llama-2-13b-hf-ggml/ggml-model-q8_0.bin -p "영화 해리포터 시리즈 배급사가 어디야"

response:

 $ make -j && ./main -m models/komt-Llama-2-13b-hf-ggml/ggml-model-q8_0.bin -p "영화 해리포터 시리즈 배급사가 어디야"
I llama.cpp build info:
I UNAME_S:  Linux
I UNAME_P:  x86_64
I UNAME_M:  x86_64
I CFLAGS:   -I.              -O3 -std=c11   -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wdouble-promotion -Wshadow -Wstrict-prototypes -Wpointer-arith -Wmissing-prototypes -pthread -march=native -mtune=native -DGGML_USE_K_QUANTS
I CXXFLAGS: -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -march=native -mtune=native -DGGML_USE_K_QUANTS
I LDFLAGS:
I CC:       cc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
I CXX:      g++ (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0

make: Nothing to be done for 'default'.
main: build = 6 (01a61bf)
main: seed  = 1692190774
llama.cpp: loading model from models/komt-Llama-2-13b-hf-ggml/ggml-model-q8_0.bin
llama_model_load_internal: format     = ggjt v3 (latest)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 512
llama_model_load_internal: n_embd     = 5120
llama_model_load_internal: n_mult     = 6912
llama_model_load_internal: n_head     = 40
llama_model_load_internal: n_head_kv  = 40
llama_model_load_internal: n_layer    = 40
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: n_gqa      = 1
llama_model_load_internal: rnorm_eps  = 5.0e-06
llama_model_load_internal: n_ff       = 13824
llama_model_load_internal: freq_base  = 10000.0
llama_model_load_internal: freq_scale = 1
llama_model_load_internal: ftype      = 7 (mostly Q8_0)
llama_model_load_internal: model size = 13B
llama_model_load_internal: ggml ctx size =    0.11 MB
llama_model_load_internal: mem required  = 13152.13 MB (+  400.00 MB per state)
llama_new_context_with_model: kv self size  =  400.00 MB
llama_new_context_with_model: compute buffer total size =   75.35 MB

system_info: n_threads = 8 / 16 | AVX = 1 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 |
sampling: repeat_last_n = 64, repeat_penalty = 1.100000, presence_penalty = 0.000000, frequency_penalty = 0.000000, top_k = 40, tfs_z = 1.000000, top_p = 0.950000, typical_p = 1.000000, temp = 0.800000, mirostat = 0, mirostat_lr = 0.100000, mirostat_ent = 5.000000
generate: n_ctx = 512, n_batch = 512, n_predict = -1, n_keep = 0


 ### instruction: 영화 해리포터 시리즈 배급사가 어디야

### Response:워너 브라더스
해리포터(Harry Potter)는 J. K. 롤링이 쓴 판타지 소설이다. 1997년부터 2007년까지 총 7권으로 발행되었고, 전 세계적으로 많은 인기를 끌었다. 영국에서는 블룸버그(Bloomsbury), 미국에서는 워너 브라더스(Warner Brothers)가 각각 출판하였다. 현재 전 세계적으로 2억 4,000만 부 이상의 판매고를 올리고 있으며, 전 세계 대부분의 문학가들에게 영향을 주었다. ### check_end_of_text [end of text]

llama_print_timings:        load time =   801.73 ms
llama_print_timings:      sample time =   108.54 ms /   308 runs   (    0.35 ms per token,  2837.66 tokens per second)
llama_print_timings: prompt eval time =  2651.47 ms /    43 tokens (   61.66 ms per token,    16.22 tokens per second)
llama_print_timings:        eval time = 120629.25 ms /   307 runs   (  392.93 ms per token,     2.54 tokens per second)
llama_print_timings:       total time = 123440.86 ms