lightblue_-_karasu-7B-gguf

RichardErkhov/lightblue_-_karasu-7B-gguf

lightblue_-_karasu-7B-gguf is a machine learning model from RichardErkhov. Use it for the machine learning task on the model card, and read the license before you ship it in a product.

Downloads · 30 days

1% of all-time downloads

All-time downloads

2.5K

Public

Repo size

91.9 GB

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.gguf106 GB · 100%

At a glance

Access: Public
Created: Jul 26, 2024
Updated: Jul 27, 2024
SHA: c3b25e9e

Model card

From the Hugging Face model README

Quantization made by Richard Erkhov.

Github

Discord

Request more models

karasu-7B - GGUF

Model creator: https://huggingface.co/lightblue/
Original model: https://huggingface.co/lightblue/karasu-7B/

Name	Quant method	Size
karasu-7B.Q2_K.gguf	Q2_K	2.92GB
karasu-7B.IQ3_XS.gguf	IQ3_XS	3.23GB
karasu-7B.IQ3_S.gguf	IQ3_S	3.39GB
karasu-7B.Q3_K_S.gguf	Q3_K_S	3.37GB
karasu-7B.IQ3_M.gguf	IQ3_M	3.48GB
karasu-7B.Q3_K.gguf	Q3_K	3.7GB
karasu-7B.Q3_K_M.gguf	Q3_K_M	3.7GB
karasu-7B.Q3_K_L.gguf	Q3_K_L	3.98GB
karasu-7B.IQ4_XS.gguf	IQ4_XS	4.13GB
karasu-7B.Q4_0.gguf	Q4_0	4.29GB
karasu-7B.IQ4_NL.gguf	IQ4_NL	4.34GB
karasu-7B.Q4_K_S.gguf	Q4_K_S	4.32GB
karasu-7B.Q4_K.gguf	Q4_K	4.54GB
karasu-7B.Q4_K_M.gguf	Q4_K_M	4.54GB
karasu-7B.Q4_1.gguf	Q4_1	4.73GB
karasu-7B.Q5_0.gguf	Q5_0	5.16GB
karasu-7B.Q5_K_S.gguf	Q5_K_S	5.16GB
karasu-7B.Q5_K.gguf	Q5_K	5.29GB
karasu-7B.Q5_K_M.gguf	Q5_K_M	5.29GB
karasu-7B.Q5_1.gguf	Q5_1	5.6GB
karasu-7B.Q6_K.gguf	Q6_K	6.09GB
karasu-7B.Q8_0.gguf	Q8_0	7.88GB

Original model description:

license: apache-2.0 datasets:

graelo/wikipedia
uonlp/CulturaX
HuggingFaceH4/ultrachat_200k language:
ja
en

Evaluation

image/png

How to use

Hugggingface

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

tokenizer = AutoTokenizer.from_pretrained("lightblue/karasu-7B")
model = AutoModelForCausalLM.from_pretrained("lightblue/karasu-7B", torch_dtype=torch.bfloat16, device_map="auto")

pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)

messages = [{"role": "system", "content": "あなたはAIアシスタントです。"}]
messages.append({"role": "user", "content": "イギリスの首相は誰ですか？"})

prompt = tokenizer.apply_chat_template(conversation=messages, add_generation_prompt=True, tokenize=False)

pipe(prompt, max_new_tokens=100, do_sample=False, temperature=0.0, return_full_text=False)

VLLM

from vllm import LLM, SamplingParams

sampling_params = SamplingParams(temperature=0.0, max_tokens=100)
llm = LLM(model="lightblue/karasu-7B")

messages = [{"role": "system", "content": "あなたはAIアシスタントです。"}]
messages.append({"role": "user", "content": "イギリスの首相は誰ですか？"})
prompt = llm.llm_engine.tokenizer.apply_chat_template(conversation=messages, add_generation_prompt=True, tokenize=False)
prompts = [prompt]

outputs = llm.generate(prompts, sampling_params)
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

Base checkpoint

augmxnt/shisa-7b-v1

Mistral-7B base
Pre-trained on 8B of MADLAD-Ja
Finetuned on Japanese instructions
Highest scoring 7B model on conversation benchmark (JA MT-Bench)

Training datasets (total ~7B)

Aozora Bunko
Japanese Law Precedent Dataset
Japanese Wikipedia
.lg.jp, .go.jp, .ac.jp domain webscrapes from CulturaX (Any documents with same first 25 characters were de-duplicated)
English Ultrachat200K-gen (So that it doesn't forget English and chatting ability learned in the base checkpoint)

Developed by

Engineers

Peter Devine

Sho Higuchi

Advisors

Yuuki Yamanaka

Atom Sonoda

Project manager

Shunichi Taniguchi

Dataset evaluator

Renju Aoki

Created: Jul 26, 2024
Updated: Jul 27, 2024