Tess-3-7B-SFT-GGUF

QuantFactory/Tess-3-7B-SFT-GGUF

This is quantized version of migtissera/Tess-3-7B-SFT created using llama.cpp

Original Model Card

<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->

<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>

<details><summary>See axolotl config</summary>

axolotl version: 0.4.1

base_model: mistralai/Mistral-7B-v0.3
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
tokenizer_use_fast: false

load_in_8bit: false
load_in_4bit: false
strict: false
model_config:

datasets:
  - path: /home/migel/ai_datasets/tess-v1.5b-chatml.jsonl
    type: sharegpt
    conversation: chatml
  - path: /home/migel/ai_datasets/Tess-3.0/Tess-3.0-multi_turn_chatml.jsonl
    type: sharegpt
    conversation: chatml
  - path: /home/migel/ai_datasets/Tess-3.0/Tess-3.0-single_turn_chatml.jsonl
    type: sharegpt
    conversation: chatml

chat_template: chatml

dataset_prepared_path: last_run_prepared_mistral
val_set_size: 0.0
output_dir: /home/migel/tess-2.5-mistral-7B-phase-1

resume_from_checkpoint: /home/migel/tess-2.5-mistral-7B-phase-1/checkpoint-440 
auto_resume_from_checkpoints: true

sequence_len: 16384
sample_packing: true
pad_to_sequence_len: true

gradient_accumulation_steps: 4
micro_batch_size: 4
num_epochs: 1
logging_steps: 1
optimizer: adamw_8bit
lr_scheduler: constant
learning_rate: 1e-6

wandb_project: mistral-7b
wandb_watch:
wandb_run_id:
wandb_log_model:

train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false

gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
saves_per_epoch: 10
evals_per_epoch: 10
save_total_limit: 3
save_steps:
eval_sample_packing: false
debug:
deepspeed: /home/migel/axolotl/deepspeed_configs/zero3_bf16.json
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
  bos_token: "<|im_start|>"
  eos_token: "<|im_end|>"
  pad_token: "<|end_of_text|>"

</details><br>

Tess-3-7B-SFT

Tess-3-7B is a finetuned version of the Mistral-7B-v0.3 base model. This version is the first phase of the final Tess-3 model, and have been trained with supervised fine-tuning (SFT) on a curated dataset of ~500K samples. The total SFT dataset contains about 1B tokens.

This model has 32K context length.

Sample code to run inference

Note that this model uses ChatML prompt format.

import torch, json
from transformers import AutoModelForCausalLM, AutoTokenizer
from stop_word import StopWordCriteria

model_path = "migtissera/Tess-3-7B-SFT"
output_file_path = "/home/migel/conversations.jsonl"

model = AutoModelForCausalLM.from_pretrained(
    model_path,
    torch_dtype=torch.float16,
    device_map="auto",
    load_in_4bit=False,
    trust_remote_code=True,
)

tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

terminators = [
    tokenizer.convert_tokens_to_ids("<|im_end|>")
]

def generate_text(instruction):
    tokens = tokenizer.encode(instruction)
    tokens = torch.LongTensor(tokens).unsqueeze(0)
    tokens = tokens.to("cuda")

    instance = {
        "input_ids": tokens,
        "top_p": 1.0,
        "temperature": 0.75,
        "generate_len": 1024,
        "top_k": 50,
    }

    length = len(tokens[0])
    with torch.no_grad():
        rest = model.generate(
            input_ids=tokens,
            max_length=length + instance["generate_len"],
            use_cache=True,
            do_sample=True,
            top_p=instance["top_p"],
            temperature=instance["temperature"],
            top_k=instance["top_k"],
            num_return_sequences=1,
            pad_token_id=tokenizer.eos_token_id,
            eos_token_id=terminators,
        )
    output = rest[0][length:]
    string = tokenizer.decode(output, skip_special_tokens=True)
    return f"{string}"

conversation = f"""<|im_start|>system\nYou are Tesoro, a helful AI assitant. You always provide detailed answers without hesitation.<|im_end|>\n<|im_start|>user\n"""

while True:
    user_input = input("You: ")
    llm_prompt = f"{conversation}{user_input}<|im_end|>\n<|im_start|>assistant\n"
    answer = generate_text(llm_prompt)
    print(answer)
    conversation = f"{llm_prompt}{answer}\n"
    json_data = {"prompt": user_input, "answer": answer}

    with open(output_file_path, "a") as output_file:
        output_file.write(json.dumps(json_data) + "\n")

Join My General AI Discord (NeuroLattice):

https://discord.gg/Hz6GrwGFKD

Limitations & Biases:

While this model aims for accuracy, it can occasionally produce inaccurate or misleading results.

Despite diligent efforts in refining the pretraining data, there remains a possibility for the generation of inappropriate, biased, or offensive content.

Exercise caution and cross-check information when necessary. This is an uncensored model.