Aquila2-70B-Expr-3.0bpw-h6-exl2

<h4 align="center"> <p> <b>English</b> | <a href="https://huggingface.co/BAAI/AquilaChat2-70B-Expr/blob/main/README_zh.md">简体中文</a> </p> </h4>

We opensource our Aquila2 series, now including Aquila2, the base language models, namely Aquila2-7B, Aquila2-34B and Aquila2-70B-Expr , as well as AquilaChat2, the chat models, namely AquilaChat2-7B, AquilaChat2-34B and AquilaChat2-70B-Expr, as well as the long-text chat models, namely AquilaChat2-7B-16k and AquilaChat2-34B-16k

The additional details of the Aquila model will be presented in the official technical report. Please stay tuned for updates on official channels.

Quick Start

1. Inference

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from transformers import BitsAndBytesConfig

model_info = "BAAI/Aquila2-70B-Expr"
tokenizer = AutoTokenizer.from_pretrained(model_info, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_info, trust_remote_code=True)
model.eval()
text = "请给出10个要到北京旅游的理由。"
tokens = tokenizer.encode_plus(text)['input_ids']
tokens = torch.tensor(tokens)[None,].to(device)
stop_tokens = ["###", "[UNK]", "</s>"]
with torch.no_grad():
    out = model.generate(tokens, do_sample=True, max_length=512, eos_token_id=100007, bad_words_ids=[[tokenizer.encode(token)[0] for token in stop_tokens]])[0]
    out = tokenizer.decode(out.cpu().numpy().tolist())
    print(out)

License

Aquila2 series open-source model is licensed under BAAI Aquila Model Licence Agreement