Qwen2-1.5_GGUF

Qwen2-1.5_GGUF

introduce

This model family contains versions q4_0 and q8_0, and the source model is Qwen2-1.5B, which has a latest date of 20240802

how to use

inder

import qwen_cpp
from typing import List
pipeline = qwen_cpp.Pipeline("your model path", "your token path", 512)

generation_kwargs = dict(
    max_length=4096,
    # max_new_tokens=args.max_new_tokens,
    max_context_length=512,
    do_sample=True,
    top_k=0,
    top_p=0.7,
    temperature=0.95,
    repetition_penalty=1.0,
    stream=True,
)

system_messages: List[qwen_cpp.ChatMessage] = []

system_messages.append(qwen_cpp.ChatMessage(role="system", content="You are a helpful assistant."))

messages = system_messages.copy()
messages.append(qwen_cpp.ChatMessage(role="user", content="你好"))
for chunk in pipeline.chat(messages, **generation_kwargs):
    print(chunk.content, sep="", end="", flush=True)
# for chunk in pipeline.generate(prompt, **generation_kwargs):
#             print(chunk, sep="", end="", flush=True)

while True:
    input_prompt = f"'Prompt': > "
    role = "user"
    try:
        prompt = input(input_prompt)
    except EOFError:
        break
    if not prompt:
        continue
    if prompt == "stop":
        break
    if prompt == "clear":
        messages = system_messages
        continue
    messages.append(qwen_cpp.ChatMessage(role=role, content=prompt))
    chunks = []
    for chunk in pipeline.chat(messages, **generation_kwargs):
        print(chunk.content, sep="", end="", flush=True)
        chunks.append(chunk)
    print()
    messages.append(pipeline.merge_streaming_messages(chunks))
print("Bye")

result:

您好，有什么我能帮到您的吗？
'Prompt': >  can you help me calcalate the result that 23 devide 4?
The result of 23 divided by 4 is approximately 5.75.
'Prompt': >  tell me about the information of Nvidia 3090?
Nvidia 3090 is a high-performance graphics processing unit (GPU) that was released in 2021. It is a successor to the 3080 and was designed to deliver superior performance in both single and multi-GPU configurations. The 3090 has a maximum memory bandwidth of 2.6 terabytes per second and can support up to 12GB of memory. It has a unique "double-snowflake" architecture that allows for greater memory bandwidth, better GPU performance, and improved scalability. It also supports up to 12 compute units per GPU, making it a versatile choice for both professional graphics applications and gaming. Additionally, the 3090 is known for its exceptional efficiency and low power consumption, making it a popular choice for professional and industrial applications. It is also a great choice for those who are looking for a high-performance GPU for their next graphics project.
'Prompt': >  stop
Bye

others