pozdgpt

PozdGPT

Peak of humanity technologies. New breath in neuroslop world.

Usage

Via llama.cpp + GGUF

# !pip install llama-cpp-python
from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="sodeeplearning/pozdgpt",
	filename="PozdGPT-Q4_K_M.gguf", # Or Q6_K, Q8_0, f16
)

AWQ + vLLM

To launch 4bit AWQ version you need to download this folder and launch your vLLM server:

# !pip install vllm

vllm serve ./PozdGPT-awq-4bit \
  --served-model-name pozdgpt \
  --quantization compressed-tensors \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.88 \
  --max-num-seqs 6 \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --api-key key \
  --port 8148

Test via telegram bot

You can test this bot in official telegram bot

Contacts

• Github
• Telegram bot
• Telegram channel