InfLLM-V2-Long-Sparse-Base-Q8_0-GGUF

AXONVERTEX-AI-RESEARCH/InfLLM-V2-Long-Sparse-Base-Q8_0-GGUF

This model was converted to GGUF format from openbmb/InfLLM-V2-Long-Sparse-Base using llama.cpp via the ggml.ai's GGUF-my-repo space. Refer to the original model card for more details on the model.

Use with Ollama

root@90dd7d73d62b:/# ollama pull hf.co/AXONVERTEX-AI-RESEARCH/InfLLM-V2-Long-Sparse-Base-Q8_0-GGUF:Q8_0
pulling manifest 
pulling 5bdae6814d6a: 100% ▕██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 8.7 GB                         
pulling a47b02e00552: 100% ▕██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏  106 B                         
pulling b78301c0df4d: 100% ▕██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏   38 B                         
pulling d03c38a6a78c: 100% ▕██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏  557 B                         
verifying sha256 digest 
writing manifest 
success 
root@90dd7d73d62b:/# ollama run hf.co/AXONVERTEX-AI-RESEARCH/InfLLM-V2-Long-Sparse-Base-Q8_0-GGUF:Q8_0
>>> hello 
Hello! How can I assist you today?

>>> Write an article about Artificial Intelligence.
Artificial Intelligence (AI) has revolutionized the way we live and work in recent years. AI refers to the ability of machines to simulate human intelligence and perform tasks that typically require 
human intelligence, such as learning, reasoning, problem-solving, decision making, perception, and language understanding. With the increasing demand for automation and efficiency, AI has become an 
essential tool across various industries, including healthcare, finance, manufacturing, transportation, and education.

AI systems are designed to analyze vast amounts of data and learn from it to make predictions or decisions. They use algorithms that can identify patterns and relationships in data, and then apply 
these insights to real-world problems. AI systems can be classified into three types: narrow AI, general AI, and superintelligence. Narrow AI is designed to perform a specific task, such as 
recognizing faces or translating languages. General AI refers to machines that have the ability to understand and learn any intellectual task that humans can do. Superintelligence refers to machines 
that are much smarter than humans in every aspect, including creativity, wisdom, and emotional intelligence.

AI has already made significant progress in several fields, with applications ranging from self-driving cars to virtual assistants like Siri and Alexa. In healthcare, AI can help doctors diagnose 
diseases more accurately and provide personalized treatment plans for patients. In finance, AI can detect fraudulent transactions and predict market trends. In manufacturing, AI can optimize supply 
chain management and improve production efficiency. In transportation, AI can enable autonomous vehicles to navigate safely on roads. In education, AI can personalize learning experiences for 
students based on their individual needs and preferences.

However, AI also raises some ethical concerns and challenges. For instance, there are concerns about job displacement as machines replace human workers. There are also issues related to privacy and 
security, as AI systems collect and analyze vast amounts of personal data. Additionally, there is a risk of bias in AI algorithms that can perpetuate discrimination against certain groups of people.

In conclusion, AI has the potential to transform various aspects of our lives, making them more efficient, effective, and personalized. However, it also presents challenges and risks that need to be 
addressed through responsible development and deployment of AI systems.

>>> Send a message (/? for help)

Chat Template

{% for message in messages %}{{'<|im_start|>' + message['role'] + '
' + message['content'] + '<|im_end|>' + '
'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
' }}{% if enable_thinking is defined and enable_thinking is false %}{{ '<think>

</think>
' }}{% endif %}{% endif %}

Use with llama.cpp

Install llama.cpp through brew (works on Mac and Linux)

brew install llama.cpp

Invoke the llama.cpp server or the CLI.

CLI:

llama-cli --hf-repo AXONVERTEX-AI-RESEARCH/InfLLM-V2-Long-Sparse-Base-Q8_0-GGUF --hf-file infllm-v2-long-sparse-base-q8_0.gguf -p "The meaning to life and the universe is"

Server:

llama-server --hf-repo AXONVERTEX-AI-RESEARCH/InfLLM-V2-Long-Sparse-Base-Q8_0-GGUF --hf-file infllm-v2-long-sparse-base-q8_0.gguf -c 2048

Note: You can also use this checkpoint directly through the usage steps listed in the Llama.cpp repo as well.

Step 1: Clone llama.cpp from GitHub.

git clone https://github.com/ggerganov/llama.cpp

Step 2: Move into the llama.cpp folder and build it with LLAMA_CURL=1 flag along with other hardware-specific flags (for ex: LLAMA_CUDA=1 for Nvidia GPUs on Linux).

cd llama.cpp && LLAMA_CURL=1 make

Step 3: Run inference through the main binary.

./llama-cli --hf-repo AXONVERTEX-AI-RESEARCH/InfLLM-V2-Long-Sparse-Base-Q8_0-GGUF --hf-file infllm-v2-long-sparse-base-q8_0.gguf -p "The meaning to life and the universe is"

or

./llama-server --hf-repo AXONVERTEX-AI-RESEARCH/InfLLM-V2-Long-Sparse-Base-Q8_0-GGUF --hf-file infllm-v2-long-sparse-base-q8_0.gguf -c 2048