Downloads · 30 days
380
9% of all-time downloads
tomlawrence/NVIDIA-Nemotron-Nano-12B-v2-VL-GGUF
NVIDIA-Nemotron-Nano-12B-v2-VL-GGUF is a image-text-to-text model from tomlawrence. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as other.
Based on NVIDIA Nemotron Nano v2 12B VL, converted to GGUF as BF16 using GGML's converthftogguf.py and then quantized to 4-bit using llama-quantize, allowing for inference on smaller consumer GPUs via Llama.cpp. <br
Downloads · 30 days
380
9% of all-time downloads
All-time downloads
4.3K
Public
Repo size
59.9 GB
Likes
0
Public
Click a slice to open those files.
.gguf33.5 GB · 100%
From the Hugging Face model README
Based on NVIDIA Nemotron Nano v2 12B VL, converted to GGUF as BF16 using GGML's convert_hf_to_gguf.py and then quantized to 4-bit using llama-quantize, allowing for inference on smaller consumer GPUs via Llama.cpp.
<br>
Nemotron Nano 12B V2 VL is a model for multi-modal document intelligence. It would be used by individuals or businesses that need to process documents such as invoices, receipts, and manuals. The model is capable of handling multiple images of documents, up to four images at a resolution of 1k x 2k each, along with a long text prompt. The expected use is for tasks like summarization and Visual Question Answering (VQA). The model is also expected to have a significant advantage in throughput. <br>
Make sure to use the latest release of Llama.cpp.
BF16
llama-mtmd-cli -m NVIDIA-Nemotron-Nano-12B-v2-VL-BF16.gguf --mmproj mmproj-BF16.gguf -p "Describe this image" --image image.jpg -c 4096 --jinja
Q4_0
llama-mtmd-cli -m NVIDIA-Nemotron-Nano-12B-v2-VL-Q4_0.gguf --mmproj mmproj-BF16.gguf -p "Describe this image" --image image.jpg -c 4096 --jinja
BF16
llama-server --host 0.0.0.0 --port 8080 -hf tomlawrence/NVIDIA-Nemotron-Nano-12B-v2-VL-GGUF:BF16 --jinja -ngl 99 --ctx-size 16384
Q4_0
llama-server --host 0.0.0.0 --port 8080 -hf tomlawrence/NVIDIA-Nemotron-Nano-12B-v2-VL-GGUF:Q4_0 --jinja -ngl 99 --ctx-size 16384
Access using the Llama.cpp Web UI on the IP address and specified port (e.g. 8080)