Downloads · 30 days
5.7K
8% of all-time downloads
NexesQuants/MIstral-QUantized-70b_Miqu-1-70b-iMat.GGUF
MIstral-QUantized-70b_Miqu-1-70b-iMat.GGUF is a machine learning model from NexesQuants. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Miqu 1 70b : a leak of Mistral Medium Alpha. Credit for this model goes to the Mistral AI company.
Downloads · 30 days
5.7K
8% of all-time downloads
All-time downloads
71.7K
Public
Repo size
837 GB
Likes
70
Public
Click a slice to open those files.
.gguf691 GB · 100%
From the Hugging Face model README
Miqu 1 70b : a leak of Mistral Medium Alpha. Credit for this model goes to the Mistral AI company.
Miqu is probably the best 70b model I could ever play with, especially as a French speaker. Smart like a good Llama 2 70b finetune, no overfitting, little censorship, reasonable alignement, and even a sufficient mastery of French language to have a proper chat in French without feeling like spreaking to a broken bot.


Requantizations with iMatrix (better quality than without) of a Q5_K_M quant of a trending 70b model without better quant/fp16 available, this through a Q8_0 intermediary step.
Miqudev provided Q5_K_M, Q4_K_M, and Q2_K on this page : https://huggingface.co/miqudev/miqu-1-70b
Here, you will find the following quants :
Full offload possible on 48GB VRAM with a huge context size :
Full offload possible on 36GB VRAM with a variable context size (up to 7168 with Q3_K_M, for example)
Full offload possible on 24GB VRAM with a decent context size.
Full offload possible on 16GB VRAM with a decent context size.
Bonus : a Kobold.CPP Frankenstein which reads IQ3_XXS models and is not affected by the Kobold.CPP 1.56/1.57 slowdown at the cost of an absent Mixtral fix. https://github.com/Nexesenex/kobold.cpp/releases/tag/v1.57_b2030 Now supperseded with another KCPP-F, with 13 different KV cache quantization lebel to chose from : https://github.com/Nexesenex/kobold.cpp/releases
Miku 70b has a theta of 1,000,000, like CodeLlama, and not 10,000, like Llama 2 models usually have. That feature singularizes it to my knowledge to ALL Llama 2 models, beside Codellamas which also have a theta of 1,000,000..
-> So, no Alpha or Rope Base Frequency change is needed up to its base 32k context, if it works as intended. And if it does, no linear/yarn rope is necessary either to reach the base 32k context.
BUT Miqu is NOT a CodeLlama 70b (released only a few days after Miqu 70b), because :
So, CodeLlama 70b is nerfed like the other CodeLlama in general benchmarks terms, while Miku is matching a FINETUNED Llama-2 expectations.
Benchs I made with the original Q2_K quant of Miku 70b, most probably made from an initial FP16 and published by Miqudev :

A graph, courtesy of Ipechman, with the TQA of WinterGooddess 32k at 39.65728274 and not 20.
Data :
Miqudev's original Q2_K quant :
miqu-1-70b.q2_K.gguf,-,Hellaswag,86.5,,1000,2024-01-29 01:40:00,,70b,Mistral_Medium,32768,,,GGUF,miqudev,miqudev,
miqu-1-70b.q2_K.gguf,-,Hellaswag,86,,2000,2024-01-29 01:40:00,,70b,Mistral_Medium,32768,,,GGUF,miqudev,miqudev,
miqu-1-70b.q2_K.gguf,-,Hellaswag_Bin,83.7,,1000,2024-01-29 01:40:00,,70b,Mistral_Medium,32768,,,GGUF,miqudev,miqudev,
miqu-1-70b.q2_K.gguf,-,Hellaswag_Bin,84,,2000,2024-01-29 01:40:00,,70b,Mistral_Medium,32768,,,GGUF,miqudev,miqudev,
miqu-1-70b.q2_K.gguf,-,Arc-Challenge,56.18729097,,299,2024-01-29 05:40:00,,70b,Mistral_Medium,32768,,,GGUF,miqudev,miqudev,
miqu-1-70b.q2_K.gguf,-,Arc-Easy,75.78947368,,570,2024-01-29 05:40:00,,70b,Mistral_Medium,32768,,,GGUF,miqudev,miqudev,
miqu-1-70b.q2_K.gguf,-,MMLU,46.96485623,,313,2024-01-29 05:40:00,,70b,Mistral_Medium,32768,,,GGUF,miqudev,miqudev,
miqu-1-70b.q2_K.gguf,-,Thruthful-QA,41.49326805,,817,2024-01-29 05:40:00,,70b,Mistral_Medium,32768,,,GGUF,miqudev,miqude
miqu-1-70b.q2_K.gguf,-,Winogrande,78.2163,,1267,2024-01-29 05:40:00,,70b,Mistral_Medium,32768,,,GGUF,miqudev,miqudev,
miqu-1-70b.q2_K.gguf,-,wikitext,4.6476,512,512,2024-01-29 01:40:00,RBF1000000,70b,Mistral_Medium,32768,,,GGUF,miqudev,miqudev,81
miqu-1-70b.q2_K.gguf,-,wikitext,4.3063,512,512,2024-01-29 01:40:00,RBF1000000,70b,Mistral_Medium,32768,,,GGUF,miqudev,miqudev,655
miqu-1-70b.q2_K.gguf,-,wikitext,3.8606,4096,4096,2024-01-29 01:40:00,,70b,Mistral_Medium,32768,,,GGUF,miqudev,miqudev,655
miqu-1-70b.q2_K.gguf,-,wikitext,3.6864,6144,6144,2024-01-29 01:40:00,,70b,Mistral_Medium,32768,,,GGUF,miqudev,miqudev,655
miqu-1-70b.q2_K.gguf,-,wikitext,4.6576,512,512,2024-01-29 01:40:00,RBF500000,70b,Mistral_Medium,32768,,,GGUF,miqudev,miqudev,81
miqu-1-70b.q2_K.gguf,-,wikitext,4.7762,512,512,2024-01-29 01:40:00,RBF100000,70b,Mistral_Medium,32768,,,GGUF,miqudev,miqudev,81
miqu-1-70b.q2_K.gguf,-,wikitext,4.8766,512,512,2024-01-29 01:40:00,RBF50000,70b,Mistral_Medium,32768,,,GGUF,miqudev,miqudev,81
miqu-1-70b.q2_K.gguf,-,wikitext,5.3367,512,512,2024-01-29 01:40:00,RBF10000,70b,Mistral_Medium,32768,,,GGUF,miqudev,miqudev,81
Benchs I made with the Q2_K I quantized from Miqudev's Q5_K_M with an intermediary Q8_0 step, and an iMatrix of 12800 tokens from wiki.train.raw :
Notice the perplexity drop brought by the iMatrix despite the requant.
Benchs I made with the Q3_K_M I quantized from Miqudev's Q5_K_M with an intermediary Q8_0 step, and an iMatrix of 12800 tokens from wiki.train.raw :
And now, the IQ3_XXS, new SOTA 3 bits quant from LlamaCPP, that I made in the same way :
Meanwhile, CodeLlama 70b Q2_K benches as such, to compare with Miqu 70B Q2_K originally quantized from FP16 by Miqudev :
And, for information, a comparable base Llama 2 70b finetuned by NousResearch for 32k context (Yarn) :
This yarn version performs closely to Llama 2 70b (but with 32k max context), and.. Much more poorly than Miqu 70b.
Also, for information, another requant from a Q4_K_S orphan of a 32k finetune of Sao10K's WinterGoddess 70b At Linear rope 2.5 (for 10k context) :
Draw your own conclusions as well !
New quants IQ1 :
V3 :
V5 :
CUSTOM QUANTS :
New quantizations strategies to bundle optimally the last tensor quantizations of Ikawrakow (LlamaCPP b2404 and beyond) and reach the hightest possible quality/size ratio. A work in progress. These quants strategies will be updated as soon as higher IQ tensor quants are available for the pertinent tensors.
Here's one new quant strategy, currently labelled IQ1_FSR, a 2% smaller quant than .IQ1_S v3/v5 with a quality slightly above IQ1_S_v3 but quite a bit lower than IQ1_S_v5 :
Second new quant strategy, currently labelled IQ1_PS, a 0.5% bigger quant than .IQ1_S v3/v5 with a quality a bit above with IQ1_S_v5 :
And 4 bigger IQ1 quants, up to 1.92 bpw :
IQ1_NS :
IQ1_MS :
IQ1_ES :
IQ1_SR :