Downloads · 30 days
11
14% of all-time downloads
BigHuggyD/jukofyork_miquplus-midnight-70b_exl2_7.0bpw
jukofyork_miquplus-midnight-70b_exl2_7.0bpw is a text generation model from BigHuggyD. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as other.
A creative writing model with 32k context, created by adding the essence of "what makes Midnight-Rose-70B-v2.0.3 different to Llama-2-70b-hf" onto miqu-1-70b.
Downloads · 30 days
11
14% of all-time downloads
All-time downloads
76
Public
Repo size
60.4 GB
Likes
0
Public
Click a slice to open those files.
.safetensors60.4 GB · 100%
From the Hugging Face model README
A creative writing model with 32k context, created by adding the essence of "what makes Midnight-Rose-70B-v2.0.3 different to Llama-2-70b-hf" onto miqu-1-70b.
Vicuna format is preferred:
USER: {prompt} ASSISTANT:
Mistral and Alpaca formats are also supported:
[INST] {prompt} [/INST]
### Instruction:
{prompt}
### Response:
miqu-1-70b-sf is a dequantized version of the miqu-1-70b model leaked from MistralAI. All miqu-derived models, including this merge, are suitable for non-commercial, personal use only.
The following YAML configuration was used to produce this model:
name: _miquplus-midnight-70b
merge_method: task_arithmetic
parameters:
normalize : false
weight: 1
models:
- model: meta-llama/Llama-2-70b-hf
- model: 152334H/miqu-1-70b-sf
- model: sophosympatheia/Midnight-Rose-70B-v2.0.3
base_model: meta-llama/Llama-2-70b-hf
dtype: float16
---
name: miquplus-midnight-70b
merge_method: linear
models:
- model: 152334H/miqu-1-70b-sf
parameters:
weight:
- filter: v_proj
value: [1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1]
- filter: o_proj
value: [1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1]
- filter: up_proj
value: [1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1]
- filter: gate_proj
value: [1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1]
- filter: down_proj
value: [1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1]
- value: 1
- model: _miquplus-midnight-70b
parameters:
weight:
- filter: v_proj
value: [0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0]
- filter: o_proj
value: [0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0]
- filter: up_proj
value: [0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0]
- filter: gate_proj
value: [0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0]
- filter: down_proj
value: [0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0]
- value: 0
base_model: 152334H/miqu-1-70b-sf
tokenizer_source: base
dtype: float16
NOTE: Run with mergekit-mega rather than mergekit as there are 2 documents in this file.
Created using Mergekit in two stages:
Midnight-Rose-70B-v2.0.3 - Llama-2-70b-hf) to miqu-1-70b-sf using task_arithmetic.v_proj, o_proj, up_proj, gate_proj and down_proj tensors were merged back into miqu-1-70b-sf.NOTE: After the second stage model is created the "donor" model can be deleted as it won't be needed again.
<details><summary>Click to see more details on the reasoning behind the merge</summary>q_proj and k_proj matrices intact:
see: https://adalkiran.github.io/llama-nuts-and-bolts/#model-diagram
and trying to blend the the q_proj and k_proj matrices of models with 100x different base RoPE frequencies is just going to contract and dilate their perceived "distance" / "time" in the text they write (so both are wrong). If you did want to insist on blending them, then the base RoPE frequency would need to be set to something in-between that is the "least wrong" for both base RoPE frequencies and the maximum context reduced accordingly...

For LLaMA2(Touvron et al., 2023b), the critical dimension dextra is 92. This implies that only the first 92 dimensions of the qt, ks vectors of LLaMA2 have seen the complete positional information during the pre-training phase and are adequately trained. In other words, the last 36 dimensions lack sufficient training, contributing to the extrapolation challenges seen in RoPE-based LLMs
see: Scaling Laws of RoPE-based Extrapolation
So even if we could try to map the 10k RoPE models embedding vectors to match the 1M RoPE models embedding vectors post-dot-product, the 1M RoPE models embedding vectors are likely fine-tuned to use these top 36 dimensions properly and have thus changed very significantly during continued pre-training.
The input_layernorm.weight, post_attention_layernorm.weight and norm.weight vectors all use LayerNorm, and:
weight – the learnable weights of the module of shape normalized_shapenormalized_shape when elementwise_affine is set to True. The values are initialized to 1.
So to combine these we would need to use the Geometric Mean for the task_arithmetic operation (and hence why the "donor" model is broken).
embed_tokens.weight and lm_head.weight tensors are not merged:The model still functions if these are merged into the second stage model, but empirically it seems to work better if we don't do this as keep the original miqu-1-70b-sf weights.