Downloads · 30 days
17
28% of all-time downloads
Fimbulvetr-40
ClaudioItaly/Fimbulvetr-40
Fimbulvetr-40 is a text generation model from ClaudioItaly. Use it when you need the model to write or continue text. It is set up for transformers.
Changing the value of kvcount from 34 to 40 indicates an increase in the number of key-value pairs in the model. These key-value pairs are mainly used to represent attention information within neural networks, particu…
All-time downloads
60
Public
Parameters
10.7B
21.5 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors21.5 GB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- Model type
- llama
- Access
- Public
- Created
- Sep 17, 2024
- Updated
- Sep 17, 2024
- SHA
- 2653389c
Try a prompt
Base models
Model card
From the Hugging Face model README
Changing the value of kv_count from 34 to 40 indicates an increase in the number of key-value pairs in the model. These key-value pairs are mainly used to represent attention information within neural networks, particularly in Transformer-type models such as LLaMA.
merge
This is a merge of pre-trained language models created using mergekit.
Merge Details
Merge Method
This model was merged using the passthrough merge method using Sao10K/Fimbulvetr-11B-v2 as a base.
Models Merged
The following models were included in the merge:
Configuration
The following YAML configuration was used to produce this model:
base_model: Sao10K/Fimbulvetr-11B-v2
merge_method: passthrough
dtype: float16
parameters:
normalize: true
slices:
- sources:
- model: Sao10K/Fimbulvetr-11B-v2
layer_range: [0, 48] # Assumi che il modello abbia 48 layer
densify:
- linear
- "rope:alpha=8192/4096" # Estende il contesto a 8192
tokens:
- source: Sao10K/Fimbulvetr-11B-v2
mode: stretch
- Task
- Text Generation
- Library
- transformers
- Type
- llama
- Created
- Sep 17, 2024
- Updated
- Sep 17, 2024