Downloads · 30 days
51
0% of all-time downloads
mzio/hedgehog-mistral_7b-alpaca_clean-smd_lora_1e_3
hedgehog-mistral_7b-alpaca_clean-smd_lora_1e_3 is a text generation model from mzio. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
This modelcard aims to be a base template for new models. It has been generated using this raw template.
Downloads · 30 days
51
0% of all-time downloads
All-time downloads
10.8K
Public
Parameters
7.3B
14.6 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors14.6 GB · 100%
From the Hugging Face model README
This modelcard aims to be a base template for new models. It has been generated using this raw template.
Configs
name: llama
model:
pretrained_model_name_or_path: 'mistralai/Mistral-7B-v0.1'
cache_dir: '/juice/scr/scr110/scr/nlp/data/neo/hub/'
return_dict: true
quantization: false
device_map: auto # null
low_cpu_mem_usage: true # false
torch_dtype: bfloat16
attn_implementation: eager # so we can load attention weights
rope_theta: 10000.0
attention:
attention_type: hedgehog_llama
feature_map: softmax_dim
feature_map_kwargs:
input_dim: 128
eps: 1e-12
# mlp: null # to set
fullspace: true
layer_idx: null # to set
learned_kernel: untied_head
learned_kernel_kwargs:
feature_dim: 128
skip_connection: false
bias: false
zero_init: false
tie_qk_kernels: false
train_qk: true
peft:
method: lora
kwargs:
r: 8 # 256
lora_alpha: 16 # 512
lora_dropout: 0.1 # 0.05
target_modules: ['self_attn.q_proj', 'self_attn.k_proj']
dataset:
name: alpaca_clean
dataset_config:
name: alpaca
path: yahma/alpaca-cleaned
chunk_size: 1024 # 2048
concat_data: true
cache_dir: '/u/scr/nlp/data/alpaca'
pretrained_model_config:
pretrained_model_name_or_path: 'mistralai/Mistral-7B-v0.1'
cache_dir: '/juice/scr/scr110/scr/nlp/data/neo/hub/'
preprocess_config: null
dataloader:
batch_size: 1
num_workers: 2
drop_last: false
pin_memory: true
optimizer:
optim: adamw_torch_fused
lr: 0.001
weight_decay: 0.0
lr_scheduler:
lr_scheduler_type: reduce_lr_on_plateau
mode: min
factor: 0.1
patience: 10
min_lr: 0.00001
trainer: # HuggingFace Trainer-like arguments
name: distill_attention
token_reduce: true
bottom_attention_only: false
reverse_kl: false
bf16: true
train_split: train
val_split: validation
num_train_epochs: 2
gradient_accumulation_steps: 8
seed: 42
batch_size: 1
load_best_model_at_end: true
greater_is_better: false
metric_for_best_model: distill/eval/loss
logging_steps: 100
evaluation_strategy: steps
max_steps: -1
eval_steps: 100
max_eval_batches: null
dataset:
name: alpaca_clean
dataset_config:
name: alpaca
path: yahma/alpaca-cleaned
chunk_size: 1024 # 2048
concat_data: true
cache_dir: '/u/scr/nlp/data/alpaca'
pretrained_model_config:
pretrained_model_name_or_path: 'mistralai/Mistral-7B-v0.1'
cache_dir: '/juice/scr/scr110/scr/nlp/data/neo/hub/'
preprocess_config: null
dataloader:
batch_size: 1
num_workers: 2
drop_last: false
pin_memory: true
optimizer:
optim: adamw_torch_fused
lr: 1e-4
weight_decay: 0.0
lr_scheduler:
lr_scheduler_type: reduce_lr_on_plateau
mode: min
factor: 0.1
patience: 10
min_lr: 0.00001
trainer: # HuggingFace Trainer-like arguments
name: default
bf16: true
train_split: train
val_split: validation
num_train_epochs: 2
gradient_accumulation_steps: 8
seed: 42
batch_size: 1
load_best_model_at_end: true
greater_is_better: false
metric_for_best_model: eval/loss # eval/rouge/geometric_mean
logging_steps: 100
evaluation_strategy: steps
max_steps: -1
eval_steps: 100
max_eval_batches: null
finetune:
method: lora
kwargs:
r: 8
lora_alpha: 16 # 32
lora_dropout: 0 # 0.05
target_modules: ['q_proj', 'k_proj', 'v_proj', 'o_proj']
[More Information Needed]
[More Information Needed]
[More Information Needed]
[More Information Needed]
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
Use the code below to get started with the model.
[More Information Needed]
[More Information Needed]
[More Information Needed]
[More Information Needed]
[More Information Needed]
[More Information Needed]
[More Information Needed]
[More Information Needed]
[More Information Needed]
Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).
[More Information Needed]
[More Information Needed]
[More Information Needed]
[More Information Needed]
BibTeX:
[More Information Needed]
APA:
[More Information Needed]
[More Information Needed]
[More Information Needed]
[More Information Needed]
[More Information Needed]