Grey-12b

<img src="https://cdn-uploads.huggingface.co/production/uploads/66dcee3321f901b049f48002/jWXtbknuetFdz5fkFn-ey.png" width="800"/>

Grey-12b

Grey-12b is a merged language model created by combining multiple models using the della_linear merge method, with Aether-12b as the base model.

Model Details 📊

• Developed by: AIXON Lab
• Model type: Merged Causal Language Model
• Language(s): English (primarily), may support other languages
• License: apache-2.0
• Repository: https://huggingface.co/aixonlab/Grey-12b

Model Architecture 🏗️

• Base model: aixonlab/Aether-12b
• Parameter count: ~12 billion
• Architecture specifics: Transformer-based language model
• Merge method: della_linear

Merged Models

1. VAGOsolutions/SauerkrautLM-Nemo-12b-Instruct
   • Weight: 0.33
   • Density: 0.4
2. cognitivecomputations/dolphin-2.9.3-mistral-nemo-12b
   • Weight: 0.77
   • Density: 0.8

Technical Specifications

• Dtype: float16
• Tokenizer source: base (aixonlab/Aether-12b)
• Merge parameters:
  • Epsilon: 0.05
  • Lambda: 1

Intended Use 🎯

As an advanced language model for various natural language processing tasks, including but not limited to text generation, question-answering, and analysis.

Ethical Considerations 🤔

As a merged model based on multiple sources, Grey-12b may inherit biases and limitations from its constituent models. Users should be aware of potential biases in generated content and use the model responsibly.

Performance and Evaluation

Performance metrics and evaluation results for Grey-12b are yet to be determined. Users are encouraged to contribute their findings and benchmarks.

Limitations and Biases

The model may exhibit biases present in its training data and constituent models. It's crucial to critically evaluate the model's outputs and use them in conjunction with human judgment.

Additional Information

For more details on the base model and constituent models, please refer to their respective model cards and documentation.

Acknowledgments 🙏

We acknowledge the contributions of:

• VAGOsolutions for the SauerkrautLM-Nemo-12b-Instruct model
• Cognitive Computations for the dolphin-2.9.3-mistral-nemo-12b model

How to Use

from transformers import AutoTokenizer, AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("aixonlab/Grey-12b")
tokenizer = AutoTokenizer.from_pretrained("aixonlab/Grey-12b")

prompt = "Once upon a time"
input_ids = tokenizer(prompt, return_tensors="pt").input_ids

generated_ids = model.generate(input_ids, max_length=100)
generated_text = tokenizer.decode(generated_ids, skip_special_tokens=True)
print(generated_text)