nano-mistral

Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->

Model Details

Model Description

<!-- Provide a longer summary of what this model is. -->

This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.

• Developed by: me
• Model type: Mistral
• Language(s) (NLP): en
• License: apache

Uses

general web text completions at extremely low resource use

Out-of-Scope Use

not an instruct model

Bias, Risks, and Limitations

trained on web text, though filtered no guarantees theres not toxic stuff in there

How to Get Started with the Model

Use the code below to get started with the model.

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("crumb/nano-mistral")
tokenizer = AutoTokenizer.from_pretrained("crumb/nano-mistral")

inputs = tokenizer(["Once upon a time,"], return_tensors="pt")
inputs = {k:v.to(model.device) for k,v in dict(inputs).items()}
outputs = model.generate(inputs, max_new_tokens=128, temperature=0.7, top_k=20, do_sample=True)
outputs = tokenizer.batch_decode(outputs)
for i in outputs:
  print(i)

Training Details

Training Data

crumb/askmistral-pile-2-15

Training Procedure

Preprocessing [optional]

[More Information Needed]

Training Hyperparameters

• Training regime: bf16 non-mixed precision <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->

Speeds, Sizes, Times [optional]

train_runtime 62541.9424

train_samples_per_second 26.557

[More Information Needed]

Evaluation

Testing Data, Factors & Metrics

Testing Data

held out set of crumb/askmistral-pile-2-15

Factors

<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->

[More Information Needed]

Metrics

<!-- These are the evaluation metrics being used, ideally with a description of why. -->

open llm leaderboard eval datasets and settings

Results

OpenLLM Leaderboard Mean Score + Stderr: (29.30, 0.42)

MMLU

value, stderr = (0.253980701754386, 0.004428598058450528)

Summary

Model Examination [optional]

its ok

Environmental Impact

<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

• Hardware Type: A6000
• Hours used: 34.74
• Cloud Provider: n/a
• Compute Region iowa
• Carbon Emitted: 4.5kg CO2eq.

Technical Specifications [optional]

Model Architecture and Objective

mistral, causal language modelling

Compute Infrastructure

what

Hardware

lambda vector 2xA6000

Software

huggingface transformers / pytorch / custom trainer

Citation [optional]

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

BibTeX:

[More Information Needed]

APA:

[More Information Needed]

Glossary [optional]

<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->

[More Information Needed]

More Information [optional]

[More Information Needed]

Model Card Authors [optional]

[More Information Needed]

Model Card Contact

[More Information Needed]