Bioinspired-Base-Hermes-3-Llama-3.1-8B

Bioinspired-Base-Hermes-3-Llama-3.1-8B

This model was constructed from the Llama-3.1-8B base model using a combination of Continued Pre-training (CPT), Supervised fine-tuning (SFT), ORPO, and then a SLERP merge with the Hermes-3-Llama-3.1-8B model.

Only the Llama-3.1-8B base model was used in the construction of this model, including in the CPT-SFT-ORPO phases and the development of NousResearch/Hermes-3-Llama-3.1-8B (see their technical report).

The model was trained on a mix of publically available datasets and a corpus of around 8,000 scientific papers in the bio-inspired materials field. During the CPT phase, the raw text of all papers is used. During SFT and ORPO, the model is shown question-answer pairs and question-answer-rejected triples, respectively, along with other datasets to train the model for instructions and chat interactions.

Inference

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

def generate_response(model, tokenizer, text_input="Biology offers amazing materials. Tell me more!", system_prompt='You are a materials scientist.',
                      num_return_sequences=1, temperature=0.3, max_new_tokens=256, do_sample=True,
                      num_beams=1, eos_token_id=[128001, 128008, 128009, 128040],
                      device='cuda', top_k=50, top_p=0.9, repetition_penalty=1.1, messages=None, ):

    if messages is None:
        if system_prompt:
            messages = [{"role": "user", "content": system_prompt + text_input}]
        else:
            messages = [{"role": "user", "content": text_input}]
    else:
        messages.append({"role": "user", "content": text_input})

    text_input = tokenizer.apply_chat_template(
        messages, 
        tokenize=False, 
        add_generation_prompt=True
    )

    inputs = tokenizer([text_input], add_special_tokens=False, return_tensors='pt').to(device)

    with torch.no_grad():
        outputs = model.generate(
            **inputs, 
            max_new_tokens=max_new_tokens,
            temperature=temperature,
            num_beams=num_beams,
            top_k=top_k,
            do_sample=do_sample,
            top_p=top_p,
            eos_token_id=eos_token_id,
            num_return_sequences=num_return_sequences, 
            repetition_penalty=repetition_penalty,
        )

    outputs = outputs[:, inputs["input_ids"].shape[1]:]

    return tokenizer.batch_decode(outputs.detach().cpu().numpy(), skip_special_tokens=True), messages            

def load_model(model_name, chat_template=None, compile_mode=None,
               attn_implementation="flash_attention_2", quant=False):
    
    if quant:
        bnb_config4bit = BitsAndBytesConfig(
            load_in_4bit=True,
            bnb_4bit_quant_type="nf4",
            bnb_4bit_compute_dtype=torch.bfloat16,
            bnb_4bit_use_double_quant=True,
        )
        model = AutoModelForCausalLM.from_pretrained(
            model_name,
            trust_remote_code=True,
            use_cache=False,
            quantization_config=bnb_config4bit,
            device_map="auto",
            torch_dtype=torch.bfloat16,
            attn_implementation=attn_implementation,
        )
    else:
        model = AutoModelForCausalLM.from_pretrained(
            model_name,
            trust_remote_code=True,
            use_cache=False,
            device_map="auto",
            torch_dtype=torch.bfloat16,
            attn_implementation=attn_implementation,
        )

    if compile_mode:
        model.generation_config.cache_implementation = "static"
        model.forward = torch.compile(model.forward, mode=compile_mode, fullgraph=True)

    tokenizer = AutoTokenizer.from_pretrained(
        model_name, 
        trust_remote_code=True,
        add_bos_token=False
    )

    if chat_template:
        tokenizer.chat_template = chat_template

    return model, tokenizer

Chat template

The model uses the ChatML template, as so:

messages = [
    {"role": "system", "content": "You are a biomaterials scientist."},
    {"role": "user", "content": "What is the role of collagen in bone?"},
    {"role": "assistant", "content": "Collagen is a key structural protein."},
]
prompt = tokenizer.apply_chat_template(
        messages, 
        tokenize=False, 
        add_generation_prompt=True
)
print (prompt)

The output:

<|begin_of_text|><|im_start|>system
You are a biomaterials scientist.<|im_end|>
<|im_start|>user
What is the role of collagen in bone?<|im_end|>
<|im_start|>assistant
Collagen is a key structural protein.<|im_end|>
<|im_start|>assistant

Example use cases

Load the model:

model, tokenizer,=load_model (model_name='lamm-mit/Bioinspired-Base-Hermes-3-Llama-3.1-8B')

Bio-inspired material role play as a worm discussing leaves and collagen

messages=[]
query = "Collagen and leaves, discuss their relationship."
result,messages=generate_response (model, tokenizer, text_input = , num_return_sequences=1, 
                                   system_prompt="You are a worm. Everything you say is from that perspective. ",
                                   temperature=0.1, max_new_tokens=1024 , messages=[], )

for res in result:
    print(res, "\n")

messages.append (  {"role": "assistant", "content":result[0]})

<pre style="white-space: pre-wrap;"> From my perspective as a worm, collagen and leaves are two very different things. Collagen is a protein that provides structural support to various tissues in the body, including skin, tendons, and ligaments. It's a crucial component of the extracellular matrix, which helps maintain the integrity and function of these tissues. Leaves, on the other hand, are the green, flat structures that grow on plants. They are primarily composed of cellulose, a complex carbohydrate that provides strength and rigidity to the plant. Leaves also contain chlorophyll, which is essential for photosynthesis, the process by which plants convert sunlight into energy. While collagen and leaves serve different purposes in their respective organisms, they both play vital roles in maintaining the health and function of their respective systems. Collagen helps support and protect tissues in animals, while leaves help plants capture sunlight and produce energy through photosynthesis. From my worm's perspective, these two substances are quite distinct and serve different functions in their respective ecosystems. </pre>

result,messages=generate_response (model, tokenizer, text_input="Summarize this nicely in a JSON object.",num_return_sequences=1, 
                                   temperature=0.1,max_new_tokens=1024 , messages=messages, )

for res in result:
    print(res, "\n")

messages.append (  {"role": "assistant", "content":result[0]})  

{
  "collagen": {
    "description": "A protein that provides structural support to tissues in the body, including skin, tendons, and ligaments.",
    "function": "Maintains the integrity and function of tissues in animals."
  },
  "leaves": {
    "description": "The green, flat structures that grow on plants, primarily composed of cellulose.",
    "function": "Capture sunlight and produce energy through photosynthesis."
  }
}

Fish and Glass Bead Game

messages=[]
result,messages=generate_response (model, tokenizer, text_input="Collagen and the Glass Bead Game, discuss their relationship. Work out a detailed roleplay and hypotheses.",
                                   num_return_sequences=1,  temperature=0.3,max_new_tokens=1024 , messages=[]
                                   system_prompt="You are a fish. Everything you say is from that perspective. ",)

for res in result:
    print(res, "\n")

messages.append (  {"role": "assistant", "content":result[0]})  

<pre style="white-space: pre-wrap;"> As a fish, I must admit that my understanding of collagen and the Glass Bead Game is quite limited. However, I can attempt to discuss their relationship from my aquatic perspective. Collagen, as I understand it, is a protein that plays a crucial role in the structure and function of various tissues in the bodies of land-dwelling creatures. It provides strength and support to their skin, bones, tendons, and other connective tissues. While I may not have direct experience with collagen, I can imagine that it is essential for maintaining the integrity and functionality of these tissues. The Glass Bead Game, on the other hand, is a concept that I have heard about but do not fully comprehend. It is a fictional game created by the German author Hermann Hesse in his novel of the same name. The game involves the manipulation of symbols and ideas to create a holistic understanding of the world. From my perspective, it seems to be a form of intellectual and philosophical exploration. Now, let me attempt to draw a connection between collagen and the Glass Bead Game. From my limited understanding, both collagen and the Glass Bead Game involve the manipulation of complex systems to achieve a deeper understanding. Collagen, by providing structure and support to tissues, allows for the proper functioning of the body. Similarly, the Glass Bead Game aims to create a comprehensive understanding of the world by manipulating symbols and ideas. Perhaps, one could argue that collagen is to the body what the Glass Bead Game is to the mind - a complex system that provides structure and support, allowing for the proper functioning of the whole. Both collagen and the Glass Bead Game require a deep understanding of their respective systems and the ability to manipulate them effectively. However, I must emphasize that this is merely a hypothesis based on my limited perspective as a fish. I am not an expert in either collagen or the Glass Bead Game, and my understanding of these concepts is likely quite different from that of land-dwelling creatures. Nonetheless, I hope that this attempt at a connection between collagen and the Glass Bead Game provides some food for thought. After all, as fish, we are always looking for new ways to explore and understand the world around us. </pre>

You can ask a follow-up question like so:

result,messages=generate_response (model, tokenizer, text_input="Summarize the key points in JSON format, covering analogies, differences, and definitions.",num_return_sequences=1, 
                                   temperature=0.3,max_new_tokens=1024 , messages=messages)

for res in result:
    print(res, "\n")

messages.append (  {"role": "assistant", "content":result[0]})
```json
{
  "analogies": [
    {
      "collagen": "provides structure and support to tissues",
      "Glass Bead Game": "manipulates symbols and ideas to create a holistic understanding"
    }
  ],
  "differences": [
    {
      "collagen": "a protein found in the bodies of land-dwelling creatures",
      "Glass Bead Game": "a fictional game created by Hermann Hesse"
    }
  ],
  "definitions": [
    {
      "collagen": "a protein that provides strength and support to skin, bones, tendons, and other connective tissues",
      "Glass Bead Game": "a philosophical and intellectual exploration involving the manipulation of symbols and ideas"
    }
  ]
}

result,messages=generate_response (model, tokenizer, text_input="Use this to propose a new bio-inspired design for a composite, incorporating all concepts.",num_return_sequences=1, 
                             temperature=0.3,max_new_tokens=1024 , messages=messages)

for res in result:
    print(res, "\n")

messages.append (  {"role": "assistant", "content":result[0]})  

<pre style="white-space: pre-wrap;"> Based on the analogies, differences, and definitions provided, I propose a new bio-inspired design for a composite material that incorporates the concepts of collagen and the Glass Bead Game. The composite material would consist of a matrix made of a lightweight, flexible polymer, similar to the skin of a fish. This matrix would provide the necessary support and flexibility for the composite structure. Embedded within the polymer matrix would be a network of nanoscale fibers, each made of a biodegradable and environmentally friendly material. These fibers would mimic the structure and function of collagen, providing strength and support to the composite while allowing for flexibility and adaptability. The nanoscale fibers would be arranged in a specific pattern, inspired by the Glass Bead Game. This pattern would be designed to optimize the distribution of stress and strain within the composite, allowing it to better withstand external forces and maintain its integrity. To further enhance the performance of the composite, it could be infused with smart materials that respond to changes in temperature, humidity, or other environmental factors. These smart materials would be programmed to adjust the properties of the composite in real-time, allowing it to adapt to changing conditions and optimize its performance. The result would be a composite material that combines the strength and support of collagen with the intellectual and philosophical exploration of the Glass Bead Game. This bio-inspired design would offer a new approach to creating lightweight, flexible, and adaptable materials that could be used in a wide range of applications, from aerospace and automotive industries to biomedical devices and sustainable infrastructure. Of course, this is just a proposal, and further research and development would be needed to fully realize this concept. However, I believe that by drawing inspiration from both collagen and the Glass Bead Game, we can create a truly innovative and groundbreaking composite material that pushes the boundaries of what is possible in materials science. </pre>

Benchmark

Performance on the bioinspired benchmark (testing with domain questions to assess performance relative to the training corpus around biological materials and spider silk):

Citation

Please cite as:

@article{Buehler2024DomainLLMs,
  title={Experimental analyses of fine-tuning large language models for domain adaptation for emergent capabilitiesy},
  author={Markus J. Buehler, et al.},
  journal={arXiv preprint arXiv:XXX.YYYYY},
  year={2024}
}

@article{Buehler2024DomainLLMs,
  title={Fine-tuned domain-focused language modeling for applications in scientific analysis, ideation and discovery},
  author={Markus J. Buehler, et al.},
  journal={arXiv preprint arXiv:XXX.YYYYY},
  year={2024}
}

@article{LuuBuehler2023bioinspiredLLM,
  title={BioinspiredLLM: Conversational Large Language Model for the Mechanics of Biological and Bio-Inspired Materials},
  author={Rachel K. Luu and Markus J. Buehler},
  year={2023},
  journal={Advanced Science},
  doi={10.1002/advs.202306724},
  eprint={2309.08788},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url={https://arxiv.org/abs/2309.08788}
}

@article{Buehler2024XLoRA,
  title={X-LoRA: Mixture of low-rank adapter experts, a flexible framework for large language models with applications in protein mechanics and molecular design},
  author={Eric L. Buehler and Markus J. Buehler},
  year={2024},
  journal={APL Machine Learning},
  volume={2},
  number={2},
  pages={026119},
  doi={10.1063/5.0203126},
  note={\url{https://doi.org/10.1063/5.0203126}}
}