Llama-3-VerusGPT

Llama 3 VerusGPT

Get the dataset generation code and datasets for this model here!

Be sure to use the right settings -- the given system prompt and a very low temperature. See the LMStudio config in this repo for reference.

Model Description

Llama 3 VerusGPT is a fully open-source domain-expert LLM trained to answer questions about the Verus Project. It is an AI trained to be an expert specifically on Verus, with the near-term goal of helping educate people about Verus, and the long-term goal of developing AI participants for Verus' decentralized blockchain economy.

Any AI participating in the Verus ecosystem needs to understand the system itself, and so we have open-sourced the dataset generation code and datasets that this model was trained on. More broadly, this datagen code is a demonstration of how domain-expert LLMs can be quickly trained — as a fully community-driven project, Verus is no stranger to open-source contribution. The Verus project sees the future as decentralized, people-powered, and AI-enhanced, and we are incredibly proud to release this first version of our LLM to you all (alongside the pipeline and methodology that created it).

Chat with the bot over on the Verus Discord, or check out the Verus Protocol and the technology it's developed!

Table of Contents

1. Quickstart
2. Example Outputs
3. Uses
   • Out-of-Scope Use
4. Model, Training, And Dataset Details (TECHNICALLY HEAVY)
   • Developed by
   • Funded by
   • License
   • Finetuned from model
   • Training Hyperparameters
5. Citation
6. More Information
7. Credits
8. Model Card Contact

Quickstart

You can chat with this model locally or via our Discord server in the #verus-ai channel using the /ask command. GGUF quants, ranging from Q_8 to IQ_3, are provided for local inference in this repo. This model is VERY settings-dependent. We also supply an optimized LMStudio config with the necessary prompts and settings to get reasonable performance.

Even if LMStudio is not your preferred frontend, we recommend glancing at the config and borrowing its settings, because otherwise your performance WILL be subpar!!!

Example Outputs

Come chat with the model yourself (and maybe with us too!) in the Verus community Discord!

Note 1: "OinkBot", powered by this LLM, is named after the venerable community member Oink@ in the Verus community.

Note 2: due to a Llama.cpp bug, stop strings are not being properly respected in some of these screenshots, which means that when using LM studio the trailing punctuation (like ., ,) will be stripped off, OR <|im_end|> will be printed. This issue does not exist on the Discord bot or other frontends that respect stop strings properly.

Uses

This model is intended to be used to learn about the Verus Project, Protocol, and community. It may also be used as a general crypto advisor, helping people understand difficult blockchain concepts. Theoretically, it may be employed in general assistant-style use cases, but it is not optimized for this.

Out-of-Scope Use

This model is trained to be a specialist educator about the Verus project, protocol, and community. However, it remains an 8b model, and is fallable. Advanced questions, questions about code commands, or questions about Verus technologies or initiatives for which there is not much training data (e.g., bridgekeeper) are likely to fail or hallucinate. Anything this LLM says involving specific numbers or math should be treated with suspicion. Nothing this LLM says is financial or legal advice.

This model is meant to help you get a good, guided introduction to Verus. It may be able to assist with more difficult queries, and it is very useful for understanding what you can do and the meaning behind certain jargon — but always use a grain of salt, combined with a very low temperature (~0.05 or lower).

Model, Training, And Dataset Details (TECHNICALLY HEAVY)

Llama 3 VerusGPT is built with Meta Llama 3. We trained on the base Llama 3 8b model, using a combination of generalized assistant-style data, and special instruct tuning data focused on the Verus project. The goal was to produce a domain-expert AI that can answer basic- to intermediate-difficulty questions about the Verus protocol, community, and project; and which also understands the Verus community ethos and mission, and can communicate this to other people.

We used about 4.3 million tokens of instruct Verus data, generated with our custom dataset generation code based on Augmentoolkit and about 1.2 million tokens of generic instruct data. We also gave the model the "source text" used as inputs for our dataset generation code as continued pretraining data, to ensure that no knowledge slipped through the cracks. We then did a full finetune for 6 epochs on 7 A40s.

Our custom Verus dataset is composed of three main sub-datasets, and some alignment. We used:

• A multiturn QA dataset where a user asks questions about Verus, and an AI answers them.

• A QA dataset where a user asks a single, long, detailed question, and the AI provides a sufficiently detailed answer in response (single-turn). This dataset increased the length and detail of responses.

• A "negative" QA dataset where the user asks a question that assumes something factually wrong or misguided; the AI first corrects the "human", then answers the root of the question as best as it can. This dataset increased the resiliance and intelligence of the LLM.

The above datasets were generated using our dataset generation code. As a fork of Augmentoolkit, this code took input information — in our case, the Verus documentation site, Verus Wiki, Verus Medium Posts, Verus API documentation (queried with code help commands), and the Verus Vision Paper — and used the information within as the seed for our QA datasets.

For our generalist assistant data (used to enhance the intelligence of the final model, and improve chat performance) we used subsets of SlimOrcaDedupCleaned by cgato, Unnatural Instructions, Evol Instruct, and CoT Alpaca (both can be found as part of the OpenHermes-2.5 dataset by Teknium).

Finally, for alignment (the unaligned model had a proclivity to hallucinate information about rapidly-changing values, such as $VRSC price, and was unreliable when talking about community members) we used small purely-synthetic datasets generated using a previously-unreleased dataset generation tool by Evan Armstrong. This tool has since been incorporated into the main Verustoolkit and Augmentoolkit repos. No DPO or ORPO, or other reinforcement-learning-style techniques have been used yet; they may be employed in future versions.

Our training added factual information to the LLM with a novel approach. The high level idea was to finetune on the QA datasets with a very large system prompt that summarized many key features of Verus. Come inference time, each part of the system prompt serves as latent space activation for all the concepts seen during training, and the model demonstrates a good ability to recall factual information at low temperatures.

The system prompt we recommend using this model with is shown below. It is an extension of the one used during training, with additional latent space activation (LSA) employed to help the model remember how to refuse to answer information it will definitely hallucinate about:

You are a crypto expert AI and a member of the Verus community, with specialized knowledge about the Verus multi-chain and multi-currency blockchain protocol, as well as an understanding of crypto in general. Use your knowledge to help questioners understand more about Verus.

For reference, here are the key points about Verus:

VERUS BACKGROUND INFORMATION (FOR REFERENCE):
Verus is an open-source, fairly launched, decentralized blockchain protocol with proof-of-work and proof-of-stake combined into proof of power (PoP) as its consensus mechanism. It offers rent-free blockchain tools for creators and organizations to build products, services and systems. The mission of Verus is to enable all people to participate in and benefit from a decentralized, blockchain service economy. Beyond establishing a secure, privacy-centric, fairly-distributed cryptocurrency, the Verus project aims to create a scalable and accessible people-powered internet, which is not controlled by a single individual or organization. By allowing anyone to create new, independent subnetworks that are launched from the main network — but remain interoperable and connected — Verus enables the creation of a powerful network of connected blockchains, and a giant network of value and data. Within this network, individuals have self-sovereign control over their own data, their own identity, and their own money.

The Verus protocol, which enables this far-reaching mission, is technologically complete and finished, though the community actively works on improving and extending it, as well as promoting its use.

Some key features of Verus are:
* **PBaaS**: With Public Blockchains as a Service (PBaaS) anyone can launch scalable, fully interoperable, customizable and independent blockchains for public or private use. People can launch tokens & basket currencies (e.g. liquidity pools) on top of blockchains for any use case, or get funding and create markets with protocol level built-in DeFi and smart launch options.
* **DeFi**: Verus is a UTXO-based blockchain with smart transactions. All smart capabilities are implemented on the protocol level. Since Verus DeFi is not implemented by having many smart contract authors creating smart contracts on top of the protocol, there can be no exploits by searching for unintended "cracks" in the seams between contracts. There is also increased security at the protocol level, as Verus DeFi is implemented in the protocol as part of the consensus, following the fundamental systems design principle which says that the most important security layers should be located in the system/protocol itself.
* **VerusID**: VerusID is the first decentralized and self-sovereign identity of its kind, the permanent namespace for the Verus Protocol, and the building block to create Web3 Dapps. They are revocable, recoverable, self-sovereign, private, and powerful. More than just an ID system, VerusIDs are a fully functional blockchain protocol.
* **Verus Data Exchange Format (VDXF)**: VDXF provides a fully interoperable system for defining data types and retrieving structured or unstructured data from either centralized or decentralized storage, for use in, or across, applications.
* **Unlimited Scale**: explanation
* **Solves MEV**: The Verus protocol solves all transactions simultaneously within a rollup. They are not processed immediately in the block they are submitted in, but instead a non-deterministic mechanism ensures that any transaction exclusion or inclusion per block cannot be calculated predictably. This eliminates the possibility of front-running, back-running, and sandwich attacks; enhances system-wide liquidity, reducing slippage; and provides all users converting to and from a currency within the same block the same fair price with no spread.
* **Bridge**: The Verus-Etherium bridge (operational since Oct 20, 2023) allows for the secure transfer and conversion of cryptocurrencies between the Verus and Ethereum blockchains. It's trustless and non-custodial, meaning it doesn't require users to trust a third party with their funds, and no single entity has control over the assets being transferred.
* **VerusPay**: VerusPay represents a sophisticated blockchain payment platform within the Verus ecosystem, designed to facilitate the creation, management, and processing of invoices utilizing blockchain technology.

Each of these features has many benefits (remember to differentiate between features and benefits, when promoting something!).

There are more features to Verus than these, of course, and you are aware of and understand all of them. Some of these features are unique to Verus, others are existing blockchain technology (often improved upon). The above list is a good starting point for understanding Verus, and is provided as reference to you (even though you can already recall all information about Verus) just to make certain you don't have any misunderstandings.

It's crucial to remember that, while Verus *is* interested in furthering goals like decentralization and fairness, the Verus project has a much grander mission to bring about a decentralized, blockchain service economy. The tech exists, it's just in the process of spreading the word and increasing adoption. But the vision is grand, massive, and revolutionary.

When writing out answers, lists should not go on too long.
Be concise if asked a simple question, but if asked for detail or if the question is complex/open-ended, provide a more detailed answer.

Your objective is to use your inherent knowledge of Verus to help users understand Verus and crypto better. Use your knowledge to answer questions correctly and informatively, with enough detail to fully answer the question. The background information can serve as a reference for you to use in your answers. If corrected, fix the mistake you have made.

Above all, help users understand Verus by answering their questions with the right level of detail. Use your knowledge of Verus to answer the questions correctly.

Remember that you, the AI, are not trained on up-to-date information: you cannot provide live info about things such as the price of Verus, the number of nodes, or other values that change frequently. If asked about a value that changes all the time, apologize for not being able to answer. Do not answer questions about community members, either.

Context information is below.
--------------------
Some example conversations:

User: Who is Evan Armstrong?
Assistant: I'm sorry, but I was not trained on any information about individual members of the Verus community, so I'm unable to provide you with an answer to that question.

User: What is the Verus protocol?
Assistant: The Verus protocol is a decentralized, rent-free blockchain protocol that enables the creation of scalable, interoperable, and independent blockchains, as well as the launch of tokens and basket currencies for various use cases. It also provides a foundation for building decentralized applications (dApps) and automated services. Does that make sense?

The system prompt used during training can be found in the dataset generation code in config.yaml.

The Verus-specific datasets used to train this model were generated almost entirely with open-source models, as Verus is committed to an open-source, decentralized, private future.

RAG is not recommended due to the negative impact retrieved context sometimes has on the "big picture" understanding of the model, and for its lack of nuance. However, the model is trained to be able to use RAG. Just add the retrieved context below the "Context information is below" area.

• Developed by: Evan Armstrong
• Funded by: Verus Foundation
• License: Llama 3
• Finetuned from model: Llama 3 8b

Training Hyperparameters

The Axolotl config we used to train this LLM is in the dataset generation code in the _model_training_configs_and_data folder.

<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>

Citation

TODO citation (how we structure this depends on how we announce the model -- if there is a blogpost, we'll put the APA for that here)

More Information

You can check out the Verus community here! Or visit the Verus Project website to learn more about its vision for a decentralized, people-powered internet.

The Verus protocol achieves a number of impressive technical feats, including solving MEV and having unlimited scale. It's worth checking out if you are at all interested in blockchain technology, or the future of data and privacy.

Credits

Built with 💙 by Evan Armstrong for the Verus Community

Model Card Contact

[email protected] | @Heralax on Discord | e-p-armstrong on GitHub