Covenant-72B

Covenant-72B

Model Overview

Covenant-72B is the largest permissionless collaboratively trained language model, trained entirely from scratch at the 72 billion parameter scale on 1.1 trillion tokens of English text.

For more details, see the technical report. This is a base model. See Covenant-72B-Chat for the instruction-tuned variant.

Covenant-72B was trained with 20+ globally distributed participants coordinated via decentralized infrastructure on the Bittensor blockchain. Unlike prior collaborative training efforts that use whitelisted compute, Covenant-72B is the first to achieve this scale with fully permissionless participation. Training used the SparseLoCo communication-efficient optimizer to reduce bandwidth requirements across distributed nodes.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "1Covenant/Covenant-72B",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("1Covenant/Covenant-72B")

input_text = "The theory of general relativity"
input_ids = tokenizer.encode(input_text, return_tensors="pt").to(model.device)
output_ids = model.generate(input_ids, max_new_tokens=100)
print(tokenizer.decode(output_ids[0], skip_special_tokens=True))

Model Details

• Compute Participants: 20+ independent contributors on Bittensor
• Minimum Compute per Participant: 8×B200 or equivalent
• Model License: Apache 2.0

Technical Specifications

Training Details:

• Dataset: DCLM-baseline
• Tokens: 1.1 Trillion
• Optimizer: SparseLoCo (communication-efficient optimizer)

Performance on Benchmarks

All results are 0-shot acc_norm (%) unless noted.

*WinoGrande uses acc; MMLU uses acc.