Downloads · 30 days
6
1% of all-time downloads
northenlab/soilfm-qwen2.5-14b-literature-cpt
soilfm-qwen2.5-14b-literature-cpt is a text generation model from northenlab. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
A domain-adapted large language model for soil science and soil microbiology, created by continued pretraining of Qwen2.5-14B-Instruct on 200,000 curated soil science text passages.
Downloads · 30 days
6
1% of all-time downloads
All-time downloads
465
Public
Parameters
14.8B
29.6 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors29.5 GB · 100%
From the Hugging Face model README
A domain-adapted large language model for soil science and soil microbiology, created by continued pretraining of Qwen2.5-14B-Instruct on 200,000 curated soil science text passages.
This model is the Language Tower component of SoilFM2, a multi-modal foundation model for soil microbiome analysis developed at Lawrence Berkeley National Laboratory.
| Base model | Qwen/Qwen2.5-14B-Instruct (14.2B parameters) |
| Method | Continued pretraining via QLoRA (4-bit NF4) |
| Format | Full merged model (LoRA weights merged into base) |
| Precision | BF16 |
| Context length | 32,768 tokens |
| Size on disk | ~28 GB |
| LoRA adapter | Also available at northenlab/soilfm-qwen2.5-14b-qlora (263 MB) |
The training corpus was assembled from four sources of soil science domain knowledge, stratified-sampled to 200,000 training examples and 10,000 validation examples (seed = 42):
| Source | Description | Proportion | Train | Val |
|---|---|---|---|---|
| PubMed Central | Full-text soil microbiology papers (39,853 articles) | 55% | 110,000 | 5,500 |
| Wikipedia | Soil science articles | 20% | 40,000 | 2,000 |
| USDA Soil Survey Manual | Official USDA technical reference | 10% | 20,000 | 1,000 |
| Wikipedia General Biology | Broad biology context to prevent catastrophic forgetting | 15% | 30,000 | 1,500 |
| Total | 100% | 200,000 | 10,000 |
Text was chunked to 1,024 tokens with 100-token overlap. The full corpus contained 329M tokens across 388,563 chunks; the 200K stratified subsample was used for this training run.
text and source fields| Parameter | Value |
|---|---|
| Quantization | 4-bit NF4 (BitsAndBytes, double quantization) |
| LoRA rank | 16 |
| LoRA alpha | 32 |
| LoRA dropout | 0.05 |
| Rank-stabilized LoRA (rsLoRA) | Yes |
| Target modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| Trainable parameters | ~263M (1.85% of total) |
| Optimizer | PagedAdamW8bit |
| Learning rate | 2e-5 (cosine schedule, 10% warmup) |
| Effective batch size | 128 (micro-batch 2, gradient accumulation 64) |
| Max gradient norm | 1.0 |
| Weight decay | 0.01 |
| Precision | BF16 mixed precision |
| Flash Attention 2 | Enabled |
| Epochs | 1 |
| Total steps | 1,500 |
A custom manual PyTorch training loop was used (rather than HuggingFace Trainer) for compatibility and control. The script is available in the project repository as train_soilfm_cpt_MANUAL.py.
| Step | Validation Loss |
|---|---|
| 500 | 1.7369 |
| 1,000 | 1.6281 |
| 1,500 | 1.6130 |
Total improvement: 7.2% over the course of training. Loss was still decreasing at the end of training with no signs of overfitting. Gradient norms remained stable in the 0.2–0.8 range throughout.
Prompt: "The role of root exudates in shaping rhizosphere microbial communities involves"
Output: The model produces coherent, technically accurate continuations using appropriate domain terminology (root exudates, rhizosphere, primary/secondary metabolites, phytohormones), demonstrating successful domain adaptation.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"northenlab/soilfm-qwen2.5-14b-literature-cpt",
torch_dtype=torch.bfloat16,
device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(
"northenlab/soilfm-qwen2.5-14b-literature-cpt"
)
prompt = "The role of mycorrhizal fungi in soil nutrient cycling"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
python3 -m vllm.entrypoints.openai.api_server \
--model northenlab/soilfm-qwen2.5-14b-literature-cpt \
--host 0.0.0.0 --port 8001
If you prefer to load the adapter separately (e.g., for 4-bit inference):
from transformers import AutoModelForCausalLM
from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained(
"Qwen/Qwen2.5-14B-Instruct",
load_in_4bit=True,
device_map="auto",
)
model = PeftModel.from_pretrained(base, "northenlab/soilfm-qwen2.5-14b-qlora")
This Language Tower works alongside other SoilFM2 components:
| Component | Description | HuggingFace |
|---|---|---|
| Language Tower (this model) | Domain-adapted LLM | northenlab/soilfm-qwen2.5-14b-literature-cpt |
| Graph Tower | Heterogeneous GNN on 2.39M-node knowledge graph | northenlab/soilfm2-graph-tower-joint-v0.1 |
| BSPR | Bayesian substrate preference model (AUC 0.94) | — |
Together these components power a prebiotic recommendation pipeline that takes 16S microbiome profiles as input and suggests soil amendments to steer community function.
This model is intended for research and non-commercial use only. The training corpus includes PubMed Central Open Access articles under various Creative Commons licenses, some of which may carry non-commercial (CC BY-NC) terms. Users should ensure their use complies with the underlying data licenses.
The base model (Qwen2.5-14B-Instruct) is released under the Apache 2.0 license.
@misc{soilfm-language-tower-2025,
title={SoilFM Language Tower: Domain Adaptation of Qwen2.5-14B for Soil Science},
author={Northen Lab, Lawrence Berkeley National Laboratory},
year={2025},
publisher={HuggingFace},
url={https://huggingface.co/northenlab/soilfm-qwen2.5-14b-literature-cpt},
note={Continued pretraining on 200K soil science literature examples via QLoRA}
}
Apache 2.0 (inherited from the base Qwen2.5-14B-Instruct model). Training data includes PubMed Central Open Access articles under various CC licenses — see Use Restrictions above.