Llama-ChemLink-Parser-8B-MTYS

Llama-ChemLink-Parser-8B-MTYS

ChemLink is a LoRA fine-tune of tokyotech-llm/Llama-3.1-Swallow-8B-Instruct-v0.3 for extracting chemical measurement values (MW, IC50, EC50, Yield) from scientific literature, with compound-name linkage for PubChem grounding and Graph RAG integration.

---

Background and Motivation

Target environment: CPU-only local hardware, no GPU required.

Chemical and pharmaceutical researchers frequently operate under security policies that prohibit cloud API usage. This model is designed to run on a standard CPU workstation (e.g., Core i7 / 24 GB RAM) via Ollama in GGUF format (q5_K_M, ~5 GB), suitable for overnight batch processing in network-restricted or air-gapped environments without any cloud dependency.

A critical requirement in this setting is compound-name linkage: downstream pipelines (PubChem grounding, Graph RAG, compound databases) need to know not just the measurement value, but which chemical compound it belongs to. This requires the model to output a compound_name field alongside each extracted value.

Two prompt conditions were evaluated:

• Condition A (no instruction): prompt requests only type / value / unit; compound_name is not mentioned.
• Condition B (with instruction): prompt explicitly requests compound_name in addition to type / value / unit.

In the Colab GPU evaluation, ChemLink outputs compound_name under both conditions. In the same Colab evaluation, all comparison models (Swallow-base, Mistral-7B) output 0% compound_name without explicit instruction (Condition A). In the local Ollama evaluation, Swallow-base also produced compound_name under Condition A; this behavior is environment- and template-dependent (see Evaluation and Limitations).

---

Key Capability

ChemLink retained compound_name output when the evaluation prompt did not explicitly request the field, across the evaluated Colab GPU and local CPU environments.

{
  "chemical_entities": [
    {
      "compound_name": "linezolid",
      "measurements": [
        {"type": "Molecular Weight", "value": 337.35, "unit": "g/mol"}
      ]
    }
  ]
}

Note: compound_name reflects the name as it appears in the source text. It is not normalized or verified against any database at inference time. Across the evaluated MW records, approximately 59–65% of ChemLink-generated compound names were directly resolvable through the PubChem REST API (see Evaluation).

This stability reduces the risk of pipeline failures where a measurement value is extracted but cannot be linked to its source compound — a risk that depends on prompt design when using baseline models.

---

Model Overview

---

Usage

Local CPU Inference (Ollama — Primary Use Case)

ollama create llama-chemlink-parser-8b-mtys -f Modelfile
ollama run llama-chemlink-parser-8b-mtys

Modelfile example (replace /path/to/ with your actual GGUF file path):

FROM /path/to/Llama-3.1-Swallow-8B-Instruct-v0.3.Q5_K_M.gguf

TEMPLATE """{{ if .System }}<|start_header_id|>system<|end_header_id|>

{{ .System }}<|eot_id|>{{ end }}<|start_header_id|>user<|end_header_id|>

{{ .Prompt }}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

{{ .Response }}<|eot_id|>"""

PARAMETER temperature 0
PARAMETER num_ctx 2048
PARAMETER num_predict 256
PARAMETER stop "<|eot_id|>"

Note: num_predict 256 is required. The default (128) causes truncation of structured JSON output.

Inference (Colab / GPU)

This repository publishes the LoRA adapter only. The base model (tokyotech-llm/Llama-3.1-Swallow-8B-Instruct-v0.3) is loaded automatically from HuggingFace.

import torch, json, re
from unsloth import FastLanguageModel
from google.colab import userdata

HF_TOKEN = userdata.get('HF_TOKEN')

SYSTEM_PROMPT = (
    "You are a chemical data extraction assistant. "
    "Extract measurements from the given text and return a JSON object. "
    "The object must have a 'chemical_entities' array. "
    "Each element must have: compound_name (string), "
    "measurements (array of objects with type/value/unit). "
    "If no target measurement is found, return {\"chemical_entities\": []}. "
    "Output only the JSON object, no explanation."
)

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name     = "MitzMitz/Llama-ChemLink-Parser-8B-MTYS",
    max_seq_length = 2048,
    dtype          = None,
    load_in_4bit   = True,
    token          = HF_TOKEN,
)
FastLanguageModel.for_inference(model)

def extract(text):
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user",   "content": text},
    ]
    input_ids = tokenizer.apply_chat_template(
        messages, tokenize=True,
        add_generation_prompt=True, return_tensors="pt"
    ).to("cuda")
    with torch.no_grad():
        output = model.generate(
            input_ids, max_new_tokens=256,
            temperature=0.0, do_sample=False,
            pad_token_id=tokenizer.eos_token_id,
        )
    return tokenizer.decode(
        output[0][input_ids.shape[1]:], skip_special_tokens=True
    ).strip()

print(extract("The compound linezolid has a molecular weight of 337.35 g/mol."))

---

Training Configuration

---

Training Data

Negative samples (1,555 records, 20.0%) contain [] as output.

Data licenses:

• ORD: CC-BY-SA 4.0
• ChEMBL: CC-BY-SA 3.0 (EMBL-EBI)
• PubChem: Public Domain (NCBI/NIH)

---

Evaluation

Dataset

Source: true_eval_all_pmid_clean.jsonl (2,963 records total; PMID-verified; no overlap was found between the fine-tuning dataset and the evaluation dataset using PMID-based matching).

This evaluation uses a stratified 500-sample subset (125 per indicator: MW / Yield / IC50 / EC50), RANDOM_SEED=42. The full 2,963-sample dataset was used to construct the source file; the 500-sample subset is drawn from it without replacement.

The tables below report results for MW only. IC50 and EC50 accuracy was not evaluated under this protocol because the structured-output format suppressed IC50/EC50 responses across the evaluated models. Yield records were included in the sampled dataset, but Yield accuracy was not evaluated because ground-truth Yield values were not extracted for scoring. PubChem-based validation is also not applicable to Yield because reaction yield is not an intrinsic molecular property.

Yield evaluation: Yield records were included in the stratified evaluation sample, but Yield performance is not reported in the present evaluation because ground-truth Yield values were not extracted and accuracy scoring was not performed. Unlike molecular weight, reaction yield is specific to the reaction, conditions, and reported yield definition, and therefore cannot be independently validated through PubChem molecular-property lookup. A separate Yield-specific evaluation dataset and scoring protocol are required before reporting Yield accuracy.

Evaluation Prompts

The following system prompts were used verbatim in all evaluations. These are identical to SYSTEM_PROMPT_NO_COMPOUND and SYSTEM_PROMPT_WITH_COMPOUND in common_chemfmt.py.

Condition A — no compound_name instruction:

You are a chemical data extraction assistant. Extract measurements from the given text and return a JSON array. Each element must have: type (IC50/EC50/MW/Yield), value (number), unit (string). If no target measurement is found, return []. Output only the JSON array, no explanation.

Condition B — with compound_name instruction:

You are a chemical data extraction assistant. Extract measurements from the given text and return a JSON object. The object must have a 'chemical_entities' array. Each element must have: compound_name (string), measurements (array of objects with type/value/unit). If no target measurement is found, return {"chemical_entities": []}. Output only the JSON object, no explanation.

The Usage section (Colab inference code) uses Condition B as the example system prompt. Condition A cannot be reproduced from the Usage section; use the prompt above.

Column Definitions

All values are computed via the PubChem REST API (queried by compound name, https://pubchem.ncbi.nlm.nih.gov/rest/pug).

JSONL field mapping (source file: mw_pubchem_eval_results.jsonl):

Denominator hierarchy:

125 source MW records
  └─ n: correctly typed MW outputs
       └─ compound_name-present records
            └─ PubChem-resolved records
                 └─ PubChem MW-consistent records

MW output coverage = n / 125. Records with no parseable output, malformed JSON, no MW measurement, or non-matching type labels are excluded from n. MW accuracy is a conditional value accuracy calculated only among correctly typed MW outputs; it is not the end-to-end success rate over all 125 records.

n differs between conditions and models because different records fail to produce a correctly-typed MW field under each prompt format and model. The source pool of 125 MW records is identical across all conditions.

Of the 81 PubChem-resolved names (ChemLink q5_K_M, with instruction), 80 had a molecular weight consistent with the extracted value within ±1% (80/81 = 98.8%), indicating that the resolved PubChem record is consistent with the intended compound. This differs from the PubChem MW match column value (80/125 = 64.0%), which uses compound_name-present records as denominator.

---

Colab GPU / NF4 — MW (n per source)

Colab GPU parameters: temperature=0.0, max_new_tokens=256, apply_chat_template. These results are provided for reference only and do not represent the local CPU deployment scenario this model targets.

Mistral-7B with instruction n=32: Only 32 of 125 MW records contained a correctly-typed MW field. Other records produced output in chemical_entities format with incorrect type labels. This is a type-label inconsistency, not truncation.

MW accuracy is conditional on correctly typed MW output and should be interpreted together with MW output coverage.

---

Local CPU / Ollama q5_K_M — MW (n per source)

Local CPU parameters: temperature=0.0, num_predict=256, num_ctx=2048, Ollama Modelfile TEMPLATE.

Mistral-7B q5_K_M with instruction n=67: Only 67 of 125 MW records contained a correctly-typed MW field. Same type-label inconsistency as Colab (less severe locally).

MW accuracy is conditional on correctly typed MW output and should be interpreted together with MW output coverage.

† Swallow-base q5_K_M no instruction: compound_name output under no-instruction condition via Ollama differs from the Colab GPU result (0%) for the same base model. The discrepancy may result from differences in prompt serialization, chat templates, quantization, or inference runtimes. A controlled ablation was not performed. This result should not be interpreted as an intrinsic model capability.

---

Limitations

compound_name reflects source text only: The model copies the compound name as written in the source document. It is not normalized or verified at inference time. Generic codes ("compound 3", "2b") common in real PubMed abstracts will be output as-is and typically fail PubChem resolution.

Mistral-7B type-label inconsistency under chemical_entities schema: Mistral-7B-Instruct-v0.2 frequently returned chemical_entities JSON, but the measurement type labels did not match the accepted MW labels (67/125 correctly typed locally; 32/125 on Colab GPU). The JSON structure itself was produced; type vocabulary was inconsistent with the schema.

Swallow-base no-instruction Ollama artifact: Swallow-base q5_K_M showed compound_name output under no-instruction condition via Ollama, not observed in Colab GPU evaluation of the same base model (0%). The discrepancy may result from differences in prompt serialization, chat templates, quantization, or inference runtimes. A controlled ablation was not performed.

IC50 / EC50: IC50/EC50 accuracy was not evaluated under this protocol. Not suitable for cross-model comparison.

Inference environment differences: Colab GPU: temperature=0.0, max_new_tokens=256, apply_chat_template. Local Ollama: temperature=0.0, num_predict=256, Modelfile TEMPLATE. Cross-environment comparisons should account for these differences.

LoRA adapter only: This repository publishes the LoRA adapter (168 MB) and tokenizer files. The base model (~16 GB) is loaded from HuggingFace at inference time. For local CPU deployment, a pre-merged GGUF file is required.

---

Intended Use

• Automated extraction of MW / Yield from chemical literature in network-restricted, CPU-only local environments
• Compound-name to measurement-value association for PubChem grounding and Graph RAG pipelines
• Overnight batch processing on CPU-only hardware without cloud API dependency

Out-of-Scope Use

• Medical diagnosis or legal judgment
• Domains outside chemistry and chemical biology
• IC50 / EC50 extraction (see Limitations)

---

Base Model Reference

---

License

Licensed under the Llama 3.1 Community License. Copyright (C) Meta Platforms, Inc. All Rights Reserved.

---

Framework Versions