llama-3.1-base-kg-extraction-full

Model Card for Llama-3.1-Argument-Triplet-Extraction

A LoRA-fine-tuned version of Llama 3.1 8B (base, not instruct) for ontology-guided knowledge graph triple extraction from historical argumentative text. Given a short passage of argumentative text and a reference ontology of ~130 (subject, predicate, object) triples, the model prioritizes matching the passage to an existing ontology triple, but may instead propose a new triple (flagged new_triple: true) when the argument expresses a valid relation the ontology doesn't cover, or return NA if no relation applies — along with a confidence score in all cases.

The model was developed for extracting argumentative knowledge graphs from historical newspaper coverage of the 1908 Messina earthquake and its international relief response.

Model Details

Model Description

This model fine-tunes Llama 3.1 8B (base) with LoRA to perform ontology-guided (not strictly closed-set) relation extraction: given an argument passage (in English, French, German, or Italian — the source corpus is multilingual historical press coverage) and a reference ontology of triples covering themes such as aid, damage, recovery, misinformation, and government response, the model outputs a JSON object identifying either the matching ontology triple, a newly suggested triple not in the ontology (new_triple: true), or NA if no relation applies — with a confidence score in all cases.

• Model type: Causal language model (decoder-only), fine-tuned with LoRA for structured/constrained information extraction
• Language(s) (NLP): English, French, German, Italian
• License: Llama 3.1 Community License (inherited from base model)
• Finetuned from model: meta-llama/Llama-3.1-8B

Uses

Direct Use

The model is intended for extracting structured knowledge graph triples from short argumentative text passages related to disaster response and relief efforts, constrained to a predefined ontology. It is designed to be used with a prompt containing the ontology as context and is not intended for open-ended or unconstrained relation extraction.

Ontology and Instruction Text

The model was trained against a fixed instruction containing the ontology below. This exact ontology text (or a functionally equivalent one) must be included in the prompt at inference time — the model was never trained to extract triples without it in context.

<details> <summary>Click to expand full ontology (130 triples)</summary>

ONTOLOGY — prioritize these triples:
- subject: aid → object: damage | predicate: insufficient for
- subject: aid → object: recovery | predicate: insufficient for
- subject: aid → object: refugees | predicate: fails to reach
- subject: aid → object: victims | predicate: saved
- subject: austria → object: aid | predicate: delivers
- subject: damage → object: aid | predicate: limits capacity of
- subject: damage → object: aid | predicate: necessitates
- subject: military → object: recovery | predicate: enables
- subject: society → object: aid | predicate: is obligated to
- subject: society → object: aid | predicate: is requested to
- subject: society → object: aid | predicate: obligates to
- subject: society → object: aid | predicate: lost interest for
- subject: aid → object: victims | predicate: reaches
- subject: austria → object: heroic | predicate: acts
- subject: austria → object: aid | predicate: effective response to
- subject: damage → object: aid | predicate: limits capacity of
- subject: damage → object: aid | predicate: prevents
- subject: damage → object: banks | predicate: limited impact
- subject: damage → object: economy | predicate: benefits financially
- subject: damage → object: economy | predicate: causes long-term to
- subject: damage → object: germany | predicate: harms economy
- subject: damage → object: italy | predicate: harms economy
- subject: damage → object: recovery | predicate: delays
- subject: population density → object: damage | predicate: increases
- subject: damage → object: epidemic | predicate: risks
- subject: damage → object: disorder | predicate: causes
- subject: disorganization → object: aid | predicate: undermines
- subject: friendship → object: aid | predicate: obligates to
- subject: italian government → object: aid | predicate: delays
- subject: italian government → object: damage | predicate: effective response to
- subject: italian government → object: order | predicate: maintains
- subject: italian government → object: recovery | predicate: enables
- subject: italian government → object: recovery | predicate: insufficient for
- subject: italian government → object: italy | predicate: harms economy
- subject: disorder → object: military | predicate: justifies
- subject: military → object: aid | predicate: delays
- subject: military → object: aid | predicate: not used enough
- subject: military → object: recovery | predicate: enables
- subject: italy → object: misinformation | predicate: spreads
- subject: aid → object: epidemic | predicate: prevents
- subject: damage → object: recovery | predicate: prevents
- subject: economy → object: recovery | predicate: enables
- subject: england → object: recovery | predicate: strategically supports
- subject: italian government → object: aid | predicate: enables
- subject: italian government → object: recovery | predicate: can efficiently handle
- subject: italian government → object: recovery | predicate: lacks plan
- subject: recovery → object: economy | predicate: hinders
- subject: aid → object: refugees | predicate: is requested for
- subject: refugees → object: aid | predicate: necessitates
- subject: refugees → object: recovery | predicate: enables
- subject: refugees → object: stereotype | predicate: challenges
- subject: royalty → object: italian government | predicate: demands action from
- subject: italian government → object: scientific | predicate: must implement
- subject: italian government → object: scientific | predicate: should implement
- subject: scientific → object: damage | predicate: explains
- subject: scientific → object: damage | predicate: prevents
- subject: scientific → object: future | predicate: predicts
- subject: society → object: damage | predicate: explains
- subject: society → object: aid | predicate: is requested to
- subject: damage → object: sympathy | predicate: evokes
- subject: sympathy → object: aid | predicate: motivates
- subject: trauma → object: verification | predicate: undermines
- subject: unity → object: recovery | predicate: enables
- subject: unity → object: aid | predicate: enables
- subject: unity → object: crisis | predicate: supports
- subject: messina → object: italian government | predicate: grateful for response
- subject: germany → object: aid | predicate: delivers
- subject: god → object: earthquake | predicate: not related
- subject: damage → object: trauma | predicate: causes
- subject: trauma → object: aid | predicate: undermines
- subject: military → object: order | predicate: maintains
- subject: military → object: victims | predicate: harms
- subject: damage → object: italian government | predicate: increases costs for
- subject: italian government → object: media | predicate: suppresses
- subject: society → object: italian government | predicate: protests
- subject: royalty → object: aid | predicate: supports
- subject: journalists → object: misinformation | predicate: spread
- subject: military → object: aid | predicate: abuses
- subject: misinformation → object: society | predicate: harms
- subject: society → object: aid | predicate: hinders
- subject: scientific → object: human ignorance | predicate: reveals
- subject: society → object: recovery | predicate: responsible for
- subject: damage → object: economy | predicate: harms
- subject: damage → object: germany | predicate: harms
- subject: italian government → object: misinformation | predicate: spreads
- subject: aid → object: organized | predicate: is well
- subject: aid → object: all cities | predicate: did not reach
- subject: international aid → object: political advantage | predicate: serves
- subject: international aid → object: recovery | predicate: not needed anymore for
- subject: material assistance → object: monetary assistance | predicate: more effective than
- subject: recovery → object: possible | predicate: not
- subject: damage → object: migration | predicate: necessitates
- subject: messina → object: recover | predicate: will
- subject: aid from germany → object: recovery | predicate: enables
- subject: aid from germany → object: superior | predicate: is
- subject: crisis management of italy → object: not working | predicate: is
- subject: disorganization → object: Italians inefficiency for crisis | predicate: proofs
- subject: disorganization → object: italian reputation | predicate: harms
- subject: italy → object: misinformation | predicate: fights
- subject: scientific → object: past earthquakes | predicate: compares to
- subject: aid → object: aid | predicate: necessitates
- subject: damage → object: recovery | predicate: hinders
- subject: damage → object: trauma | predicate: creates
- subject: inaction → object: aid | predicate: delays
- subject: messina → object: aid | predicate: delays
- subject: italian government → object: narrative | predicate: controls
- subject: italian government → object: misinformation | predicate: spread
- subject: italian government → object: recovery | predicate: delays
- subject: italian government → object: recovery | predicate: must prioritize
- subject: italian government → object: recovery | predicate: lacks money
- subject: royalty → object: aid | predicate: enables
- subject: royalty → object: recovery | predicate: enables
- subject: military → object: aid | predicate: enables
- subject: journalists → object: disorganization | predicate: reveal
- subject: society → object: italian government | predicate: protests
- subject: society → object: aid | predicate: obligates to
- subject: aid → object: italy | predicate: misused by
- subject: damage → object: navigation | predicate: prevents

</details>

Full instruction template used during training and required at inference:

Extract the knowledge graph triple from the argument below.

ONTOLOGY — prioritize these triples:

{ONTOLOGY_TEXT — the full block above}

RULES:
1. If the argument matches a triple from the ontology above, use it exactly as listed.
2. If the argument contains a valid knowledge graph relation NOT in the ontology, suggest a NEW triple using the same format (subject, predicate, object). Set 'new_triple' to true.
3. If no argument or relation exists, return NA for subject, predicate, object.

Return ONLY a JSON object with these keys: subject, predicate, object, confidence, new_triple.
- confidence: float 0.0-1.0
- new_triple: true if this is a new suggested triple not in the ontology, false otherwise
No explanation, no markdown, just the JSON.

Downstream Use

Trained for a specific historical use case for knowledge graph extraction.

Out-of-Scope Use

• Not intended for extracting triples outside the specific ontology it was trained on (~130 triples related to earthquake relief discourse); it will attempt to force-fit or hallucinate near-ontology values for out-of-domain text.
• Not intended as a general-purpose chat or instruction-following model. It is a base model fine-tuned narrowly for this extraction task and has not been aligned or safety-tuned.

Bias, Risks, and Limitations

• The training and validation data derive from a specific historical press corpus (1908 Messina earthquake coverage); the model's behavior on other disasters, time periods, or domains is untested and likely to degrade.
• Evaluation loss (eval_loss) reached its minimum around epoch ~3 of training and increased through epochs 4–5, suggesting mild overfitting in later epochs; however, output evaluation performed well with the last epoche.

Training Details

Training Data

• Training set: 1,000 examples
• Validation set: 200 examples
• Each example pairs a short argumentative passage (extracted from historical press coverage) with an instruction containing the fixed ontology and the expected output triple (or NA).
• [More Information Needed] on dataset licensing, provenance, and annotation process.

Training Procedure

Base model weights were loaded in fp16 with device_map="auto", gradient checkpointing enabled, and fine-tuned with LoRA rather than full fine-tuning.

Preprocessing [optional]

Training examples were formatted using an Alpaca-style prompt template:

### Instruction:
{instruction}

### Input:
{input}

### Response:
{output}

Tokenized with truncation=True, padding="max_length", max_length=2048.

Training Hyperparameters

• Training regime: fp16 mixed precision
• Base model: meta-llama/Llama-3.1-8B
• PEFT method: LoRA — r=16, lora_alpha=32, lora_dropout=0.05, bias="none", target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Trainable parameters: 41,943,040 (0.52% of 8,072,204,288 total)
• Epochs: 5
• Batch size: 1 per device, gradient accumulation steps = 4 (effective batch size 4)
• Learning rate: 2e-4 (peak), cosine schedule, warmup ratio 0.05
• Max gradient norm: 0.5
• Weight decay: 0.0
• Optimizer: AdamW (torch)

Speeds, Sizes, Times [optional]

• Total training steps: 4,285
• Total training time: ~5 hours 29 minutes
• Hardware: 1x NVIDIA A100 (a100-large, Hugging Face Jobs)
• Throughput: ~0.867 samples/sec, ~0.217 steps/sec
• Final train loss (mean over run): 0.1636

Testing Data

857-example held-out validation split from the same source corpus as the training data (see Training Data above).

Metrics

Token-level cross-entropy loss (eval_loss), computed over full input+output sequences (not response-only; the ontology/instruction text is included in the loss calculation, which somewhat compresses the reported values since a large, fixed portion of every sequence is trivially predictable).

Results

Summary

Validation loss decreased steadily through the first ~3 epochs, reaching a minimum (0.2801–0.2818) around epoch 2.3–3.0, then increased through epochs 4–5 while training loss continued to fall — indicating mild overfitting in the later epochs. The released model corresponds to the final epoch-5 checkpoint rather than the epoch-3 minimum. See Bias, Risks, and Limitations above.

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

• Hardware Type: NVIDIA A100 (a100-large)
• Hours used: ~5.5
• Cloud Provider: Hugging Face Jobs

Model Architecture and Objective

Decoder-only causal language model (Llama 3.1 8B architecture) with LoRA adapters applied to attention and MLP projection layers, trained with a standard next-token-prediction (causal LM) objective on Alpaca-formatted instruction/response pairs.

Compute Infrastructure

Hardware

1x NVIDIA A100 (a100-large tier), via Hugging Face Jobs.

Software

• transformers
• peft
• datasets
• bitsandbytes
• accelerate
• torch 2.6.0
• CUDA 12.4, cuDNN 9
• Base image: pytorch/pytorch:2.6.0-cuda12.4-cudnn9-devel