Mistral-Small-3.2-24B-LoRA-ref-parsing

<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->

<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>

<details><summary>See axolotl config</summary>

axolotl version: 0.13.0.dev0

base_model: mistralai/Mistral-Small-3.2-24B-Instruct-2506
processor_type: AutoProcessor

# Enable to use mistral-common tokenizer
tokenizer_use_mistral_common: true
chat_template_strategy: tokenizer

deepspeed: zero1.json

load_in_8bit: false

# axolotl lora optimizations
# lora_mlp_kernel: true
# lora_qkv_kernel: true
# lora_o_kernel: true

# must include for mistral model
skip_prepare_dataset: true
remove_unused_columns: false
sample_packing: false

datasets:
  - path: yurui983/reference-parsing-lora
    split: train_single
    type: chat_template
    roles_to_train: [assistant]

test_datasets:
  - path: yurui983/reference-parsing-lora
    split: valid_single
    type: chat_template
    roles_to_train: [assistant]


dataset_prepared_path: last_run_prepared

output_dir: ./outputs/mistral-out-single-1103

adapter: lora
lora_model_dir:

sequence_len: 4096
pad_to_sequence_len: false

lora_r: 64
lora_alpha: 64
lora_dropout: 0.05
lora_target_linear: true
# lora_target_modules: 'model.language_model.layers.[\d]+.(mlp|cross_attn|self_attn).(up|down|gate|q|k|v|o)_proj'

wandb_project: mistral-lora-ref-parsing
wandb_entity: zhuyurui0323-odoma
wandb_watch: gradients
wandb_name: mistral-small-3.2-24B-lora-ref-parsing-single
wandb_log_model: 'false'

gradient_accumulation_steps: 2
micro_batch_size: 4
num_epochs: 1
optimizer: adamw_torch
lr_scheduler: cosine
learning_rate: 5e-5

bf16: auto
tf32: false

gradient_checkpointing: true
logging_steps: 5
# flash_attention: true

loss_watchdog_threshold: 5.0
loss_watchdog_patience: 3

warmup_ratio: 0.1
evals_per_epoch: 4
saves_per_epoch: 3
weight_decay: 0.0
special_tokens:


do_causal_lm_eval: true
eval_causal_lm_metrics: ["sacrebleu", "perplexity"]

max_grad_norm: 1.0

# save_first_step: true  # uncomment this to validate checkpoint saving works with your config

</details><br>

Model description

This repository provides a LoRA adapter fine-tuned from mistralai/Mistral-Small-3.2-24B-Instruct-2506 on the odoma/reference-parsing-lora dataset, for bibliographic reference parsing. Given reference strings as input, the model generates a schema-constrained JSON object with bibliographic fields such as authors, title, publication date, venue/publisher, and pages (when available). The adapter is optimized for robust parsing under SSH-realistic citation conditions, including multilingual references, abbreviated forms, and heterogeneous formatting (e.g., footnote-style citations). Compared with the base model in a consistent single-call setup, this fine-tune improves structured output stability and field-level accuracy, making it suitable as a drop-in parser component in citation indexing and linking pipelines.

It achieves the following results on the evaluation set:

• Loss: 0.3481
• Memory/max Active (gib): 53.66
• Memory/max Allocated (gib): 53.66
• Memory/device Reserved (gib): 70.06

Intended uses & limitations

Intended uses

• Reference parsing in citation indexing pipelines: convert already-extracted reference strings into structured JSON for downstream linking (e.g., OpenAlex/Wikidata matching) and analytics.
• SSH-oriented citation processing, including multilingual and stylistically diverse references (e.g., humanities monographs, footnote-like formats, abbreviated venues/publishers).
• Batch parsing of reference lists when each reference string is provided separately (recommended: one reference per call).

Limitations

• Not a reference extractor: it does not detect or segment references from raw PDFs or full-text documents. Use a layout/OCR or extraction step first.
• Schema sensitivity: the adapter is optimized for a specific target JSON schema; changing field names or required fields may reduce quality unless prompts are updated (or the adapter is re-tuned).
• Underspecified citations: when key information is missing or ambiguous in the input string, the model may output partial JSON or infer fields (hallucinations). For high-precision applications, apply validation rules (e.g., required fields, date formats) and consider human review on low-confidence cases.
• Formatting noise: line breaks, hyphenation, OCR artifacts, or multiple references concatenated into one string can degrade performance.

Training and evaluation data

The adapter was fine-tuned on odoma/reference-parsing-finetuning, a supervised dataset of (reference string → JSON) pairs formatted in an instruction/chat style. The data is curated from three complementary gold standards that reflect different citation regimes:

• CEX: English-language scientific articles with relatively regular bibliography formatting.
• EXCITE: German/English SSH documents with end-section, footnote-only, and mixed citation regimes.
• LinkedBooks: humanities references with strong stylistic variation and multilinguality; the schema coverage is more limited (typically authors/title/date/place).

Evaluation uses a schema-constrained parsing setup aligned with training, and reports both generation quality (e.g., structured-field accuracy / F1 in downstream benchmarking) and training-time metrics (loss and memory footprint as shown above).

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• distributed_type: multi-GPU
• num_devices: 2
• gradient_accumulation_steps: 2
• total_train_batch_size: 16
• total_eval_batch_size: 8
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 161
• training_steps: 1611

Training results

Framework versions

• PEFT 0.17.1
• Transformers 4.57.1
• Pytorch 2.7.1+cu126
• Datasets 4.0.0
• Tokenizers 0.22.1

Credits

The dataset is being developed by Yurui Zhu (Odoma). This work is carried out in the context of the EU-funded GRAPHIA project (grant ID: 101188018).