mistral-7b-academic-metadata-extractor

Mistral-7B Academic Metadata Extractor

A fine-tuned Mistral-7B-Instruct-v0.3 model for structured metadata extraction from academic documents.

The model extracts metadata from academic text and produces structured JSON output.

Extracted Metadata

The model extracts the following fields:

• Title
• Keywords
• Authors
• Affiliations
• Publisher
• Journal
• Publication Year

Performance

The fine-tuned model achieved an overall field-level accuracy of 91.8% on the test set.

Training Dataset

A dataset of 10,000 academic PDF documents was prepared for the project.

The final dataset was divided into:

• Training: 9,000 documents
• Validation: 900 documents
• Testing: 100 documents

Approximately 3,000 documents were manually reviewed to complete or correct missing metadata fields.

Training Configuration

Output Format

The model is designed to return metadata in JSON format:

{
  "title": "",
  "keywords": [],
  "authors": [
    {
      "name": "",
      "affiliations": [""]
    }
  ],
  "publisher": "",
  "year": "",
  "journal": ""
}

Research

This model was developed as part of the research:

Developing AI Agents That Extract Metadata Using The RAG Method With Fine-Tuned Large Language Models

Authors: Tamam Alhasan and Savaş Takan
Affiliation: Artificial Intelligence and Data Engineering Department, Ankara University

The research compared RAG-based metadata extraction with supervised fine-tuning. Mistral-7B-Instruct-v0.3 achieved 74.43% overall accuracy in the RAG setup, while the fine-tuned model improved the overall accuracy to 91.8%.

Intended Use

This model is intended for extracting structured bibliographic metadata from academic and scientific documents.

Potential applications include:

• Academic repositories
• Digital libraries
• Research databases
• Automated document indexing
• Scientific knowledge management

Limitations

The model was trained and evaluated primarily on English-language academic documents. Performance may vary for documents with unusual layouts, incomplete metadata, OCR errors, or domains significantly different from the training data.

Authors

Tamam Alhasan
Artificial Intelligence and Data Engineering
Ankara University, Türkiye