baoore-dataset-1

Baoore Dataset 1 (French-Mooré)

Dataset Description

• Repository: mooredataset/baoore-dataset-1
• Languages: French (fra) - Mooré (mos)
• Total sentences: 233,945

This dataset is a comprehensive parallel corpus for Machine Translation between French and Mooré. It is aggregated from multiple sources including dictionary entries, specific curated translations, generic translated data, and synthetic/filtered datasets. The dataset has been systematically curated and deduplicated for fine-tuning modern machine translation models such as NLLB.

Dataset Structure

The data is provided in JSONL format. Each line contains a JSON object representing a translation pair and its metadata.

Data Instances

A typical example from the dataset looks like this:

{
  "translation": {
    "fra": "Il mange une pomme.",
    "mos": "A rita pomme.",
    "origin": "traduction"
  }
}

Data Fields

• translation: A dictionary containing:
  • fra: The French sentence (string).
  • mos: The Mooré translation (string).
  • origin: The source origin of the data (string). Possible values:
    • laion-coco-nllb: Generic data from LAION-COCO-NLLB.
    • jw: Biblical data from JW.
    • french_moore: Human translation (D1).
    • traduction: Human translation (D2).
    • jw: Biblical data from JW.
    • mafand: Data from the MAFAND dataset.
    • wibonary: Data from the Webonary dictionary.

Corpus Statistics

The total corpus comprises 230,343 aligned sentence pairs (excluding SMOL data). The composition of the dataset by configuration (config_name) is detailed below. The statistics for the combined subsets are aggregated from their respective base subsets.

Note: 'existing' data groups the previously available datasets (MAFAND, Webonary, and Jehovah's Witnesses). 'trad-generique' is the combination of 'traduction' and 'generique'. 'trad-existing' is the combination of 'traduction' and 'existing'. Percentages reflect the proportion relative to 'all-data'.

Subsets

The repository is structured into different subdirectories containing various subsets of the data, each corresponding to a specific configuration:

• all_data: Combinaison de toutes les données (config : all-data).
• gen: Données génériques de laion-coco-nllb (config : generique).
• trad: Données de traduction humaine (config : traduction).
• old: Données qui étaient disponibles (MAFAND, Webonary, JW) (config : existing).
• trad_gen: Combinaison de traduction et generique (config : trad-generique).
• trad_old: Combinaison de traduction et existing (config : trad-existing).

Preprocessing & Filtering

• Deduplication: The corpus has undergone exact deduplication across French and Mooré sentence pairs.
• Filtering: Datasets (such as the LAION subset) have been filtered using an alignment confidence threshold (e.g., > 0.80) based on Sentence-Transformers (LaBSE).
• Normalization: Basic character normalization is applied for the Mooré language to standardise orthography and maintain data quality.

Intended Use

This dataset is intended to be used for training and fine-tuning neural machine translation systems for the French-Mooré language pair, supporting the preservation and digitalization of the Mooré language.