jstor_classifier_colonial

Colonial JSTOR Article Classifier

Model Description

jstor_classifier_colonial is a fine-tuned roberta-large model that has been trained to identify scientific articles published on colonial or ethnological topics. We take the initialized roberta-large weights and fine tune on a custom dataset of titles of scientific article drawn from JSTOR's corpus between 1800-1900.

Usage

This model can be used with HuggingFace Transformer's Pipelines API for text classification:

from transformers import pipeline

classifier = pipeline(
"text-classification",
 model='imperial-science/jstor_classifier_colonial',
 device=0,
 truncation=True,
 max_length=512
)

example = """
The Races of Central Asia
"""

def classify(title):
    """
    Classify text(s) as either 'COLONIAL' or 'NOT'.

    Parameters:
    article (str or list of str): The text(s) to be classified. Can be a single string
                                  or a list of strings.

    Returns:
    list of dict: A list of dictionaries, each containing:
                  - 'label' (str): Either 'COLONIAL' or 'NOT'
                  - 'score' (float): The confidence score of the classification
  
    """
    if isinstance(article, str):
        article = [article]
    
    results = classifier(article)
    
    output = []
    for result in results:
        label = "COLONIAL" if result["label"] == "LABEL_1" else "NOT"
        output.append({'label': label, 'score': result["score"]})
    
    return output


print(classify(example))

Training data

The custom dataset consists of 5,584 titles of scientific articles drawn from JSTOR's corpus for 1800-1900, available upon request via Constellate.

Labels were generated by GPT-4o with the following prompt and a subsample of one-third was checked by a graduate student:

You are an expert in labelling for text classification, trained to analyze academic journal article titles from JSTOR for the 19th century.
        
The journal article titles may have OCR errors. 
        
Your task is to classify a given title based on its focus.
        
Specifically, you should determine if the title suggests that it is an article on topics relating to the British Empire, British imperialism, scientific exploration, colonists and settlers, indigenous people, race, ethnology, and anthropology, or other colonial topics relating to Britain (including those related to Ireland).
        
If the title is vague e.g. "OBSERVATIONS", then it is not an article relating to our topics of interest.
        
Do not provide an explanation. Simply respond in JSON format, so that if the title suggests an article on the topics from above, respond with 'response' of ‘yes’. If it is ‘not’, set 'response' to ‘no’.
        
Here is the title text:

We make no attempt to correct for OCR errors in any way beforehand, so that the model is exposed to these errors in training.

Our class balance for the labels was: 25.7% 'yes', and 74.3% 'no'.

The data was randomly split 60-20-20 (train-val-test), and fine tuning was done with a batch size of 110 and learning rate of 2e-5 for 40 epochs.

Evaluation

The F1 on the test set was 94.9%.