religious_org_v1

Technical Specifications Document is available at: Technical Specifications

Non-Profit Mapping Project Documentation: Religious Orgs Segmentation

Author: Zilun Lin - GivingTuesday Data Commons
Note for external readers: Some Databricks links in this document point to internal notebooks and may not be accessible to people outside GivingTuesday.

1. Approach

Definition

We use the following definition for categorizing religious orgs provided in the academic literature:
“Religious organizations are organizations whose identity and mission are derived from a religious or spiritual tradition and which operate as registered or unregistered, nonprofit, voluntary entities.” (Source: Montclair.ed)
This definition is operationalized in how we prompt GPT 4 to classify the training and testing datasets. Namely, we give it information on the name, mission statement and key activities and prompt it to find mentions/wording/terminology that reveal an org’s religious affiliations.

Religious Recipient Orgs

Using the 990 CN file and BERT classifier:
(1) Categorise orgs by religion (if any) using name (23F990-LINE-C), mission statement (23F990-PART-03-LINE-1) and activities (20F990-PART-03-LINE-4A, 4B, 4C).

Using the 990 EZ file and BERT classifier:
(1) Categorise orgs by religion (if any) using name (23F990-LINE-C), primary exempt purpose (F9_03_PZ_MISSIODESCES) and program accomplishment description(F9_03_PZ_PRSRACACDEES).

2. Code documentation

The code has two parts. The first consists of a notebook where the fine-tuning and testing datasets are constructed and uploaded to huggingface. This latter notebook is stored on databricks. The second consists of a Google Colab notebook where the actual fine-tuning of the model is done and its accuracy is tested. We use Google Colab because fine-tuning libraries can’t seem to be run on my databricks environment and I couldn’t figure out how to fix it. Google Colab also has the upside of giving me access to the much more powerful A100 GPUs (after paying a small fee) that is 10x quicker for fine-tuning.

All of the notebooks should be reasonably documented. Please message Zilun Lin if there are any mistakes or missing documentation.

Classifying and formatting datasets

This notebook randomly samples from the 990 datamart and classifies the sample orgs using GPT4. It also generates a curated dataset of artificial orgs that are associated with under-represented religions. These two datasets are combined, formatted into an appropriate instruct-prompt-output format for fine-tuning and uploaded to HuggingFace. The final dataset has over 2k examples for training and validation, and 500 examples for testing.

Link to EDA Notebook (Databricks)

Fine-tuning the LLM and testing for accuracy

We downloaded the fine-tuning dataset from HuggingFace and fine-tuned a set of LLMs. The resulting models are uploaded to HuggingFace. We also test these model’s accuracy on an unseen testing dataset.

(Llama Models)
Llama Model Fine-tuning (Google Collab)

(Bert models)
BERT Model Fine-tuning (Google Collab)

3. Outputs and Results

The two notebooks’ final outputs are two models:

• BERT base curated
• Llama 3.2 3B curated

The curated models are fine-tuned using the combined dataset of actual and artificial orgs. The curated models should have better accuracy on under-represented orgs.

The following are the accuracy measures (higher is better) for the fine-tuned/trained models.

• BERT base curated:
  • Weighted F1 score: 0.93
  • Macro F1 score: 0.76
• Llama 3.2 3B curated:
  • Weighted F1 score: 0.85
  • Macro F1 score: 0.27

The following are the speed of inference measures (lower is better) for the fine-tuned/trained models.

• BERT base curated time taken for 500 inferences: 3 seconds.
• Llama 3.2 3B curated time taken for 500 inferences: ~10 minutes

When comparing models for text classification, BERT stands out for its fast inference speed and strong classification performance, as indicated by its F1 scores. The F1 score is a measure that balances precision and recall, offering a robust metric for evaluating classification tasks. BERT performs better than LLaMa in both the weighted and the macro F1 metrics, suggesting that it has a high accuracy in general and is adept at categorizing less frequent religious affiliations as well.

Aside from less accurate classification, the larger Llama 3.2 3B model also has a major drawback: its inference speed is significantly slower, and hosting the model is resource-intensive. For example, a linear extrapolation suggests that categorizing 200,000 organizations with Llama 3.2 3B could take approximately 4,000 minutes. This makes it less practical for scenarios where processing speed is a priority.

In comparison, BERT is much faster for inference, thanks to its streamlined model architecture, which is specifically optimized for tasks like classification. This makes BERT a more practical choice for applications requiring both speed and great classification performance.

4. Deployment

The curated BERT model is now hosted on MLFlow (Databricks) in the model registry under the name `religious_orgs_model`, and has been released to the public under the apache-2 license on Huggingface. The processed data will be available for download in a data mart or API.

The API endpoint will output five fields, three for BERT classification and two based on 1023 EZ data availability:
BERT Natural Language Outputs:
(1) Religious Classification (and classification probability)
(2) Classification probability for the designated religious affiliations (and classification probability)
(3) Classification probability for whether the organisation is religious or not (and probability)

5. Example usage

This is a minimal-code example of how to use this model to classify texts as related to a religion (or none). This model is public and doesn't require login to download. Use snapshot_download to get the latest version of the full model.

repo_id = 'GivingTuesday/religious_org_v1'
from huggingface_hub import snapshot_download
snapshot_download(repo_id)

# snapshot_download will return / print a location for the model. Pass that into the function below.
from transformers import pipeline
# 1. Define the local path to your downloaded model directory
# Replace './my_local_model_directory' with the actual path on your machine
model_path = './my_local_model_directory' # copied from output of previous cell

# 2. Load the text classification pipeline from the local directory
# The 'pipeline' function automatically loads the model, tokenizer, and config
# from the specified local path.
try:
    classifier = pipeline(
        task="text-classification",
        model=model_path,
        tokenizer=model_path
    )
    print(f"Model loaded successfully from {model_path}")
except Exception as e:
    print(f"Error loading model: {e}")
    print("Please ensure the directory contains the model weights, config, and tokenizer files.")
    exit()

# 3. Define the string(s) you want to classify
sentences = [
    """St. John's Episcopal is a really beautiful community that preaches "faith in action" -- they welcome and assist migrants, partner with community organizations, run a housing first apartment building for unhoused folks, and they are part of the Together Colorado coalition of faith-based organizations that lobbies for anti-poverty legislation. Bonus: the cathedral is gorgeous and historic.""",
    """I’m atheist but rehearse in a community choir at Christ Church United Methodist and they have pro LGBTQ+ stuff everywhere. They even do a reconciliation Sunday service every year in June during pride month""",
]

# 4. Run the classification on the input string(s)
results = classifier(sentences)

# 5. Print the results
for i, result in enumerate(results):
    print(f"\nText: '{sentences[i]}'")
    print(f"Classification Results:")
    # Results can be a list of dictionaries if top_k is used
    if isinstance(result, list):
        for label_info in result:
            print(f"- Label: {label_info['label']}, Score: {label_info['score']:.4f}")
    else:
        print(f"- Label: {result['label']}, Score: {result['score']:.4f}")

Output (actual reddit comments):

Text: 'St. John's Episcopal is a really beautiful community that preaches "faith in action" -- they welcome and assist migrants, partner with community organizations, run a housing first apartment building for unhoused folks, and they are part of the Together Colorado coalition of faith-based organizations that lobbies for anti-poverty legislation. Bonus: the cathedral is gorgeous and historic.'
Classification Results:
- Label: Christianity, Score: 0.8822

Text: 'I’m atheist but rehearse in a community choir at Christ Church United Methodist and they have pro LGBTQ+ stuff everywhere. They even do a reconciliation Sunday service every year in June during pride month'
Classification Results:
- Label: Christianity, Score: 0.9899