NTEE_category_tagging

• Dataset: Fine-tuning + Validation
• Primary Labels: All Labels-Weights=1.0
• Secondary Labels: Foreign Affairs & National Security and Mutual & Membership Benefit-Weight=0.5
• Training Data: The Dataset used had over 3.6k examples for training and validation
• Accuracy: Weighted F1 (fine-tuning)=0.95; Macro F1 (Holdout test)= 0.67
• Technical Specifications Document: Technical Specifications Link

---

Note

Important clarification on taxonomy and labels

This model does not implement the full official NTEE taxonomy across all levels. Instead, it predicts only a custom set of 28 cause-area categories, which are inspired by (but not identical to) NTEE “Level 2” major group categories. The 28 labels used here may differ from the canonical NTEE v2.0 definitions and counts (for example, NTEE’s 26 major groups).

References in this card to NTEE Level 1, Level 3, and Level 5 codes are provided for background and context only. The model does not directly predict those codes. All model outputs are limited to the 28 cause-area labels explicitly listed below under “Labels” / “Level 2 – Major Group Categories.”

Example usage

from transformers import BertTokenizer, BertForSequenceClassification
import torch
tokenizer = BertTokenizer.from_pretrained("GivingTuesday/NTEE_category_tagging")
# OR this works too:
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("GivingTuesday/NTEE_category_tagging")

# num_labels: 28 because this will assign one of 28 NTEE codes
model = BertForSequenceClassification.from_pretrained("GivingTuesday/NTEE_category_tagging", num_labels=28) 
model.eval()

# example texts
text = """NATIONAL CHURCH RESIDENCES OF SOUTH,PROVIDE HOUSING FOR LOW AND MODERATE INCOME PERSONS.,"THE SOLE PURPOSE IS TO PROVIDE ELDERLY PERSONS AND HANDICAPPED PERSONS WITH HOUSING FACILITIES AND SERVICES SPECIALLY DESIGNED TO MEET THEIR PHYSICAL, SOCIAL, AND PSYCHOLOGICAL NEEDS, AND TO PROMOTE THEIR HEALTH, SECURITY, HAPPINESS AND USEFULNESS IN LONGER LIVING, THE CHARGES FOR SUCH FACILITIES AND SERVICES TO BE PREDICATED UPON THE PROVISION, MAINTENANCE, AND OPERATION THEREOF ON A NONPROFIT BASIS."""
text = """NORTH CAROLINAS EASTERN ALLIANCE,"TO PROMOTE AND ENCOURAGE ECONOMIC DEVELOPMENT WITHIN EASTERN NORTH CAROLINA BY FOSTERING DEVELOPMENT PROJECTS TO PROVIDE LAND, BUILDINGS, FACILITIES, PROGRAMS, INFORMATION AND DATA SYSTEMS, AND INFRASTRUCTURE REQUIREMENTS FOR BUSINESS AND INDUSTRY WITHIN EASTERN NORTH CAROLINA.","HELPS TO RECRUIT NEW BUSINESSES INTO THE REGION, AS WELL AS HELP EXPAND EXISTING BUSINESSES BY ASSISTING WITH SITE LOCATIONS AND GRANTS. THEY ALSO WORK WITH THE LOCAL COMMUNITY COLLEGES TO EDUCATE THE CITIZENS OF THE REGION SO TO MAKE THE AREA MORE APPEALING TO POTENTIAL NEW BUSINESSES."""
inputs = tokenizer(text, return_tensors="pt")
#encoded_output = tokenizer.encode(text)
#print(tokenizer.convert_ids_to_tokens(encoded_output))

with torch.no_grad():
    output = model(**inputs)
    probabilities = torch.softmax(output.logits, dim=1)
    predicted_class_index = torch.argmax(output.logits, dim=1)
    predicted_label = model2.config.id2label[predicted_class_index.item()]
    print(predicted_label)

Easier line-by-line code available here: Notebook with Usecase

Non-Profit Mapping Project Documentation: Cause Area Segmentation

Author: Edward Moore - GivingTuesday Data Commons
Note for external readers: Some Databricks links in this document point to internal notebooks and may not be accessible to people outside GivingTuesday.

Introduction

The cause area segmentation uses a multi-stage NLP system to assign nonprofit organizations to detailed cause categories using mission statements and program descriptions. It supports top-level NTEE industry group codes and major group categories. The segmentation uses a classifier that integrates human-in-the-loop GPT labeling and fine-tunes a transformer model (BERT) to support multi-label prediction.

1. Approach

1.1 Definitions

1.1.1 Cause Area

A cause area refers to a structured, human-interpretable label that reflects the primary social mission of a nonprofit organization. This taxonomy draws from the National Taxonomy of Exempt Entities (NTEE) codes, version 2.0.

• Level 1 - Industry Group Codes
  1. ART - Arts, Culture, and Humanities
  2. EDU - Education
  3. UNI - Universities
  4. ENV - Environment and Animals
  5. HEL - Health
  6. HOS - Hospitals
  7. HMS - Human Services
  8. IFA - International, Foreign Affairs
  9. PSB - Public, Societal Benefit
  10. REL - Religion Related
  11. MMB - Mutual/Membership Benefit
  12. UNU - Unknown, Unclassified
• Labels / Level 2 - Major Group Categories
  1. Arts, Culture & Humanities (ART)
  2. Education - Universities (B40, B41, B42, B43) (UNI)
  3. Education - All Others (EDU)
  4. Environment (ENV)
  5. Animal-Related (ENV)
  6. Health Care - Hospitals (E20, E21, E22, E24) (HOS)
  7. Health Care - All Others (HEL)
  8. Mental Health & Crisis Intervention (HEL)
  9. Voluntary Health Associations & Medical Disciplines (HEL)
  10. Medical Research (HEL)
  11. Crime & Legal-Related (HMS)
  12. Employment (HMS)
  13. Food, Agriculture & Nutrition (HMS)
  14. Housing & Shelter (HMS)
  15. Public Safety, Disaster Preparedness & Relief (HMS)
  16. Recreation & Sports (HMS)
  17. Youth Development (HMS)
  18. Human Services (HMS)
  19. International, Foreign Affairs & National Security (IFA)
  20. Civil Rights, Social Action & Advocacy (PSB)
  21. Community Improvement & Capacity Building (PSB)
  22. Philanthropy, Voluntarism & Grantmaking Foundations (PSB)
  23. Science & Technology (PSB)
  24. Social Science (PSB)
  25. Public & Societal Benefit (PSB)
  26. Religion-Related (REL)
  27. Mutual & Membership Benefit (MMB)
  28. Unknown (UNU)
• Level 3 - Division / Level 4 - Subdivision
  • ~650 categories
• Level 5 - Organization Type Codes
  • RG - Regular Nonprofit
  • AA - Alliance/Advocacy Organizations (*formerly 01*)
  • MT - Management and Technical Assistance (*formerly 02*)
  • PA - Professional Societies/Associations (*formerly 03*)
  • RP - Research Institutes and/or Public Policy Analysis (*formerly 05*)
  • MS - Monetary Support - Single Organization (*formerly 11*)
  • MM - Monetary Support - Multiple Organizations (*formerly 12*)
  • NS - Nonmonetary Support Not Elsewhere Classified (N.E.C.) (*formerly 19*)

1.1.2 Primary vs. Secondary Classification

• Primary Classification: The single best match for the organization’s mission and activities.
• Secondary Classification: An additional relevant category.

1.2 Variables

The variables that are used in this segmentation include:

• FILERNAME1 (Header - Line C - Name of Charity Line 1)
• FILEREIN (Header - Line D - EIN)
• MISSION (990 Missions Data Mart: Part I, Line 1 - Mission & Most Significant Activities)
• ACTIVIDESCRI1 (990 Programs Data Mart: Part III, Line 4a - Program 1 Free Text)
• ACTIVIDESCRI2 (990 Programs Data Mart: Part III, Line 4b - Program 2 Free Text)
• ACTIVIDESCRI3 (990 Programs Data Mart: Part III, Line 4b - Program 3 Free Text)
• OPERATHOSPIT (990 Basic Fields Data Mart: Part IV-20-a - Organizations that operate one or more hospital facilities)
• MAONINLOEXXP (990 Basic Fields Data Mart: Part V-16 - Organizations that are educational institutions subject to the section 4968 excise tax)

1.3 Approach

1.3.1 Overview

The classifier operates in three main phases:

1. Embedding-based candidate selection using vector similarity.
2. GPT-assisted annotation for human aligned soft labeling.
3. Supervised fine-tuning of a BERT- based model with weighted loss.

1.3.2 Data Preprocessing

A sample of 10,000 organizations from 2019 990 filings were selected for model training and evaluation. For each organization, a single text input was created combining EIN, Name, Mission, and up to three Activities.

Embedding-Based Candidate Selection:

1. Label Definition Embedding:
   For each of the 28 supported cause areas, definitions were curated based on IRS Activity Code Definitions. The definitions were embedded using the all-mpnet-base-v2 sentence-transformers model, which maps text to a 768 dimensional dense vector space.
2. Organization Text Embedding:
   For each organization, the mission and activities were concatenated into a single input string, which was then embedded using the all-mpnet-base-v2 sentence-transformers model.
3. Top 3 Label Retrieval:
   Cosine similarity was computed between the organization embedding and each label embedding to identify the top three most semantically similar cause areas.

1.3.3 GPT-Enhanced Labeling

Using an organization’s EIN, Name, Mission, and Activities along with top 3 candidate labels from the embedding model, OpenAI’s GPT-4-turbo model was prompted to choose the primary and secondary cause area classification. The top 3 candidate labels from the embedding model were included in the prompt to improve the accuracy and efficiency by providing context and guidance to the LLM. The system and user prompts used are as follows:

system_prompt = """
    You are an expert in classifying nonprofits into the correct cause area category based on their mission statement and program activities.
   
    Given a nonprofit's mission and activities, you must:
    1. Choose the **primary** category code (from A-Z, BB, EE) that best matches.
    2. Choose the **secondary** category code that could also apply (or null if none).

    Special rules:
    - If the nonprofit supports causes outside the US, primary must be 'Q'.
    - If the nonprofit fits definitions for 'Y' (Mutual & Membership Benefit), primary must be 'Y'.
    - Always return exactly **two codes**: primary, secondary.
    - If no clear primary match: primary is 'Z' (Unknown).
    - If no clear secondary: secondary is 'null'.

    Respond only with the two codes, separated by a comma, e.g., 'B, P' or 'Z, null'.

    Category Codes:
    {cause_areas_map}

    Category Definitions:
    {category_definitions_text}
    """

user_prompt = """
    Mission Statement and Activities:
    {full_text}

    Top {top_n} likely categories from embedding model: {top_matches}

    What are the primary and secondary categories?
    """

1.3.3 Dataset Splitting

The annotated dataset of 10,000 organizations was split in two phases:

1. Holdout Test Set (20% of 10,000):
   A stratified 80/20 split based on the primary label was performed to create a test set of 2,000 organizations, held out entirely for final model evaluation.
2. Fine-Tuning / Validation (80% of 10,000)
   The remaining 8,000 organizations were further split into 90/10 using stratification based on the primary label, resulting in 7,200 organizations for model fine-tuning and 800 organizations for validation during training.

1.3.4 Fine-Tuning the BERT Classifier with Weighted Loss

The goal of this stage was to fine-tune a pre-trained BERT model (bert-base-uncased) to classify nonprofit organizations into one of 28 possible cause area labels.

Model Input:

• Each annotated organization contained a primary and secondary label
• A weighted loss approach was used to incorporate both primary and secondary labels into the fine-tuning, allowing the model to set higher importance on the primary labels
• Several iterations were performed using different approaches to weighting the primary and secondary labels (see Section 3 - Outputs and Results)
• A custom subclass of Hugging Face’s Trainer was implemented to apply the sample weights and the weights were preserved using a custom data collator.

Model Output:
The fine-tuned classifier outputs raw logits over the 28 cause area classes, which are converted to the top-2 predictions (primary and secondary), along with confidence scores (softmax probability). The validation dataset (800 orgs) is used to produce a weighted F1 score for evaluation of the fine-tuning.

Model Evaluation
The model was evaluated on the hold-out test dataset (2,000 orgs) using both macro and weighed F1 score.

2. Code documentation

2.1 Notebooks

1. Dataset Creation
   Dataset Creation Notebook (Databricks)
   In this notebook, a random sample of nonprofits is taken from the datamarts to create a training dataset and a testing dataset. Embeddings are computed using NTEE definitions and org descriptions to identify top-3 semantically similar cause areas. GPT then uses the semantic top matches to return a structured classification.
2. BERT-Training
   BERT Training Notebook (Databricks)
   In this notebook, the bert-base-uncased model is fine-tuned on the GPT-labeled dataset. The top-2 cause area predictions are produced, along with confidence, and F1 scores (macro and weighted) are computed.
3. IRS Validation
   IRS Validation Notebook (Databricks)
   In this notebook, the predictions of the test dataset (embedding, GPT, and LLM) are compared to the IRS-assigned cause areas from the Business Master File. This step gives us an understanding of how often the IRS-assigned codes are inaccurate and is essential for future work of identifying orgs that serve underserved populations.

2.2 Tools Used

• Sentence-transformers embedding model (all-mpnet-base-v2)
  • Provides vector representation of NTEE definitions and org descriptions to identify top-3 semantically similar cause areas.
• OpenAI GPT-4-Turbo
  • Refines semantic top matches and returns a structured classification (primary/secondary group).
• HuggingFace Transformers (bert-base-uncased)
  • Fine-tuned for multiclass classification into 28 cause areas. Predictions used for evaluation and potential deployment.
• HuggingFace Datasets
  • Used to store and serve fine-tuning and test datasets.
• MLflow + Databricks
  • For experiment tracking, training orchestration, and model registry.

3. Outputs and Results

4. List of approaches that did not work

• Zero shot classification
• Using only the mission statement (excluding activities). The mission statement is often short and broad, whereas the activities include much more detail
• Using the activity code definitions to fine tune an LLM
• Keyword Matching Only: Too imprecise for broad categories like “Human Services” or “Public Benefit.”
• Embedding-Only Classification: Performed reasonably but lacked the reasoning nuance of GPT.
• GPT Classification Without Embedding Context: Sometimes misclassified due to lack of grounding in NTEE definitions.
• Training without Label Consistency: Early attempts used inconsistent labeling schemes, which reduced model performance. Switching to GPT-labeled, semantically grounded training examples improved results.

6. Plan for Deploy, Scale, Archive, and Expose-data steps

• The fine-tuned LLM can be hosted publicly on a GT huggingface account
• The final output can be a table of organizations with the following rows:
  • EIN
  • Tax Year
  • Cause Area 1
  • Cause Area 1 Likelihood
  • Cause Area 2
  • Cause Area 2 Likelihood

5. Future work

1. Run entire dataset through the classification model
2. Add OPERATHOSPIT (990 Basic Fields Data Mart: Part IV-20-a - Organizations that operate one or more hospital facilities) and MAONINLOEXXP (990 Basic Fields Data Mart: Part V-16 - Organizations that are educational institutions subject to the section 4968 excise tax) variables to the final classified dataset to improve identification of Universities and Hospitals.
3. Expand the training dataset to include an equal quantity of organizations for each of the 28 labels, most likely achieved by generating mission statements and activities for undersampled labels with the help of GPT (current version undersamples some labels due to the natural distribution of cause areas in the 990 dataset). This would likely improve the F1 scores, most significantly the macro F1.