category-of-needs-classification-bge-m3-v2

category-of-needs-classification-bge-m3-v2

A SetFit multi-label classifier fine-tuned on BAAI/bge-m3 to identify the categories of human need expressed in natural-language search queries on the Connect211 social-services search platform.

Other models in this family

Three backbone variants were trained and evaluated side-by-side. Choose based on your accuracy vs. latency vs. memory trade-offs:

bge-m3-v2 is the largest model in the family. Despite its size, it scores lower on accuracy metrics in our evaluation than the two lighter variants. It may be worth exploring for use cases where the richer multilingual embedding space of BGE-M3 is advantageous, but for most Connect211 production workloads mpnet-v2 is the recommended choice.

---

Purpose

When a person searches on Connect211 they often express a need in plain, informal language — "I need a place to stay tonight", "help with my electric bill", "my kid needs shots". This model classifies those raw queries into one or more of 79 structured need categories (derived from the AIRS-aligned Gravity L2 taxonomy), enabling Connect211 to:

1. Navigate its AIRS/211 service taxonomy automatically.
2. Augment semantic search with category-level signals to surface more relevant results.
3. Track population-level need trends over time.

---

Labels (79 categories)

The model was trained to predict all 79 unique categories in the Gravity L2 taxonomy. The gold standard evaluation below covers 55 of those 79 categories (spanning 18 real-world need scenarios). The remaining 24 categories were not included in the eval set but are fully supported by the model.

Evaluated (55):

Cash and Income Help          Checkups and Screenings       Child Care Programs
Childcare Cost Help           Clothing and Apparel          Criminal Justice Support
Daily Living Care Help        Devices and Technology        Digital Skills Training
Disability Support Help       Disaster Response Help        Drug Safety Programs
Education Programs            Emergency Shelter Now         Finding a Job
Food Benefits Programs        Free Food Programs            Free Health Supplies
Free Legal Services           General Safety Resources      Getting Free Rides
Getting Free Supplies         Getting Healthcare            Health Education Support
Help Paying Legal Costs       Help Paying Utility Bills     Help Paying for Care
Help Paying for Food          Help Paying for Rides         Help with Housing Costs
Help with Legal Costs         Home Items and Goods          Home Repairs and Safety
Housing Search Help           Housing with Extra Support    Internet Access Help
Job Skills Training           Keeping Your Home Stable      Learning New Skills
Legal System Navigation       Low-Cost Housing Programs     Lower Your Energy Costs
Medical Cost Help             Mental Health Treatment       Parenting and Family Help
Personal Finance Help         Personal and Family Safety    Phone and Internet Help
Stop Utility Shutoff          Substance Use Resources       Substance Use Treatment
Temporary Housing Help        Transportation Cost Help      Utility Bill Assistance
Vehicle Repair Help

Trained but not yet evaluated (24):

Consumer Rights Help          Digital Access Assistance     Disaster Preparedness
Disaster Recovery Help        Education Financial Aid        Funeral and Burial Help
Health and Safety Items       Healthy Eating Support        Help Paying for School
Keeping Your Home Safe        Long-Term Care Services       Mental Health Resources
Pet Care Assistance           Physical and Speech Therapy   Public Use Facilities
Safety Gear and Equipment     School Success Support        Small Business Support
Social Groups and Community   Tax Preparation Help          Travel Safety Programs
Unemployment Support          Work Costs and Supplies       Workplace Safety Help

---

Performance

All metrics were produced by a dedicated FastAPI inference server that wraps the fine-tuned SetFit model and applies a calibrated probability threshold per class. The server was containerised and load-tested independently; numbers below reflect end-to-end API latency, not raw model inference.

Accuracy — English gold standard (160 queries × 18 scenarios, 55 of 79 categories covered)

Note: bge-m3-v2 has the highest mean probability cosine similarity across all 38 languages (0.801 vs 0.647 for mpnet and 0.558 for minilm), suggesting its embedding space is more internally consistent across languages — even though its classification accuracy on this task is lower.

Multilingual — translated gold standard (38 languages, 160 queries each)

Languages evaluated include: Arabic, Amharic, Cantonese, Dari, Dutch, Fijian, Filipino, Finnish, French, German, Gujarati, Hawaiian, Hindi, Hmong, Haitian Creole, Japanese, Khmer, Korean, Lao, Malay, Mandarin (Simplified & Traditional), Nepali, Oromifaa, Persian, Polish, Portuguese, Punjabi, Romanian, Russian, Rwandan, Samoan, Somali, Spanish, Swahili, Tagalog, Ukrainian, Vietnamese.

Latency & resource usage (API server, containerised)

Idle memory: ~1,433 MB (largest of the three variants)

---

Training

• Framework: SetFit (few-shot contrastive fine-tuning)
• Base model: BAAI/bge-m3
• Task: Multi-label classification over 55 social-service need categories
• Training data: Synthetically generated labeled queries covering real-world search patterns observed on Connect211, augmented with human-reviewed examples

---

Usage

from setfit import SetFitModel

model = SetFitModel.from_pretrained("connect211/category-of-needs-classification-bge-m3-v2")

predictions = model.predict(["I need help paying my rent", "looking for food bank near me"])
print(predictions)

---

About Connect211

Connect211 is a social-services search platform that helps individuals and caseworkers find health and human services programs across the United States. The 211 taxonomy is maintained in alignment with AIRS standards.