sbert-obe-csematch

sbert-obe-csematch — Continual Learning v3.0

OBE AI System | Course Outcome (CO) Semantic Matching Engine for Computer Science Engineering

This is a domain-specific Sentence-BERT model fine-tuned on Outcome-Based Education (OBE) data for Computer Science & Engineering courses. It maps exam questions and course outcome descriptions into a shared 384-dimensional dense vector space, enabling high-accuracy semantic matching of exam questions to their corresponding Course Outcomes (COs).

---

🚀 What's New in v3.0 (Continual Learning Release)

• Warm-started from existing fine-tuned weights (domain knowledge preserved)
• Trained on the expanded & balanced OBE_Augmented_Dataset (7,009 rows vs 6,387 previously)
• Top-1 Accuracy improved from 54.6% → 82.4% (+50.9% relative gain)
• Macro F1-Score improved from 47.9% → 79.6% (+66.1% relative gain)
• Cosine Separation Margin improved from 0.089 → 0.214 (+139.6% discriminative boost)
• All embeddings remain 384-dimensional — zero breaking changes to downstream APIs

---

📋 Model Details

Architecture Modules

SentenceTransformer(
  (0): Transformer[BertModel] — 6 layers, 12 heads, 384 hidden
  (1): Pooling — mean pooling strategy
  (2): Normalize — L2 normalization
)

---

🏋️ Training Configuration

Data Split Strategy

Groups are defined by (Course Name, CO) pairs, ensuring all augmented variants of the same exam question land entirely in train OR validation — zero data leakage guaranteed by assertion.

---

📊 Benchmark Results (N=500, seed=42)

Evaluated on 500 randomly sampled questions (23 courses) from Unique_Master_Questions. Each question is matched only against the valid COs of its own course.

Cosine Separation Margin = Mean Positive Similarity − Mean Negative Similarity. A higher margin means the model is more discriminative between correct and distractor COs.

---

💻 Usage

Basic CO Matching

from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer("obe-ai-system/sbert-obe-csematch")

# Exam question
question = "Implement a Binary Search Tree and explain its traversal methods."

# Course outcome candidates
course_outcomes = [
    "CO1: Implement fundamental data structures including trees, graphs, and heaps.",
    "CO2: Analyze time and space complexity of common algorithms.",
    "CO3: Apply sorting and searching algorithms to solve computational problems.",
    "CO4: Design recursive and iterative solutions to complex programming problems.",
]

# Encode and compute similarity
q_emb  = model.encode(question, normalize_embeddings=True)
co_emb = model.encode(course_outcomes, normalize_embeddings=True)
scores = util.cos_sim(q_emb, co_emb)[0]

# Rank by similarity
ranked = sorted(zip(course_outcomes, scores.tolist()), key=lambda x: x[1], reverse=True)
for co, score in ranked:
    print(f"{score:.4f}  {co[:60]}")

Batch Evaluation

import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("obe-ai-system/sbert-obe-csematch")

questions = ["...", "..."]  # list of exam questions
co_descs  = ["...", "..."]  # corresponding CO descriptions

q_embs  = model.encode(questions, normalize_embeddings=True, batch_size=64)
co_embs = model.encode(co_descs,  normalize_embeddings=True, batch_size=64)

# Pair-wise cosine similarity
similarities = np.einsum("ij,ij->i", q_embs, co_embs)
print(f"Mean cosine similarity: {similarities.mean():.4f}")

---

🗂️ Repository Files

---

🔌 Integration — OBE AI System FastAPI Service

This model powers the /api/v1/co-match endpoint of the OBE AI System:

# POST /api/v1/co-match
{
    "question": "Explain the concept of virtual memory and page replacement.",
    "course_code": "CSE 311",
    "top_k": 3
}

# Response
{
    "top_matches": [
        {"co_id": "CO4", "similarity": 0.8721, "description": "Analyze working procedure and structure of main and virtual memory."},
        {"co_id": "CO3", "similarity": 0.7423, "description": "..."},
        {"co_id": "CO2", "similarity": 0.6912, "description": "..."}
    ],
    "embedding_dim": 384,
    "model_version": "sbert-obe-csematch-v3.0"
}

---

📚 Training Data

The model was trained on the OBE Augmented Dataset covering 15+ CSE courses:

Data types: Original Exam (2,049) + Class Test (211) + Augmented (4,749) = 7,009 total rows

---

⚙️ Technical Notes

• Backward compatible: All embeddings are 384-dimensional — no changes to downstream vector stores, similarity pipelines, or APIs required
• Warm-start training: v3.0 built upon v2.x fine-tuned weights using continual learning (domain knowledge preserved, catastrophic forgetting minimized)
• Group-aware data split: (Course Name, CO) grouping prevents augmented variants of the same question from leaking across train/val splits
• In-batch hard negatives: MultipleNegativesRankingLoss treats other CO descriptions in the same batch as hard negatives, improving discriminative ability

---

📈 Training Logs

---

🧰 Framework Versions

---

📖 Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}