setfit_baai_newrelic_gpt-4o_improved-cot-instructions_chat_few_shot_generated_remove_

Netta1994/setfit_baai_newrelic_gpt-4o_improved-cot-instructions_chat_few_shot_generated_remove_

setfit_baai_newrelic_gpt-4o_improved-cot-instructions_chat_few_shot_generated_remove_ is a text classification model from Netta1994. Use it when you need a label for a piece of text. It is set up for setfit.

This is a SetFit model that can be used for Text Classification. This SetFit model uses BAAI/bge-base-en-v1.5 as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

Downloads · 30 days

9% of all-time downloads

All-time downloads

Public

Parameters

109M

1.8 GB on disk

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.safetensors438 MB · 100%

At a glance

Task: Text Classification
Library: setfit
Model type: bert
Access: Public
Created: Sep 19, 2024
Updated: Sep 23, 2024
SHA: b802d9f3

Try a prompt

Base models

BAAI/bge-base-en-v1.5

Model card

From the Hugging Face model README

SetFit with BAAI/bge-base-en-v1.5

The model has been trained using an efficient few-shot learning technique that involves:

Fine-tuning a Sentence Transformer with contrastive learning.
Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

Model Type: SetFit
Sentence Transformer body: BAAI/bge-base-en-v1.5
Classification head: a LogisticRegression instance
Maximum Sequence Length: 512 tokens
Number of Classes: 2 classes

Model Sources

Repository: SetFit on GitHub
Paper: Efficient Few-Shot Learning Without Prompts
Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Label	Examples
1	<ul><li>'Reasoning: \nThe answer provided is directly relevant to the question about how to cancel a trade in Yahoo Fantasy Football after it has been accepted. It states the specific requirement that 4 out of 10 team owners need to vote to veto the trade within 48 hours for it to be cancelled. This information is well-supported by the document provided. The answer is concise and to the point, addressing the specific question without adding unnecessary details.\n\nFinal result:'</li><li>'Reasoning:\nThe answer correctly identifies Joan Gaspart as the individual who resigned from the presidency of Barcelona after the team's poor showing in the 2003 season. This is directly supported by the document, which explicitly states that "club president Joan Gaspart resigned, his position having been made completely untenable by such a disastrous season on top of the club's overall decline in fortunes since he became president three years prior." The answer is concise and directly relevant to the question without including any extraneous information.\n\nEvaluation:'</li><li>"Reasoning:\n1. Context Grounding: The answer addresses the support for employee wellness and highlights the Headspace app, which matches the content in the documents that describe ORGANIZATION reimbursing Headspace subscriptions for employees.\n2. Relevance: The answer directly addresses the question by discussing the support systems in place for employee wellness, particularly the Headspace subscription.\n3. Conciseness: The answer is concise and directly provides the relevant information without unnecessary details.\n4. Direct Response: The answer does not attempt to extrapolate beyond what is provided in the document and sticks to the information available.\n5. Specificity: The answer covers the specifics about the Headspace subscription being reimbursed monthly, aligning well with the document's details.\n6. Additional Information: While the answer is correct, including a mention of the fitness subscription as another wellness support option would have been beneficial if provided in the document.\n\nOverall, the answer is well-supported, relevant, and concise, thus appropriately addressing the question based on the provided document.\n\nFinal annotation: \nEvaluation:"</li></ul>
0	<ul><li>'Reasoning:\nThe answer provided is mostly well-supported by the document and directly addresses the question of how to care for snails. It includes several key points such as providing a well-ventilated tank, using moist peat or soil as substrate, adding climbing objects, and providing calcium sources like cuttlefish bone or eggshells. It also mentions feeding snails fresh fruits and vegetables and maintaining a small water dish—all of which align with the document. \n\nHowever, the answer introduces a point that snails enjoy listening to classical music, which is not mentioned or supported by the document. This inclusion is not grounded in the provided information, and it introduces unnecessary and possibly distracting information.\n\nFinal Evaluation: \n\nEvaluation:'</li><li>'Reasoning:\nThe answer "James Marcia divides questions into five categories" is incorrect according to the information provided in the document. The document clearly states that James Marcia's questions are divided into three categories: occupation, ideology, and interpersonal relationships. Thus, the response does not accurately reflect the document's content and fails to address the specific question asked.\nEvaluation:'</li><li>'Reasoning:\nThe answer provided is "The Kroc Institute at Notre Dame focuses on International Peace Studies." This response relates to information provided in the document, specifically about the Kroc Institute. However, the question asked is about "What company did Ray Kroc own?" The answer does not address the specific question and insteadprovides irrelevant information.\n\nEvaluation:'</li></ul>

Label

Examples

<ul><li>'Reasoning: \nThe answer provided is directly relevant to the question about how to cancel a trade in Yahoo Fantasy Football after it has been accepted. It states the specific requirement that 4 out of 10 team owners need to vote to veto the trade within 48 hours for it to be cancelled. This information is well-supported by the document provided. The answer is concise and to the point, addressing the specific question without adding unnecessary details.\n\nFinal result:'</li><li>'Reasoning:\nThe answer correctly identifies Joan Gaspart as the individual who resigned from the presidency of Barcelona after the team's poor showing in the 2003 season. This is directly supported by the document, which explicitly states that "club president Joan Gaspart resigned, his position having been made completely untenable by such a disastrous season on top of the club's overall decline in fortunes since he became president three years prior." The answer is concise and directly relevant to the question without including any extraneous information.\n\nEvaluation:'</li><li>"Reasoning:\n1. Context Grounding: The answer addresses the support for employee wellness and highlights the Headspace app, which matches the content in the documents that describe ORGANIZATION reimbursing Headspace subscriptions for employees.\n2. Relevance: The answer directly addresses the question by discussing the support systems in place for employee wellness, particularly the Headspace subscription.\n3. Conciseness: The answer is concise and directly provides the relevant information without unnecessary details.\n4. Direct Response: The answer does not attempt to extrapolate beyond what is provided in the document and sticks to the information available.\n5. Specificity: The answer covers the specifics about the Headspace subscription being reimbursed monthly, aligning well with the document's details.\n6. Additional Information: While the answer is correct, including a mention of the fitness subscription as another wellness support option would have been beneficial if provided in the document.\n\nOverall, the answer is well-supported, relevant, and concise, thus appropriately addressing the question based on the provided document.\n\nFinal annotation: \nEvaluation:"</li></ul>

<ul><li>'Reasoning:\nThe answer provided is mostly well-supported by the document and directly addresses the question of how to care for snails. It includes several key points such as providing a well-ventilated tank, using moist peat or soil as substrate, adding climbing objects, and providing calcium sources like cuttlefish bone or eggshells. It also mentions feeding snails fresh fruits and vegetables and maintaining a small water dish—all of which align with the document. \n\nHowever, the answer introduces a point that snails enjoy listening to classical music, which is not mentioned or supported by the document. This inclusion is not grounded in the provided information, and it introduces unnecessary and possibly distracting information.\n\nFinal Evaluation: \n\nEvaluation:'</li><li>'Reasoning:\nThe answer "James Marcia divides questions into five categories" is incorrect according to the information provided in the document. The document clearly states that James Marcia's questions are divided into three categories: occupation, ideology, and interpersonal relationships. Thus, the response does not accurately reflect the document's content and fails to address the specific question asked.\nEvaluation:'</li><li>'Reasoning:\nThe answer provided is "The Kroc Institute at Notre Dame focuses on International Peace Studies." This response relates to information provided in the document, specifically about the Kroc Institute. However, the question asked is about "What company did Ray Kroc own?" The answer does not address the specific question and insteadprovides irrelevant information.\n\nEvaluation:'</li></ul>

Evaluation

Metrics

Label	Accuracy
all	0.7313

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("Netta1994/setfit_baai_newrelic_gpt-4o_improved-cot-instructions_chat_few_shot_generated_remove_")
# Run inference
preds = model("Reasoning:
1. **Context Grounding**: The answer appropriately uses information from the document to address the query. The email provided (finance@ORGANIZATION_2.<89312988>) matches the reference in Document 1.

2. **Relevance**: The answer is relevant to the question and does not deviate into unrelated topics. It directly addresses the contact point for questions on travel reimbursement.

3. **Conciseness**: The answer is clear and to the point, effectively communicating the necessary information without unnecessary details.

4. **Specificity**: The email address given in the answer is taken directly from the provided document, ensuring the response is specific and accurate.

5. **Overall Completeness**: The answer meets all the criteria mentioned and does not attempt to address unrelated issues or give overly general information.

Final evaluation:")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	33	97.7083	289

Label	Training Sample Count
0	127
1	137

Training Hyperparameters

batch_size: (16, 16)
num_epochs: (2, 2)
max_steps: -1
sampling_strategy: oversampling
num_iterations: 20
body_learning_rate: (2e-05, 2e-05)
head_learning_rate: 2e-05
loss: CosineSimilarityLoss
distance_metric: cosine_distance
margin: 0.25
end_to_end: False
use_amp: False
warmup_proportion: 0.1
l2_weight: 0.01
seed: 42
eval_max_steps: -1
load_best_model_at_end: False

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0015	1	0.2055	-
0.0758	50	0.2629	-
0.1515	100	0.2514	-
0.2273	150	0.2486	-
0.3030	200	0.2145	-
0.3788	250	0.1535	-
0.4545	300	0.0866	-
0.5303	350	0.0335	-
0.6061	400	0.0101	-
0.6818	450	0.0041	-
0.7576	500	0.0027	-
0.8333	550	0.0024	-
0.9091	600	0.0022	-
0.9848	650	0.0024	-
1.0606	700	0.002	-
1.1364	750	0.0018	-
1.2121	800	0.0016	-
1.2879	850	0.0016	-
1.3636	900	0.0015	-
1.4394	950	0.0014	-
1.5152	1000	0.0015	-
1.5909	1050	0.0013	-
1.6667	1100	0.0013	-
1.7424	1150	0.0013	-
1.8182	1200	0.0012	-
1.8939	1250	0.0013	-
1.9697	1300	0.0013	-

Framework Versions

Python: 3.10.14
SetFit: 1.1.0
Sentence Transformers: 3.1.1
Transformers: 4.44.0
PyTorch: 2.4.0+cu121
Datasets: 3.0.0
Tokenizers: 0.19.1

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}

Task: Text Classification
Library: setfit
Type: bert
Created: Sep 19, 2024
Updated: Sep 23, 2024