sum-flan-t5-base-synthetic-v1

Fine-tuned FLAN-T5 for Child Helpline Case Summarization

This model is a fine-tuned version of google/flan-t5-base specifically optimized for summarizing child helpline case call transcripts. It has been trained on domain-specific data to better understand and summarize conversations involving child protection issues.

Model Description

• Base Model: google/flan-t5-base
• Architecture: T5ForConditionalGeneration
• Language: English
• Parameters: 248M
• Task: Text Summarization
• Domain: Child Protection/Helpline Conversations

Key Improvements Over Base Model

Domain Specialization

• Base Model: Generic text-to-text transformer trained on diverse internet content
• Fine-tuned Model: Specialized for child helpline case summarization with understanding of:
  • Child protection terminology and concepts
  • Helpline conversation patterns and structures
  • Sensitive case reporting protocols
  • Legal and procedural references specific to child welfare

Enhanced Performance

• Contextual Understanding: Better comprehension of child welfare scenarios including child labor, forced marriage, abuse cases
• Structured Summaries: Generates concise, actionable summaries that capture key information:
  • Caller identity and location
  • Nature of the concern/issue
  • Action items and referrals provided
• Sensitive Content Handling: Trained to appropriately summarize sensitive child protection cases while maintaining essential details

Technical Specifications

Usage

from transformers import T5ForConditionalGeneration, T5Tokenizer
import torch

# Load model and tokenizer
model = T5ForConditionalGeneration.from_pretrained("openchs/sum-flan-t5-base-synthetic-v1")
tokenizer = T5Tokenizer.from_pretrained("openchs/sum-flan-t5-base-synthetic-v1")

# Generate summary
def generate_summary(text: str, max_length: int = 256) -> str:
    input_text = f"Summarize the following child helpline case call transcript:{text}"
    
    inputs = tokenizer(
        input_text,
        max_length=1024,
        padding='max_length',
        truncation=True,
        return_tensors='pt'
    )
    
    with torch.no_grad():
        outputs = model.generate(
            input_ids=inputs['input_ids'],
            attention_mask=inputs['attention_mask'],
            max_length=max_length,
            num_beams=4,
            length_penalty=2.0,
            early_stopping=True,
            no_repeat_ngram_size=2
        )
    
    summary = tokenizer.decode(outputs[0], skip_special_tokens=True)
    return summary

# Example usage
transcript = """
Hi, is this 116? Yes, thank you for calling. Who am I speaking to? 
My name is John, I'm from Mwanza. I've got a serious concern about my 12-year-old sister. 
She's being forced into child labor at a local factory...
"""

summary = generate_summary(transcript)
print(summary)
# Output: "John reported a case of child labor involving his 12-year-old sister in Mwanza. 
# The counselor advised him to report it to the local Labor Office and police, with follow-up from the helpline."

Example Outputs

Child Labor Case

Input: Complex transcript about 12-year-old forced into factory labor Output: "John reported a case of child labor involving his 12-year-old sister in Mwanza. The counselor advised him to report it to the local Labor Office and police, with follow-up from the helpline."

Child Marriage Case

Input: Conversation about forced marriage prevention Output: "Mariam reported a case of child marriage involving her dad in Kisauni. The counselor advised reporting the issue to the children's office and police, and offered follow-up support."

Training Data

The model was fine-tuned on a curated dataset of child helpline call transcripts, focusing on various child protection scenarios including:

• Child labor cases
• Child marriage prevention
• Abuse reporting
• General child welfare concerns
• Referral and follow-up procedures

Intended Use

This model is specifically designed for:

• Child Protection Organizations: Automated summarization of case calls for documentation
• Helpline Services: Quick generation of case summaries for follow-up and reporting
• Social Workers: Efficient case documentation and handover summaries
• Research: Analysis of child protection case patterns and trends

Limitations

• Domain Specific: Optimized for child helpline conversations and may not perform well on other text types
• Language: Currently trained only on English transcripts
• Context Window: Limited to 1024 input tokens (approximately 700-800 words)
• Sensitive Content: While trained on sensitive material, human review is recommended for critical cases

Ethical Considerations

• This model handles sensitive information about child welfare cases
• Outputs should be reviewed by qualified professionals before use in official documentation
• Privacy and confidentiality protocols must be maintained when using this model
• The model is intended to assist, not replace, human judgment in child protection cases

Evaluation Metrics Comparison

Performance on Child Helpline Case Summarization Test Set

Domain-Specific Evaluation Metrics

Human Evaluation Results

Evaluated by child protection professionals on 100 test cases

Model Performance

Compared to the base FLAN-T5 model, this fine-tuned version shows significant improvements across all evaluation metrics:

Key Improvements:

• ** ROUGE Scores**: 50-84% improvement across ROUGE-1, ROUGE-2, and ROUGE-L metrics
• ** Domain Accuracy**: 94% accuracy in using child protection terminology (vs 52% for base model)
• ** Information Extraction**: 91% success rate in identifying key case details (vs 68% for base model)
• ** Action Item Detection**: 87% accuracy in identifying referrals and follow-up actions (vs 45% for base model)
• ** Professional Assessment**: 4.4/5 overall quality rating from child protection professionals (vs 2.8/5 for base model)

Performance Highlights:

• Relevance: Better identification of key information in child protection contexts
• Conciseness: More structured and actionable summaries with appropriate length
• Domain Accuracy: Proper use of child protection terminology and procedures
• Consistency: More reliable output format across different case types
• Professional Quality: Summaries meet standards for official case documentation

Citation

If you use this model in your research or applications, please cite:

@misc{flan-t5-child-helpline-summarizer,
  title={Fine-tuned FLAN-T5 for Child Helpline Case Summarization},
  author={openchs},
  year={2024},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/openchs/sum-flan-t5-base-synthetic-v1}}
}

License

This model inherits the Apache 2.0 license from the base FLAN-T5 model. Please ensure compliance with local data protection and child welfare regulations when using this model.

Contact

For questions about this model or its applications in child protection work, please contact [[email protected]].

Performance Metrics

Evaluation Results