indobert-ner-citizen-reports

IndoBERT NER for Citizen Reports

This project aims to build a Named Entity Recognition (NER) system to automatically extract key information from citizen reports related to public infrastructure issues in Indonesia.
The trained NER model is designed to be integrated into a web-based reporting application, enabling automatic classification of user reports into multiple entity categories.

---

🚀 Key Features

• Domain-specific NER for public infrastructure reports
• Entity Categories:
  • LOC → Location
  • INFRA → Infrastructure type
  • PROB → Problem type
  • TIME → Time
  • DESC → Additional description
  • SEV → Severity level
• Dataset preprocessing from real citizen reports
• Automatic annotation using dictionary-based entity labeling
• CoNLL dataset format for NER training
• Fine-tuned IndoBERT (taufiqdp/indonesian-sentiment)
• Evaluation metrics: F1, Precision, Recall
• Ready-to-use model for backend integration (Flask, FastAPI, or Streamlit)

---

📊 Model Training

• Base model: IndoBERT (taufiqdp/indonesian-sentiment)
• Task: Named Entity Recognition (token classification)
• Dataset: Citizen reports on public infrastructure issues (preprocessed & automatically annotated)
• Format: CoNLL with BIO tagging scheme
• Evaluation Results:
  • Precision: 0.88%
  • Recall: 0.91%
  • F1-score: 0.89%

---

💻 Usage

from transformers import AutoTokenizer, AutoModelForTokenClassification, pipeline

# Model repo on Hugging Face
model_name = "Ysfii-Dev/indobert-ner-citizen-reports"

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForTokenClassification.from_pretrained(model_name)

# Create NER pipeline (using aggregation to merge tokens into entities)
ner_pipeline = pipeline(
    task="ner",
    model=model,
    tokenizer=tokenizer,
    aggregation_strategy="simple"  # "simple" merges tokens into full entities
)

# Example input text
text = "Lampu jalan di Jl. Sudirman mati sejak malam kemarin."

# Run prediction
entities = ner_pipeline(text)

# Print results
for entity in entities:
    print(entity)

Secara default, pipeline ner di Hugging Face akan mengembalikan hasil prediksi per-token. Jadi kalau teksmu panjang, misalnya:

"Lampu jalan di Jl. Sudirman mati sejak malam kemarin."

tanpa aggregation_strategy, hasilnya bisa seperti ini (per kata/token):

[
  {'entity': 'B-INFRA', 'word': 'Lampu', 'score': 0.98},
  {'entity': 'I-INFRA', 'word': 'jalan', 'score': 0.97},
  {'entity': 'B-LOC', 'word': 'Jl.', 'score': 0.95},
  {'entity': 'I-LOC', 'word': 'Sudirman', 'score': 0.96},
  {'entity': 'B-PROB', 'word': 'mati', 'score': 0.92},
  {'entity': 'B-TIME', 'word': 'sejak', 'score': 0.91},
  {'entity': 'I-TIME', 'word': 'malam', 'score': 0.90},
  {'entity': 'I-TIME', 'word': 'kemarin', 'score': 0.89},
]

Jadi potongan entitas masih terpisah-pisah per kata.

Nah, kalau kamu tambahkan parameter:

pipeline(..., aggregation_strategy="simple")

Maka hasilnya akan digabung jadi satu entitas utuh, misalnya:

[
  {'entity_group': 'INFRA', 'word': 'Lampu jalan', 'score': 0.975},
  {'entity_group': 'LOC', 'word': 'Jl. Sudirman', 'score': 0.955},
  {'entity_group': 'PROB', 'word': 'mati', 'score': 0.920},
  {'entity_group': 'TIME', 'word': 'sejak malam kemarin', 'score': 0.900},
]

---

📌 Applications

• Automated extraction of structured information from unstructured citizen reports
• Public infrastructure monitoring systems
• Government service platforms for faster issue resolution

---

📜 License

This model is released under the MIT License.

---

🙌 Acknowledgements

• IndoBERT pretrained model by taufiqdp
• Hugging Face Transformers
• Citizen report datasets on public infrastructure issues