etl-error-explainer-v2

ETL Error Explainer v2

Model Description

ETL Error Explainer v2 is a domain-specific instruction-tuned language model designed to analyze ETL and data pipeline failures.

Given an ETL error message and execution context (environment, severity, cloud provider), the model produces a structured JSON response containing:

• category
• root_cause
• immediate_fix
• long_term_fix

The model is intended to assist Data Engineers, Analytics Engineers, Platform Engineers, and DevOps teams during troubleshooting and incident response.

---

Model Details

Developed by

Fırat Çelik

Model type

Instruction-tuned causal language model

Base model

Qwen/Qwen2.5-1.5B-Instruct

Fine-tuning method

QLoRA (4-bit NF4 Quantization + LoRA)

Language

English

License

Apache-2.0

---

Model Sources

Hugging Face Repository

https://huggingface.co/firfircelik/etl-error-explainer-v2

Dataset

Synthetic ETL troubleshooting instruction dataset created by the author.

---

Intended Uses

Direct Use

The model is designed for structured ETL troubleshooting.

Example applications include:

• Spark error explanation
• SQL exception analysis
• Databricks job failures
• Azure Data Factory pipeline failures
• Kafka ingestion errors
• Snowflake query failures
• Schema evolution issues
• Authentication and permission errors
• Data validation failures

Input:

Context:
environment=prod
severity=High
cloud=Azure

Analyze the following Databricks failure:

org.apache.spark.sql.AnalysisException:
cannot resolve 'customer_email'

Output:

{
  "category": "...",
  "root_cause": "...",
  "immediate_fix": "...",
  "long_term_fix": "..."
}

---

Downstream Uses

The model can be integrated into:

• AI Data Engineering copilots
• Internal troubleshooting assistants
• Incident response systems
• RAG applications
• Chatbots
• IDE assistants
• Knowledge base generation
• Support automation

---

Out-of-Scope Uses

The model is not intended for:

• General-purpose chat
• Code generation
• SQL execution
• Production system diagnosis
• Security analysis
• Legal advice
• Medical advice

Outputs should always be reviewed before applying fixes to production environments.

---

Bias, Risks and Limitations

The model has been trained specifically for ETL troubleshooting.

Limitations include:

• It does not inspect live systems.
• It may suggest common fixes that are not applicable in every environment.
• It cannot validate infrastructure configurations.
• It may hallucinate missing context if insufficient information is provided.
• It should complement—not replace—human expertise.

---

Recommendations

Best performance is achieved when prompts include:

• Full error messages
• Stack traces
• Cloud provider
• Environment (dev/test/prod)
• Severity level
• Relevant SQL or Spark snippets

---

Getting Started

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "firfircelik/etl-error-explainer-v2"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

prompt = """
Context:
environment=prod
severity=High
cloud=Azure

Analyze the following Databricks failure:

org.apache.spark.sql.AnalysisException:
cannot resolve 'customer_email'
"""

inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=256
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

Training Details

Base Model

Qwen/Qwen2.5-1.5B-Instruct

Fine-Tuning Method

QLoRA

• 4-bit NF4 quantization
• LoRA adapters
• PEFT

LoRA Configuration

Optimizer

paged_adamw_8bit

Training Hyperparameters

---

Training Data

The model was trained on a synthetic instruction dataset covering realistic ETL failures.

Covered domains include:

• Apache Spark
• Databricks
• SQL
• Azure Data Factory
• Snowflake
• Kafka
• Delta Lake
• Data validation
• Schema evolution
• Authentication
• File ingestion
• Cloud storage

Each sample contains:

• execution context
• instruction
• structured target JSON

---

Evaluation

A held-out validation split (10%) was used during training.

Additionally, the model was manually tested using unseen ETL failure scenarios.

Evaluation focused on:

• JSON validity
• Root cause correctness
• Practical remediation quality
• Instruction following

---

Environmental Impact

Hardware

NVIDIA Tesla T4

Platform

Kaggle Notebooks

Training Method

QLoRA (4-bit)

Using QLoRA significantly reduces GPU memory usage and energy consumption compared to full fine-tuning.

---

Technical Specifications

Architecture

• Transformer Decoder
• Causal Language Modeling

Objective

Instruction-following generation of structured ETL troubleshooting responses.

Output format:

{
  "category": "...",
  "root_cause": "...",
  "immediate_fix": "...",
  "long_term_fix": "..."
}

---

Supported Domains

---

Citation

@misc{celik2026etlexplainer,
  author = {Fırat Çelik},
  title = {ETL Error Explainer v2},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/firfircelik/etl-error-explainer-v2}}
}

---

Contact

GitHub

https://github.com/firfircelik

Hugging Face

https://huggingface.co/firfircelik