aws-enterprise-assistant-qwen2.5-7b-qlora

AWS Enterprise Assistant — Qwen2.5-7B QLoRA

A domain-specific AI assistant fine-tuned on official AWS documentation using QLoRA (Quantized Low-Rank Adaptation). Answers questions about AWS services with grounded, accurate responses and refuses out-of-scope questions.

---

Model Details

Model Description

This model is a QLoRA fine-tune of Qwen2.5-7B-Instruct, trained on 407 instruction-following Q&A pairs generated from official AWS documentation. It is designed to answer questions about AWS cloud services accurately, with built-in domain refusal for non-AWS questions. The full system includes FAISS-based retrieval (RAG) and a FastAPI backend.

• Developed by: Debarun Das (Debarun12)
• Model type: Causal Language Model (instruction-tuned, QLoRA adapter)
• Language: English
• License: Apache 2.0
• Finetuned from: Qwen/Qwen2.5-7B-Instruct
• PEFT Version: 0.13.0

Model Sources

• Repository: https://github.com/Debarun12/aws-enterprise-ai-assistant
• HuggingFace: https://huggingface.co/Debarun12/aws-enterprise-assistant-qwen2.5-7b-qlora

---

Uses

Direct Use

Load the adapter on top of Qwen2.5-7B-Instruct to answer AWS-related questions about EC2, S3, Lambda, IAM, RDS, VPC, CloudWatch, and DynamoDB.

Downstream Use

Plug into a RAG pipeline with a FAISS vector store built from AWS documentation chunks for grounded, source-cited answers. A FastAPI backend with streaming support is included in the project repository.

Out-of-Scope Use

This model is not intended for general-purpose conversation, code generation, or non-AWS topics. It will refuse questions outside the AWS domain by design.

---

How to Get Started with the Model

from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel
import torch

base_model_name = "Qwen/Qwen2.5-7B-Instruct"
adapter_name = "Debarun12/aws-enterprise-assistant-qwen2.5-7b-qlora"

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(adapter_name, trust_remote_code=True)

# Load base model in 4-bit
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
)

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, adapter_name)
model.eval()

# Inference
messages = [
    {"role": "system", "content": "You are an expert AWS cloud assistant."},
    {"role": "user", "content": "What is the difference between IAM roles and IAM users?"},
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.inference_mode():
    outputs = model.generate(**inputs, max_new_tokens=300, temperature=0.3, do_sample=True)

answer = tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(answer)

---

Training Details

Training Data

• Source: Official AWS documentation (publicly available at docs.aws.amazon.com)
• Services: EC2, S3, Lambda, IAM, RDS, VPC, CloudWatch, DynamoDB
• Pages scraped: 53 documentation pages
• Chunks: 211 text chunks (~460 words average)
• Q&A pairs: 407 instruction-following examples
• Generation model: Qwen2.5-7B via Ollama (local)
• Question types: Factual (211), Procedural (196)

Training Procedure

Preprocessing

Raw HTML pages were scraped from AWS documentation, cleaned with BeautifulSoup, and chunked into 512-word overlapping segments. Each chunk was passed to a local Qwen2.5-7B model to generate factual and procedural Q&A pairs in Alpaca instruction format. Pairs were formatted using Qwen2.5's chat template via apply_chat_template.

Training Hyperparameters

• Training regime: bf16 mixed precision
• LoRA rank (r): 8
• LoRA alpha: 16
• LoRA dropout: 0.05
• Target modules: q_proj, v_proj
• Quantization: 4-bit NF4 with double quantization
• Optimizer: paged_adamw_8bit
• Learning rate: 2e-4
• LR scheduler: cosine
• Epochs: 3
• Batch size: 1 (effective 16 with gradient accumulation)
• Max sequence length: 512 tokens
• Warmup ratio: 0.05

Speeds, Sizes, Times

• Hardware: NVIDIA GeForce RTX 5050 (8.5GB VRAM)
• Training time: ~17 minutes
• Adapter size: ~9.8MB (adapter_model.safetensors)
• Trainable parameters: 2,523,136 / 7,618,139,648 (0.03%)

---

Evaluation

Testing Data

9 held-out questions covering all 8 AWS services, plus 3 out-of-scope questions for refusal testing.

Metrics

Confidence scores are derived from cosine similarity between query embeddings and retrieved documentation chunks (FAISS IndexFlatIP with normalized vectors).

Results

Summary

The model achieves consistent confidence scores of 0.68–0.78 on AWS domain questions and correctly refuses 100% of out-of-scope questions. Answers are grounded in retrieved AWS documentation via FAISS RAG.

---

Bias, Risks, and Limitations

• AWS documentation changes over time; the training data reflects a snapshot from June 2026.

Recommendations

Always verify critical AWS configuration decisions against the official AWS documentation. Use the confidence score as a signal — responses below 0.5 should be treated with caution.

---

Environmental Impact

• Hardware: NVIDIA GeForce RTX 5050 (laptop GPU)
• Hours used: 0.28 hours (~17 minutes)
• Cloud Provider: None (local training)
• Carbon Emitted: Negligible (local, short training run)

---

Technical Specifications

Model Architecture

• Base: Qwen2.5-7B-Instruct (decoder-only transformer)
• Adapter: LoRA on q_proj + v_proj attention layers
• Inference: 4-bit NF4 quantized base + bf16 LoRA computation
• RAG: FAISS IndexFlatIP with sentence-transformers/all-MiniLM-L6-v2 embeddings

Software

• PyTorch 2.12.0 (CUDA 12.8)
• Transformers 4.46.0
• PEFT 0.13.0
• TRL 0.9+
• FAISS 1.14.3
• Sentence-Transformers 5.5.1
• FastAPI 0.112+

---

Model Card Authors

Debarun Das — GitHub | HuggingFace

Framework Versions

• PEFT 0.13.0
• Transformers 4.46.0