llama32-1b-python-docstrings-qlora

Model Card for llama32-1b-python-docstrings-qlora

A parameter-efficiently fine-tuned adapter on top of meta-llama/Llama-3.2-1B-Instruct for generating concise one-line Python docstrings from function bodies.

Model Details

Model Description

• Developed by: Abdullah Al-Housni
• Model type: Causal language model with LoRA/QLoRA adapters
• Language(s): Python code as input, English docstrings as output
• License: Same as meta-llama/Llama-3.2-1B-Instruct (Meta Llama 3.2 Community License)
• Finetuned from model: meta-llama/Llama-3.2-1B-Instruct

The model is trained to take a Python function definition and generate a concise, one-line docstring describing what the function does.

Uses

Direct Use

• Automatically generate one-line Python docstrings for functions.
• Improve or bootstrap documentation in Python codebases.
• Educational use for learning how to summarize code behavior.

Typical usage pattern:

• Input: Python function body (source code).
• Output: Single-sentence English description suitable as a docstring.

Out-of-Scope Use

• Generating full, multi-paragraph API documentation.
• Security auditing or correctness guarantees for code.
• Use outside Python (e.g., other programming languages) without additional fine-tuning.
• Any safety-critical application where incorrect summaries could cause harm.

Bias, Risks, and Limitations

• The model can produce incorrect or incomplete summaries, especially for complex or ambiguous functions.
• It may imitate noisy or low-quality patterns from the training data (e.g., overly short or cryptic docstrings).
• It does not understand project-specific context, invariants, or business logic; outputs should be reviewed by a human developer.

Recommendations

• Use the model as an assistive tool, not an authoritative source.
• Always review and edit generated docstrings before committing to production code.
• For non-Python or highly domain-specific code, consider additional fine-tuning on in-domain examples.

How to Get Started with the Model

Example with 🤗 Transformers and PEFT (LoRA adapter):

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

base_model_id = "meta-llama/Llama-3.2-1B-Instruct"
adapter_id = "Abdul1102/llama32-1b-python-docstrings-qlora"

tokenizer = AutoTokenizer.from_pretrained(base_model_id)
model = AutoModelForCausalLM.from_pretrained(base_model_id, device_map="auto")
model = PeftModel.from_pretrained(model, adapter_id)

def make_prompt(code: str) -> str:
    return
        f'Write a one-line Python docstring for this function:\n\n{code}\n\n"""'

code = "def add(a, b):\n    return a + b"
inputs = tokenizer(make_prompt(code), return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=32, do_sample=False)
text = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(text)

Training Details

Training Data

• Dataset: Python subset of CodeSearchNet (Nan-Do/code-search-net-python)
• Inputs: code column (full Python function body)
• Targets: First non-empty line of docstring
• A filtered subset of ~1,000–2,000 examples was used for efficient QLoRA fine-tuning

Training Procedure

• Objective: Causal language modeling (predict the docstring continuation)
• Method: QLoRA (4-bit quantized base model with LoRA adapters)
• Precision: 4-bit quantized weights, bf16 compute
• Epochs: 1
• Max sequence length: 256–512 tokens

Training Hyperparameters

• Learning rate: ~2e-4 (adapter weights only)
• Epochs: 1
• Optimizer: AdamW via Hugging Face Trainer
• LoRA rank: 16
• LoRA alpha: 32
• LoRA dropout: 0.05

---

Evaluation

Testing Data, Factors & Metrics

Testing Data

Held-out test split from the same CodeSearchNet Python dataset, using identical code → one-line docstring mapping.

Factors

• Function size and complexity
• Variety in docstring writing styles
• Presence of short or noisy docstrings

Metrics

• BLEU (sacreBLEU): strict n-gram overlap, sensitive to paraphrasing
• ROUGE (ROUGE-1 / ROUGE-2 / ROUGE-L): better for short summaries

Results

Approximate performance on ~50 held-out samples:

• BLEU: ~12.4
• ROUGE-1: ~0.78
• ROUGE-2: ~0.74
• ROUGE-L: ~0.78

Summary

The model frequently reproduces or closely paraphrases the correct docstring. Occasional failures include echoing part of the prompt or returning an empty string. Strong performance for a 1B model trained briefly on a small dataset.

---

Model Examination

Not applicable.

---

Environmental Impact

• Hardware Type: Google Colab GPU (T4/L4)
• Hours Used: ~0.5–1 hour total
• Cloud Provider: Google Colab
• Compute Region: US
• Carbon Emitted: Not estimated (very low due to minimal training time)

---

Technical Specifications

Model Architecture and Objective

• Base model: Llama 3.2 1B Instruct
• Architecture: Decoder-only transformer
• Objective: Causal language modeling
• Parameter-efficient fine-tuning using LoRA (rank 16)

Compute Infrastructure

Hardware

Single Google Colab GPU (T4 or L4)

Software

• Python
• PyTorch
• Hugging Face Transformers
• PEFT
• bitsandbytes
• Datasets

---

Citation

Not applicable.

---

Glossary

Not applicable.

---

More Information

See the Hugging Face model page for updates or usage examples.

---

Model Card Authors

Abdullah Al-Housni

---

Model Card Contact

Available through the Hugging Face model repository.