drywall-qa-clipseg

<p align="center"> <h1 align="center">Prompted Segmentation for Drywall QA</h1> <p align="center"> Text-conditioned binary mask prediction for construction defect detection </p> </p>

<p align="center"> <a href="#1-methodology">Methodology</a> &bull; <a href="#2-data-preparation">Data Preparation</a> &bull; <a href="#3-results">Results</a> &bull; <a href="#4-failure-cases--potential-solutions">Failure Cases</a> &bull; <a href="#quick-start">Quick Start</a> &bull; Full Report (PDF) </p>

---

Feed a construction photo and a text prompt. Get a binary segmentation mask back.

Two tasks — crack detection and drywall taping/joint detection — both driven by natural language at inference time. Change the prompt, change what gets segmented. No class heads, no retraining.

Input:  image.jpg  +  "segment wall crack"
Output: image__segment_wall_crack.png   (binary mask, {0, 255})

---

1. Methodology

Model: CLIPSeg

We fine-tune CLIPSeg (Luddecke & Ecker, CVPR 2022) — a text-conditioned segmentation model built on CLIP. The entire CLIP backbone (149.6M params) stays frozen. Only a lightweight 3-block transformer decoder with U-Net skip connections (1.13M params) is trained.

The model takes an RGB image and a text prompt. The CLIP vision encoder (ViT-B/16) and text encoder independently produce embeddings. The decoder fuses these via cross-attention and generates logits at 352x352, which are thresholded at 0.5 to produce binary masks.

<details> <summary><b>Why CLIPSeg over Grounded SAM, SEEM, X-Decoder?</b></summary> <br>

CLIPSeg is the only architecture that gives direct text-to-mask conditioning without bounding box intermediates, fine-tunes reliably on small datasets, and runs on consumer hardware with mature HuggingFace support.

</details>

Training Configuration

<details> <summary><b>Why BCEDiceLoss instead of standard BCE?</b></summary> <br>

Standard BCE alone fails on thin structures like cracks — the severe foreground/background imbalance means BCE happily predicts "all background" at low loss. Dice loss directly optimizes overlap, forcing the model to find crack pixels. The 50/50 blend gives gradient stability (BCE) and overlap-awareness (Dice).

</details>

Training Pipeline

Training converged at epoch 11 (val mIoU 0.1605). The remaining 7 epochs showed no improvement before early stopping triggered at epoch 18.

All hyperparameters: configs/train_config.yaml

---

2. Data Preparation

Sources

Two datasets from Roboflow Universe, downloaded manually in COCO format:

Note: The cracks dataset had 0 generated Roboflow versions — the owner never created an exportable version, making API download impossible. The raw export was downloaded directly from the website.

Mask Rendering

• Cracks: COCO polygon annotations rendered to pixel-accurate binary masks using pycocotools.mask. Some annotations had empty segmentation fields (edge case) — handled with try/except fallback to bounding box rendering.
• Taping: Only bounding box annotations available. Filled rectangles used as mask approximations. This is a known limitation — the rectangles include substantial background, which affects training signal quality.

Prompt Augmentation

5 synonyms per class, randomly sampled each training iteration. This forces the decoder to learn semantic meaning from the text encoder rather than memorize exact strings:

Pipeline

Splits

Stratified by class (taping vs cracks), seed 42:

Preprocessing code: src/data/preprocess.py · Dataset class: src/data/dataset.py

---

3. Results

Best Predictions

The model's strongest predictions reach IoU 0.78 on both cracks and taping:

Test-Set Metrics (985 samples)

Taping outperforms cracks because filled-rectangle masks provide a stronger supervision signal (larger contiguous regions) compared to thin crack annotations where minor spatial offsets cause disproportionate IoU drops.

Inference

---

4. Failure Cases & Potential Solutions

Worst Predictions

The model's worst predictions (IoU near zero) reveal systematic failure patterns:

What's going wrong in these examples:

• Cracks (rows 1–3): The model activates over broad wall regions instead of tracing the thin crack lines. Fine cracks disappear at 352x352 resolution, and the frozen CLIP backbone has no features for hairline construction defects. The predictions show the model "knows something is there" but can't localize it precisely.
• Taping (rows 4–6): The model predicts large rectangular blobs that don't match the actual joint locations. This directly traces back to the filled-rectangle training masks — the model learned to predict rectangles because that's what it was supervised on.

Root Causes

Proposed Solutions

---

Repo Structure

<details> <summary><b>File-by-file listing</b></summary> <br> </details>

---

Quick Start

Prerequisites: Python 3.11+, uv, Homebrew (macOS)

brew install graphviz plantuml typst d2
uv sync

1. Get the data

Download both datasets from Roboflow Universe in COCO format → place under data/raw/:

data/raw/
├── taping/          # drywall-join-detect (COCO export)
│   ├── train/
│   └── valid/
└── cracks/          # cracks-3ii36 (COCO export)
    └── train/

2. Preprocess

uv run python -m src.data.preprocess

3. Train

uv run python -m src.train

4. Evaluate

uv run python -m src.evaluate

5. Predict on a single image

uv run python -m src.predict path/to/image.jpg "segment crack"

6. Build the report

d2 reports/diagrams/pipeline.d2 reports/diagrams/pipeline.png
plantuml -tpng reports/diagrams/training.puml
uv run python reports/diagrams/architecture.py
typst compile reports/report.typ reports/report.pdf

---

Reproducibility

• All random state seeded with 42 (data splits, PyTorch, NumPy).
• Hyperparameters: configs/train_config.yaml.
• Per-epoch training logs: outputs/logs/.

---

<p align="center"> <a href="https://huggingface.co/youngPhilosopher/drywall-qa-clipseg/blob/main/reports/report.pdf"><b>Read the full report (PDF)</b></a> </p>