Segformer85Mv1

Segformer85M — Apple Orchard Semantic Segmentation

Segformer-B5 (85M parameters) fine-tuned for 8-class semantic segmentation of outdoor apple orchard scenes captured from a robotic platform.

This repo contains two checkpoints:

Quick Use

from huggingface_hub import hf_hub_download
from transformers import SegformerForSemanticSegmentation
import torch, cv2, numpy as np
import torch.nn.functional as F

# 1. Download weights — pick v1 OR v2
ckpt_path = hf_hub_download(repo_id="WEN0256/Segformer85Mv1", filename="Segformer85Mv2.pt")
#                                                                       ^^^^^^^^^^^^^^^^^
#                                                                       use v2 by default

# 2. Init architecture from base + load fine-tuned weights
NAMES = ["tree","ground","person","sky","road","mountain","building","background"]
model = SegformerForSemanticSegmentation.from_pretrained(
    "nvidia/segformer-b5-finetuned-ade-640-640",
    num_labels=8,
    id2label={i:n for i,n in enumerate(NAMES)},
    label2id={n:i for i,n in enumerate(NAMES)},
    ignore_mismatched_sizes=True,
).cuda().eval()
model.load_state_dict(torch.load(ckpt_path, map_location="cuda")["model"])

# 3. Inference
img = cv2.imread("your_image.jpg")
H, W = img.shape[:2]
H32, W32 = (H//32)*32, (W//32)*32
rgb = cv2.cvtColor(cv2.resize(img, (W32, H32)), cv2.COLOR_BGR2RGB).astype(np.float32) / 255.0
mean = np.array([0.485, 0.456, 0.406]); std = np.array([0.229, 0.224, 0.225])
x = torch.from_numpy(((rgb - mean) / std).transpose(2,0,1)).unsqueeze(0).float().cuda()

with torch.no_grad():
    logits = model(pixel_values=x).logits
    logits = F.interpolate(logits, size=(H, W), mode="bilinear", align_corners=False)
    pred = logits.argmax(1)[0].cpu().numpy()  # H x W, values 0..7

A ready-to-use predict.py is included in this repo.

Classes (id → name)

Architecture & Preprocessing

Performance

v1 (Segformer85Mv1.pt) — original training only

Validated on a temporally-disjoint hold-out from the same recording (frames 4501+, no leakage):

v2 (Segformer85Mv2.pt) — v1 + Orchard Navigation fine-tune ⭐

Same v1 hold-out → no regression on old domain:

NEW orchard hold-out (different camera, autumn season — Aug+Sep capture):

Visual qualitative: v1 sometimes misclassifies autumn foliage as person (red); v2 cleanly segments it as tree. See samples/ for side-by-side examples.

v1 per-class IoU (8-class, no leak)

Training Data

v1 base

• ~5300 frames from a single oak_0415_oneRadar_1 recording (spring, single camera)
• Initial annotations from 3 separate Roboflow projects (SAM-assisted polygons), merged + class-aligned (vines→tree, moutain→mountain typo fixed)
• Pseudo-labels generated by an earlier model to fill SAM annotation gaps
• Temporal split: frames <=4500 train (5177 samples), frames >4500 validation (155 samples) — no neighbor leakage

v2 fine-tune (NEW)

• +311 images from "Orchard Navigation" dataset:
  • 178 frames from a Sep-16 recording (autumn season)
  • 134 frames from a Windows webcam capture (Aug 23, different camera/sensor)
• Tree-only polygon annotations
• Mixed with 500 sampled v1 images (full 8-class masks) to prevent forgetting
• Non-tree pixels in new images set to ignore_index=255 so the model only adapts its tree decisions, leaving other classes untouched

Training Recipe

v1

v2 fine-tune (delta from v1)

Limitations

This model was trained on a single Korean apple orchard (spring 2024) with a single robot platform, plus a small fine-tune on a second autumn capture. Expect degradation on:

• ⚠️ Different orchards (different tree species, layouts, training systems)
• ⚠️ Different cameras (different FOV, color profiles, sensors)
• 💀 Different seasons not in training (winter dormant trees)
• 💀 Different lighting (rain, dawn/dusk, night)
• 💀 Aerial / drone perspectives

For deployment in a new context, plan to fine-tune on 100-300 in-domain images.

Files in This Repo

License

Apache 2.0