asl-unified-yolo11n-mp

ASL Unified Gesture Detector (YOLO11n, MediaPipe boxes)

This model detects American Sign Language (ASL) alphabet gestures (A–Z) and digits (0–9) in images and video. It is intended for use in gesture-controlled systems, such as commanding a mobile robot swarm.

Model details

• Architecture: YOLO11n (Ultralytics)
• Input size: 512×512
• Classes (36):
  • Digits: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9
  • Letters: A, B, C, D, E, F, G, H, I, J, K, L, M, N, O, P, Q, R, S, T, U, V, W, X, Y, Z
• Training framework: Ultralytics YOLO
• Box generation: MediaPipe HandLandmarker-based bounding boxes

Dataset

Trained on a unified YOLO-format dataset derived from:

1. ASL Alphabet Dataset (Kaggle)
   (A–Y, no J/Z, no digits)
2. ASL Dataset (Kaggle)
   (0–9 + A–Z, including J and Z)

Both datasets were converted to YOLO detection format with MediaPipe-based bounding boxes, 80/20 train/val split, and unified class mapping (36 classes). Filenames were prefixed by source and class to avoid collisions.

Details about the conversion and class mapping are available in the associated GitHub repository.

Metrics

Validation metrics (val split, 1,006 images, 36 classes):

• Precision: 0.942
• Recall: 0.940
• mAP50: 0.961
• mAP50–95: 0.911

Key command classes

These classes are particularly important for robot-swarm control:

Full per-class metrics are available in the training logs.

Usage

With Ultralytics (PyTorch)

from ultralytics import YOLO
import cv2

model = YOLO("asl_unified_yolo11n_mp_best.pt")

cap = cv2.VideoCapture(0)
while True:
    ret, frame = cap.read()
    if not ret:
        break
    results = model(frame, conf=0.7, iou=0.45, verbose=False)
    boxes = results.boxes
    if len(boxes) > 0:
        cls = int(boxes.cls)
        conf = float(boxes.conf)
        symbol = model.names[cls]
        print(f"Detected: {symbol} (conf={conf:.2f})")
    cv2.imshow("frame", frame)
    if cv2.waitKey(1) & 0xFF == ord("q"):
        break

cap.release()
cv2.destroyAllWindows()

With the gesture_swarm package

from gesture_swarm.detector import GestureDetector

detector = GestureDetector(
    "asl_unified_yolo11n_mp_best.pt",
    conf_thresh=0.7,
)

# frame: OpenCV BGR image (e.g., from cv2.VideoCapture)
symbol = detector.detect_from_frame(frame)
if symbol is not None:
    print("Detected symbol:", symbol)

You can integrate this detector into a pipeline that accumulates symbols into words or command sequences (e.g., ["B", "2"] → “move forward 2 units”) and maps them to structured swarm commands.

Intended use

• Real-time gesture recognition for human–robot interaction.
• Mapping ASL letters and digits to commands for a mobile robot swarm.
• Prototyping gesture-based interfaces in research, education, and robotics projects.

Limitations

• Trained primarily on cropped, relatively controlled hand images; performance may drop in very cluttered, low-light, or highly dynamic environments.
• Single-hand detection; multi-hand scenarios are not explicitly modeled.
• Some classes (e.g., 0, 2, 6, O, M) have lower mAP than others; consider adding domain-specific data if these are critical for your application.
• Not evaluated for safety-critical applications.

How to cite

If you use this model in a paper, project report, or demo, please cite it as:

@misc{jdanoff2026aslunifiedyolo11nmp,
  author       = {Alexander Jdanoff},
  title        = {{ASL Unified Gesture Detector (YOLO11n, MediaPipe boxes)}},
  year         = {2026},
  howpublished = {\url{[https://huggingface.co/ajdanoff/asl-unified-yolo11n-mp](https://huggingface.co/ajdanoff/asl-unified-yolo11n-mp)}},
  note         = {YOLO11n model for ASL alphabet (A–Z) and digits (0–9) gesture detection, trained with MediaPipe-based bounding boxes.}
}

In text, you can refer to it as:

“We use the ASL Unified Gesture Detector (YOLO11n, MediaPipe boxes) [Jdanoff, 2026], available at https://huggingface.co/ajdanoff/asl-unified-yolo11n-mp.”

License & acknowledgements

Model weights and code: MIT License (see LICENSE).
Datasets: see the original Kaggle dataset licenses and attribution requirements.

This project uses:

• Ultralytics YOLO
• MediaPipe