timesformer-dairy-cows

TimeSformer — Dairy Cow Behaviour Recognition

Fine-tuned TimeSformer for seven-class behaviour recognition in dairy cattle, developed as part of the MooAnalytica digital twin framework at Dalhousie University.

This model forms the behavioural perception layer of an AI-driven dairy cattle digital twin, converting continuous barn surveillance video into structured per-cow behavioural state streams used for nutrition modelling and welfare monitoring.

---

Model Description

• Base model: facebook/timesformer-base-finetuned-k400 (Kinetics-400 pretrained)
• Architecture: TimeSformer with divided space-time attention
• Input: 12-frame clips at 1.6 fps (10-second windows), 224×224 px crops
• Output: 7-class behaviour label per clip

Behaviour Classes

---

Training Data

• Source: Dalhousie University Ruminant Animal Centre (RAC), Truro, Nova Scotia — March 2025
• Setup: 3 overhead CCTV cameras, 17 Holstein dairy cows
• Annotated clips: 4,964 manually annotated video clips
• Augmented training set: 9,600 clips (oversampling of minority classes)
• Validation set: 689 clips (held-out, no augmentation)
• Split: Clip-level random train/val split (80/20)

---

Performance

Evaluated on the held-out validation set (n = 689 clips):

Per-class performance highlights: Feeding & Standing (F1=0.944), Lying (F1=0.912), Drinking (F1=0.905). Lowest: Standing (F1=0.720) and Ruminating & Standing (F1=0.782) due to postural similarity.

---

Usage

import torch
import numpy as np
from transformers import TimesformerForVideoClassification, AutoConfig
from huggingface_hub import hf_hub_download

# Load model
model_id = "shreyayayay/timesformer-dairy-cows"
config = AutoConfig.from_pretrained(model_id)
model = TimesformerForVideoClassification.from_pretrained(
    model_id,
    config=config,
    ignore_mismatched_sizes=True
)
model.eval()

ID2LABEL = {
    0: "Standing",
    1: "Lying",
    2: "Drinking",
    3: "Feeding & Standing",
    4: "Feeding & Lying",
    5: "Ruminating & Standing",
    6: "Ruminating & Lying",
}

IMNET_MEAN = np.array([0.485, 0.456, 0.406], dtype=np.float32)
IMNET_STD  = np.array([0.229, 0.224, 0.225], dtype=np.float32)

def preprocess_clip(frames):
    arr = np.stack(frames).astype(np.float32) / 255.0
    arr = (arr - IMNET_MEAN) / IMNET_STD
    arr = arr.transpose(0, 3, 1, 2)
    return torch.from_numpy(arr).unsqueeze(0)

with torch.no_grad():
    pixel_values = preprocess_clip(frames).to(next(model.parameters()).device)
    logits = model(pixel_values=pixel_values).logits
    pred = int(torch.argmax(logits, dim=-1))
    print(f"Predicted behaviour: {ID2LABEL[pred]}")

Loading weights directly

import torch
from huggingface_hub import hf_hub_download

weights_path = hf_hub_download(
    repo_id="shreyayayay/timesformer-dairy-cows",
    filename="last.pt"
)
checkpoint = torch.load(weights_path, map_location="cpu")

---

Integration with MooAnalytica Pipeline

Video → YOLOv11 detection → ByteTrack tracking → TimeSformer classification → Nutrition model (M-M DMI estimation)

Full pipeline code: github.com/raoshreya17/dairycow-digitaltwin

---

Training Details

---

Limitations

• Trained on a single barn environment (RAC, Truro NS); cross-farm generalisation not yet validated
• Nighttime performance is 6.7 percentage points lower than daytime due to infrared noise
• Identity persistence across camera cuts relies on ByteTrack; brief identity switches (~8% of interrupted tracks) may introduce short temporal labelling noise
• Lying and Ruminating & Lying are occasionally confused due to near-identical posture

---

Citation

@article{rao2026mooanalytica,
  title     = {An AI-Driven Behavioural Perception and Nutrition Modelling Framework
               as the Behavioural State Layer of a Dairy Cattle Digital Twin},
  author    = {Rao, Shreya and Garcia, Eduardo and Neethirajan, Suresh},
  journal   = {npj Veterinary Sciences},
  volume    = {1},
  number    = {3},
  year      = {2026},
  publisher = {Nature Publishing Group},
  doi       = {10.1038/s44433-026-00004-x}
}

---

License

MIT — see LICENSE

Contact

Shreya Rao — MCS candidate, Dalhousie University
Supervisor: Dr. Suresh Neethirajan
GitHub: raoshreya17