human-activity-recognition

Human Action Recognition: Hybrid Deep Learning Framework

This repository contains the source code, models, and deployment configurations for a robust hybrid deep learning system designed for Human Action Recognition (HAR)[cite: 1]. This project was developed as an Open Ended Lab for the Deep Learning (CS – 405) course at the Military College of Signals, National University of Sciences & Technology (NUST)[cite: 1].

For a comprehensive, detailed breakdown of the methodology, experimental setup, and architectural choices, please refer to the official document: DL OEL Report.pdf.

📌 Project Overview

The primary objective of this project is to address the temporal modeling limitations of traditional CNNs and recurrent models by combining spatial feature extraction with Transformer-based temporal modeling[cite: 1]. The framework supports multimodal learning through distinct processing branches for video and sensor data[cite: 1].

Key Features

• Hybrid Architecture: Integration of CNNs (for spatial/local features) and Transformers (for long-range temporal dependencies)[cite: 1].
• Temporal Attention: Custom attention mechanisms allow the network to dynamically focus on the most informative video frames or sensor timesteps[cite: 1].
• Computational Efficiency: Leverages dilated convolutions, frozen early CNN layers, and label smoothing to maximize performance while minimizing computational overhead[cite: 1].
• Production Deployment: Deployed as a scalable RESTful API bridging deep learning research with a real-world web application[cite: 1].

---

📊 Datasets

The architecture was trained and evaluated on two distinct benchmark datasets to represent visual and sensor modalities:

1. Video Modality (UCF101): Features over 13,000 video clips categorized into 101 diverse action classes[cite: 1]. Processing involves extracting 16 uniform frames per clip, resized to 112x112 pixels for efficiency[cite: 1].
2. Sensor Modality (WISDM): Contains time-series data tracking 18 distinct daily activities[cite: 1]. Raw accelerometer and gyroscope data were merged into a 6-channel vector and segmented using overlapping sliding windows of 200 timesteps[cite: 1].

---

🧠 Model Architecture

Video Branch (CNN-Transformer)

• Spatial Feature Extraction: A pre-trained ResNet50 backbone (with early layers frozen) extracts a 2048-dimensional feature vector per frame[cite: 1].
• Temporal Modeling: Features pass through a 1D Dilated Convolutional layer to widen the receptive field, followed by a 2-layer Transformer Encoder[cite: 1].
• Pooling & Classification: A Soft Temporal Attention mechanism dynamically highlights important frames, passing a weighted sum to an MLP classification head[cite: 1].

Sensor Branch (1D-CNN & Transformer)

• Convolutional Stem: A lightweight 1D-CNN utilizes decreasing kernel sizes and dilated convolutions to capture local rhythmic motion patterns[cite: 1].
• Temporal Modeling: Processed maps are fed into a 2-layer Transformer Encoder coupled with Temporal Attention pooling and a dense classification head[cite: 1].

Multimodal Late Fusion

The architecture includes a Late Fusion strategy that dynamically projects the probability distributions of both modalities into a shared 256-dimensional embedding space[cite: 1]. Note: Because the UCF101 and WISDM datasets are unpaired and feature different taxonomies, true mathematical fusion was constrained; hence, the models were ultimately evaluated as independent branches[cite: 1].

---

📈 Performance & Results

• Video Model (UCF101): Achieved 97.07% validation accuracy across 101 classes[cite: 1]. The model comprises 31.95 Million parameters[cite: 1].
• Sensor Model (WISDM): Achieved 61.31% validation accuracy, excelling at rhythmic activities like walking (F1-score of 0.9836) and jogging (F1-score of 0.9967)[cite: 1]. The highly lightweight model contains only 0.51 Million parameters, achieving fast inference speeds of ~3.23 ms for edge device suitability[cite: 1].

---

🚀 System Deployment

To demonstrate practical application, the trained models are deployed as a full-stack web application for real-time inference:

• Backend: Built with Flask for real-time video prediction. Containerized via Docker (python:3.13.5-slim) utilizing headless OpenCV, and hosted on Hugging Face Spaces[cite: 1].
• Frontend: A ReactJS interface providing a clean drag-and-drop environment for video uploads and structured result visualization[cite: 1].
• Live Web App: https://har-demo.vercel.app[cite: 1]
• API Endpoint: https://ahmedjaved812-human-action-recognition.hf.space/predict[cite: 1]

---

---

📜 License

MIT License

Copyright (c) 2026 Ahmed Javed, Abdul Rehman, Mushaf Ali Meesum

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.