sgm-lnd-det-v1

Image Segmentation for Improved Landmark Detection in Lower Limb X-Rays

This repository provides PyTorch model weights for the deep learning networks described in the paper Optimization and Benchmarking of Image Segmentation for Improved Landmark Detection in Lower Limb X-Rays and Accurate Coronal Plane Alignment of the Knee Classification. Refer to the associated paper for full methodological details and achieved performance.

Repository Purpose

This repository hosts pre-trained model weights only, which are intended to be used in conjunction with the accompanying repository that includes the complete inference pipeline, model definitions, and usage explanation to automatically analyze full-leg radiographs for lower-limb malalignment assessment.

If you use any of the provided models, please cite:

@article{sanchez2025optimization,
  title={Optimization and benchmarking of image segmentation for improved landmark detection in lower limb X-rays and accurate coronal plane alignment of the knee classification},
  author={Sanchez, Sebastian Amador and Zarghami, Ashkan and Van Overschelde, Philippe and Vandemeulebroucke, Jef},
  journal={IEEE Access},
  year={2025},
  publisher={IEEE}
}

---

Training Data

The models were trained using the dataset described in the associated publication. The training dataset is not distributed as part of this repository.

---

Available Models

For each region of interest (hip, knee, and ankle joints), deep learning models were trained following four different landmark localization paradigms:

1. Image segmentation
2. Heatmap regression
3. Coordinate regression
4. Segmentation-guided coordinate regression (SGR)

For each anatomical region, the following numbers of landmarks and coordinates are predicted:

Employed architectures

Depending on the landmark localization technique, a different deep learning architecture was used.

• U-Net: U-Net architecture with a VGG-16 pre-trained backbone on its encoder, where the final layer contains one output channel per landmark.
• VGG-16: VGG-16 pre-trained architecture, where the last fully-connected layer has a number of nodes equivalent to the number of coordinates to be estimated.
• SGRNetwork16: Segmentation-guided coordinate regression architecture. The segmentation branch is based on a U-Net architecture, whose encoder topology is equivalent to the coordinate regression branch, and both branches use a VGG-16 backbone.

Repository structure

The weights are located in the Weights directory. The complete repository is structured as:

Weights/
    ├── Heatmap                      → Weights for heatmap regression
        ├── Ankle                    → Ankle joint model
            ├── ankle_weights.pth
        ├── Femur                    → Hip joint model
            ├── femur_weights.pth
        ├── Knee                     → Knee joint model
            ├── knee_weights.pth
    ├── Regression                   → Weights for coordinate regression with similar organization as heatmap regression
    ├── Segmentation                 → Weights for image segmentation with similar organization as heatmap regression
    ├── SGR                          → Weights for SGR with similar organization as heatmap regression
    ├── ROI_Detection
        ├── fll_detection_joints.pth → Weights for joint detection via a Faster-RCNN model
BuildModels.py
Detection.py
SGR.py
UNet.py
Weights.py

Description

• BuildModels.py — Builds the segmentation and regression models.
• Detection.py — Builds the Faster R-CNN model.
• SGR.py — Implementation of the SGR coordinate regression architecture.
• UNet.py — U-Net architecture and its related functions.
• Weights.py — Dictionary that defines the paths to the weights

---

Requirements

Developed and tested with Python 3.9.12. The following dependencies are required:

torch==2.2.2
torchvision==0.17.2
huggingface_hub==1.8.0

---

Usage Examples

A. Pre-trained Landmark Localization Models

I. Expected input and outputs:

• Input:
  • Cropped and pre-processed regions of full lower-limb X-ray images:
    1. Intensity scale between [0, 1]
    2. Size resampled to 512 x 512
  • Images must be provided as 3-channel tensors of shape (N, 3, 512, 512). Grayscale radiographs should be replicated across the three channels before inference.
• Output:
  • Depending on the landmarks localization technique:
    • Image segmentation and heatmap regression: 2D probability maps from where pixel coordinates can be derived through different centroid computation algorithms.
    • Coordinate regression and SGR: 2D landmark coordinates (x, y) in image pixel space.
  • Since the outputs are in the 512 x 512 coordinate system, these have to be transformed to the original coordinate system for accurate results.

II. Typical inference pipeline:

1. ROI detection (hip, femoral diaphysis, knee, ankle)
2. Cropping and pre-processing of detected regions (not implemented in the current repository)
3. Landmark localization using the pre-trained models
4. Downstream post-processing and alignment metric computation (not implemented in the current repository)

III. Loading weights and running inference:

import torch
from BuildModels import create_regression_model, create_segmentation_model, create_sgr_model

device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# # # # # # # # # # # # # # # # # # # Image segmentation / heatmap regression usage # # # # # # # # # # # # # # # # # #
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #

# Paths to pretrained model weights (select segmentation or heatmap)
path_to_hip_weights = 'Weights/Segmentation/Best_Segmentation_Femur.pth'
path_to_knee_weights = 'Weights/Segmentation/Best_Segmentation_Knee.pth'
path_to_ankle_weights = 'Weights/Segmentation/Best_Segmentation_Ankle.pth'

# Hip model
torch_hip_images = torch.rand(1, 3, 512, 512)  # Format: batch, channel, height, width
hip_unet = create_segmentation_model(number_landmarks=1)
hip_unet.load_state_dict(torch.load(path_to_hip_weights, map_location=device))
hip_unet.eval()  # Estimate the landmarks
with torch.no_grad():
    hip_landmarks = hip_unet(torch_hip_images)  # Output landmarks: batch, 1, 512, 512

# Knee model
torch_knee_images = torch.rand(1, 3, 512, 512)  # Format: batch, channel, height, width
knee_unet = create_segmentation_model(number_landmarks=5)
knee_unet.load_state_dict(torch.load(path_to_knee_weights, map_location=device))
knee_unet.eval()  # Estimate the landmarks
with torch.no_grad():
    knee_landmarks = knee_unet(torch_knee_images)  # Output landmarks: batch, 5, 512, 512

# Ankle model
torch_ankle_images = torch.rand(1, 3, 512, 512)  # Format: batch, channel, height, width
ankle_unet = create_segmentation_model(number_landmarks=2)
ankle_unet.load_state_dict(torch.load(path_to_ankle_weights, map_location=device))
ankle_unet.eval()  # Estimate the landmarks
with torch.no_grad():
    ankle_landmarks = ankle_unet(torch_ankle_images)  # Output landmarks: batch, 2, 512, 512
    
# Execute post-processing and centroid computation depending on the landmark localization technique 

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# # # # # # # # # # # # # # # # # # # # # # # # Coordinate regression # # # # # # # # # # # # # # # # # # # # # # # # #
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #

# Paths to pretrained model weights
path_to_reg_hip_weights = 'Weights/Regression/Best_Regression_Femur.pth'
path_to_reg_knee_weights = 'Weights/Regression/Best_Regression_Knee.pth'
path_to_reg_ankle_weights = 'Weights/Regression/Best_Regression_Ankle.pth'

# Hip model
torch_hip_images = torch.rand(1, 3, 512, 512)  # Format: batch, channel, height, width
hip_vgg = create_regression_model(number_coordinates=2)
hip_vgg.load_state_dict(torch.load(path_to_reg_hip_weights, map_location=device))
hip_vgg.eval()  # Estimate the landmarks
with torch.no_grad():
    hip_coordinates = hip_vgg(torch_hip_images)  # Output landmarks: [batch, 2]

# Knee model
torch_knee_images = torch.rand(1, 3, 512, 512)  # Format: batch, channel, height, width
knee_vgg = create_regression_model(number_coordinates=10)
knee_vgg.load_state_dict(torch.load(path_to_reg_knee_weights, map_location=device))
knee_vgg.eval()  # Estimate the landmarks
with torch.no_grad():
    knee_coordinates = knee_vgg(torch_knee_images)  # Output landmarks: [batch, 10]

# Ankle model
torch_ankle_images = torch.rand(1, 3, 512, 512)  # Format: batch, channel, height, width
ankle_vgg = create_regression_model(number_coordinates=4)
ankle_vgg.load_state_dict(torch.load(path_to_reg_ankle_weights, map_location=device))
ankle_vgg.eval()  # Estimate the landmarks
with torch.no_grad():
    ankle_coordinates = ankle_vgg(torch_ankle_images)  # Output landmarks: [batch, 4]
    
# Execute post-processing and centroid computation depending on the landmark localization technique 

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# # # # # # # # # # # # # # # # # # # # # # # # # # # # SGR # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #

# Paths to pretrained model weights
path_to_sgr_hip_weights = 'Weights/SGR/Best_SGR_Femur.pth'
path_to_sgr_knee_weights = 'Weights/SGR/Best_SGR_Knee.pth'
path_to_sgr_ankle_weights = 'Weights/SGR/Best_SGR_Ankle.pth'

# Hip model
torch_hip_images = torch.rand(1, 3, 512, 512)  # Format: batch, channel, height, width
hip_sgr = create_sgr_model(number_coordinates=2)
hip_sgr.load_state_dict(torch.load(path_to_sgr_hip_weights, map_location=device))
hip_sgr.eval()  # Estimate the landmarks
with torch.no_grad():
    _, hip_sgr_coordinates = hip_sgr(torch_hip_images)  # Output landmarks: [batch, 2]

# Knee model
torch_knee_images = torch.rand(1, 3, 512, 512)  # Format: batch, channel, height, width
knee_sgr = create_sgr_model(number_coordinates=10)
knee_sgr.load_state_dict(torch.load(path_to_sgr_knee_weights, map_location=device))
knee_sgr.eval()  # Estimate the landmarks
with torch.no_grad():
    _, knee_sgr_coordinates = knee_sgr(torch_knee_images)  # Output landmarks: [batch, 10]

# Ankle model
torch_ankle_images = torch.rand(1, 3, 512, 512)  # Format: batch, channel, height, width
ankle_sgr = create_sgr_model(number_coordinates=4)
ankle_sgr.load_state_dict(torch.load(path_to_sgr_ankle_weights, map_location=device))
ankle_sgr.eval()  # Estimate the landmarks
with torch.no_grad():
    _, ankle_sgr_coordinates = ankle_sgr(torch_ankle_images)  # Output landmarks: [batch, 4]
    
# Execute post-processing and centroid computation depending on the landmark localization technique 

Note: Input images should be preprocessed consistently with the training setup (e.g., normalization, resizing). Refer to the main repository for details.

---

B. ROI (Joint) Detection

import torch
from Detection import get_model_instance_segmentation

device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')

path_to_weights = 'Weights/ROI_Detection/fll_detection_joints.pth'

# Classes: background, hip, femoral diaphysis, knee, ankle
num_classes = 5

detection_model = get_model_instance_segmentation(num_classes)
detection_model.load_state_dict(torch.load(path_to_weights, map_location=device))
detection_model.to(device)

---

Limitations

The provided models are intended for research purposes only and have not been validated for clinical use. They must not be used for diagnostic or medical decision-making.

---

License

The model weights in this repository are licensed under the Creative Commons Attribution Non Commercial Share Alike 4.0 License. See the LICENSE file for the full legal text.