gummybear_hierarchical_fusion

tbhugging/gummybear_hierarchical_fusion

What this model does

Given 6 illumination settings × 36 camera-orbit views of 128×128 projections from the GummyBear Tomography dataset, the model predicts

• particle_x
• particle_y
• particle_z

coordinates of the embedded particle.

Architecture

Per-view GAP (pooled) CNN trunk → light sin/cos tokens fused within each camera → camera sin/cos tokens fused across the orbit (hierarchical_pooled_light_then_camera_fusion).

Final Report M10 Step 3 / 10_2 — pooled hierarchical head (model_pooled).

• Hub: tbhugging/gummybear_hierarchical_fusion
• Companion dataset: tbhugging/gummybear-tomography
• Runnable report: GummyBearTomography_Final_Report.ipynb (M10 Step 3 — 10_2 pooled ladder)
• Source study checkpoint: checkpoints/m10/m10_hierarchical_light_then_camera.pt (model_pooled / m10_2_hierarchical_pooled_light_then_camera_fusion)

Training configuration

• Protocol: 10_2 (pooled GAP hierarchical; excludes Fourier 10_2)
• Backbone: pooled_gap (pooled trunk)
• Input field: anomaly_ref
• Normalisation: per_image_zscore
• Illumination orbit: 0°, 60°, 120°, 180°, 240°, 300° (6 lights)
• Camera orbit: 36 views on 10° stride (0°–350°) (36 views)
• Flat layout: light_major
• Geometry: sin_light, cos_light, sin_camera, cos_camera
• Fusion: hidden 128, depth 1, camera latent 128
• Targets: particle_x, particle_y, particle_z
• Builder: HierarchicalLightThenCameraFusionLocalizer.for_10_2_pooled()
• Variant: m10_2_hierarchical_pooled_light_then_camera_fusion
• Trainable parameters: 748038
• Stage-B learning rate: 0.0003

Evaluation Results

Structured scores for the Hub widget are declared in the YAML model-index / metrics metadata (Model Cards — Evaluation Results).

Testing Data

• Dataset: tbhugging/gummybear-tomography (config m8_1)
• Splits: validation, test
• Protocol: Final Report M10 Step 3 / 10_2 (6 lights × 36 cameras, anomaly_ref, per_image_zscore, GAP pooled trunk only, hierarchical light-then-camera fusion; excludes Fourier 10_2)
• Source checkpoint: checkpoints/m10/m10_hierarchical_light_then_camera.pt (model_pooled only)

Metrics

Reported error is Euclidean RMSE over particle (x,y,z): d_i = ||pred_i - y_i||_2, then RMSE_total = sqrt(mean_i d_i^2). Hub metric id: rmse (display name RMSE_total (Euclidean xyz)).

Results

Scores match the Final Report M10 Step 3 10_2 pooled GAP hierarchical bar.

Source: Final Report M10 Step 3 / 10_2.

Load

import torch
# libraries from https://github.com/tbgitoo/gummybear-tomography
# Historical class name — use .for_10_2_pooled() for this GAP checkpoint only.
from tomography_ml.localization.localize_multiview import (
    HierarchicalLightThenCameraFusionLocalizer,
)

n_lights = 6
n_cameras = 36
light_angles_deg = [0.0, 60.0, 120.0, 180.0, 240.0, 300.0]
camera_angles_deg = [0.0, 10.0, 20.0, 30.0, 40.0, 50.0, 60.0, 70.0, 80.0, 90.0, 100.0, 110.0, 120.0, 130.0, 140.0, 150.0, 160.0, 170.0, 180.0, 190.0, 200.0, 210.0, 220.0, 230.0, 240.0, 250.0, 260.0, 270.0, 280.0, 290.0, 300.0, 310.0, 320.0, 330.0, 340.0, 350.0]
model = HierarchicalLightThenCameraFusionLocalizer.for_10_2_pooled(
    n_cameras=n_cameras,
    n_lights=n_lights,
    camera_angles_deg=camera_angles_deg,
    light_angles_deg=light_angles_deg,
    flat_layout='light_major',
)
views = torch.zeros(1, n_lights, n_cameras, 1, 128, 128)
model(views)  # materialise lazy layers
state = torch.load('pytorch_model.bin', map_location='cpu', weights_only=True)
model.load_state_dict(state)
model.eval()
xyz = model(views)

Input tensor shape: [batch, n_lights, n_cameras, channels, height, width] (flat light_major layout also supported). Do not load with .for_10_2() (that builds the Fourier 10_2 trunk).

Inference

For an example with worked download, model instanciation and inference, see: 11_3_test_gummybear_hierarchical_fusion.ipynb in the github.com/tbgitoo/gummybear-tomography repository.