Downloads · 30 days
11
38% of all-time downloads
rxhxn2904/arqgan-indian-temple-restoration
arqgan-indian-temple-restoration is a image-to-image model from rxhxn2904. Use it when you need one image transformed into another. The card lists the license as mit.
A pipeline that restores damaged Indian temple imagery and projects the result onto a 3D model. Rebuilt from the ARQGAN concept (2D restoration GAN) for Kalinga/Nagara temple architecture, driven from an HBIM model.
Downloads · 30 days
11
38% of all-time downloads
All-time downloads
29
Public
Repo size
5.5 GB
Likes
0
Public
Click a slice to open those files.
.pt3.8 GB · 78%
From the Hugging Face model README
A pipeline that restores damaged Indian temple imagery and projects the result onto a 3D model. Rebuilt from the ARQGAN concept (2D restoration GAN) for Kalinga/Nagara temple architecture, driven from an HBIM model.
rxhxn2904AchyutAcharya182 / AchyutAcharya13
Input: a temple render/photo with synthetic damage applied (cracks, erosion, missing sections) + its segmentation map (per-architectural- element mask, e.g. shikhara / mandapa / amalaka / plinth). Output: the same view with damage removed, texture and fine ornament detail restored by the trained HD generator.
Try it live: ArqGAN demo Space — upload your own damaged temple photo and restore it in the browser.
| Temple | v2 | HD | XL | Notes |
|---|---|---|---|---|
checkpoints/ (Temple_1) | ✅ | ✅ | ✅ | fully trained baseline |
checkpoints_Area_3_Temple_8/ | ✅ | ✅ | ✅ | full v2/HD/XL run, fine-tune-cluster baseline (test SSIM 0.9902) |
Only best_*.pt checkpoints are included (not last_*, which are
redundant duplicates from the same runs).
Dataset (3DITA source, point clouds, synthesized training triples) and an interactive demo live separately:
3DITA point cloud + labels (per temple) 40-50 photos (per temple,
│ varying angle/lighting)
▼ │
dita_loader.py photo_alignment.py
Poisson mesh + per-face labels lighting normalize +
│ pose registration (PnP)
│ │
└──────────────────┬───────────────────────────────┘
▼
synthesize_pairs.py — thousands of (damaged, segmentation,
intact) training triples per temple
▼
train ArqGAN (v2 / HD / XL)
▼
apply to a damaged temple: mirror intact→broken (geometry),
GAN repaints rendered views, project back onto the mesh (texture)
▼
textured 3D model, viewable from any angle
Geometry comes from the point cloud, not the photos. 3DITA already provides real, labeled geometry for all 47 temples — reconstructing it again from photos would throw that away to rebuild a worse version. Photos only ever contribute texture/appearance, after being told where they were standing via classical pose registration.
| Folder | What's in it |
|---|---|
geometry/ | Point cloud → mesh, photo → pose registration. Run this first if using 3DITA |
core/ | Base pipeline: data generation, the v2 model, training, HBIM loading, metrics, preflight check |
hd/ | The 18-hour coarse-to-fine model (512→1024, perceptual + feature-matching losses) |
xl/ | The bigger model (157M params, self-attention) + the ablation tool that checks whether bigger actually helped |
to3d/ | 2D→3D texture projection: renders the mesh, runs the GAN, bakes colour onto the surface |
docs/ | Per-component READMEs, fix logs, and the overfitting/data guide |
Root-level: smoke_test.py (post-install check), _paths.py (import
bootstrap — every script now runs from any working directory).
core/callbacks.py holds the shared training callbacks (early stopping,
checkpointing, the two overfitting detectors, LR schedule, logging).
pip install -r requirements.txt
python smoke_test.py # 12 CPU checks, under a minute, no data needed
smoke_test.py verifies the plumbing — component transforms, the
validation split, all three model families, occlusion, the texture bake,
the pose round-trip. It says nothing about model quality; it catches the
failures that otherwise surface eighteen hours into a run.
Two optional-but-recommended extras:
embreex — accelerates occlusion testing. Occlusion is correct
without it, just slow; the code says which backend answered.
(An earlier version of this file recommended rtree here. That was
wrong — rtree is a spatial index, not a ray tracer, and trimesh's ray
casting never used it. See docs/05_BUGFIXES.md.)xatlas — enables sharp UV texture maps (falls back to vertex colours
if absent OR if the build crashes; some builds segfault — the code
isolates it in a subprocess so a crash is survived, not fatal)cd core
python preflight.py --mesh /path/to/lingaraj.glb
Checks GL context, file format, component names, and does a test render. If it passes, everything downstream will work. Export your HBIM to GLB first — IFC/RVT are not readable (preflight tells you this).
cd geometry
python3 -c "
from dita_loader import load_dita_temple, mesh_to_trimesh
from photo_alignment import register_photo_set
o3d_mesh, face_labels = load_dita_temple('lingaraj_points.txt')
tri_mesh = mesh_to_trimesh(o3d_mesh)
tri_mesh.export('lingaraj.glb') # feed this into Step 1 below
# photos = [...your 40-50 PNGs...]
# registered, unregistered = register_photo_set(photos, tri_mesh)
"
Confirm your point cloud's column format first — see
geometry/README.md. Run on one temple before all 47.
cd core
python synthesize_pairs.py --mesh /path/to/lingaraj.glb \
--out_dir ../data --n_views 3000
One temple → 3000 training triples. Inspect names first with
--inspect if unsure they'll classify.
Standard (v2, ~4–6h total, fits 6h budget):
cd core
python train.py --data_root ../data --max_hours 6 --batch_size 12
Every epoch logs gen_gap (train minus val SSIM) and d_gap (D's score
on training reals minus held-out reals). Those two numbers are how you
tell memorisation from a hard task — the loss curves cannot. Training
ends with a pass on test wedges never used for early stopping or
checkpoint selection; that is the number to report, not best val.
See docs/06_OVERFITTING.md.
High-detail (HD, ~18h, better on fine ornament):
cd hd
python train_hd.py --data_root ../data --total_hours 18
Use best_G.pt (v2) or best_local.pt (HD) for inference.
Train XL the same way (identical protocol, so the comparison is fair):
cd xl
python train_xl.py --data_root ../data --total_hours 18
Then compare validation curves:
cd xl
python ablation_compare.py \
--v2 ../checkpoints/history.json \
--hd ../checkpoints_hd/history.json \
--xl ../checkpoints_xl/history.json
Tells you whether extra capacity helped or just memorised. Don't ship XL's numbers without running this.
cd to3d
python pipeline_2d_to_3d.py \
--mesh /path/to/restored_geometry.glb \
--checkpoint ../checkpoints_hd/best_local.pt \
--out textured.glb \
--tex_size 2048 --lighting flat
Renders the mesh, restores each view with the GAN, projects colour back
onto the surface. --lighting flat bakes clean albedo. Omit
--checkpoint for a passthrough test of the geometry path alone.
| Task | VRAM | Time |
|---|---|---|
| Data generation (3000 views) | CPU only | 2–3 h |
| Train v2 | ~28 GB @ bs12 | 1.5–3 h (early-stops) |
| Train HD | ~18 GB @ 1024/bs2 | ~14–16 h |
| Train XL | close to 32 GB | slower/epoch |
| 2D→3D per temple | modest | 20–30 min |
Everything targets a single 16–32 GB GPU. Drop batch size if you OOM.
Estimates, not measured — no training was run in the build environment.
| Metric | Range | What it measures |
|---|---|---|
| Geometry whole-object F1@1% | 90–96% | 3D shape (from mirroring + hard-restore) |
| Geometry restored-region F1@1% | 55–90% | the repaired region only; depends on damage symmetry |
| Texture SSIM (v2 / HD) | 0.78–0.90 / 0.85–0.93 | surface appearance from the GAN |
Report whole-object AND restored-region side by side — the whole-object figure is high mainly because measured geometry is preserved untouched.
All accuracy figures assume synthetic damage. Validate on real.
Geometry accuracy assumes symmetry holds — fails where both sides are lost or the temple is intentionally asymmetric.
One-temple style bound. More reference temples = better coverage.
Nothing here was trained or benchmarked on a GPU in the build environment. Verify the first epoch's timing and the 1024px forward pass yourself before committing to a long run.
The 40–50 photos must be REAL photographs, not Blender renders.
They are the only place real-world surface appearance enters the
system; rendering them returns the mesh's own texture and the
pipeline becomes circular. Training triples, by contrast, are
correctly synthetic — paired damaged/intact ground truth cannot be
photographed. Full reasoning in docs/06_OVERFITTING.md.
Data generated before the correctness pass is invalid. Two bugs
silently corrupted it: component transforms were dropped (multi-part
HBIM exports collapsed to the origin) and segmentation maps were
rendered with shading (label colours varied by surface orientation).
Both are fixed; regenerate rather than reuse. Details in
docs/05_BUGFIXES.md.
See docs/ for detailed per-component notes and the fix logs.
docs/05_BUGFIXES.md is the one to read before running anything long.