PR-ENDO: Physically Based Relightable Gaussian Splatting for Endoscopy

Joanna Kaleta1,2,*,   Weronika Smolak-Dyżewska3,*,   Dawid Malarz3,   Diego Dall'Alba2,4,   Przemysław Korzeniowski2,   Przemysław Spurek3
1Warsaw University of Technology, 2Sano Centre for Computational Medicine, 3Jagiellonian University, 4University of Verona
*equal contribution
Accepted to MICCAI 2025!
arXiv (pre-peer review) Code Full paper (camera ready) Supplementary videos
PR-ENDO

Abstract

Endoluminal endoscopic procedures are essential for diagnosing colorectal cancer and other severe conditions in the digestive tract, urogenital system, and airways. 3D reconstruction and novel-view synthesis from endoscopic images are promising tools for enhancing diagnosis. Moreover, integrating physiological deformations and interaction with the endoscope enables the development of simulation tools from real video data. However, constrained camera trajectories and view-dependent lighting create artifacts, leading to inaccurate or overfitted reconstructions. We present PR-ENDO, a novel 3D reconstruction framework leveraging the unique property of endoscopic imaging, where a single light source is closely aligned with the camera. Our method separates light effects from tissue properties. PR-ENDO enhances 3D Gaussian Splatting with a physically based relightable model. We boost the traditional light transport formulation with a specialized MLP capturing complex light-related effects while ensuring reduced artifacts and better generalization across novel views. PR-ENDO achieves superior reconstruction quality compared to baseline methods on both public and in-house datasets. Unlike existing approaches, PR-ENDO enables tissue modifications while preserving a physically accurate response to light, making it closer to real-world clinical use.

Pipeline

Pipeline Overview

Simulate body movement

Decomposition

Change colon properties (roughness)

Change light properties

Reconstruction - Qualitative Comparison

BibTeX

If you consider this work useful, please cite:

@misc{kaleta2024prendo,
  title={PR-ENDO: Physically Based Relightable Gaussian Splatting for Endoscopy}, 
  author={Joanna Kaleta and Weronika Smolak-Dyżewska and Dawid Malarz and Diego Dall'Alba and Przemysław Korzeniowski and Przemysław Spurek},
  year={2024},
  eprint={2411.12510},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2411.12510}
}