GS-IR: 3D Gaussian Splatting for Inverse Rendering

Zhihao Liang1*, Qi Zhang2*, Ying Feng2, Ying Shan2, Kui Jia3

1South China University of Technology, 2Tencen AI Lab,
3School of Data Science, The Chinese University of Hong Kong, Shenzhen

Paper Code


We propose GS-IR, a novel inverse rendering approach based on 3D Gaussian Splatting (GS) that leverages forward mapping volume rendering to achieve photorealistic novel view synthesis and relighting results. Unlike previous works that use implicit neural representations and volume rendering (e.g. NeRF), which suffer from low expressive power and high computational complexity, we extend GS, a top-performance representation for novel view synthesis, to estimate scene geometry, surface material, and environment illumination from multi-view images captured under unknown lighting conditions. There are two main problems when introducing GS to inverse rendering: 1) GS does not support producing plausible normal natively; 2) forward mapping (e.g. rasterization and splatting) cannot trace the occlusion like backward mapping (e.g. ray tracing). To address these challenges, our GS-IR proposes an efficient optimization scheme that incorporates a depth-derivation-based regularization for normal estimation and a baking-based occlusion to model indirect lighting. The flexible and expressive GS representation allows us to achieve fast and compact geometry reconstruction, photorealistic novel view synthesis, and effective physically-based rendering. We demonstrate the superiority of our method over baseline methods through qualitative and quantitative evaluations on various challenging scenes.

Contributions & Method

In the baking stage, we cache the occlusion from the pretrained 3D Gaussians and store it in volumes.

In the decomposition stage, we conduct masked-trilinear interpolation to recover the occlusion from volumes.

Decompsoition & Relighting Results


Mip-NeRF 360

Relighting Results of Point Lighting

We implement the shadow map with reference to the point shadow on LearnOpenGL.
We visualize the cases of rotating the point light source with a fixed camera and rotating the camera with a fixed point light source respectively.