Deep Real-time Volumetric Rendering Using Multi-feature Fusion

ACM SIGGRAPH 2023 (Conference Track)

Jinkai Hu, Chengzhong Yu, Hongli Liu, Ling-qi Yan, Yiqian Wu, Xiaogang Jin*

Abstract:

We present Multi-feature Radiance-Predicting Neural Networks (MRPNN), a practical framework with a lightweight feature fusion neural network for rendering high-order scattered radiance of participating media in real time. By reformulating the Radiative Transfer Equation (RTE) through theoretical examination, we propose transmittance fields, generated at a low cost, as auxiliary information to help the network better approximate the RTE, drastically reducing the size of the neural network. The light weight network efficiently estimates the difficult-to-solve in-scattering term and allows for configurable shading parameters while improving prediction accuracy. In addition, we propose a frequency-sensitive stencil design in order to handle non-cloud shapes, resulting in accurate shadow boundaries. Results show that our MRPNN is able to synthesize indistinguishable output compared to the ground truth. Most importantly, MRPNN achieves a speedup of two orders of magnitude compared to the state-of-the-art, and is able to render high-quality participating material in real time.

Recommended citation:

@inproceedings{10.1145/3588432.3591493,
author = {Hu, Jinkai and Yu, Chengzhong and Liu, Hongli and Yan, Lingqi and Wu, Yiqian and Jin, Xiaogang},
title = {Deep Real-time Volumetric Rendering Using Multi-feature Fusion},
year = {2023},
isbn = {9798400701597},
publisher = {Association for Computing Machinery},
booktitle = {ACM SIGGRAPH 2023 Conference Proceedings},
articleno = {61},
numpages = {10}
}