Geometry-aware Single-image Full-body Human Relighting

ECCV 2022


Chaonan Ji1, Tao Yu2, Kaiwen Guo2, Jingxin Liu2, Yebin Liu3,

1Tsinghua University    2Meta Reality Labs    3Guangdong OPPO Mobile Telecommunications Corp., Ltd

Abstract


Responsive image

Single-image human relighting aims to relight a target human under new lighting conditions by decomposing the input image into albedo, shape and lighting. Although plausible relighting results can be achieved, previous methods suffer from both the entanglement between albedo and lighting and the lack of hard shadows, which significantly decrease the reality. To tackle these two problems, we propose a geometry-aware single-image human relighting framework that leverages single-image geometry reconstruction for joint deployment of traditional graphics rendering and neural rendering techniques. For the de-lighting, we explore the shortcomings of UNet architecture and propose a modified HRNet, achieving better disentanglement between albedo and lighting. For the relighting, we introduce a ray tracing-based per-pixel lighting representation that explicitly models high-frequency shadows and propose a learning-based shading refinement module to restore realistic shadows (including hard cast shadows) from ray-traced shading maps. Our framework is able to generate photo-realistic high-frequency shadows such as cast shadows under challenging lighting conditions. Extensive experiments demonstrate that our proposed method outperforms previous methods on both synthetic and real images.


Overview


Responsive image

Figure 2.Illustration of our framework architecture. There are two stages in our method: de-lighting and relighting. The de-lighting stage takes the input image and outputs estimated albedo (Section 4). For the relighting stage (Section 5), a full-body 3D model and a face 3D model are estimated by 3D Recon Module and then are sent to the renderer to render coarse shading maps (Section 5.1). The Refine Module takes the coarse shading maps and the inferred ambient occlusion map as input and produces the final shading map (Section 5.2). Finally, the estimated albedo map and final shading map are dot product to produce the relit image.


De-lighting


Responsive image


Re-lighting


Responsive image



Demo video



Citation


@article{ji2022relight,
  title={Geometry-aware Single-image Full-body Human Relighting},
  author={Ji, Chaonan and Yu, Tao and Guo, Kaiwen and Liu, Jingxin and Liu, Yebin},
  booktitle={European Conference on Computer Vision (ECCV)},
  year={2022}
  month={October},
  
}


Acknowledgement


This paper is supported by National Key R\&D Program of China (2021ZD0113501) and the NSFC project No.62125107, No.62171255 and No.61827805.