CVPR 2023

FitMe: Deep Photorealistic 3D Morphable Model Avatars

Alexandros Lattas, Stylianos Moschoglou, Stylianos Ploumpis,
Baris Gecer, Jiankang Deng, Stefanos Zafeiriou
Imperial College London
Paper arXiv Video Rendering Code


FitMe turns arbitrary facial images to relightable facial avatars, directly usable in common gaming and rendering engines.


Abstract

In this paper, we introduce FitMe, a facial reflectance model and a differentiable rendering optimization pipeline, that can be used to acquire high-fidelity renderable human avatars from single or multiple images.

The model consists of a multi-modal style-based generator, that captures facial appearance in terms of diffuse and specular reflectance, and a PCA-based shape model. We employ a fast differentiable rendering process that can be used in an optimization pipeline, while also achieving photorealistic facial shading. Our optimization process accurately captures both the facial reflectance and shape in high-detail, by exploiting the expressivity of the style-based latent representation and of our shape model.

FitMe achieves state-of-the-art reflectance acquisition and identity preservation on single ``in-the-wild'' facial images, while it produces impressive scan-like results, when given multiple unconstrained facial images pertaining to the same identity. In contrast with recent implicit avatar reconstructions, FitMe requires only one minute and produces relightable mesh and texture-based avatars, that can be used by end-user applications.


Video

Method

In two steps, we optimize the latent vector of the generator, the shape and rendering parameters, by combining 3DMM fitting, GAN inversion and GAN tuning.


The optimization is guided by accurate diffuse and specular differentiable rendering.

Single-image Results

Single Image Results

FitMe can be optimized for arbitrary facial images, from a single internet sourced picture, to multiple selfie photographs.

Selfie Results



Even with three unconstrained quick selfies of a subject, in an arbitrary environment and illumination, FitMe produces high-fidelity relfectance and shape that can be photorealistically relighted.

BibTeX

@InProceedings{Lattas_2023_CVPR,
      author    = {Lattas, Alexandros and Moschoglou, Stylianos and Ploumpis, Stylianos and Gecer, Baris and Deng, Jiankang and Zafeiriou, Stefanos},
      title     = {FitMe: Deep Photorealistic 3D Morphable Model Avatars},
      booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
      month     = {June},
      year      = {2023},
      pages     = {8629-8640}
  }