9 min read UIUC

Face migration in classical Artwork

Face migration in classical Artwork
This is an edited version of my final project report, a result of my Computational Photography (CS 445, Fall 2023) class at UIUC, taught by Prof. Derek Hoiem. Special thanks to his lectures and his systematic and creative ways of teaching such a fundamental and core computer vision course.

I. Motivation

In project 3 (a project on the topic of gradient domain editing), we implemented texture style transfer, learning how to transfer the "drawing style" from one picture to another based on pixel intensity. Given the creative perspective of this notion, we want to incorporate additional techniques not yet covered in class projects, such as object averaging (morphing), image warping, matting, and hole-filling, to create a pipeline for “Face Migration in Classical Artwork.”

Our inspiration comes from a creative rendition of Vermeer’s “Girl with a Pearl Earring” (shown in Figure 1), where the girl's face is replaced with that of an oil-paint cat. This gave us the idea of seamlessly merging objects from one classical artwork into the scene or objects of another. Our primary objective is to facilitate face-to-face swaps that do more than just blend faces; we also want to apply texture transfer to match the drawing style and morph only into the face region, preserving the integrity of the original artwork. Figure 2 shows another example we aim to achieve, swapping the two faces in “American Gothic” by Grant Wood.

Figure 1 (left). The creative rendition of Vermeer’s “Girl with a Pearl Earring”. Figure 2 (right). “American Gothic” 

II. Approach

Figure 3. Our project pipeline.

Our approach comprises several steps, each of which is covered in the class. The pipeline shown above provides an overview of our method (refer to Figure 3), and here are steps with more detailed descriptions:

  1. Image Selection: We first obtain two sets of images. One set consists of pictures of a "cat," the other set includes numerous classical portrait artworks.
  2. Source and Target Images: Select one image from each set, labeling them as the 'source' and 'target' images, respectively.
Figure 4. Example selected source image and target image.
  1. Facial Landmarks Extraction: Extract facial landmarks from both images using the standard 68 landmarks model.
    • For human faces, we automatically extract the landmarks using dlib’s detector and predictor, which is a mature technology.
    • For cat faces, we manually extract the landmarks using a Python tool we developed, taking the positions of dlib’s predictor landmarks as reference (Elmahmudi, Ali, et. al.). For more details, see the README file in the code repo.
Figure 5. Facial Landmarks Extraction
  1. Triangulation: Triangulate the facial landmarks of each face to match the same mesh structure. This is achieved using a convex hull and the Delaunay algorithm.
Figure 6. Proper triangulation for facial meshes
  1. Affine Transformation: Perform an affine transformation from the stylized source face to the target face, or vice versa. Obtain the mask of each face.
Figure 7. Face affine transformation between two human faces.
  1. Poisson Blending: Apply Poisson or Mixed blending using the source face's region of interest, the target artwork, and the source face mask.
Figure 8. Three different output images with different blend techniques.

III. Results

Figure 9. Cat-human Face Example 1 (w/ Poisson Blending) Source: unknown, internet. Target: “Girl with a Pearl Earring” by Vermeer
Figure 10. Cat-human Face Example 2 (w/ Mixed Blending) Source: unknown, internet. Target: “Portrait of Giovanni di Nicolao Arnolfini” by Jan van Eyck Source face is well-fitted into the target face, but this time it also maintains the target face structure as well
Figure 11. Cat-human Face Example 4 (w/ Poisson Blending) Source: generated with DALL-E, internet. Target: “Man With a Ring 1617” by Werner van den Valckert Source face is well-fitted into the target face in terms of the facial contour. As this cat face is more abstract compared to others, the mouth seems to be misaligned, due to the improper annotation, as well as the unmatched color profile of two faces.
Figure 12. Human-human Face Example 1 (w/ Poisson Blending) Source & Target: “American Gothic” by Grant Wood
Figure 13: The final outputs of two different blending techniques are compared: Poisson blending, which tends to maintain more of the source face, and Mixed blending, which more effectively blends the source face into the target face, resulting in an interesting stylistic look.

IV. Implementation

For this project, Python was our primary programming language, complemented by a range of open-source libraries. Below is a list of these libraries in our project:

  • cv2 (OpenCV): 
    • General Functions: Used for reading images, converting color spaces, visualizing landmarks on images, and calculating the bounding box coordinates.
    • Convex Hull and Subdiv2D: Used for creating convex hulls and Delaunay triangulation, essential in the process of meshing facial landmarks.
    • cv2.getAffineTransform(), cv2.warpAffine(): These functions perform affine transformations for morphing faces from source to target images.
    • cv2.fillConvexPoly(): Fills a convex polygon, which we used for masking and blending regions in the image.
    • Poisson Blending: we opted for the cv2.seamlessClone() function instead of the functions built during project 3 for its efficiency and effectiveness in the blending step. 
  • dlib: Used for its robust facial detector and predictor, providing the facial landmarks necessary for our transformations.
  • Pillow: Used for opening and handling the content and style images from their respective directories.
  • Others: matplotlib, numpy, json

For image datasets, some were acquired from the internet (e.g., Google Arts, Google Search), while others were generated using GAN technology (e.g., DALL-E). The sources of these images will be listed in the reference section.

V. Problem

The main problem was achieving a seamless blend of the source face with the target artwork's color palette and style. We initially tried mixed gradient blending, which sometimes caused the source face to lose its unique features. We then experimented with poisson gradient blending, which retained more of the source face structure and was ultimately chosen for our project. We provide users with both blending options, allowing them to select based on their preference.

Figure 14. Poisson blending is impacted by the mustache on the target face, resulting in darker blended areas. Mixed blending achieves a more stylistically consistent result.
Figure 15. Due to the source face's brightness compared to the target, neither blending technique used here yields an ideal result.

For future improvements, we hope to explore other techniques that account for variations in lighting, saturation, and color profiles. Potential solutions include:

  • Implementing a multi-scale blending approach for preserving finer details while ensuring a smooth transition. 
  • Incorporating color correction algorithms to match the source face's color tone with the target artwork.
  • Enhancing triangulation, especially around key facial features, to achieve more detailed and accurate morphing.
Figure 16. Distorted areas in the source face when morphed into the target face, due to a limited number of control points.

Our initial goal of facilitating general object migration between artworks, time constraints led us to focus solely on facial swaps. An automatic segmentation method could be incorporated into the project to handle the different shapes, colors, and textures present in various objects, which would be quite interesting to observe as objects migrate through each other across different artworks. Recent developments with Semantic Automatic Matting (SAM) (Kirillov et al., 2023) seem to be a promising option to be integrated in the future.

VI. Acknowledgements

This final project was completed by a group of three students, led by Tommy. The other two members include Rishit C., who focused on applying style transfer to the source image, and Taeseok R. As mentioned earlier, this document is an edited version of the project report, and as such, some content, such as details on the style transfer process, has been omitted here.

Tommy Liu

  • Project report writing and the finalized version
  • Implemented a tool to manually annotate the 68 landmarks for cat face, as well as save/load the landmark file to be used in the Notebook
  • (Main notebook) implemented the parts for landmark extraction, triangulation w/ convex hull and Delaunny algorithm, face blending, the final output and comparing section, as well as the final code integration
  • Ran demo on various source and target images to showcase the project's capabilities
  • Wrote the README file in the GitHub repo

The full code is located here on GitHub.

Reference

List of images used

  • van Eyck, J. (c. 1438). Portrait of Giovanni di Nicolao Arnolfini. [Painting].Berlin.
  • Christus, P. (c. 1465-1470). Portrait of a Young Girl. [Oil-on-oak panel]. Gemäldegalerie, Berlin.
  • van den Valckert, W. (1617). Man With a Ring. Location Unknown.
  • Vermeer, J. (c. 1665). Girl with a Pearl Earring. [Oil painting]. Mauritshuis museum, The Hague.
  • Repin, I. (1884). Portrait of Vsevolod Mikhailovich Garshin. [Oil on canvas]. Metropolitan Museum of Art.
  • Wood, G. (1930). American Gothic. [Painting]. Location Unknown.
  • Elmahmudi, A. and Ugail, H., 2021. Identification of facial landmarks using Dlib [Digital Image]. In A. Elmahmudi and H. Ugail, "A framework for facial age progression and regression using exemplar face templates," The Visual Computer, vol. 37. doi: 10.1007/s00371-020-01960-z. Available: https://www.researchgate.net/figure/dentification-of-facial-landmarks-using-Dlib-a-Facial-landmarks-b-The-position-and_fig2_343699139
  • Adam88xx. (2014, May 23). Cat Kitten Gaze Regard Glance Look Head. [Digital Image]. IStock. Available: www.istockphoto.com/photo/cat-gm493250135-40361554. Accessed 6 Dec. 2023.
  • Aleksandrov, V. Red Cat. [Digital Art]. ArtPal. Available: www.artpal.com/VladimirAleksand?i=238586-12. Accessed 6 Dec. 2023.
  • Brooks, K. (2015, Sept. 3). Cats Are Taking over Famous Western Artworks and We’re Definitely Not Mad about It. HuffPost. Available: www.huffpost.com/entry/cats-are-taking-over-famous-western-artworks_n_55e75737e4b0aec9f355c018. Accessed 6 Dec. 2023.
  • Flood, S. Orlando. [Painting]. Year and Location Unknown.

List of readings

  • A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollár, and R. Girshick, "Segment Anything," 2023. arXiv:2304.02643 [cs.CV]. [Online]. Available: https://arxiv.org/abs/2304.02643
  • D. King, "Dlib's demo code for using 68 landmarks predictor," dlib.net. [Online]. Available: http://dlib.net/face_landmark_detection.py.html. [Accessed: Dec. 05, 2023].
  • OpenCV, "seamlessClone() - OpenCV 3.4 documentation," docs.opencv.org. [Online]. Available: https://docs.opencv.org/3.4/df/da0/group__photo__clone.html#ga2bf426e4c93a6b1f21705513dfeca49d. [Accessed: Dec. 05, 2023].
  • OpenCV, "Subdiv2D() - OpenCV 3.4 documentation," docs.opencv.org. [Online]. Available: https://docs.opencv.org/3.4/df/d5b/group__imgproc__subdiv2d.html. [Accessed: Dec. 05, 2023].
  • L. R. Finka et al., "Geometric morphometrics for the study of facial expressions in non-human animals, using the domestic cat as an exemplar," Scientific Reports, vol. 9, no. 1, Art. no. 9883, Jul. 2019, doi: 10.1038/s41598-019-46330-5.
  • G. Martvel, I. Shimshoni, and A. Zamansky, "Automated Detection of Cat Facial Landmarks," 2023. arXiv:2310.09793. [Online]. Available: https://arxiv.org/abs/2310.09793 [cs.CV]. [Accessed: Dec. 05, 2023].
  • G. Martvel, N. Farhat, I. Shimshoni, and A. Zamansky, "CatFLW: Cat Facial Landmarks in the Wild Dataset," 2023. arXiv:2305.04232. [Online]. Available: https://arxiv.org/abs/2305.04232 [cs.CV]. [Accessed: Dec. 05, 2023].
  • Tech4Animals Lab, "CatFLW: Cat Facial Landmarks in the Wild Dataset," Tech4Animals, 2023. [Online]. Available: https://www.tech4animals.org/catflw. [Accessed: Dec. 05, 2023].
  • Piętka, Ola. “Face Swapping & Style Transfer.” GitHub, 14 Nov. 2023, github.com/OlaPietka/Portrait-Style-Transfer. Accessed 7 Dec. 2023.
  • Q. Zhang, "Landmark Detection for Animal Face and 3D Reconstructions," zhangtemplar.github.io, 2023. [Online]. Available: https://zhangtemplar.github.io/animal-keypoints/. [Accessed: Dec. 05, 2023].
  • S. Mallick, "Face Swap using OpenCV," learnopencv.com, 2023. [Online]. Available: https://learnopencv.com/face-swap-using-opencv-c-python/. [Accessed: Dec. 05, 2023].