Portrait Animations from Videos using VToonify – GitHub code

VToonify is an AI software that converts a portrait video to a stylized animation. It is built on StyleGAN. The paper is here.

1. First download the code from VToonify GitHub repository. Open a dos prompt, in the directory where you want to install, type the command:
>git clone https://github.com/williamyang1991/VToonify.git
2. Download checkpoints. You should read README.md in the “VToonify\checkpoint” directory. It tells you the folder structure of checkpoints.
3. Go to hugging face or Google drive to download “directions.npy”, “encoder.pt”, “faceparsing.pth” and “shape_predictor_68_face_landmarks.dat.” Put them under “checkpoint” root directory.
4. There are a few pre-trained models for different styles, such as cartoon, comic, illustration, pixar, etc. You can see the samples of each style at huggingface space. I suggest you try one of them first. “vtoonify_d_cartoon” has the best results. Go to vtoonify_d_cartoon at hugging face, download four files there and put under your “checkpoint\vtoonify_d_cartoon” directory.
5. You need to configure a virtual environment to run the code. If you haven’t installed Anaconda3, go to install Anaconda3.
6. Setup a conda environment with the instruction. If the instruction confuses you, you can download a ready-for-use stylegan3_env at Gumroad (compatible with CUDA 11.8). Unzip it and put it at your anaconda3 installation under “envs” directory.
7. If you are going to run with Nvidia GPU on windows, you need to install VS2022 buildtools. Set the path of cl.exe in the environment variables. The instruction and issues are explained here.
8. I suggest you run the code using CPU. The speed is acceptable. In order to run on CPU, you change 3 files: “VToonify\util.py”, “VToonify\model\simple_augment.py”, “VToonify/model/stylegan/model.py.” In these files, change model.stylegan.op to model.stylegan.op_cpu. See the detailed instruction here.
9. Prepare a short video mp4 or gif file with a clear face movement. Put it in “data” directory. You can also use the existing input file in “data” as well, for example “651.mp4.”
Note: VToonify works only with face (portrait) videos.
10. Open an Anaconda Prompt. Run command:
>conda activate stylegan3_env
10. Still in the Anaconda prompt, go to the directory “VToonify” and run command:
>python style_transfer.py --scale_image --content ./data/651.mp --video
11. When it finishes, the new video is saved at “output” directory.
12.  If you cannot play the mp4 file due to encoding differences, import the image sequence to After Effects or other video editing tools to render as a video.
13. (Optional) Now you can download other pre-trained models and check out their results.