Generate Images of Your Face with Stable Diffusion and DreamBooth

One of the most impressive capabilities of the latest wave of generative image models is how well they can create variations of real people. Models like Stable Diffusion are trained on image datasets that include famous faces, and some of the results are surprisingly good.

For the last few weeks I have been experimenting with DreamBooth, a Google technique for fine-tuning. In practice, that means retraining a Stable Diffusion model so it learns a new concept, whether that is an artistic style, a new object, or your own face.

With the notebook below you can follow the full process and end up with your own Stable Diffusion model capable of generating new images where you appear.

• Training notebook on Google Colab

Prepare your photos

To start, you need around 10 to 20 photos of yourself. To avoid distortions and help the model learn a stable representation of your face, it is useful to keep them square and vary the backgrounds, clothing, and poses.

Time to train

To fine-tune the model we use Google Colab, which gives us access to a cloud GPU for free.

1. Check that the notebook has GPU enabled.
2. Grant access to Google Drive so the trained model can be saved.
3. Install the required libraries.
4. Download the base model from Hugging Face using your own token.
5. Configure fine-tuning with Training_Subject, With_Prior_Preservation, SUBJECT_TYPE, and INSTANCE_NAME.
6. Download regularization images.
7. Train the model and save the resulting .ckpt.
8. Test the fine-tuned model directly from the notebook.

Inference

Once the model is trained, you do not need to repeat all the previous steps every time you want new images. You can use a second notebook focused only on inference:

• AUTOMATIC1111 inference notebook