projects

Art made with artificial intelligence

Explorations with VQGAN + CLIP, prompt engineering and the Rusty Roboz visual universe.

6/1/20223 min

In the summer of 2021, a Google Colab notebook came into my hands, which using VQGAN + CLIP was capable of generating images from a text sentence. The results were quite impressive, and there was something magical in writing a sentence and seeing how the AI interpreted it and generated what for me was undoubtedly a work of art.

I kept testing for months, using initial images that the AI used as a starting point, mixing text inputs so that the image was a combination of two sentences, and experimenting with a new concept for me: prompt engineering. Some typical examples were ending the sentence with "top of artstation" to give it a professional drawing style or "rendered by Unreal Engine 4" so that the image looks like it was generated by that engine.

Some of these images can be seen in the following Instagram account, where I uploaded some of my first tests, using titles of books, movies, comics and video games as prompts:

Instagram / rustyroboz

At one point I started using as a starting image a sketch I made some time ago of Jax, the little robot that appears in Ted Chiang's The Lifecycle of Software Objects, and I also began including "Rusty Roboz" in all the prompts. The idea was to see if from the same initial image and the words "Rusty Roboz" interesting variations could be created.

In this way I started to create a lot of Rusty Robozs, which took a long time since running these models requires a good graphics card, which I did not have, so I had to settle for using Google Colab. It is an excellent service that provides free access to a notebook development environment with access to graphics cards, even with its time constraints and limited hardware.

Little by little you start to get a kind of intuition about which prompt will generate the most interesting output. After a while I had a considerable number of images and videos of different Rusty Robozs and inevitably it occurred to me to make a collection of NFTs.