As mentioned in a previous post, I have been excited to experiment with AI Art Generators. That post primarily focused on OpenAI’s Dalle-2. However, lately I have been obsessed with an open-source alternative to Dalle-2 that can be installed locally and run on a GPU. I am of course talking about StabilityAI’s Stable Diffusion.
Perhaps one of the most advanced text-to-image models available at the time of this writing, Stable Diffusion is a deep learning model released in 2022 by a collaboration of Stability AI, CompVis LMU, Runway, EleutherAI and LAION. Stable Diffusion is based on latent diffusion models, a kind of deep generative neural network that can learn to produce high-quality images from complex data distributions.
WHAT IS CONTROLNET?
In my opinion, one of the most game-changing developments in AI Art is a new extension for Stable Diffusion called “ControlNet”. This Extension allows the user to upload a hand drawing sketch or reference photo and generate a completely finished Artwork from it. This gives the Artist far greater control of the output generated than was previously possible. Before you could generate a hyper-realistic looking person, which was incredible enough as it is. But now with ControlNet, you can control the pose and composition of that final image.
ControlNet is an extension for Stable Diffusion that allows users to control human pose and composition in image generation. It works by adding an extra conditioning layer to the original Stable Diffusion model, which can detect edges or human poses from an input image.
ControlNet is a powerful and versatile tool that enhances Stable Diffusion’s capabilities for generating realistic and diverse human images. It offers more control over the pose and composition of the output image than traditional text-to-image methods. It also enables users to explore different styles and genres of art using AI.
ControlNet is useful for creating artworks that require specific poses or compositions, such as portraits, comics, or illustrations. Users can use ControlNet along with any Stable Diffusion models, such as DistilGPT2 Stable Diffusion or Deforum1. Users can also use ControlNet with any image on the web as the input source.
To use ControlNet, users need to install an extension for AUTOMATIC1111’s Stable Diffusion web UI, which is a browser-based interface that allows users to access Stable Diffusion models online. Users can also use Google Colab notebooks to run ControlNet on their own devices. Users can then select an input image and a target model, and generate new images that copy the pose and composition of the input image.
ControlNet is a powerful and versatile tool that enhances Stable Diffusion’s capabilities for generating realistic and diverse human images. It offers more control over the pose and composition of the output image than traditional text-to-image methods. It also enables users to explore different styles and genres of art using AI.
In this post, I will show you just how easy it is to generate Hyper Realistic AI Generated Art from a simple Ink Drawing using Stable Diffusion and an extension called ControlNet:
BRING A SIMPLE DRAWING TO LIFE WITH STABLE DIFFUSION + CONTROLNET
The first thing I did for this project was make the simple Ink Drawing. Which I have included below:
Next step was to load Stable Diffusion and enter my Prompt and other settings:
After that it was time to scroll down to the ControlNet Section, upload my sketch, and set the settings:
After that all I had to do was click the big “GENERATE” button at the top and wait a few minutes for my Art to be generated. Here are the final results:
Even though I knew what I was trying to achieve, I was still pretty amazed with the results. This kind of technology was unimaginable to me just a few months ago. I still maintain that this will not replace artists but help creative work get done fast and lead to better creations overall.
If you are interested in learning more about this technology, I have included some helpful links below.
