Fantasia - Real-time Generative AI

Mar 19, 2024

Within the Devine research group, we’ve got a research project around Generative AI. To help communication around this project, I developed an interactive installation featuring real-time generative AI.

Beginning of January, my colleague Giliam Ganzevles showed me this video, which showcased super-fast generation of imagery within TouchDesigner using StreamDiffusion.

Inspired by that technical showcase, he came up with the following concept brief:

Fantasia: Conduct the Water Dance
Dive into a magical world and conduct your own Disney-inspired adventure with interactive visuals that are developed live by Generative Artificial Intelligence. Be inspired by the timeless classic Fantasia from 1940 and use your movements to control the water to the rhythm of the music. Watch as your magical powers come to life in the world around you!

Before tinkering with the StreamDiffusion TouchDesigner plugin, I needed to get some more background of TouchDesigner, as this was my first time working with it. I followed the great TouchDesigner Fundamentals series, which gave me a good understanding of the software.

StreamDiffusion is able to either do text-to-image or image-to-image in a fast way. You lose some image quality (because of the usage of LCM), but get back around 30fps in return.

In a first tryout, I tried feeding it the webcam input directly. While this worked, it was pretty hard to get some consistent results that responded well to the user input. Luckely, TouchDesigner comes with a built-in nvidia flex physics solver, which I used to create a water simulation, controlled by a body tracking system. This resulted in a much more consistent and visually appealing result.

left: Webcam image, center: physics solver, right: StreamDiffusion output

touchdesigner flow — Touchdesigner project flow

Of course, we are not limited to generate images with “waves”. As this is a diffusion model, we can change the text prompt to create other graphics:

different prompts — The impact of a different prompt on the output

Learned a ton doing this project, and I’m looking forward to expanding it with some more presets and color options.

You can play with the installation in the Budafabriek Kortrijk, during the Howest info days on Saturday 4 May 2024 (10:00 to 13:00), Saturday 29 June 2024 (10:00 to 16:00), Saturday 31 August 2024 (10:00 to 16:00) and Friday 13 September 2024 (17:00 to 19:30).