In an era that is heading rapidly toward ‘personalized-everything’ (ads, movies, medicine, finances, luxury products, etc.), why not have personalized stories? And this is what an experience presented at Venice Immersive 2023 is trying to do.
Tulpamancer is an immersive experience that is uniquely tailored to each user. It uses AI-tools such as ChatGPT or Stable Diffusion to generate unique immersive stories (both visuals and narration) for each participant, in the same main theme.
So, how can the creators get to know you so fast to generate a tailored story? The experience introduces a storyline to collect information that will be used to generate images and narratives. The user is invited to the first room, where an old-generation computer asks you questions about your past and present, making you think about (maybe) forgotten memories of your childhood, but also about your present. All questions are the same for all participants, but based on their responses, the stories are different. In the second room, an immersive experience is generated, that makes you reflect on your past and possible futures. A unique storyline, a unique visual in the same main theme – as such showing the multiple and infinite possibilities available by using AI, even in this incipient and not so perfect stage of both technology development and new storyline productions.
Fascinated by the idea behind Tulpamancer, TechvangArt talked with the creators Matthew Niederhauser and Marc da Costa about their vision for the future regarding stories using XR and AI, challenges in creating stories using AI, but also about memories, and what collective memories could be erased 😀 😀
Matthew Niederhauser has been taking computers apart since he was a kid. The first time he encountered VR was in the 90s, playing video games when he always wanted to strap a monitor to his head. He worked as a photojournalist for a while and eventually came back towards creating new media projects, especially VR, when he co-founded the studio Sensorium. He caught up with his friend from college, Marc, and started to plan immersive works and artificial intelligence.
Marc de Costa is an anthropologist by training, but he has always been interested in the relationship between technology, society, and human experience. In addition, he has done a lot of work with AI and data, being interested in how VR and AI can open up new ways for storytelling.
OK, but before, let us explain what is the Tulpa.……
The ‘‘Tulpa” – an idea rooted in Tibetan Buddhism but popularized by the theosophical thinker Annie Bessant (1847-1933) – refers to the physical manifestation of thought through spiritual practice and intense concentration. A transformation in society now appears to loom with the emergence of AI, a technology that derives its power by concentrating intensely on enormous volumes of online text and images – the digital traces of our collective unconscious. The ‘‘Tulpa” was a way of making sense of and providing alternative frames for modernity. This work thus provides participants an occasion to explore alternative histories and futures for AI.
TVA: You are working at the intersection of XR and AI, and for me, it is very interesting, because it can offer new forms of storytelling, but how do you see yourself as creators in this type of intersection, why did you choose to use AI?
Matthew Niederhauser: We’ve seen a very large seismic shift in the quality of large machine learning models in terms of emulating language, and speech. We felt it was compelling as artists to begin experimenting with it, especially since we have always been interested in how it can try to engage and talk with you and then tell you a story. And so in the purest sense, that is how we approached Tulpamancer and of course, there’s a lot of structure and inside story, but we wanted to take this very particular moment in terms of the development of the technology to expose both its limitations, but also to point to what I personally feel is going to be a new realm of storytelling: very personalized, based on your data, and it’s not only going to be only images and video, but immersive as well.
Marc Da Costa: I would add that one of the interests that we have in AI is the fact that it is increasingly being woven into our everyday lives in a lot of ways. It’s certainly still quite new and emergent what’s happening right now. But, we’ve seen in the last 10 years the emergence of big data, data analytics, and the ways in which it’s converged with things like surveillance, capitalism, etc. We’re now at a moment where we approach this as experimental artists who are trying to understand not how this can be applied commercially but to understand what the implications are for real human experience in everyday life and how we can potentially find new ways to apply it as a creative medium.
TVA: And as I saw Tulpamancer – for me, it was kind of an experiment to do personalized storytelling, which is interesting, but it has also its limitations due to technology, but also because it is done maybe for the first times, how do you see this evolving in a couple of years?
Matthew Niederhauser: It’s been interesting days because we worked really to get it ready up to the last minute so even you were some of the first 30 or 40 people that we were able to interact with after coming out of the experience. I think that people that came out of the experience, it compelled them to think more deeply about the technology itself, which is probably been the most important impact. It gave people pause, to take a moment to think about the implications of what the technology is going to do. And I don’t think it’s going to be next year, but definitely, within the next 5 to 10 years, you’re going to see this type of personalized content, especially through video, podcasts, and all sorts of different mediums – and quite frankly, the time necessary to produce that content is getting shorter and shorter. And there’s a part of me that thinks on a longer timescale – in the decade or two if we think about a Metaverse or large 3D immersive and interactive worlds, which take lots of labor; AI is probably one of the perfect tools for creating large immersive environments for this type of interaction.
Marc Da Costa: And I would say the other side of that coin, which is very important to keep track of, is the fact that there are very powerful financial incentives for companies to implement it; and also potentially, companies might abuse this technology in a lot of ways. There are a lot of early concerns about how it will be applied, and people worry about misinformation. But also there’s certainly going to be huge transformations in a lot of labor markets. If we think of a country, such as the Philippines, where a lot of the middle class has jobs such as outsourced animation work, architectural rendering or call centers – these sorts of jobs will really change. When we talk about AI, it’s important for everyone who’s engaged in a political citizenship sort of way to really ask a lot of questions about it and track how it’s being implemented. Because of everything that happened over the last 10 years (with data analytics, for example), it would not be unfair to anticipate something similar and we have to keep an eye out for it.
Matthew Niederhauser: Also, building a Metaverse with this technology can potentially be a form for the most invasive forms of data collection. And the ability to use the power of immersive storytelling to shape behaviors. This also can get us to the discussion that some of these tools are open source and who owns them, can use them. That is why it’s important to know how this technology is rolling out right now, as well.
TVA: These are really important conversations, indeed…But getting back to your experience, you used a lot of things related to memory. You put the users in a situation to dig into their memory, and then talk about personal memories, in front of/with a computer, you collect these memories, and then force users to re-memorize them, in the form of an immersive experience. Why did you choose the topic of memory?
Marc Da Costa: From my perspective, there are two levels. The first one is at the beginning of the experience when you’re sitting down and going through this chatting or interviewing process where users are recovering different memories. Here, we were interested in creating a context for people. By moving through different timescales and different parts of your life that have happened and your dreams for the future, the hope was to help people touch memories that were even a bit forgotten or quiet. And as such to have openness to the experience that follows because the interest in the work is to give people a very personal and intimate encounter with the help of AI technologies in a way that equips them to understand or to question or to reflect. So for us, memory gives someone a good basis of experience to approach it.
We were also interested in the longer historical context in which new technologies have emerged and been taken up by creative communities in different ways. This project is also inspired by some movements that were happening in the early 20th century, which was itself a time of great technological change that gave birth to abstract painting and there were changes in aesthetic styles, and through our conversations and work, we are also interested in giving a point of reflection on this contemporary technology that can hearken back to a previous moment where changes happened.
Matthew Niederhauser: I would briefly add that in terms of an installation or an experience like this, the moments leading up to being in a headset are sometimes the most important, such as priming the person to be potentially open, and pulled into a form of media consumption. Marc did a lot of the design of the objects in that first room which can make people feel nostalgic to it in terms of the chat and it puts people into a place where they feel ready to engage with what’s going to come in – as being in a headset really impacts you. This can potentially reinforce or even help users reimagine their own memories of these very specific times and even potentially give them a new basis for thinking about their future as well. So, it just felt like the right direction in terms of the theme that I feel…
TVA: What was the most challenging part in setting up the experience? I mean, it was the technical part, to get running it, to build custom experiences, or..?
Marc Da Costa: These are difficult things to build because they’re totally custom software. And because we’re really at the very leading edge of all of these tools. We’re the first ones discovering a lot of the problems, a lot of the bugs, and so on. There’s certainly an implementation challenge to things that always need our attention. And on top of that, certainly working with the indeterminacy of AI is challenging. So for instance, unlike Photoshop, where you apply a filter, and you kind of know exactly what you’re going to get, because AI is statistical, you don’t know, so you have to develop intuitions for managing all that chaos.
Matthew Niederhauser: We were lucky that our technical director, Aaron Santiago, a very talented coder, had been exploring the space in the same way as us. So he understood a lot of the ideas that we were trying to bring to fruition. But the biggest challenge is that as much as the systems that you build around it, you’re eventually sending something into the abyss of the model itself. And, you do have consistency to a certain extent in its responses. But, particularly as a storyteller who has worked in this medium for a while, and you are making very deliberate decisions about script, and animation art, then suddenly when involving AI you have to ‘let it go’. Especially when creating an open-ended responsive piece like this, because we have used AI to create, we would go through 1000s of iterations to find the one that we like, and then edit it and move through it. And it’s a different way of using AI as a tool, it is structured so differently, as a a creator, you just have to approach it in a new manner.
TVA: From a process perspective, how did you work, you create different scenarios or puzzles, prompt structures, or..? Marc Da Costa: It’s a work with a four-act structure that takes you through a journey from the intimacy of a bedroom to the ineffable unknown. There’s definitely sort of a framework put in to carry you through these 12 scenes across four arcs. So it’s sort of defined in archetype, but algorithms are fitting in – it’s almost like a coloring book where you have the outlines, and then the AI is doing the coloring.
Matthew Niederhauser: There is something called prompt engineering. And that is finding a larger structure in which we are taking the responses from the participants and filtering them through to create a certain tone and structure. And so in that sense, there is definitely an architecture in it. A lot of time was spent thinking about the structure, even though every single time it is very different and based on the responses of the participants. And quite frankly, it really is the first time we are showing it to a lot of different people, which is one of the more exciting parts about participating here. There’s just been so much more to think about after, now just showing it to the public for the first few times, but for me, we’ve been very focused on making it stable and presentable to the public. We’ve had a tonne of new ideas just from seeing a response.
As we have become accustomed at the end of the intervieTECHVANGART asked some Crazy Questions
Shot Questions:
TVA: A book that you will save if the world is collapsing:
Matthew Niederhauser: Slaughterhouse-Five, by Kurt Vonnegut
Marc Da Costa: Don Quixote, by Miguel de Cervantes
TVA: False memories. If you both have the power to induce to your co-creator a false memory, what would it be?
Matthew Niederhauser: Probably trivial things, such as I was much cooler at college than he potentially thought I was. Marc Da Costa: This is a very scary idea, I don’t know if I can do that with other people’s heads. That is my answer
TVA: What would you ask the psychologist about memory?
Matthew Niederhauser: Is it the amygdala or other parts of the brain that make me wake up every morning
Marc Da Costa: What do you find most surprising about memory?
TVA: If you had the power, what would you erase from out collective memory?
Matthew Niederhauser: I mean, to be sincere, I would probably erase some of the larger collective acts of violence that have occurred… But, at the same time, that’s the only way we learn… So, I don’t know… Marc Da Costa: steam engine, learning how to bust fossil fuels
TVA: OK, that is a very contemporary topic :)))
MATTHEW NIEDERHAUSER CREATOR | Matthew Niederhauser is an artist and photojournalist. His work pushes the limits of emerging AI and XR technologies within a wide range of mediums. He studied anthropology at Columbia University before earning his MFA in Art Practice from SVA while also a Pulitzer Center Grantee, Visiting Artist at the MIT Center for Art, Science, and Technology (CAST), and Member of New Inc. At New Inc he co-founded Sensorium, an experiential studio working at the forefront of immersive storytelling. When not focusing on projects that have premiered at Sundance New Frontier, Tribeca Storyscapes, and IDFA DocLab, he teaches at NYU Tisch and Tanden. Most recently, Matthew became a Studio Fellow and Technical Director at ONX Studio in New York. matthewniederhauser.com |
MARC DA COSTA CREATOR | Marc Da Costa is an artist and anthropologist whose work explores the relationship between emerging technology and lived experience. Da Costa’s artistic research and interactive installations examine how data and technical infrastructures focus our attention on the world in particular ways and, in so doing, shape the structures of experience available to us. Da Costa’s anthropological scholarship has explored these themes through studies of placemaking practices in the Anthropocene, with a particular focus on Antarctic research expeditions and critical cartography. Da Costa’s work has been exhibited widely in the US and Europe and his writing on the intersection of data and society has appeared in The New York Times, The Guardian, Vice, and elsewhere. marcdacosta.com |
 MICHELLE OLIVER GUTIERREZ INSTALLATION DESIGN | Michelle Oliver Gutierrez is an architect and experiential designer. Her work shapes physical settings to amplify storytelling through unique interactive experiences. She is typically involved in all aspects of design, from concept and art directing, 3D modeling, and rendering, down through coordinating fabrication – often serving as the bridge between every design aspect and production partners. Michelle mainly focuses on new media activations and interactive spaces that integrate digital components within story-driven settings, both in the real world as well as in virtual environments. oliver.studio |
 AARON SANTIAGO TECHNICAL DIRECTOR & LEAD DEVELOPER | Aaron Santiago is an XR new media artist based in NYC. Aaron has years of experience in XR including live VR installations, performances, and experiential VR work. Aaron’s work has been shown internationally in New York Live Arts, Boston Cyberarts, and The Shed. Aaron works across a wide range of technology with a cross-disciplinary practice of combining software engineering across a range of artistic mediums. Aaron’s work is collaborative and hybrid, including websites, VR, AR, and computer sculpture. aaron.work |
All photos are copyrighted by TULPAMANCER and 80 th VENICE IMMERSIVE and may be used by the press only for the purpose of news or editorial coverage
