October 2022

Virtual Fusion: voices of cyborgs and other beings

What constitutes a voice and how are different voices represented in digital worlds?

This question shapes a new collaboration between Idun Isdrake [student member, Concordia] and Jean-Ambroise Vesac [co-investigator member, UQAT] who got acquainted with each other’s work at the Materia Labo Public hosted by Hexagram and ELEKTRA in the fall of 2022.

Isdrake is a game designer, film director and cyborg artist, currently doing PhD research at Concordia University, and a member of the Hexagram network. Vesac is a professor of experimental and event design, head of l’Espace O Lab at UQAT Rouyn-Noranda. They both have experience in new media arts and audiovisual production, and Vesac has created an infrastructure for virtual scenes and performances, including a collaboration with opera singer Caroline Gélinas. Together, the three artists plan to create a performance art video game based on the convergence of their distinct voices. A fusion of electronic and classical music, with hybrid beings and environments.

At the beginning of their residency at UQAT Rouyn-Noranda, Isdrake got to experience materials that Vesac and Gélinas had already created. The three artists had met over Zoom to share their artistic process and discuss how to combine their styles. During the one week residency, Isdrake and Vesac worked with the motion capture studio at UQAT. Their explorative artistic process led them to research layers of meaning and fusion of styles in virtual space. They worked on the creation of Isdrake’s avatar which will inhabit the mixed reality space together with the avatars of Vesac and Gélinas. Motion capture techniques were used to record the artists movements, as well as photogrammetry which is a process for obtaining data from photographs to create 3D models.

The residency gave the opportunity for Isdrake and Vesac to walk, document and share stories connected to specific places that interacted together and inspired musical dialogue. The forest in Rouyn-Noranda became part of the backdrop in this initial state of the collaboration and the worldbuilding of the game. The artists also experimented with musical creation, and Isdrake showcased their innovative cyborg interface which was also included in the motion capture sessions.

Isdrake’s voice is based on a cyborg interface and audiovisual materials related to ice and the rapid melting of icebergs, as well as their Norse ancestry and homeland shaped from the body of an ice giant. This voice testifies to the recordings of glaciers that have become smaller and smaller each time Isdrake visited them over the past two decades and, together with scientific data, is the main content for the current version of this performative work and interface based on Near Field Communication (NFC) technology. Ten small NFC implants are located in Isdrake’s upper body, right under the skin on their hands and chest. A musical instrument based on a Raspberry Pi computer connected to Arduino boards holding NFC readers with antennas can read the implants when they are held within 1 cm from the skin. The magnetic field from the reader or any device featuring a NFC reader, like some tablets and mobile phones, can power a NFC implant to transfer a small amount of data, in the same way a metro card is scanned when entering the metro station.

Isdrake started thinking of non intrusive cyborg interfaces around 2011 when researching non-verbal communication, neurodiversity and selective mutism (specifically the inability to scream from pain or other disabling contexts). They got their first NFC implant as soon as it became available at a few piercing studios in Sweden around 2015. Isdrake’s work and research investigates how the physical and digital intertwine, and how to design diverse and inclusive interfaces with planetary accountability. Influenced by metal music as well as the sounds of melting ice, resembling screams, Isdrake creates versions of voices mediated through their body in symbiosis with technology, allowing them to play a scream and compose other audio from files added to the instrument.

Personal expression is grounded in the body and because avatars frame the embodied self they become a visual metaphor for identity and belonging. Digital recreation and the reclaiming of body appearance and identity marks are powerful tools for tribes, communities and societies where identity or culturally important artefacts have been forbidden or destroyed by religious and territorial colonialism. In this context, fire is another element Isdrake works with, symbolising the deaths of mainly women, who were tortured, raped, murdered and burnt by Christian priests during historical events like The Great Noise in Sweden in the late 1600s, and countless hunts of non-monotheistic peoples globally. Technological development, archives of data and stories are traversed and controlled by biased power structures. How can digital worlds and physical body modification for empowerment intertwine, give voice to people and other beings who are silenced by those power structures?

Body presence is important when working with avatars and mixed realities, and the freedom of movement we get in virtual worlds introduces new sensations. A sense of space is defined by our body, digital layers mixed with physical, makes us reimagine and redefine space, time and ourselves. Mirror neurons give us the possibility to feel connected to our avatars. When looking at your avatar in real time, making the same movements as your own body, the border between the two is erased and it feels like you are looking at yourself. Working with motion capture is a very interesting experience in terms of body and space perception. By using all levels of the ‘room’ Isdrake worked with both floor and air movement when directing the future movement patterns of their avatar within the game world which is planned to include both live and pre-recorded movement.

They also use Artificial Intelligence systems and tools to highlight and limit bias in datasets and to imagine hybrids of humans, cyborgs and ice. Some of this research will influence Isdrake’s avatar in this virtual fusion project as well. Below is an example of how Isdrake, through the Midjourney model, imagines the fusion of their avatar body with ice.

The next steps for this project is to design the game world, the three artists’ interactions with it, as well as a fourth character, the player of the game. Maybe they will shape the environment with echolocation mechanics? or compose new music from the different styles of the artists avatars?
The artists will continue to discuss their distinct voices and the way their bodies interact with the setting they are in. In the same way a bat uses echolocation to navigate a space, a singer bounces their voice on the walls of a concert hall to feel the room. A dancer bounces on floors, walls, in the air, a musician plays instruments, a cyborg channels screams of ice giants. Anything is possible, as long as there is diverse voices and languages.

Cette publication est également disponible en : Français (French)