NUAGE: Design and evaluation of a tangible audiovisual interface
The design and evaluation of the tangible audiovisual interface NUAGE is part of the research project of Marie-Eve Bilodeau, a master’s student in electrical engineering at the École de technologie supérieure (ÉTS).
Marie-Eve Bilodeau’s master’s degree is rooted in the field of human-computer interaction. Her thesis aims to rethink the design of interfaces used by audiovisual performance artists.
How can we make the stage presence of audiovisual artists more interesting and interactive, both for themselves and for their audience?
NUAGE experimentation, test session at the SAT, Photo credit: Yan Breuleux, Visuals: Rémi Lapierre
The interfaces traditionally used for the creation of audiovisual performative works are mainly software tools. They are graphical user interfaces that offer little possibility of interaction to the artist, in addition to making him/her invisible.
Bilodeau therefore proposes the use of “tangible” interfaces as an alternative to traditional computers, screens, keyboards and mouse. Her thesis presents a new design methodology anchored in the principles of “tangible interaction” [1] to guide the design of audiovisual performative interfaces.
In order to test and implement the proposed design methodology, the NUAGE interface prototype was designed and then tested in a real context of use.
Résumé de la méthodologie
The design methodology proposed by Bilodeau is based on 4 main principles of tangible interaction [2]:
Tangibility and materiality
Physical embodiment of data
Bodily interaction
Real space embeddedness
In other words, interfaces must be physical, represent data in a physical way, allow for bodily interaction and fit into real physical space.
The methodology identifies the design intentions specific to audiovisual performative interfaces. Thus, a tangible interface should meet the following objectives:
Expressiveness
Performativity
Participation
Aesthetics
Engagement
Finally, the methodology proposes a taxonomy of the different types of audiovisual tangible interfaces:
Interactive Surface and Tabletops
Constructive Assemblies
Token and Constraints
Pressure Based Interfaces
Wearables
Motion Based Interfaces
Augmented Common Instruments
For more details on the methodology, see Marie-Eve Bilodeau’s master’s thesis “Méthodologie de conception des interfaces tangibles audiovisuelles et artistiques” [3].
Design of the NUAGE interface
The NUAGE interface aims to meet all the design intentions of the methodology. It implements two of the interface types, Pressure Based Interface and Motion Based Interface.
The NUAGE (Cloud) is a 3D silicone structure. Its seven hollow and independent bubbles give it the appearance of a cumulus. Each bubble is equipped with a barometric pressure sensor. The central bubble is equipped with an orientation sensor. The structure of the NUAGE was made by pouring liquid silicone into a mold designed in 3D printing. The sensors inserted in the NUAGE are connected to an electronic prototyping platform which processes the data received and retransmits them by the OSC communication protocol via WiFi.
The NUAGE is a control interface. Just like a keyboard or a MIDI keypad would be. The OSC data it transmits can be used by content generation software, such as TouchDesigner, to create visual and sound effects.
The use of the NUAGE is instinctive. It is a matter of taking it in your hands, pressing on the bubbles and moving it freely in the air. A simplistic use of the NUAGE would be to assign each of its bubbles to a musical note.
Interface NUAGE, Photo credits: Marie-Eve Bilodeau and Yan Breuleux
Interface evaluation
The NUAGE was evaluated during the creation and interpretation of an audiovisual work in front of an audience. A context close to its real future use. An experimentation was carried out with participants in the fall of 2021.
Artist-participants were given the mandate to create a 5-minute audiovisual work controlled by the NUAGE. The visuals were developed by artist-programmer Rémi Lapierre. He developed a procedural 3D animation generation system using TouchDesigner. Yan Breuleux [co-researcher, UQAC-NAD] realized the sound part of the work in collaboration with Claude Asselin. The sound content was created using GRM Tools.
The data collection session took the form of a live performance of the work in front of audience participants at the Society for Arts and Technology (SAT) in Montreal.
NUAGE experiment, November 2021, Photo credits: Marie-Eve Bilodeau and Yan Breuleux, Visuals: Rémi Lapierre
Likert-type questionnaires were given to audience members and performers before and after the performance. An informal discussion followed. The questions and discussions focused on the perception of the interface and the performance in relation to the design intentions listed above. For example, the level of visibility of the interface, the perceived level of control and the aesthetics of the interactions were evaluated.
The following video presents the work “Close to me”, performed by Yan Breuleux, during the experimental session that took place on November 30th at the SAT.
NUAGE experimentation, November 2021, Performer: Yan Breuleux, Visuals: Rémi Lapierre
This experimentation allowed us to make recommendations about the physical characteristics of the interface, but also about the scenography of a work using it. For example, the need to make the interface more robust, more sensitive and more visible was noted.
Towards new collaborative projects
From this first collaboration with Professor Yan Breuleux was born the desire to pursue her research on the narrativity of tangible interfaces in the context of immersive performance. Thus, a second round of creation took place in spring 2022 with an improved version of NUAGE.
Embodied Meaning : A/V performance Clouds in Fulldome format using a tangible interface (SUR_Exposition series), Photo credits: Yan Breuleux, Visuals: Vénus Grivegnée (NAD), Rémi Lapierre.
References:
[1] Hornecker, E. & Buur, J. (2006). Getting a grip on tangible interaction: a framework on physical space and social interaction. Proceedings of the SIGCHI conference on Human Factors in computing systems, pp. 437–446.
[2] Hornecker, E. (2015). Tangible interaction. The Glossary of Human Computer Interaction.
[3] Bilodeau, M. (2022). Méthodologie de conception des interfaces tangibles audiovisuelles et artistiques (Mémoire de maîtrise, École de technologie supérieure)
Marie-Eve Bilodeau is currently completing her master’s degree in electrical engineering at ÉTS. Her research focuses on human-computer interaction and the design of tangible interfaces for the audiovisual arts. She holds a technical degree in audiovisual electronics and a bachelor’s degree in electrical engineering. Marie-Eve worked as an FPGA designer at Grass Valley and as a system designer at Thinkwell Studio Montreal. Marie-Eve’s master’s degree is directed by Ghyslain Gagnon, professor in the electrical engineering department at ÉTS and co-directed by Yan Breuleux, professor at NAD-UQAC.
