Immersed Body in VR Environments: Auditory Wanderer

In contemporaneity, immersive digital worlds create an experimental background that consists of body-movementinterconnections and augment our experiences with innovative oneiric fluctuations. Immersive environmentsprovide a dynamic context for investigating the relation between interaction engagement types and disembodiment,focusing on multi-sensorial stimuli.

The sense of immersion is in fact the definition of Virtual Reality; in particular, the feeling of being presentin the virtual environment simulates the sensation of the physical space, generating an illusion of presence inthe virtual world ¹. The embodied hybridization between the physical and digital body-avatar is a result ofthe induced

immersion due to alternative optic and acoustic environments. According to recent research in a variety ofcontexts, the senses of embodiment, body movement, and body ownership, i.e., being represented by an avatar or avirtual body that feels like it is the user’s body, as well as agency, orientation, and self-location,play an important role in amplifying the illusion of immersion in the VR experience ².

In order to analyse disembodiment in VR, we should delineate the concept of embodiment. We could defineembodiment as a phenomenological element where the body - digital or physical - is part of the cognition andperception of the world. Embodiment as a term includes diverse notions: the sensorimotor state of a body, itsdimensions and geometries (its morphology), and the mental representation of it in the overall perception ofreality. Grabarczyk and Pokropski refer to the subtle issue of embodiment in VR as something "hard to imagine(let alone design) a truly disembodied VR experience" ³. This is the reason that the duality of the embodiedand disembodied condition is vital and inseparable in the VR experience; it is a lived emotion that changes backand forth from one feeling to another interdependently.

Although it is not totally demonstrated, the relation between embodied cognition, simulation modes, and immersioncan provide a fertile speculative perspective for researching VR environments. In our investigation, digitalartists, especially those who work with sound or music, have explored a more metaphoric meaning of immersion inVirtual Reality. Despite having a virtual body representation that feels like it is our own, embodiment comesfrom the affordances of interacting with the environment, as defined by Gibson’s theory of ecologicalapproach in perception ⁴.

Interestingly, in this metaphoric mode, immersion is sometimes more connected with a sense of a more dream- like,surreal feeling of disembodiment," such as the sensation of falling¹. In this mode, the swingbetween embodiment and disembodiment is shifted from the visual representation of the virtual body to theauditory and kinesthetic perceptual terrain. This inherent contradiction between embodiment and disembodimentbreeds a disconnect or mismatch between the experiences that the physical body and the virtual body are havingat each given time (i.e., the VR environment simulates that my body is flying, falling, or floating, yet mykinesthetic senses indicate that I am not, which is registered as a disembodied experience).

The recreated soundscape shifts unmeasurable and intangible qualities along with the wandering that occurs in thevirtual reality environment. In the following paragraphs, we will attempt to emphasise the importance of 3Daudio as an immersive factor during the stage of disembodiment in the VR experience.

Two Ears – Three Dimensions in VR

In our effort to explore how disembodiment is perceived and how binaural² audio improves immersion inVR experiences, it would be useful to refer to binaural audio as the most widespread and accessible version ofvirtual environments’ sound today, delivered to the user through a normal pair of headphones. Alternativetechnologies such as wave-field synthesis ⁶ or higher-order ambisonics^{3, 7} using tens or hundredsof loudspeakers and typically associated with large projection or cave systems are not considered here, due totheir extreme technical and hardware requirements.

Even if it dates back to the late 1800’s, binaural audio became the object of intense research during the'90s ⁸,⁹ along with a wider early boom in immersive media and virtual reality research. The applicationfocus at the time was driven mainly by the transmission of information to the user through immersion intosimulated environments for engineering, scientific applications, audio-only spatialized⁴ sonificationof data ⁹, or auralization of simulated acoustics ¹⁰. Research on binaural technologies has continuedundiminished ever since ¹¹. At present, binaural audio is the dominant immersive audio technology in VirtualReality applications, combined with a highly visually stimulating environment. In this mode, 3D audio enhancesthe effectiveness of a VR experience significantly, filling out the sensorial gap that has been created by thestereophonic method. For 3D vision, there is now 3D audio included in one experience.

A three-dimensional sound environment provides an engaging experience for one to several participants within oneVR environment. 3D sound is synthesised by simulating spatial cues in natural hearing. In 3D audio, thedepth-of-field of the recreated soundscape changes dramatically, providing a better correlation with theacoustics of the virtual space. Binaural audio, which uses HRTF (Head-Related Transfer Function), is a responsegiving the impression that sound is coming from a particular point in space.

Basically, it is a transfer function that shows how a sound from a specific direction is modified acoustically bythe head and shape of the ears before reaching the eardrum (generally the outer ear) ¹². Essentially, thevirtual acoustics are integrated with binaural delivery in modern audio engines in one step: e.g., the reverbcorresponding to the virtual space is calculated in real-time, based on the position of the sound source, andthe listener. Then, each echo path is spatialized (binaurally) through its own HRTF for the corresponding echodirection with reference to the listener. After the initial distinct echoes, later room reverberation becomestoo random for each echo to be spatialized separately, and the respective binaural signals are approximated“on average” to have the right spectrum and timbre depending on the overall reverberation profile ofthe room. Early and late reverberant binaural signals are finally summed together by the algorithm. The binauralfiltering in itself has no capability to imprint "depth-of- field"; it only imprints directional cues to theaudio (where the sound is coming from, not how far it is). The combination, however, of the binaural filteringand the spatial reverberation gives us strong cues of distance: how far or close sound sources are. Thissimulation reinforces the immersive audio experience and occasionally gives the participants the impression thatit is identical to the physical space that the soundscape was recorded in. Thus, through the addition of 3Daudio, the feeling of presence in space is magnified, increasing the impact and profundity of a VR experience¹³.

The revival of binaural audio is justifiable if we think that a 3D audio environment fits in a virtual soundspace. Audio in VR is gradually becoming established in academia through a variety of applications (e.g.,gaming, media arts, and production), either in the form of specialised audio tools or as part of VR engines suchas Unity or Unreal Engine. This approach is effective to a great extent regarding the full-body reception ofsonic stimuli that propagate in physical space. The participants travel through a 3D visual setting; the bodybecomes a mediator of potentiality and multiplication, while 3D audio intensifies the immersive experience.It could be argued that VR that does not involve 3D audio but some non-immersive, non- spatially coherentversion in tandem with the visuals, such as stereo, actually degrades immersion ¹⁴. At this point, it would bewise to think of 3D audio as an alternative option that deserves to be explored in artistic VR works.

Three VR works – Three Perspectives of Sonic Immersion

While exploring the various modalities in which 3D audio is utilised in VR works, we were drawn to Evolver byMarshmallow Laser Feast, The Jellyfish by Mélodie Mousset, and Space Walk by Andrea Mancianti, Sebastian

J. Schlecht, Vesa Välimäki, Riku Jarvinen, and Esa Kallio. These three works⁵ introduceimmersion by utilising sound in different ways. In our effort to bring to the surface the connections between 3Daudio, immersion, and disembodiment, we follow a speculative approach by collocating these three works withinthe “audio in VR” context. The triptych of 3D audio, immersion, and disembodiment is investigatedbelow on a reflective level for further research. Therefore, it is necessary to present the works from ananalytical and technical point of view so that the context in which this investigation takes place is as clearand concise as possible.

Evolver: A Virtual Reality of Life and Breath by Marshmallow Laser Feast (2022) is a large-scale VR workthat involves a list of contributors from a variety of backgrounds: Marshmallow Laser Feast, an experientialcollective based in London, is behind the production, which is usually inspired from the intersection ofscience, art, and technology. They created the project in an attempt to take us on a dream-like trip into thedepths of human existence. An in-breath is visualised as it travels into the human body, which is scaled up toseventy feet. The vascular system becomes a dendroidal ecosystem that follows the cycle of respiration. Theexploration of the self occurs hypodermically, where the oxygenated, constant blood flow forms river deltas,ripples, and whirlpools, leading to a single “breathing” cell ¹⁵. The concept emphasises the ideaof breath, life and nature, interconnecting each of those elements with the cycle of respiration, a functionthat is given an ontological dimension. The interdependent connection between humans and the natural world isevident as the human cardiovascular system resembles an extensive branching system in full detail. It couldbe said that the participant enters into an endless network that looks like a human ecosystem building, “agiant branching human 100 feet tall with the size of an oak tree”⁶.

Through Evolver, a single breath transcends the visitor to realms of human existence never experienced beforewhile maintaining its significance as an integral part of our world. The participant is immersed in the journeyof oxygen flow through the body as it becomes a din of tiny particles that formulate patterns that we encounterin nature, like mycelium, pollinators, and bacteria. More details regarding the sonic part of the work will beaddressed below in the next chapter, where members of Marshmallow Laser Feast describe the design, process, and3D audio implementation. The Jellyfish (2021) is an artwork that invites the participant to sink intoan imaginary underwater state. In this VR work, Mélodie Mousset (HanaHana), Edo Fouilloux, and ChristianHeinrichs introduce us to glowing lifeforms who are beckoning for the participants to sing with them. In thisghostly marine setting, the glowing jellyfishes become the medium, an instrument that the participant candevelop and use in order to establish a communicative model. The audience dives into the deep waters of theirconsciousness and interacts with the jellyfishes. The involvement of the participant in this soundscape ofserenity is a major element that emphasises the immersiveness of the aquatic environment. If the participantlooks at one of the creatures, then they approach him, getting closer and trying to make contact. Eye contact,which is a basic factor of communication between animals, is perceived by the artists as a compositional tool, away to convey a naturalistic feel to the work. The participant’s gaze triggers the jellyfishes tocome closer to them and establish a connection.

Moreover, gross tactile-kinesthetic cues generate sensations which present in the work as the jellyfish movesaccording to the participant’s head and body movements. The body becomes a perceptual frame of reference,a reference point according to which the distances of objects are measured. Nevertheless, perception of thehuman body may vary at times, which might be an inconsistent factor for the design of VR applications. The waythe environment is perceived is as subjective as the perception of one’s own body. This idea isinterpreted in the artwork where the jellyfish mirrors the participant’s actions and it traces the body,suggesting ways that the participant’s body is differentiated from the aquatic environment ¹⁶.

Regarding the technical aspect that shapes the work’s soundscape, a hybrid-made synthesiser is triggered byreal- time voice analysis. The synthesiser measures the immediate pitch amplitude, tonality, and phrasing of theparticipant’s voice while shaping the soundscape, the underwater colour setting, and the animation of thejellyfishes and their environment. The harmonic elements and spatial traits are also controlled through thesynthesiser. The rhythmic elements are concurrently distributed through all headsets in the event that there isa group of participants. A system, which is developed following synaesthetic associations, matches a differentcolour to each note on the scale. In this way, it enables the participants to visualise their voice and gaincontrol of their vocal tone, which affects the colour of the virtual environment ¹⁷. The project is anengaging quest into the newly developing and upcoming world of VRMI (Virtual Reality Musical Instruments) ¹⁸,in which the possibilities are endless.

Space Walk (2020) is a virtual planetarium artwork that primarily has an educational character,providing detailed information on our solar system. It is based at Aalto University, at the Aalto VirtualPlanetarium, with the aim of investigating the application of 3D numerical space weathersimulations⁷. The second installment of Space Walk, beginning in 2020, focused on the improvement ofsound design and a stand-alone VR platform intending to improve the audiovisual experience while promoting theuse of an up-to-date affordable VR headset, ideal for users, libraries, art spaces, and academia.

The work involves visualisations of scientific data like magnetic field lines and atmospheric phenomena, followedby explanatory texts. The user travels from one planet to another through a navigational interface, while thesoundtrack develops impromptu, matching each celestial object sonically with the aim of accompanying theresultant data visualisation layers. The sound functions either in isolation or in combination with the layers,maintaining a consistent musical communication through the use of themes that are inspired by sci-fi film orvideo game music while giving prominence to the scientific context. Parts of sound are renderedstereophonically, while others involve immersive sound spatialization techniques.

Data plays a significant role in this work. Every planet connects to a description text box, and each userchooses with separate handheld controllers the elements that will be audible and visible. The latest version ofSpace Walk features physical parameters that are visualized near various solar system objects, like solar windplasma, interplanetary and planetary magnetic field lines and equal density surfaces of the solar wind and thegases. These gases are getting away from the atmospheres of Venus, Mars, and the comet 67P, which constitutes anephemeral evolution of the Sun’s interior magnetic field and also a wide distribution of radio waves inthe Earth’s ionosphere.

The musical part is composed according to aspects such as the quick shift from a deep and wide overview of thesolar system to a detailed perspective of a single celestial object. The fusion of experimental music genres(involving contemporary, electronic, and noise), constitutes a significant component in the creation of asonically rich soundtrack that blurs the boundaries between scoring and sound design. The themes, thesoundbeds⁸, and static soundscapes are looped and can be interrupted or rehashed in order to balancethe undetermined period of time that each user might spend at each possible observation point and the variousdynamic locations of the planets. The sonification of the planets is a process involving their physicalparameters but also a more abstract interpretation that draws from popular and literary sources. The Sun, forexample, is the primary source of energy in our universe, is represented by percussive sounds and polyrhythmicpatterns ¹⁹.

The technical part of the work involves the implementation of sound in order to achieve immersion through apractical approach; pre-binauralised stereo tracks boost control over the spatial fidelity and overall soundmix. This is noticeable when the users zoom out from the direct proximity that they have acquired with aspecific planet, and the stereo mix is dynamically rendered as a stereo sound source spatially located close tothe planet. Eventually, leaving or approaching a planet is sonically strengthened by a zoom sound effect, wherethe sound corresponds to the planet’s changing visual size. The real-time sound spatialization is executedby the spatializer included in the Oculus integration toolkit, while the stage before binauralization isprocessed through research-based plugins (SPARTA and 3D artificial reverberator) ²⁰, ²¹.

In conversation with the 3D Audio Artists of the Works

In our effort to address the way that immersion serves disembodiment and the way that our perception is affectedby the utilisation of 3D audio in VR artworks, Henrik Oppermann and James Bulley, who worked on the spatializedaudio part of Evolver; Christian Heinrichs, who specialises in procedural audio⁹ and created thespatialized audio for The Jellyfish, and Andrea Manciati with his team, who designed and implemented the soundfor Space Walk, provided answers to a couple of questions through online correspondence with the authors. In allcases, 3D audio takes on the role of an interpreter of the virtual environment, shifting the immersive aspect onan unprecedented scale and extending the contemporary body to unmapped territories of human perception.

Henrik Oppermann and James Bulley were interviewed for their contributions to Evolver. They have workedwith spatialized sound as a core part of their practice. For them, 3D sound is an important aspect of theoverall work in virtual reality environments. Especially, in Evolver, they used a mixture of thirdorder ambisonic soundbeds and mono sound sources that are spatialized. These were then rendered dynamically tobinaural audio in headphones depending on the position of the listener.

Regarding the use of 3D audio in VR and the possibility to further implement it in their work, they argue that itgives a much closer experience to how we listen to sounds in real life. They are able to recreate a very naturalexperience while also having the ability to make sounds hyperreal and more present. Adding to this, Bulleystated that there is huge potential in head/body position tracking and ambisonics, especially in hybridscenarios such as augmented reality: creating platforms for audiences to experience the world around them andengage deeply with more-than-human realities and extra-sensory experiences through spatial sound.

When they were asked if 3D audio is a better expressive medium in comparison to stereo, Oppermann asserted thatwith stereo mixes, the users hear all the sounds inside their heads without any kind of externalisation. Thiscan really break immersion, as it takes them out of the experience. A powerful audio engine that gives a lot ofoptions on how to control the sound is apparently preferable. Bulley was more analytical, explaining thatdifferent configurations like mono, stereo, conventional spatial formats, and 3D sound all have their places andcan be very expressive in different ways, since it depends on aesthetic conceptual choices and also on thecontext and environments that the work is for and presented in. At the same time, 3D sound has huge potentialfor truly immersive experiences that can engage with space in a way that stereo often cannot.

He also stressed that in terms of working with predefined spatialization features in game engines (or similar) orworking with handmade spatialization features, both can have their place, and there is much fertile ground inbetween. He refers to some exceptionally interesting audio engines and plugins that are currently available(such as the IEM plugin suite, Blue Ripple Sound, SPAT, Sound Particles, DearVR, etc.), that provide unique andpowerful scenarios to work with. Though he added that in many cases, where the work is driven by particularconcepts and needs, a bespoke, handmade solution (or a collaborative approach using some of these plugins oraudio engines) can be the only truly progressive way to work.

With reference to the subject of disembodiment and its relation to 3D audio, they were asked if 3D audiostrengthens the essence of presence from the user's perspective or the elements that the user meets in the VRenvironment. They both agreed that in essence, stereo brings disembodiment to the overall experience.Nevertheless, with 3D audio, users can connect the aural level with the visual level. 3D audio can create astrong sense of embodied presence within real, virtual, and augmented worlds that differs from other establishedsound formats.

An example of this condition in VR is where there are interactive sounding parameters at play (for example, handmovements linked to subtle particle sounds using motion tracking, or real-time monitoring of someone’sheartbeat linked to elements of a sound score). This embodied presence can, in fact, connect the aural andvisual elements in time and space, mapping a territory that Michel Chion usefully describes as‘synchresis’, allowing the play and interrogation of the spatial magnetization of sound with thevisual stimuli. They also underlined that works that involve 6DOF¹⁰ can create very interestingand novel aspects of embodiment for audiences moving in space, especially when teamed with ambisonic soundbedsthat move with the listener or can be “discovered” in the space by a listener.

For Christian Heinrichs, who has been incorporating 3D audio into his work since his undergraduate studiesbetween 2006 and 2010, the integration of the audio part of The Jellyfish was a much easier experience. Thespatialization plugins existed already (in this case he used Google’s Resonance SDK), and they allowed himto focus much more on the design of the sound, music and interactional aspects. In The Jellyfish, thisexperience is sonically quite unique compared to other experiences because it is so driven by the player’svoice and ears, literally disembodying the user’s voice and allowing them to control the environment.

Concerning 3D audio in VR and the potentialities of further implementation in his work, Heinrichs said thatinteractivity is the key to making next-level experiences. The fact that the user can look around and listen outfor different elements in a specific scene and be able to move towards or away from certain musical or sonicelements are vital key-points for him. Moreover, he emphasised that by establishing these two attributes, theusers can go much further, especially if they follow the path of hyper- embodiment or inhabitation; making themost subtle qualities of their very being in space have an effect on their surroundings, and so forth.

Heinrichs distinguished 3D audio and stereo as fundamentally different elements in VR creation. He claimed thatstereo is a great format for bringing powerful audio experiences across since it is already capable of giving astrong illusion of depth. However, he insisted that the users (arguably) do not “inhabit” a stereoimage, in the same way that they inhabit a spatial sound field. This opinion is verified if we take headrotations and positional movement into account. Heinrichs argued that it is the equivalent of comparing a videogame on a screen to a VR experience in a headset.

Regarding automated versus handmade 3D audio game engine qualities, he pointed out that there are aspects thatwill always be “automated” in spatial audio, especially in VR. For example, the soundfield has to berotated in real- time in order to respond to the users’ head movements and give them the impression thatthey are inside the sonic scene. In order to place a sound at a given location in space, the 3D audio designerneeds the audio engine (usually working jointly with the game engine) to encode a mono or stereo source to berendered at that particular location in space.

Another issue to consider is the distance attenuation, and other real-time effects that help mimic the way sound

behaves in the real world, along with the invention of sonic behaviours that might not be possible in the realworld, or their exaggeration, so that there follows a formation of a hyperrealistic space. Heinricks concludedthat sometimes it is worth designing a spatial soundfield outside of the game engine using ambisonic plugins andso forth, in order to have a static three-dimensional “bed” of sound that is decoded at runtime as asoundfield, again following the users’ head movements. Ultimately, these elements can be mixed andmatched. During such experiences, the 3D audio designer might also want to use straight stereo playback. Forexample, to contrast user interface features or other content that is rendered “inside the head”with 3D content in the scene, which is rendered “outside the head”.

In respect of the relationship between 3D audio and disembodiment, he argued that 3D audio is a huge contributorto the feeling of immersion and presence inside VR experiences; 3D audio can help the users perceive elementsthat they cannot actually see (e.g., behind them). The novelty of it alone can really trick people’s mindsinto thinking that they are really inhabiting the virtual environment.

Composer, performer, and media artist Andrea Manciati gave us more information regarding Space Walk. Thesonic part is divided into two categories: the sounds constituting the soundtrack of each planet, which uses thebinaural spatializer in Unity, mainly to compensate for the listener's head rotation and allow for a "credible"localization of sources. The second category involves some "baked" binaural tracks within the composition (i.e.,where the binauralization is embedded in the sound file rather than synthesised in real-time) layered withstereo ones to add lively and more "diffused" elements.

Manciati acknowledged that the potential of 3D audio in VR is very high. Though the tools available within gameengines are still in their infancy compared to the techniques that are available in more traditional audioenvironments (from DAWs to node-based programming environments), they have not yet been implemented in gameengines. On the other hand, he noticed a tendency to approach 3D audio often only as a technical way to enhancerealism, rather than a powerful creative possibility that could be based on a very different paradigm thanreality. He underlined that this tendency could limit its capabilities and risk keeping 3D audio in thesecontexts a bit superficial.

He addressed the complexity that sound in VR possesses by recognising the different implementations that stereoand 3D audio might have; the techniques applied are never neutral, and even if some could deliver objectivelymore details or information, it is not guaranteed that the creators will choose this direction in all cases. Hestated that the tendency to "engineerize" artistic choices (i.e., if the choice is more powerful,performant, newer, or more realistic, then it is automatically better), is a bit reductionist and does not helpthe cause of artistically relevant, expressive uses of certain technologies. Contrarily, he mentioned that heprefers non-automated 3D audio, where he has pure control over the making process, leaving a unique mark on hiscraft. He concluded that VR in general could be extremely powerful at evoking a sense of disembodiment, makingthis element one of the main axes of artistic experimentation. Certainly, attentive and creative strategiesusing 3D audio technologies could contribute to this aspect. Still, he highlighted that using a certaintechnology to define and measure such culturally loaded ideas as immersion and presence can run the risk ofsimplifying the concepts at stake. Lastly, he agreed that VR is powerful at reflecting on the ideas of presenceand embodiment, not so much because of its realism or because it makes us believe in something, but ratherbecause of the frictions it creates in the experience; in the dissonances and sense of estrangement it canintroduce.

Discussion

In our research, we explored the potentialities of disembodiment in artistic VR works and the modes in which 3Daudio functions in order to improve these experiences. 3D audio in the digital arts has been developing in thepast few years, along with the explosion of 3D audio utilisation in other sectors like entertainment (e.g.,video games). All three works that were presented belong to the category of 3D audio in immersive VR, and theyare useful to further explore the sense of disembodiment. The audio experience in all the works is a mixture ofrealistic and hyperreal sonic stimuli. The spatialized sound of the oxygenated blood flow in Evolver,the surrounding environment in The Jellyfish, and the majestic orchestral soundtrack of SpaceWalk form a type of hyperrealistic atmosphere, almost creating a sense of flotation.

Additionally, in all three case studies, head/body position tracking is the core component of the overallimmersive experience. In Evolver, the user explores the dendroidal cardiovascular system according tohis movements. In The Jellyfish, the invertebrates move according to the participant’s head,voice, and body movement, and in Space Walk, the user transports instantly from one celestial object tothe other, overcoming factors like distance and time via room-scale body movement.

Furthermore, sound localization is an important aspect regarding immersion and disembodiment; 3D audio is appliedby obtaining a peculiar localizational trait with the moving spatialized sound of the swirling blood flow inEvolver, the distant and then gradually close calls of the invertebrates in The Jellyfish, andthe corresponding soundtracks for each celestial body, where the sound object’s apparent width correspondsto the planet’s changing visual size in Space Walk. The users’ physical

cues and constraints are fully adjusted to specific 3D audio attributes which decode distance and direction ofsound in space, sometimes in an engaging and often extra-sensory way.

It is important to emphasise the fact that the selected artworks present a high aesthetic conceptual quality, andboth context or environment, combined with the use of 3D audio, can achieve a truly immersive experience.Although, as stated, spatialization plugins existed in the past, the applied technology of 3D audio in VR hastaken the experience to an innovative level. The aural and visual levels are interconnected and perceived by thecreators as a totally interactive container, triggering a diversity of sound parameters that emerge and becomeestablished as factors of immersion depth. It could be stated that in all works, the user's inhabitation(spatial and aural) and the so-called hyper-embodiment are attributes that connect while immersion occurs.Acoustic stimuli can be enjoyed in the virtual pace, amplifying the spatial sensations. Consequently, when asonic hyperrealistic framework is shaped, revolutionary, and intriguingly singular experiences emerge, enhancingthe bizarre and intense sense of immersion in these VR environments.

Conclusion

The body seems to defy gravity, in an almost disembodied state, as it experiences paths that the mind merelyindicates. The perception shifts qualities connected to a 3D audiovisual alternative reality. An embodiednon-human avatar is the leading actor who interacts with a dreamlike environment while expanding abstracttactile and kinesthetic cues. In these environments, there are no certain tasks given other than themesmerising, welcoming, and sometimes mysterious, immersive environments.

Through the analytical approach that was given by the 3D audio artists of the works that we explored, there is aclear sensical stance that binaural audio or spatialized three- dimensional audio may improve the feeling ofimmersion in virtual environments over a simple stereophonic audio delivery. Researching the implementation of3D audio, we demonstrated the affordances and the role that disembodiment acquires in the works, serving thefeeling of immersion in virtual reality environments, with a main focus on binaural audio.

Ultimately, sound becomes environmental, ecological, and universal. 3D sound acquires an omnipresent attribute,interconnecting embodiment, presence, and immersion in Virtual Reality. Thus, sound embeds the corporeal body invarious literal or metaphorical contexts where the interaction between embodiment and disembodiment results inan ontological shift; the participant objectifies the experience through these opposite sensations. Directionalcues that belong to the traditional stereo or even surround set-up could be less immersive compared to thespatial sonic advancements provided by 3D audio.

Binaural audio maintains its interaural cues by recreating the density and shape of a human head and by usinggross tactile-kinesthetic cues. Although the development of 3D sound in these contexts is still a nascent field,it is vital to elaborate more in depth through research studies that explore embodiment and disembodiment inrelation to immersive sound in VR environments. Undoubtedly, this impressive field and its advancements willhave even more implications if and when VR experiences become more prevalent in mainstream culture. For the timebeing, 3D audio is a sea of opportunity and a very exciting time to be working in and participating in thisfield for creators and researchers alike. In that fashion, 3D audio in VR can introduce a new revolutionaryconcept that reaches higher planes of existence in the embodiment/disembodiment interplay. It would be rewardingto come across more studies on the implementation of 3D audio in artistic VR works and research further on itsimmersive aspect in the near future.

Appendix

. Questionnaire on 3D audio, immersion, and embodiment/disembodiment in VR artistic works.

Evolver by Marshmallow Laser Feast, by Henrik Oppermann and James Bulley. Full answers by HenrikOppermann and James Bulley

On the utilisation of 3D Audio in VR.

1. Have you ever experienced 3D audio in VR before? Does "Evolver" involve normal stereo, or isit using binaural delivery?

HO: Yes. Many times. :D

JB: Yes – both of us work with spatialized sound as a core part of our practice. Within VR, 3D sound isvery much a key part of making work within the medium. A recent piece that Henrik and I have been collaboratingon, Evolver, with the art collective Marshmallow Laser Feast, is a current example of this.Evolver uses a mixture of third order ambisonic soundbeds and mono soundsources that are spatialized.These are then rendered down dynamically to binaural in headphones dependent on the position of the listener.

On immersion and interactivity

2. What are the potentialities of 3D audio in VR and how would you further implement them inyour work?

HO: It is not only a potential. It gives a much closer experience to how we listen to sounds in real life. We areable to recreate a very natural experience, whilst also having the ability to make sounds hyperreal and morepresent. JB: Adding to this, I think there is huge potential in head/body position tracking and ambisonics,especially in hybrid scenarios such as augmented reality: creating platforms for audiences to experience theworld around them and engage deeply with more-than-human realities and extra-sensory experience through spatialsound.

On sound design

3. Do you think that 3D audio is a better expressive medium in comparison to stereo? If yes,would you like the part of 3D audio to be automatically designed from a game engine or handmade, that is, tobe specially made from you as an exclusive feature (control all sonic attributes in the scenes of thework)?

HO: Yes. 3D audio is 100% necessary for immersive experiences. With stereo mixes you will hear all the soundsinside your head without any kind of externalisation. This can really break immersion and take you out of theexperience. A powerful audio engine that gives you a lot of options on how to control the sound is preferable.

JB: For me, I think different configurations like mono, stereo, conventional spatial formats and 3D sound allhave

their places and can be very expressive in different ways – it very much depends on aesthetic conceptualchoices, and also on the context and environments that the work is for and presented in. That said, 3D sound hashuge potential for truly immersive experiences that can engage with space in a way that stereo often cannot. Interms of working with predefined spatialization features in game engines (or similar) or working with handmadespatialization features, both can have their place, and there is much fertile ground in between. There are someexceptionally interesting audio engines and plugins that are currently available (such as the IEM plugin suite,Blue Ripple Sound, SPAT, Sound Particles, DearVR, etc) that provide unique and powerful scenarios to work with,but in many cases, where the work is driven by particular concepts and needs, a bespoke, handmade solution (or acollaborative approach using some of these plugins / audio engines) can be the only truly progressive way towork.

On embodiment/disembodiment

4. Since there is a kind of modality with embodiment/disembodiment connected to 3D audio, in what respect doyou think that 3D audio enhances the feeling of presence and contributes to the creation of a "disembodiedfeeling"?
In other words, does 3D audio strengthen the essence of presence from the user'sperspective or the elements that the user meets in the VR environment?

HO: Stereo really brings disembodiment to the overall experience. With 3D audio you can connect the aural levelwith the visual level.

JB: Discussions of embodiment and disembodiment in relation to experiences of 3D audio are complex andmultifaceted. There is no doubt in my mind that 3D audio, explored sensitively (whether heard over headphones,or from conventional multichannel speakers’ systems or technologies like wavefield synthesis), can createa strong feeling of embodied presence within real, virtual and augmented worlds that differs to otherestablished sound formats. One simple example of this in VR is where there are interactive sounding parametersat play (for example hand movements linked to subtle particle sounds using motion tracking, or real-timemonitoring of someone’s heartbeat linked to elements of sound score). As Henrik mentions, this can reallyconnect the aural and visual in time and space, mapping a territory that Michel Chion usefully describes as‘synchresis’ and allowing play and interrogation of the spatial magnetization of sound tovisual.Works that involve 6DOF (6 degrees of freedom), can create very interesting and novel aspects ofembodiment for audiences moving in space, especially when teamed with ambisonic sound beds that move with thelistener, or can be ‘discovered’ in the space by a listener. The development of 3D sound in thesecontexts is still a nascent field, and it is vital for there to be more in depth research studies exploring

embodiment and disembodiment in relation to immersive sound (particularly in VR environments).

B. Questionnaire on 3D audio, immersion, and embodiment/disembodiment in VR artistic works.

The Jellyfish by Mélodie Mousset (HanaHana) and Edo Fouilloux and Christian Heinrichs. Fullanswers by Christian Heinrichs.

On the utilisation of 3D audio in VR.

1. Have you ever experienced 3D audio in VR before? Does "Jellyfish" involve normal stereo, oris it using binaural delivery?

CH: I have been integrating 3D audio into my work since my undergraduate studies in Glasgow between 2006 and2010. Back then it was harder to get things like ambisonics and HRTFs to work and… In 2012 I developed afull audiovisual rendering and interaction system during a project placement at BBC R&D. There were noproper VR headsets and game engines were not as advanced and user friendly as they are today. This system used aportable projector mounted on a helmet, pointing at a curved retro- reflective curtain that would reflect theprojection straight into the direction of the wearer’s eyes. I used a Kinect to follow and detect bodymovements, Blender Game Engine to render the visuals and Puredata and some external libraries to do all thesound rendering. There were so many steps involved in getting all the bits of technology to talk to each other(without crashing) that it became difficult to go into a lot of details with the demo experiences. However, Idid manage to reach a sense of disembodiment in the “Beach Demo”, which involved swimming both overand underwater, exploring different parts of a beach. https://vimeo.com/manage/videos/59663582 (apologies forthe low framerate on the POV part of the video – the experience was in fact smooth, but the screen capturewas impossibly slow) https://vimeo.com/manage/videos/59354963 (remake of ascene from the point and click adventure “full throttle”).

Since then, things have become easier by orders of magnitude. This is largely thanks to the rebirth of HeadMounted Displays from 2013 onwards and the prevalence of accessible game engines such as Unity and Unreal. Thus,making the Jellyfish experience was much easier in terms of setting up 3D audio. The spatialization pluginsexisted already (in this case we used Google’s Resonance SDK, but there are others), and it allowed me tofocus much more on the design of the sound, music and the interactional experience with each one. I think thisexperience is sonically quite unique compared to other experiences because it is so driven by the player’svoice and ears, literally disembodying your voice and allowing you to control the environment. A similar piecethat’s worth checking out is “Breathe”: https://headspacestudio.com/projects/breathe/

On immersion and interactivity

2. What are the potentialities of 3D audio in VR, and how would you further implement them inyour work?

CH: The potential for using 3D audio in VR is huge, and relatively unexplored in my opinion. I really think thatinteractivity is the key to making next-level experiences. The fact that you can look around and listen out fordifferent elements in the scene. That we can then move towards or away from certain musical or sonic elements.Once you’ve established those two things, you can go much further, especially if you follow the path ofhyper-embodiment / inhabitation, making the most subtle qualities of your very being in space have an effect onyour surroundings, and so forth.

On sound design

3. Do you think that 3D audio is a better expressive medium in comparison to stereo? If yes, would you likethe part of 3D audio to be automatically designed from a game engine or handmade, that is, to be speciallymade from you as an exclusive feature (control all sonic attributes in the scenes of the work)?

CH: No, I don’t think “better” is the right word. It is, however, fundamentally different, inmy opinion. Stereo is a great format for bringing across powerful audio experiences and already is capable ofgiving a strong illusion of depth. However, you (arguably) don’t “inhabit” a stereo image, inthe same way you inhabit a spatial sound field. Especially once you take head rotations and positional movementinto account. It is the equivalent of comparing a video game on a screen to a VR experience in a headset.Regarding automated vs handmade qualities– there are aspects that will always be “automated”in spatial audio, especially in VR. For example, the soundfield has to be rotated in real-time in order torespond to your head movements and give you the impression that you’re inside the sonic scene. In order toplace a sound at a given location in space, you need the audio engine (usually working in tandem with the gameengine) to encode a mono or stereo source to be rendered at that particular location in space. You also have tothink about distance attenuation, and other real-time effects that help mimic the way sound behaves in the realworld (or do things that might not be possible in the real world or exaggerate them to create a hyperrealisticspace). Sometimes it’s worth designing a spatial soundfield outside of the game engine using ambisonicplugins and so forth, in order to have a static three-dimensional “bed” of sound that is decoded atruntime as a soundfield, again following the player’s head movements. These things can be mixed andmatched. During such experiences you might also want to use straight stereo playback, for example to contrastuser interface elements or other content that are rendered “inside the head” with 3D content in thescene, which is rendered “outside the head”.

On embodiment/disembodiment

4. Since there is a kind of modality with embodiment/disembodiment connected to 3D audio , in what respect doyou think that 3D audio enhances the feeling of presence and the way it contributes to the creation of a"disembodied feeling"? In other words, does 3D audio strengthen the essence of presence from the user'sperspective or the elements that the user meets in the VR environment?

CH: I believe I answered this question above. I think 3D audio (and you will hear many people say this) is a hugecontributor to the feeling of immersion and presence inside VR experiences. 3D audio can help you perceive thatwhich you cannot see (e.g. behind you). The novelty of it alone, when done right, can really trickpeople’s minds into thinking that they are really inhabiting the virtual environment. It will beinteresting to see how this develops, if/when VR experiences become more prevalent in mainstream culture. Rightnow, it’s a sea of opportunity and a very exciting time to be working and participating in this field.

Here’s a mini “sneak peek” I had done of The Jellyfish that gives a better impression than allthe other material that’s available of how the experience works and feels like: https://drive.google.com/file/d/1wzhPLrMJAZ7Ep9W5yqd1g950FPknGifl/view?usp=sharing

Here’s a non-public dev video where I’m singing into the audio system. Might give you a better senseof how the different audio layers work together: https://drive.google.com/file/d/1X4i5rNntwx5Ygu7Fx5oKPNv36C74svXL/view?usp=sharinghttp://dolphinclub.website/reveries/

C. Questionnaire on 3D audio, immersion, and embodiment/disembodiment in VR artistic works.

Space Walk by Andrea Mancianti, Sebastian J. Schlecht, Vesa Välimäki, Riku Jarvinen, and EsaKallio. Full answers by Andrea Manciati.

On the utilisation of 3D Audio in VR.

1. Have you ever experienced 3D audio in VR before? Does "Space Walk" involve normal stereo, or is it usingbinaural delivery?

AM: Yes, I have experienced a few VR works using so called 3D or "immersive sound" before, but admittedly used ingeneral pretty cosmetically, for reasons that might be related to the tools available. This particular pieceuses two main strategies: the sounds constituting the soundtrack of each planet are using the binauralspatializer in Unity,

mainly to compensate for the listener's head rotation and allow for a "credible" localization of sources. At thesame time within the music itself there are some "baked" binaural tracks (i.e. where the binauralisation isembedded in the sound-file, rather than synthesised in real-time) layered with stereo ones, to add lively andmore "diffused" elements. More details on the implementation can also be found in the paper.

On immersion and interactivity

2. What are the potentialities of 3D audio in VR and how would you further implement them in your work?

AM: Overall the potential is very high, but the tools available within game engines are still in their infancy,compared to the techniques that are available in more traditional audio environments (from DAW to node-basedprogramming environments), but not yet implemented in game engines. On the other hand, I see a tendency to thinkof spatial audio often only as a technical way to enhance realism rather than a powerful creative possibility,that could be based on a very different paradigm than reality. This tendency I think could limit its potentialand risk keeping 3D audio in these contexts a bit superficial. In my work the relationship between space andsound is central, so I definitely will use such techniques in the future, but the majority of my personalartistic work happens outside of VR because of the limitations I mentioned.

On sound design

3. Do you think that 3D audio is a better expressive medium in comparison to stereo? If yes, would you likethe part of 3D audio to be automatically designed from a game engine or handmade, that is, to be especiallymade from you as an exclusive feature (control all sonic attributes in the scenes of the work)

AM: I think the idea of better or worse is definitely tricky when talking about expressive means. I think theyare simply different. The techniques are never neutral and even if some could deliver objectively more detailsor information is not granted this would need to be preferred in all cases. I think this tendency to"engineerize" artistic choices (i.e. if it is more powerful, performant, newer or more realistic isautomatically better) is a bit reductionist and doesn't help the cause of artistically relevant, expressive usesof certain technologies. And for the second part of the question, no I would definitely not want those to beautomated. Hand making these aspects is totally part of my craft and I wouldn't trade for a script.

On embodiment/disembodiment

4. Since there is a kind of modality with embodiment/disembodiment connected to 3D audio, in what respect doyou think that 3D audio enhances the feeling of presence and contributes to the creation of a"disembodied
feeling"? In other words, does 3D audio strengthen the essence of presence from theuser's perspective or the elements that the user meets in the VR environment?

AM: I think VR in general could be extremely powerful at modulating this sense of embodiment/disembodiment, andthat could be one of the main axes of artistic experimentation with this medium. And of course, attentive andcreative strategies using 3D audio technologies could contribute to this aspect. But once again, (and I might bemisinterpreting the question here) I feel that using a technology to define and measure such culturally loadedideas as those of immersion and presence, can run the risk of simplifying the concepts at stakes. VR is in myopinion powerful at making us reflect on the ideas of presence and embodiment, not so much because of itsrealism, or because it makes us believe, but rather because of the frictions it creates in the experience. Inthe dissonances and sense of estrangement it can introduce.

D. Questionnaire on 3D audio, immersion and embodiment/disembodiment in VR artistic works.
Full answersregarding In the Eyes of the Animal by Abandon Normal Services and Marshmallow Laser Feast, by NatanSiningaglia.

On the utilisation of 3D Audio in VR.

1. Have you ever experienced 3D audio in VR before? Does "Evolver" involve normal stereo, or is it usingbinaural delivery?

NS: We used binaural delivery On immersion and interactivity

2. What are the potentialities of 3D audio in VR and how would you further implement them in your work?

NS: 3D binaural audio in VR has a huge impact on the level of immersion of a VR experience.

Think about how much information you can get, with closed eyes, just from listening to the sounds surroundingyou. You can understand what kind of dynamic you are in the middle of, and you get an idea of the qualities ofthe space (size, type,...)..In VR especially, the sound can dialogue in such a meaningful way with the visual,giving a "physicality" and a sense of consistency to objects in the view and even outside the view! By the soundof a virtual object, we can have a deep intuition about how we can interact with it, since the sound isconnected to the physical body of the object, it's consistency, mass, temperature... we can guess howdangerous/soft/pleasant/hard/... it would be to touch it. this kind of influence is subconsciously very powerfuland can be used to determine the psychological dimension of a user in a space. Audio can be even used to directthe attention of the user. this compensates a bit for the loss of the visual role of the (movie) director (beingVR a medium where is the

user that decides the shot, the framing, the view over the reality)

On sound design

3. Do you think that 3D audio is a better expressive medium in comparison to stereo? If yes, would you likethe part of 3D audio to be automatically designed from a game engine or handmade, that is, to be speciallymade from you as an exclusive feature (control all sonic attributes in the scenes of the work)?

NS: I don't think 3d audio is a better expressive medium than stereo.

I think that 3d audio is an expressive medium in which (in a much more effective and profound way, compared tostereo) specialization and immersion are key elements of artistic expression and methodology. Said so, from theperspective of a virtual scene creator, it's crucial to have control of the sound engine as much as the visualcounterpart, to be able to make use of all the sound generation-perception techniques in the composition of theexperience.

On embodiment/disembodiment

4. Since there is a kind of modality with embodiment/disembodiment connected to 3D audio, in what respect doyou think that 3D audio enhances the feeling of presence and the way it contributes to the creation of a"disembodied feeling"? In other words, does 3D audio strengthen the essence of presence from the user'sperspective or the elements that the user meets in the VR environment?

NS: I replied above.