Performance/Installation:

Mesh is a single person audio-visual interactive installation to be experienced in a small, enclosed room. The person’s body and movements are tracked as they face a large visual projection of the interaction. This project was created during COVID-19 lockdown and was installed in my living room – the furnishings were draped in white bedding to allow for a more accurate tracking of bodily movements. Unfortunately, the video of the screen and audio recording lags and stutters. Running Quicktime to record was a bit too much for my computer while it also ran Max, openFrameworks, PoseNet, two webcams.

Ideally, Mesh would be installed in a completely dark room with a Kinect tracking the body. The dark, small enclosed space and surrounding sound are to achieve an encompassing experience.

Mesh is about a private experience with the surrounding space and feeling entangled and enveloped with the environment. It’s about playfully exploring the sonic virtual and physical space, letting it move you and you it, to experience the feedback and recursivity of the cybernetic system which the human is a part of not outside from.

Technology:

Skeleton Tracking With PoseNet:
PoseNet is a machine learning model that allows for Real-time Human Pose Estimation.

I used node to run javascript out of the browser, I ran a p5 sketch locally to detect the x and y values of key points of the body with a webcam and ml5’s PoseNet model. These values were then sent to Max via OSC messaging. To help PoseNet detection I covered the living room with white sheets, paper and other white bedding, lit the space up with a large sad lamp and wore tight fitting black clothing for a stark contrast. I still had issues with the smoothness and accuracy of the values from PoseNet. With Max, Posenet, two webcams and openFrameworks running at the same time processing could be slow and the tracking could jump around creating “noisey” values. To smooth these out I used the bucket and line objects in Max.

*Ideally Mesh would be deployed with a Kinect to track the skeleton and body. The infrared tracking allows for Mesh to function in a dark room. My access to a Kinect fell through so PoseNet was the trusty backup.

Sound (MAX 8):
There are three main patches, each generating sounds using sound waves, modulation and envelopes. The modulations and amplitude levels of the sounds are controlled by the position of different assigned body parts relative to the width and height of the web cam image capture.

1. Frequency Shifting Signal & Percussion with Line~
Control: OSC RIGHT WRIST
Frequency shift amount is altered by the position of the right wrist. Amplitude of the patch is mapped to the y axis position of the right wrist. When the wrist is above (less than) 500 on the y axis the sounds are audible.

2. Detuning Additive Inharmonics
Control: OSC LEFT WRIST
The inharmonics are created by multiplying the fundamental frequency by non-whole numbers. Frequency shift amount is altered by position of the left wrist along the x axis. Amplitude of the patch is mapped to the y axis position of the left wrist. When the wrist is below (greater than) 600 on the y axis the sounds are audible.

3. Speed + Low Hum + OSC Out
Control: OSC RIGHT SHOULDER + FULL BODY MOVEMENT
There are two static “low hummer” sounds. One uses a frequency modulation and the other an additive synth. There is a third sound of a higher frequency range which is set by the position of the right shoulder on the x axis. The amplitude of this sound is modulated by the speed of movement calculated using frame differencing. The right shoulder position along the x axis also controls the amplitude of the whole patch – standing at one edge of the screen is louder than the other. This patch also sends the speed and amplitude values to openFrameworks with OSC.

Vision (openFrameworks):
The aim for the visuals was to create something resembling a viscous texture that could hold the features of the person interacting and the audio to appear as one substance. This was the inspiration and for further development lighting and a uniform material could be added to the project:

To reach the stage presented I first began with generating voronoi diagram. The ofxVoronoi
addon is used to create the vein like patterns.
github.com/madc/ofxVoronoi

The next step was sending the frame to the buffer the add blur shaders, convert the FBO to an image and then bind it to a texture to draw on a plane and then displace the plane based on the shades of the image. This means the height of the plane changes based on the levels of light and dark.

Next I applied the voronoi to the web cam image. They are draw based on points detected by the computer vision (ofxCv) addon as good features to track. This creates the feature of the lines drawing around the person’s body.
ofxCv
ofxOpenCv

The oF project receives the speed and amplitude values from the Speed MAX patch. The speed values effect the rotation of the plane and the height scale of the displacement map. The amplitude effects the amount of blur added to the image, though this is subtle, the higher the amplitude the greater amount of Gaussian blur.
ofxOsc

Hardware:

– MacBook Pro (Mojave 10.14.6)

– In-built camera

– USB Camera

– Lumie Sad Lamp

– Projector

– Sennheiser Headphones with 3.5 metre cord.

Theoretical Foundation:

I researched the topic of Human Computer Interaction, cybernetics and subjectivity for another project. Below are adapted excerpts from an essay written with Jonas Grünwald that have been the basis for this project. The full essay can be accessed here: http://www.marisadimonda.com/wp-content/uploads/2021/02/beyond-invisibility-rethinking-computer-interfaces.pdf

Cybernetics and Boundaries
Cybernetics is the study of the flow of information, messages and signals between humans and non-humans defined through feedback and information processing.1 It was initially established in 1947 by Norbert Wiener, an MIT mathematician. Cybernetics has had social, cultural, and theoretical implications on subjectivity because of its tendency to reconfigure boundaries when constructing a framework in which to place humans and non-humans. N. Katherine Hayles, has presented the history of cybernetics through three distinct phases.2 First-order cybernetics (1943-1960) was focused on information flowing through the system with the observer standing outside the boundary of that system. However, the observer could be drawn in to the system being observed through a feedback loop. Second-order (1960-1985) brought the observer from outside the boundary and fundamentally into the system thereby breaching the boundary between human and machine. The observer has a role within the system and thus can alter the flow and transmission of information.3 The observer enters the system they are observing therefore becoming an observer observed. With this redrawing of boundaries came the notion of reflexivity – that which was seen to generate a system becomes part of the system.4 From the self-referencing system emerges recursivity, meaning, the observer inside the loop of the system can adjust the system based on feedback moving from a linear mechanism of inputs and outputs to spiral pattern.5 Third-order (1980 – present) is concerned with virtuality. It holds the observer and the system within a complex environment which is networked, adaptive, coevolving and emerging.6 Concern became not just with how systems could replicate themselves but how systems could serve as the “springboard to emergence” 7. For example, growth within an artificial life computer program can evolve spontaneously in directions the programmer may not have anticipated.

Reflexivity and the Implications on Western Subjectivity
From the period of reflexivity we see the blurring boundaries of inside and outside challenging the Western idea of the subject. The Western subject is anthropocentric – an autonomous, private, stand-alone agent acting on the world where consciousness is the seat of human identity. The consequence of this is the use and abuse of those considered non-human and the misunderstanding of the entangled nature of the human/non-human milieu.8 Cybernetics breaks this Western tradition with it’s systems approach to “human-machine networks and collective consciousness” – within the system humans and non-humans are considered equal where the subject is constituted as a node which is part of a whole.9 Outside is inside and inside is outside. We have here, what Hayle’s coined, the “posthuman”, that is, the post-individualised-human where the human persists through machines.10 The study of cybernetics created a cultural shift which bled into media art practices that reflected upon and appropriated the tenets of cybernetics; information, feedback, and systems.

Cybernetic video artist and Radical Software contributor, Paul Ryan, used the analogy of the looping Möbius strip in his work to demonstrate how cybernetic reflexivity strips the barrier of inside and outside.

In Ryan’s work, Everyman’s Mobius (1969-1972), gallery visitors were invited to follow instructions to record themselves performing certain actions then watch this videotape recording before it was then erased by the next user.12 Ryan called the direct video experience of watching oneself “self-processing”. The viewing of yourself on video, for Ryan, dissolved the idea of being a singular isolated subject.13

With video we can know the difference between how we intend to come across and how we actually do come across. What we put out, what is taken by the tape, is an imitation of our intended image…Taking in your own outside with video means more than just tripping around the moebius strip in private. One can pass through the barrier of the skin-pass through the pseudo self to explore the entirety of one’s cybernet.14

Ryan was working in the early days of the introduction of portable cameras and home video recording. It allowed for instant feedback and new possibilities for making art – video was a new medium. Now recording yourself with a smart phone is part of the everyday. As a result the users of these technologies can have a hyper-awareness of themselves in media spaces, an awareness which was new and liberating for Ryan.

In an article discussing the appropriation of cybernetics by the Radical Software community, Carolyn L. Kane uses the notion “radical subjectivity” to define the new human-machine subjectivities brought about by reflexivity. Our interpretation of what Kane means by radical is taken from Herbert Marcuse. Marcuse set out the notion of radical subjectivity as referring to the “development of a form of self-consciousness that finds present social and economic conditions intolerable. The radical act is a refusal of these conditions and an orientation toward social transformation”.13 Cybernetics can be considered radical in that it challenges the dominant construct of subjectivity – the Western subject – by drawing wider boundaries and creating frameworks where machines, humans and animals are treated as equals. The subject is no longer a stand-alone agent acting on the world but within loops with ceaseless recursive flows and exchanges of information.14 This subject, however, has not prevailed.

Computer and Body as Media
What happens when we see the computer not as a tool but as media, like film, radio, or television? What happens when we see the body as media? From a cybernetic perspective human and animal bodies are media because of their capacities to store, transmit and process information.15 Positing the body as an informational medium brings it closer to, and entangles it with, computational media such as the Internet and Web. Further entangling occurs through the overlay of physical objects with virtual reality, for example, through mobile phones, GPS technology, and bio-sensors, which have allowed for physical and virtual realms to merge.16 In a collaborative virtual reality project, Diane Gromola, explores embodiment and the merging of the virtual and physical with Dancing with the Virtual Dervish: Virtual Bodies (1994-2003). The VR environment was constructed from MRI scans of Gromala’s body. Rather than use the “unfamiliar sensation and cognitive disruption” features of the VR experience to escape and disembody, Gromala wanted to look inwards to reconfigure and enhance her experience of her body as a way to deal with chronic pain – a re-embodiment through technology and the virtual realm.17 The disruption and immersiveness of VR allows for the user to potentially experience a disruptive embodied subjectivity with the data and images of Gromala’s body.18 This experience dismantles the Western notion of subjectivity by loosening the individual’s solipsistic subjectivity to be embodied with the machine outside.

Hayles defines the merging of physical and virtual realms as the fourth phase of cybernetics where the actual and the virtual are integrated to form mixed reality. The entanglement of the body and machine under the cybernetic framework allows for emergence and the ability to coevolve, thus, Hayles claims, it is part of the cybernetic impulse to perceive computers as cognisant and human bodies as media, blurring the line between nature and culture.19 However, Hayles does not wish to collapse the feedback loops connecting bodies and computational media, “these recursive feedback loops between culture and computation create a coevolutionary dynamic in which computational media and humans mutually modify, influence, and help to constitute one another in the phenomenon known as technogenesis.”.20 This idea is returning to the kind of radical subjectivity envisioned by Radical Software where the human subject is integrated into the system with the machine. Within the cybernetic system the outside becomes inside and the inside becomes the outside along the Möbius strip. Technogenesis brings the technology from outside to the subjectivity inside, and in doing so, affects it while recursively affecting each other. In this picture the human subjectivity is integrated with its non-human environment configuring something vastly different to the traditional singular isolated subject of the West.

Examining human computer interaction and subjectivity through the history of cybernetics helps us to understand the long standing potential to deconstruct and dismantle the human-machine divide and the Western notion of the individual subject.

References:
1 Kane, Carolyn L. “The Tragedy of Radical Subjectivity: From Radical Software to Proprietary Subjects.” Leonardo, vol. 47 no. 5, 2014, p. 481.
2 N.K. Hayles, “Cybernetics,” in Critical Terms in Media Studies, ed. W.J.T. Mitchell and Mark B.N. Hansen. University of Chicago Press, 2010, pp. 147-149.
3 Kane [1] p. 482.
4 Hayles, N. Katherine. How We Became Posthuman: Virtual Bodies in Cybernetics, Literature, and Informatics. University of Chicago Press, 2010, p. 8.
5 Lovink, Geert. “Cybernetics for the Twenty-First Century: An Interview with Philosopher Yuk Hui.” e-Flux, 2019, www.e-flux.com/journal/102/282271/cybernetics-for-the-twenty-first-century-an-interview-with-philosopher-yuk-hui/.
6 Hayles [2] p. 149.
7 Hayles [4] p. 11.
8 Rae G. “Anthropocentrism”. Encyclopedia of Global Bioethics. Springer, Cham. 10th. ed. 2016, p. 3.
9 Kane [1] p. 482.
10 Ibid.
11 Greene, Rachel. Internet Art. Thames & Hudson, 2004. p. 10.
12 Everyman’s Moebius Strip, Paul Ryan.” Electronic Arts Intermix, http://www.eai.org/supporting-documents/694.
13 Kane [1] p. 482.
14 “Herbert Marcuse” Stanford Encyclopedia of Philosophy, https://plato.stanford.edu/entries/marcuse/.
15 Kane [1] p. 485.
16 Hayles [2] p. 148
17 Ibid.
18 Gromala, Diane. “Re-Embodiment Dancing with the Virtual Dervish: Virtual Bodies.” Riding the Meridian, 2003, www.heelstone.com/meridian/gromala/gromala.html.
19 Ibid.
20 Hayles [2] p. 153.