An Artist Uses an iPhone to Visualize Sounds in A.R.Zach Lieberman is making sounds. “Click.” “Psh.” “Ah.” “Oorh.” “Eee.” With every noise, an amorphous white blob bursts onto a screen, leaving a trail of shapes lingering in the air. As Lieberman moves his phone backward through the cloud of blobs, the noises replay in reverse as if he were rewinding a vinyl record.
With this environmental information in hand, Lieberman can capture sound through the phone’s mic, process and visualize it with the app he built, and then map the illustrations to an exact location in the room. You can leave a snap in one corner, a word in another, and watch a trail of sound waves appear exactly where the phone first captured them. Move the phone along the 3-D path and it’ll work like a scrubber, replaying the audio forward and backward in real time. “It plays that chunk of audio that’s closest to where the devices is,” Lieberman says.
Still, the app provides one of the most obvious and transparent illustrations of how augmented reality works and what it can potentially do. Lieberman, like many of the other artists using augmented reality to produce new experiences, views his app as more of an experiment than anything. It’s a way to move past the commercialized experiences that inevitably follow hyped technology and think more critically about how it can be used. “It’s like, OK, we have a camera or microphone in space, so what does that mean?” he says. “It’s important to think about how we can push the interaction with that in different directions.”augmented realityARApple