Overview and Inspiration
Amidst a snacking session interrupted by an unavoidable YouTube ad, I pondered the inconvenience of interacting with technology while hands were messy. This sparked my exploration into Natural User Interfaces (NUIs) and Graphical User Interfaces (GUIs), seeking seamless interaction without physical contact. But why stop at functionality? I delved into gestural interfaces, envisioning a world where drawing and VJ-ing were possible with mere gestures, aiming to infuse surprise and delight into user experiences.
Challenges and Discovery
Using the handpose library from ml5.js, I developed an interactive system where users could manipulate particles and generate music simply by moving their hands. The system detected the position of the user's index finger to control the particle movement on screen, creating dynamic visual effects. Additionally, hand gestures such as pinching triggered acoustic guitar sounds, synchronizing with the particle movements.
While the prototype faced limitations in complex music creation and occasional gesture misinterpretation, it demonstrated the exciting possibilities of using hand gestures for creative expression and interaction with technology.
Technology and Design Execution
While experimenting with hand gestures, I discovered the Leap Motion Controller, an infrared-based device for precise gesture recognition. I explored using it to create Gesture Video Jockeys (GVJs) – a concept aimed at visually enhancing DJ performances and fostering immersive audio-visual experiences.
I made it work using a leap motion controller to be able to detect gestures which would then talk to TouchDesigner to control the audio visuals being generated in TD. Music played a huge part in this installation, the genre of choice, Techno!
What does the future hold?
Gesture-controlled interfaces have vast real-world applications:
Simplifying video calls with intuitive hand gestures.
Translating sign language for seamless communication.
Enhancing driving safety with gesture-controlled functions.
Enabling device interaction without touching screens.
Improving workplace safety by remotely controlling machinery.
Facilitating intuitive 3D modeling for artists.
This technology holds immense potential for creative expression and innovation across various sectors, promising a future where interactions with technology are more natural and intuitive.