This project was part of a 1 week workshop at IAAC, utilising the institute’s two ABB IRB6700 industrial robots. The workshop incorporated Dr. Gannon’s interaction workflow which enabled safe, real-time control of the industrial robots.
Our project Strings of Life is a dual robotic recreation of a marionette, where 2 participants would use their hands to interact with a camera to manipulate the robot’s position, making the puppet move.
For participants to get smooth and meaningful movements from the marionette system they had to communicate with one another to coordinate their hand movements. Though most of the time it ended in chaos…
Interaction logic
Over the week, time was spent developing the interaction logic for the participants to interact with the system. With consideration for where the participants would be located in relation to the robots was a key factor in how the data from the hand would control the robots axises.
With the participants located behind the robots and after some simulated tests it made sense for the participants hand positions to manipulate the robots axis as shown in the image below:
Workspace & control
System architecture
To make this process of interaction possible we developed a python script that incorporated the libraries of openCV and media-pipe to obtain a digital skeleton of the participants hand. From there we were able to compute the positions of the hand and fingers, and then couple this data to a specific axis of the robot before publishing it to ISAAC SIM.
End effector & puppet design
The end effector was designed from 40mm square section of timber with a lap joint at the cross section. The puppet was made from the same timber and connected to the end effector with elastic cord.
Credits
Strings of life is a project of IAAC, Institute for Advanced Architecture of Catalonia developed in the Master in Robotics & Advanced Construction 01 – 2023-2024
Project team: Feargal Murphy, Geetham Pasumarty, Arun Prasad, Prottay Roy Chowdhury and Vasista Monavarthy
ferg.works 