Context:
In the context of sound design relying on the use of an analog modular synthesizer, we propose a robotic solution to assist the setup of audio wire connections between the different sound modules. Analog modular synthesizers were introduced in the 1960s like the Moog Modular 55 system synthesizer developed by the pioneer Robert Moog. These first synthesizers revolutionized the creation of novel artificial sounds and are still today used to create new musical genres because their possibilities of electronic sound exploration are almost unlimited.
Today, software audio plugins are used by music producers to emulate the sound of modular synthesizers. Nevertheless, the real hardware modular synthesizers are still used due to their ergonomic properties and their possibility to be interfaced with other hardware devices.
With a real modular analog synthesizer, the creation of a sound consists of interconnecting the inputs/outputs of several sound modules (oscillators, filters, etc.) with several wires and adjusting a multitude of parameters using analog potentiometers and switches. The resulting configuration of a sound is called a "patch". However, a major limitation of these old machines is that they have no memory and it is therefore impossible to save their settings to find them again when the musician explores a new sound after changing the previous wiring.
Objectives:
Therefore, in this PhD thesis, we propose to control a robotic manipulator to autonomously retrieve a sound patch that was previously created manually by a musician. The advantage of this robotic solution is to preserve the integrity of the instrument without any hardware modifications.
The objective of the robotic task is to automatically insert all the electric wires required on the different modules of the modular synth and to turn the potentiometers and switches to retrieve the sound of interest. The approach we plan is to develop a visual servoing control strategy that will rely on the use of data provided by an RGB-D camera onboard the robot.
From the scientific point of view, the subject is quite ambitious because the manipulation of linear deformable objects like electric cables is not trivial due to their deformable behavior in opposite to rigid objects. Moreover the control strategy of the robot will have to consider several constraints such as for example, the occultation of the camera field of view at the time of the insertion of the cables, and the mechanical limits of the potentiometer/switches of the synthesizer when adjusting their values in order to not damage them.
Note that this concept of autonomous wiring is not limited to the sole application in the musical context. Wiring is a delicate task, involving gripping abilities, handling of deformable objects, cable routing, and complex actions such as connector insertion which are generally performed by skilled human operators. These tasks are particularly difficult to robotize, and are present in many industrial fields: installation of cable harness in the automotive field, wiring of electrical or control cabinets.
This proof of concept will be deployed on a 7 degrees-of-freedom robotic manipulator equipped with a RGB-D camera and a gripper (see picture at the top), by developing the following key aspects:
- Tracking of the wire and its connectors from visual observation (RGB-D), required for the grasping task.
- Visual tracking of the synthesizer panel for detection and localization of the different patching points.
- Development of a visual servoing approach for grasping one connector of a cable (stored in a rack).
- Positioning of the grasped connector at the insertion location on the synthesizer panel by visual servoing.
- Inserting the connector based on force control.
- Control the cable deformation to optimize the visibility of the synthesizer panel controls (potentiometers/switches) and patching points.
- Visual estimation of the potentiometers and switches current position setting.
- Grasping of the potentiometers/switches and adjustment of their values to the desired position.
- Investigation of an audio-serving for fine-tuning the potentiometers, by analyzing the sound produced, to get closer to the reference sound.
[1] R Lagneau, A Krupa, M Marchal - Automatic shape control of deformable wires based on model-free visual servoing IEEE Robotics & Automation Letters, 2020, vol 5, no. 4, pp 5252– 5259.
[2] J Zhu, B Navarro, R Passama, P Fraisse, A Crosnier, A Cherubini - Robotic manipulation planning for shaping deformable linear objects with environmental contacts IEEE Robotics and Automation Letters, 2019, vol 5, no. 1, pp 16-23.
[3] A Sengupta, A Krupa, E Marchand - Visual tracking of deforming objects using physics-based models IEEE International Conference on Robotics and Automation, pp 14178–14184, Xi’an, China, 2021.
[4] L. Smolentsev, A. Krupa, F. Chaumette - Shape visual servoing of a tether cable from parabolic features In IEEE Int. Conf. on Robotics and Automation, ICRA'23, London, UK, May 2023.