Bot-thoven: A Robot Musician

Bot-thoven performs Beethoven and Other Famous Piano Works (Final Demo)

Plan

I will begin working with David to combine his MIDI file converting code with my robot. 

Procedures

I will connect my xylophone-playing robot and run David’s MIDI file converting code through Arduino and Processing software. 

Progress

We were able to upload the code onto the Arduino board and check if it would work. Currently, the code is not functioning perfectly as intended because it is unable to take a MIDI file data and have the robot play it. 

Problems

For next week, I will continue working with David to make his MIDI file converting code compatible with my robot. 

Plan

I will need to finish connecting the rest of my servo-mallet systems to a servo driver with my second order for the PCA9685.

Procedures

I will connect the servo-mallet systems with the servo driver and run my Arduino code to verify that the system can strike the xylophone key at varying dynamics.

Progress

The second servo driver has arrived and I was able to finish connecting the remaining 15 of my servo mallet systems to the servo driver for testing. I was able to run my Arduino code successfully through the servos, allowing me now to have the ability to perform a 30-note range passage with musical expression.

Here is a link to Bot-thoven playing across the xylophone (scale) in 3 different musical styles.  

Problems

For next week, I will need to begin working with David to combine his MIDI file converting code with my robot.

Plan

I will need to connect my servo-mallet systems to the PCA9685 servo driver.

Procedures

I will connect the servo-mallet systems with the servo driver and run my Arduino code to verify that the system can strike the xylophone key at varying dynamics.

Progress

I have finished taking 15 out of 30 of my servo mallet systems and attaching it to the servo driver for testing. I was able to run my Arduino code successfully through the servos, allowing me now to have the ability to perform a 15-note range passage with musical expression.

Problems

For the upcoming weeks, I will need to continue connecting the rest of my servo-mallet systems to a servo driver whenever my second order for the PCA9685 arrives.

Problems Faced During Research

One problem I faced while pursuing my research was the use of zip-ties as a fastener between the servo head and xylophone mallet. While I initially believed that using zip-ties would be secure enough to attach the mallet to the servo motor, the zip-ties kept slipping off. To address this problem, I used rubber bands as an extra fastener for my second prototype of the servo-mallet system, which resulted in the mallet to stop slipping off of the servo horn and become much more secure. 

Another problem I faced while pursuing my research was integrating my work in the Robotics Lab with that of my partner David’s in the Computer Systems Lab. While we initially wanted to use computer vision to convert an image of sheet music into data for my robot to perform, we realized that this method may be too difficult to achieve within our timeline. Instead, we decided to take a more simple approach by using a MIDI file and converting it into data for my robot to perform.

A final obstacle I faced was the long delivery time for my parts to arrive. My order for the second PCA9685 servo driver, a part crucial to finish the next step in my project, has still not arrived yet. To address this problem, multiple orders of the servo driver have been placed in hopes that the part will arrive very soon. 

Plan

I will need to finish building the remaining 15 servo-mallet systems.

Procedures

I will attach the mallets onto all of the servos and use a 3D printer to print the stands to hold up the servo-mallet system.

Progress

I have finished printing the final 15 stands needed for my servo-mallet system. Using zip-ties and rubber bands, I also completed attaching the mallets to the servo horn.

In addition, as shown above, the new components I had ordered arrived. This includes the power supply, the power adapter, and the servo extension cables. 

Problems

For next week, I will need to continue building the final set of servo-mallet systems by attaching the servo to the stands I have finished 3D printing. 

Plan

I will need to begin building the remaining 15 servo-mallet systems.

Procedures

I will screw the long servo horns on to all of the servos and use a 3D printer to print the stands to hold up the servo-mallet system.

Progress

I have finished attaching the long servo horns to the 15 servos. The stands are currently printing right now through the 3D printer and should be finished printing by Tuesday.

 Problems

For next week, I will need to continue building the final set of servo-mallet systems by attaching the mallet using a zip tie.

Practical Need or Problem Trying To Be Solved

Creating a musical instrument-playing robot will allow anyone to listen to a personalized live performance at any time or location. The robot will be able to perform any given repertoire with musical expression like a human performer as long as a MIDI file of the music is prepared for the program to read. While our project currently focuses on a single instrument – a xylophone, our work on how to have robotic instruments play with dynamics and musical techniques has the potential to be applied for a variety of instruments. The expansion of our research to other instruments will give listeners even more freedom in personalization. In addition, music has been shown to help dementia and Alzheimer’s patients in helping regain memory loss. Our technology has the potential to be used in the field of music therapy – where Bot-thoven can be set up in nursing homes to help residents use the power of music for positive recovery. 

Another reason why this project is important is because it will help develop fine motor control of a robot. Bot-thoven requires precise servo movement when controlling a mallet to strike the xylophone with varying musical emotions. The project’s main goal of “playing an instrument expressly” requires meticulous motor control, expanding Bot-thoven’s application to fields beyond music. Since motor control is an important aspect of any robotics project, the methods used to develop fine servo motor control through Bot-thoven will serve as important research and information for all roboticists. 

Plan

I will need to attach the servo-mallet systems I have created to the stands I have 3D printed and connect it to the servo driver/Arduino to test if the servo-mallet systems function as intended. 

Procedures

I attached the servo-mallet systems to the stands and then ran my code to test each individual one to make sure they can strike the xylophone key as intended. 

Progress

I initially took the wooden frame my partner built and assembled it with the xylophone (shown below).

After attaching my servo-mallet systems to the stands, I placed each of the systems in their proper location. Each of the systems were able to strike a xylophone key at varying dynamics while running my code, so I now have 15 functioning servo-mallet systems prepared for use. 

Problems

For next week, I will need to continue building the final set of servo-mallet systems so that I can test my code with all the keys on the xylophone. 

Plan

I will need to build more servo-mallet systems so that I can test my code with more keys (and therefore play more interesting musical passages).

Procedures

I used the 3D Printer to print additional copies of the stand.

Progress

I was able to print additional copies of the stand for the servo-mallet system using the 3D printer. An image is shown above depicting the additional stands which were printed. 

In addition, I was able to create additional servo-mallet systems so that I will be able to test my robot with passages consisting of more notes. An image is shown above depicting the additional servo-mallet systems created. I achieved my progress report goal which I set to accomplish by the end of January of “creating at least 15 servo-mallet systems.”

Problems

For next week, I will need to attach the servo-mallet systems I have created to the stands I have printed and connect it to the servo driver/Arduino to test if the servo-mallet systems function as intended. 

Plan

I will need to take the musical elements I’ve been able to add to my code and test it on new musical passages. 

Procedures

I used the code I had written last week to hit a xylophone key with different musical expressions and tested it on a new musical passage.  

Progress

I was able to add an additional servo-mallet system to run with my servo driver. This allowed me to experiment with three-note tunes like Mary Had a Little Lamb. 

A video can be found here where my robot is playing Mary Had a Little Lamb with a mix of single notes and chords (two notes at once). The passage is played three times, with the articulation changing from a lot of ringing to a muted sound, the tempo (speed) getting slower, and the dynamics becoming softer.

Problems

I will now need to build more servo-mallet systems so that I can test my code with more keys (and therefore play more interesting musical passages), as well as try using a rubber band to fasten the mallet on to the servo horn instead of a zip tie. 

Plan

I will need to take what I have learned from familiarizing myself with the PCA9685 library and write code to have the servos appropriately strike a xylophone key. 

Procedures

I used the example “Servo” code from the Adafruit Library as a baseline to add upon it and write my own code. I had to find the correct PWM values for my servos to correctly strike the keys and update the code to play a tune at varying dynamics/articulations. 

Progress

After familiarizing myself with the PCA9685 library last week, I was able to write a code to control multiple servos (2 as of now) using a single Arduino pin. After being able to strike a xylophone key with my servo-mallet system, I added improvements to my code to achieve the following:  

• Playing a tune consisting of two notes
• Playing a key with different articulation (lots of ringing vs. stiff/muted)
• Playing a key at different tempo (fast vs. slow)
• Playing a key with different dynamics (loud vs. soft)
• Playing multiple notes at the same time (chords)

A video can be found here where a robot is playing a two-note passage with a mix of single notes and chords (two notes at once). The passage is played four times, with the articulation changing from a lot of ringing to a muted sound, the tempo getting slower, and the dynamics becoming softer.

In terms of my budget, I have used $114.70 so far. The table above shows the details of my purchases I have made for my project as of now. Once I order an additional PCA9685 in the Spring, my total budget should sum up to $129.65.

Problems

I will now need to build more servo-mallet systems so that I can test my code with more keys (and therefore play more interesting musical passages). While testing, I also realized that the zip ties may not be the best option to attach the mallets on the servo, so I will need to try using other methods such as a rubber band.