Wired and Wireless Versions

There are two versions of for a wired setup, and a bluetooth wireless setup. The wired setup uses a single Raspberry PI zero which collects EEG data and sends MIDI messages over the USB:MIDI converter to the synthesizer. This is problamatic, as there is potential for a lot of noise which can effect the signal from the EEG electrodes. Considering the EEG singal is in the range of a millionths of a voltage, any noise can disrupt the signal and make it unusable. Therefore, a pair of python scripts for a wireless version have been provided. The msc_client.py is run on the raspberry pi which handles the MIDI messages, so can sit on the table away from the user. The msc_server.py is run on the rapsberry PI zero with the Pinaps, being worn by the user.


Hardware

In our setup, we used the following equipment

* Roland AIRA SYSTEM-1 PLUG-OUT Synthesizer - https://www.roland.com/us/products/system-1/
* USB to Midi cable - https://www.amazon.co.uk/Interface-OIBTECH-Upgrade-Professional-Converter/dp/B07FQSFMRZ
* Raspberry pi Zero - https://www.amazon.co.uk/Raspberry-Pi-Zero-Wireless-Essentials/dp/B06XCYGP27
* Pinaps -  https://www.blino.io/product-page/pi-zero-eeg-hat

Additionally, for the wireless version, a Raspberry PI Model 3 B is used manage sending MIDI messages to the synthesizer. Any raspberry PI should be able to take the place of this though, as long as it supports Bluetooth.


Software

This project involves the installation of exisiting python packages and python scripts which run on either the single wired Raspberry PI, or the pair of Raspberry PIs in the wireless setup.

The python software packages involved in this project include the following: EEG Pinaps (http://docs.blino.io), serial port interfacing (https://github.com/mido/mido) and for the wireless versions PyBlueZ (https://github.com/pybluez/pybluez).

These packages are used in the python script which is run on the Raspberry PI zero with the Pinaps. The wireless setup involves two python scripts addtionally using PyBlueZ for bluetooth.
For the full set of instructions on setting up and running the this project as designed. See the guide section.

Python Code

Let's dive into the python code of the wired setup. The wireless setup follows the same design here but breaks the python code into two scripts and involves the management of a bluetooth connection. These details will not be covered here but are available in the source code for inspection: https://github.com/Harri-Renney/Mind_Control_Synth/tree/master/wireless_version

Constants

A number of constants are defined globally as:

# Constants
CTRL_LFO_PITCH = 26
CTRL_LFO_RATE = 29
MIDI_MESSAGE_PERIOD = 1

The first to constants preceded by CTRL are control constants for the Roland low frequency pitch and rate/speed respectively. The values for these indices are found in the Roland control numbers are documented in the SYSTEM-1 Midi document: https://www.roland.com/uk/support/by_product/system-1/owners_manuals/

The MIDI_MESSAGE_PERIOD is the time in seconds between updating the Roland synth with new conrol parameters. Currently set to one second.

Mido

Setting up Mido on the USB port with an initialization control message can be found below:

print(mido.get_output_names())  # Prints list of output ports. If port below doesn't open, see which USB port is available here.
port = mido.open_output('USB Midi:USB Midi MIDI 1 20:0')
msgModulate = mido.Message('control_change', control=CTRL_LFO_PITCH, value=100)
port.send(msgModulate) 

The first line prints out all available output ports. This is not necessary, but useful if you need to list the ports and find the name of the USB output port you require. The next line creates the mido port. Here, the name of the USB port found by printing the output port previously is used. Notice it refers to it with USB in name. The last two lines create a mido message for controlling the low frequency oscillators pitch (CTRL_LFO_PITCH) with a value of 100. This message is then sent over the usb port using port.send(msgModulate). As attention values are read from the Pinaps after, the msgModulate will be re-created with a new 'value'.

Control Parameters

As will be covered later, there are a few variables used to control the change in vibrato in response to the attention values based on a equation of motion. The variables for position and velocity will change in the equation of motion, acceleration will determine the speed at which they will change.

VibratoPos = 0
vibratoVel = 0
vibratoAcc = 2

Blino Pinaps

Getting the calculated attention values from the Pinaps is very straightforward. All we need is a few lines of python code to initialize the Pinaps in a default setup:

#Pinaps setup.
pinapsController = PiNapsController()
pinapsController.defaultInitialise() 

Note, that to operate/initialize in the default mode, the Pinaps will need to have been setup in the default hardware configuration. See http://docs.blino.io/devices_pi/ for more details.

Attention Updates

To parse the latest Pinaps EEG data, the script will need to continue to parse data from the pinaps. To do this, the python code executes the following code:

while True:    
    data = pinapsController.readEEGSensor()    
    aParser.parse(data, parserUpdateVibrato) 

The function "parserUpdateVibrato" is given to the parser. This is a function which reads the attention value and updates the vibrato according to some equations. These will be covered in the next section. This callback is repeatedly called, parsing the latest Pinaps data.

Updating Vibrato Position

Of course, we could just add a constant value to the current vibrato level depending on the attention value read, but that's boring. Let's use a second order differential equation, one of Newton's laws of motion. This involves initial position, initial velocity and acceleration.

"""    
    Equation of motion used to modify virbato.
"""
def positionStep(position, velocity, accel):    
    position = position + velocity * 2 + (1/2) * accel * 4    
    velocity = accel * 2 + velocity 

Using this equation will build up a feeling of momentum in relation to the attention values over time. Don't worry about this too much, it is used to just update the 'position' of vibrato when you give it a 'velocity' and 'accel' values, which you can control in the program.

if(blinoParser.attention > 50):    
    positionStep(VibratoPos, vibratoVel, acc)    
    VibratoPos = 0 if VibratoPos < 0 else VibratoPos
else:    
    positionStep(VibratoPos, vibratoVel, -acc)    
    VibratoPos = 100 if VibratoPos > 100 else VibratoPos

MIDI Messages

Mido works by using the following function to create new MIDI messages as a python object:

msgModulate = mido.Message('control_change', control=CTRL_LFO_PITCH, value=vibratoStrength) 

Here, the first parameter is the 'Message type'. This is used to describe what kind of message we are sending. In the case of our program, the goal is to use the Pinaps calculated attention values to control the modulation of pitch (Vibrato). Therefore, set the channel variable to be used in the message to CTRL_LFO_PITCH. We define CTRL_LFO_PITCH as 26 previously.

The final variable set is the value to set the selected control parameter to. In this case, we set it to the vibratoStrength variable, which is derived from the changing attention values.

Now the new message has been set, with an update value for the control parameter for the LFO pitch, the message can be sent across the port using:

port.send(msgModulate) 

Timing MIDI Messages

It is redundant and poor practice to send as many MIDI messages as the raspberry Pi can whilst also parsing EEG data. Most of the time, the raspberry pi will have read and parsed all the latest data available from the Pinaps, and therefore will spend many cycles in the while loop sending the same MIDI messages. To avoid this, an interval period is setup to send MIDI messages in a controlled manner.

time.sleep(MIDI_MESSAGE_PERIOD)

The attention values typically update every second. Therefore, setting this interval period to one second should send updated vibratoStrength messages as the attention values have changed.

Overall Program

https://github.com/Harri-Renney/Mind_Control_Synth.py

import time
import mido

from pinaps.piNapsController import PiNapsController
from NeuroParser import NeuroParser

"""
	Equation of motion used to modify virbato.
"""
def positionStep(pos, vel, acc):
    return pos + vel * 2 + (1/2) * acc * 4

def velocityStep(vel, acc):
    return acc * 2 + vel

CTRL_LFO_PITCH = 26
CTRL_LFO_RATE = 29

MIDI_MESSAGE_PERIOD = 1

vibratoPos = 0
vibratoVel = 0
vibratoAcc = 4
def parserUpdateVibrato(packet):
    global vibratoPos
    global vibratoVel
    global vibratoAcc
    if(packet.code == NeuroParser.DataPacket.kPoorQuality):
        print("Poor quality: " + str(packet.poorQuality))
    if(packet.code == NeuroParser.DataPacket.kAttention):
        print("Attention: " + str(packet.attention))

        ##Change in vibratoStrength depending on meditation values##
        ##@ToDo - Change to include more momentum build up etc##
        if(packet.attention > 50):
            vibratoPos = positionStep(vibratoPos, vibratoVel, vibratoAcc)
            vibratoVel = velocityStep(vibratoVel, vibratoAcc)
            vibratoPos = 100 if vibratoPos > 100 else vibratoPos
            vibratoPos = 0 if vibratoPos < 0 else vibratoPos
        else:
            vibratoPos = positionStep(vibratoPos, vibratoVel, -vibratoAcc)
            vibratoVel = velocityStep(vibratoVel, -vibratoAcc)
            vibratoPos = 100 if vibratoPos > 100 else vibratoPos
            vibratoPos = 0 if vibratoPos < 0 else vibratoPos
        
def main():
    #Init USB:MIDI interface.
    #print(mido.get_output_names())                                                     #Used to originally find correct serial port.
    port = mido.open_output('USB Midi:USB Midi MIDI 1 20:0')
    msgModulate = mido.Message('control_change', control=CTRL_LFO_PITCH, value=100)
    port.send(msgModulate)

    #Init Pinaps.
    pinapsController = PiNapsController()
    pinapsController.defaultInitialise()
    pinapsController.deactivateAllLEDs()

    aParser = NeuroParser()

    #Parse all available Pinaps EEG data. Calculate vibrato value and send as MIDI message.
    while True:
        data = pinapsController.readEEGSensor()
        aParser.parse(data, parserUpdateVibrato)

        print("Message vibrato strength: ", vibratoPos)
        msgModulate = mido.Message('control_change', control=CTRL_LFO_RATE, value=vibratoPos)
        port.send(msgModulate)

        #Sleep for defined message period.
        time.sleep(MIDI_MESSAGE_PERIOD)

if __name__ == '__main__':
    main()

Challenges

Pretty straightforward! If you're still itching to keep going, here are some things you can investigate.