BCI-based assisted living and robot avatar system

Overview

The robot-avatar will have data sent/received using TCP/IP. The data we will be sending/receiving will comprise: commands for motor control (forward, left, right, back, control nerf gun), streaming video & audio from control computer to robot-avatar, and streaming video & audio from robot-avatar to control computer. Text-to-speech will be done by the communication software module on control computer and sent as audio to the robot-avatar.

Outline of main parts required:

• Raspberry Pi Model 3
• Arduino
• H-Bridge Motor Shield for Arduino
• Robot platform with wheels and DC motors
• Raspberry Pi Camera (or another HD camera ->USB with microphone e.g. GoPro)
• Ultrasonic distance sensors
• Stereo Amplifier and speakers
• 24" LED HDMI monitor

Overview schematic for the Robot-Avatar

Motherboard: 970A-DS3P AMD 970A (Socket AM3+) DDR3 [AMD 970 Chipset, x4 DDR3, x2 PCI-E x16, x3 PCI-E x1, x2 PCI, x6 SATA 6GB/s, x1 Gigabit LAN, x2 USB 3.0, x12 USB 2.0] £53

CPU: AMD Piledriver FX-4 Quad Core 4300 3.80GHz £60

RAM: Kingston HyperX Savage Red 16GB (2x8GB) DDR3 PC3-19200C11 2400MHz £78

GFX: MSI Radeon R7 240 "Low Profile" 2048MB GDDR3 PCI-Express Graphics Card £45

HDD Solid State : PM951 256GB M.2 PCI-e 3.0 x 4 NVMe £80

Monitor: G2260VWQ6 22" 1920x1080 TN FREESYNC 1ms Gaming Widescreen £100

Various: £100

Total cost: £516

1. Electrode placement for p300 and SSVEP https://www.researchgate.net/figure/283728220_fig2_Fig-3-Electrode-placement-and-SSVEP-signal-spectrum-a-EEG-electrodes-placed-in

2. Comparative Study of SSVEP- and P300-Based Models for the Telepresence Control of Humanoid Robots https://www.researchgate.net/publication/283728220_Comparative_Study_of_SSVEP-_and_P300-Based_Models_for_the_Telepresence_Control_of_Humanoid_Robots

3. P300 brain computer interface: current challenges and emerging trends http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3398470/

4. A Low-Cost EEG System-Based Hybrid Brain-Computer Interface for Humanoid Robot Navigation and Recognition http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3762758/

5. A Review of P300, SSVEP, and Hybrid P300/SSVEP BrainComputer Interface Systems [pdf] http://cdn.intechopen.com/pdfs/44907/InTech-A_review_of_p300_ssvep_and_hybrid_p300_ssvep_brain_computer_interface_systems.pdf

As it's rather difficult and long-winded to describe the functioning of SSVEP and p300-based BCIs using written language, I've included these videos of other people's projects as examples [skip to fourth to see a robot!]:

In this first video, an SSVEP-based speller is shown. As you can see, each letter flickers at a different frequency..

In this second video, an SSVEP-based cursor is shown. As you can see, each direction choice flickers at a different frequency..

In this third video, a p300-speller is shown..

In this fourth video, somewhat more excitingly, Nao is controlled by SSVEP. The user attends to the flickering lights on the frame which correspond to which of the four commands they desire to send to Nao (e.g. walk forward)

* Communication Module

p300 or SSVEP speller

* Wheelchair Module

Can be a flickering light box for direction (SSVEP) and stop; p300 menu for fave locations (photo of kitchen, front door, lounge, find friend* etc]

*friend can have RFID tag so we can find them in the house easily

* Assistive Living Environment Module

Rooms broadcast to BCIALERAS (brain computer interface assisted living entertainment and robotic avatar system) so we know where user is

-> Adjust lighting / turn on off including lamps (photo menu of lights)

-> Text-to-voice using SSVEP or p300 speller to answer door

-> Adjust temperature either home-wide or local room (e.g. AC / fans)

-> Fave food p300 menus to communicate with friends which food/drink would like easily! These could be sent to smart phone of friend so they can be purchased or brought as required.

* Robotic Avatar Module

When confined to one room and only able to move slowly in wheelchair (if at all when in bed) it could be frustrating to hear noises etc. and wonder what they are. Or perhaps may wish to explore the garden and look at plants, ponds etc. Or perhaps watch TV with friend whilst they are in sitting room and user is in bed. This is accomplished using a robot avatar. Simply a tracked robot with a 1m vertical extension featuring video cameras (including zoomable) and voice synth (for SSVEP/P300 speller to speech) and microphone. It could additionally feature a screen broadcasting video of the user.

- Robot can be controlled directly, or move semi-autonomously to fave locations.

- An automated rescue feature for robot to reach safety in case of getting stuck in corners and frustrating the user.

The robot could feasibly leave the house and be accompanied by friend on visits to shops, parks, galleries, museums car rides. Communicating via 4G to user stuck at home.

* Entertainment Module

Top Menu with links [Reading / Video / Audio / Internet ]

-> Reading -> Control of Kindle App using KDK [turn page/close/select book]

Net-> Web Browsing -> Cursor for Hyperlinks + pop up keyboard for URLs/fields

Net-> Twitter -> + pop up keyboard

Net -> Video -> Control using Amazon Video API and YouTube

Net -> Audio -> local library, iTunes?, streaming services

? keyboard option of p300 speller?

? p300 speller to choose books

Net -> Email control