3D Haptic Vest for Visually Impaired and Gamers

The code to run the motors from the Parallax Propeller is here. This code is for the Adafruit 24-Channel 12-bit PWM LED Driver - SPI Interface (TLC5947). The following is an explanation of how it works. To run the LED drivers first send out a 0 signal to the clock and the latch. Then send out 12 bits of information to the driver and shift them in by sending a pulse to the clock. Repeat that process 24 times (one for each LED or Motor). When all the data is in, the program sends a pulse to the latch which starts the PWM. Then the program constantly repeats this PWM process.

Note: There is a big difference for shifting in information between the Adafruit 24-Channel 12-bit PWM LED Driver - SPI Interface (TLC5947) which I got to work and the Adafruit 12-Channel 16-bit PWM LED Driver - SPI Interface (TLC59711) which did not work for me.

The difference is that the TLC5947 http://www.adafruit.com/datasheets/tlc5947.pdf has an input for the latch that the microcontroller can physically be attached to unlike the TLC59711 http://www.adafruit.com/datasheets/tlc59711.pdf. The TLC59711’s latch is controlled through sending a specific code then shifting in data for LED groups and individual pins for PWM of the 12 output pins. The speed for sending in this information exceeds the speed of Python running on the Raspberry Pi at 700 MHz (the default speed). If I used C or C++ on the Raspberry Pi or SPIN on the Parallax Propeller to send out the information I believe it would have worked properly.

To run FFT on the Raspberry Pi I downloaded and modified the FFT Program. This allowed me to run real-time FFT. In that code they put down 2^11 as the minimum chunk (a piece of audio) size over the regular 44,100 Hz audio setting. Originally I used 2^13 for the real-time FFT because it worked on the Pi. In order to speed up the process so that the motors would react instantaneously to the noise I lowered the setting from 2^11 to 2^8 with a rate of 14,400 Hz. In order to lower the rate of audio sampling for playing the Call of Duty game I used a USB microphone instead of using a male-male adapter for the microphone slot in the USB sound card. The other one of the male-to-male adapters connects to the audio output of the Xbox.

Lowering the Hz to 14,400 and the chunk size to 2^8 enables the Pi to handle the FFT computations which intern increases the speed proportionally from 5 times to 56 times each second. I was shocked that I was able to get this far with the processing power of the Raspberry Pi. In the process of making the program faster I send out 4 control bytes and a 2-character return byte for each FFT iteration to control the vest. This program takes up about 60% of the computational power of the Raspberry pi excluding the GPU  while still running the python program with c libraries. I was also shocked when I found that out. So in the future I will be making a better pattern recognition program and will hopefully remove the parallax propeller from the equation to have only one main microcontroller running the whole process. I also wrote code for running the vest at different frequencies so that anyone can use it for different games by changing the settings of the code a little. The Raspberry Pi required extra libraries PyAudio, PySerial, NumPy to run the code. This gamer vest is so fun! 

I researched Fast Fournier Transforms (FFT) on the Raspberry Pi and decided to implement it in this vest for gamers. I tested the vest using the Call of Duty game. The wearer can now feel the recoil when they shoot their gun and other bass sounds like heart thumps from low health, enemy’s loud shots, grenades and rocket launchers. It’s really fun! The vest can be used in other games and also for watching movies. It would be fantastic if the motor array was embedded into the back of movie theater seats to add another dimension to the movie viewing experience. Kind of like the Mickey's PhilharMagic® show/movie at Disney World where you get to smell a scent or feel something run across the room by having your ankles tickled with air.

FFT enables the Pi to make spectrograms of the audio from any sound. A pattern recognition program is used to determine when there are audio cues. Then a signal is sent through a USB cable to the Parallax Propeller which then sends information to the Adafruit 24 PWM LED drivers which turns on the motors in the vest.

The vest uses LED Drivers to control 48 vibration motors. I researched and couldn't find a good solution so I originally used shift registers to do the pulse width modulation. But then I thought of LED drivers and they worked much better. Coincidentally both of them have similar properties.

I will publish the code soon.

9/28/14 Update:  The code can be found here.

I wanted to use the Raspberry Pi in this project because it is a miniature computer that could replace the big laptop I use to control the vest. Originally, because I wanted to make another box, I made a Python program to control Adafruit 12 PWM port LED Drivers but Python on the Raspberry Pi is too slow to run them. So for my second attempt to make the Raspberry Pi (that I purchased from Element14) control the vest I sent the startup and controller commands to the Parallax Propeller and let it do the hard work to control the Adafruit 24 PWM LED drivers in the original box.

Currently my Raspberry Pi can control the vest and pretty soon I will have code written for it for gaming. I have also 3d-printed a case for the Pi to prevent me from frying it when I handled it. 

9/28/14 Update:  See the journal entry "Gamer Vest - Feel like you are in the game!" and code and video updates for more on the gamer vest.

To my knowledge a vest like this has not been done for the visually impaired. My vest has 48 vibration motors which is a whole lot more compared to other gaming vests.

I also use an innovative way of suppressing the transfer of vibrations from motor to motor. I use a neoprene vest with neoprene pads on which the motors are mounted. This reduces the vibration transfer between motors and also reduces the transfer of vibrations between the wires and the motors.

Fast Fournier Transforms (FFT) on the Raspberry Pi:  I used Python to make calls to multiple libraries (such as NumPy, PyAudio, PySerial) to implement FFT 56 times per second at sample rates of 14,400 Hz from the audio stream.  This was unusual since other people using the Raspberry Pi without the GPU could only get the FFT to 5 times per second.  But what was extra special was the process only took about 60% of the computing power. This gave me really good response time with no noticeable lag.  Game playing wearing the vest is awesome!  

I added a button to the vest so that the visually impaired person can send an email to someone who can help them in case they have fallen or need help. The email address and message are preconfigured. It uses Wi-Fi to make the connection.

9/28/14 Update:  The additional button on the vest, when pushed, now sends an email to a remote assistant that includes a picture taken by the Kinect. This way the assistant can call the wearer and tell them what is in the picture.

I demonstrated my vest for Dr. Joe Fontenot of the Community Services for Vision Rehabilitation (CSVR) in Mobile, AL. on September 2. (See http://csvrlowvision.org/ for more information.) Dr. Fontenot, who is legally blind, also tried on the vest. He gave some positive feedback. We also discussed the challenges that visually impaired people face in everyday life. He was impressed by the vest and was not aware of anything like it in the marketplace. He liked the idea of using it to communicate to a remote assistant. If I can make the vest send emails with pictures then a remote assistant could call and help identify items in the surrounding environment. The remote assistant email could be configured to be that of a relative or a friend.

9/28/14 Update:  I was able to get this to work. See the Connectivity log entry.

To test this project I got a dozen volunteers to use the vest while blindfolded and some of them listened to loud rock music.  They did not have any direct collision when put into a maze of obstacles.  They all took about a minute or less each to learn how to use the vest.  So it is very intuitive.  What is directly in front of the sensor is displayed on the belly and the peripherals are displayed on the left and right portions of the back.  High objects can be determined from low objects because of the 4 rows of motors.  Higher objects are on the higher rows and lower objects are on lower rows.  Soon I will try the vest on legally blind people.

If anyone is searching for a cheaper parallax propeller board.  When I last checked,  I think I saw that Adafruit sells them for less than parallax does(lol).

This project is not limited to only using the propeller board, others can most definitely use Arduino with Adafruits open source led driver code.

The vibration motors should be bought in bulk so they cost less.  I bought mine off of ebay.

The neoprene sports vest was also bought from ebay.