SonoSight: The AR glass for hearing impaired

For our holographic projection system we went with a beam splitter assembly,  With the DLP microprojector behind us. The prototype had to be confined to this tight space due the space constraints of the beaglebone cape. We have been in talks with supplier who can hook us up with a compact SPI protocol DLP2000 module including the optical assembly. 

But without further ado here is the build log of our custom projection system for the HMD.

(Figure: Manoj wearing the AR HMD setup)

(Figure: HMD in action)

The HMD did work well with the right angle beam splitter assembly (Despite some ghosting issues). The ghosting issues however dont show up when viewed in first person by the user.

(Figure:Side view of the projection setup)

Now over to how we did it, We used some very simple components to pull this off. 

1. Polycorbonate sheet (3mm thick) (15cm*8cm) 

2. DLP2000 projection cape with beaglebone black

3. Double Corrugated cardboard box

Now over to the actual nitty gritty build process, We take the Polycardbonate sheet and we cut it to a 15cm*8cm rectangle using a strong pair of shears. 

We also need to make a hinge mechanism to adjust the viewing operation of the device. For this we use a CD case as a hinge. We bind it to the cardboard cutout using some E7000 adhesive. 

The beaglebone black with the projection unit and LiPo boost circuit is placed on the top of the cardboard box. A convex lens from the google cardboard setup is used to bring the focus near to the viewer.  I'll share a bit about the optics in a later post but for now, Here are some images and demo of our first version.

(Figure: Setup with the holographic projection system.)

Speech recognition is the process of converting an acoustic signal, captured by a microphone to a set of words.  Some speech recognition systems use "training" where

an individual speaker or user reads text or isolated vocabulary into the system.  Voice User Interface are included in Speech Recognition off late in all smartphones with applications like

voice dialing (e.g. "Call home"), domestic appliance control, call routing (e.g. "I would like to make a collect call") , search, simple data entry. With recent efforts in multi layer deep learning the accuracy has gone way up and is now offered as a service.

(Figure: Block diagram of typical speech recognition software)

This application is based on using the Google voice and speech API’s which google provides for free. The voice is recorded using the Microphone. Then using Google speech API’s it is converted to text. The text is compared with previously configured commands in
configuration command file. We also set several control command keywords which would control the interface. If the text matches with any of the commands, thus it performs the specific task. After finding the required commands the bash command for particular task is executed.

To setup speech api, an account is created with the google cloud and a json file is obtained which is an authentication file. To export the authentication file the following command is used . 
export GOOGLE_APPLICATION_CREDENTIALS "pathname"

export GOOGLE_APPLICATION_CREDENTIALS "pathname"

Once installation is done, recognition can be obtained by simple api calls to the cloud. 
--For examples please have a look at this. ----

Since the recognition we need is a real time which is like listening to a audio and simultaneously it should be able to update. There should be some way of getting the inputs from the mic and storing it and sending it to the cloud in a proper way. To do this Pyaudio module is used. 

Augumented Reality is all the rage today thanks to the constant battle between the giants Apple, Google and Microsoft. Despite each of them trying to come up with their own AR glasses along with startups like META there is still no wide market AR glass. While starting this project we looked for affordable compact AR displays that were ready to buy and we found none. So thanks to the work chip manufacturers like TI are putting in, We can make our own AR glass. But before we get into the nitty gritty of AR, Let's discuss AR.

Augmented reality in it's simplest definition actually a mixture of real life and virtual reality, A mixed reality. Augumented Reality is a gateway to making all our cyberpunk wishes a reality by projecting it to our retinas. AR has already found a loyal following from artists, designers and gamers. With a bigger push it is sure to reach a wider audience with day to day consumers. The push is on it's way, The reason why no big name consumer lever AR glass is out is because of the cost involved with the nascent technology. Microsoft Hololens costs 3000$ (Granted it comes with a micro PC in the headset) and the standalone Meta dev kit costs 1500$. The big giants are hard at work with the photonic lightfield optics technologies the AR break has yet to arrive yet. 
(Figure: AR glass projection system)

While low cost options would enter after the high end ones come out, We needed to get a budget AR system out. Somewhat like a godsent, Texas Instruments got out a 99$ DLP pico projector beaglebone cape. 
(Figure: DLP pico projector)

By making a custom optical system, We could channel the DLP2000 projection to Augumented Reality display and possibly even stereo 3D?

I came across this idea after seeing amazing assistive technologies being developed for visual impairment. I was also inspired by the amazing work Dr.David Eagleman did at Stanford on Synesthesia. 


Synesthesia is a unique condition a few people have around the world by simulating  two or more sensory perception and ties one to another. (Eg: Hearing colors, Visual tactile). While I found the whole concept very interesting and appealing, I saw some potential in the way we could treat assistive devices. While people with sensory disability in sight, hearing or smell often find it difficult at start a lot of them begin to rely on their other senses to guide them through daily life. People with visual impairment often rely frequently on sound to assist them in crossing a road or using smell to identify people they regularly meet. A recent research found using Magnetic Resonance Imaging that other sensory processing regions in people afflicted with blindness were more active and showed evidence of remapping (link). I personally found the concept to be beautiful and representing the resilience of the brain. 

When I conversed about it with my friends Vignesh and Bala, We eventually struck on an idea to represent external sensory input in sensory impaired people in a way providing Artificial Synesthesia. The concept while appealing from a neuroscience perspective also had wide implications for people with such impairments. We decided on developing an artificial synesthesia system for the hearing impaired by mixing external sounds and visual stimulus. The reason was that existing efforts which took a similar idea in a different perspective.  David Eagleman's group had designed a vest to solve a similar problem by combining external audio signals with mechanical actuators to give a tactile feedback. 

(Figure from Scott Novich and David Eagleman)

While we did love the concept and the amazing work behind it, We wanted to take a new approach which would combine Augumented Reality based overlay with external sensors. 
The benefits of such an approach are:

1) Ability to provide quasi real time voice to text service around the hearing impaired
2) Lower cost due to the advent of low cost DLP projectors
3) Additional services provided by the AR glass
4) Information overlay would be intuitive and would require limited training.

While there are concerns already about the ethics of cochlear implants in the hearing impairment community, We want to make it clear that we see our device as an addition to amazing deaf community and their intricate sign language. Assistive technologies often walk on a tight rope so as to not take away from the amazing culture the communites have built and we would be willing to hear feedback to this end on how we can adapt in addition with it.