Optimote | Bio-Adaptive Media Controller

Over the past several evenings, I have developed a standalone web app to replace the VLC media player component of the Optimote project. Check out the video below for a demo.

 Calm mode and thrill mode work well, but I plan to eventually evolve this into a more general purpose tool to adapt media and UI elements on the web browser or across an OS, not just media player apps.   User testing will be helpful to gather feedback in both raw bio-data and user feedback forms in order to assess and add more features the technology.   In my country collecting health data such as the heart rate data requires a waiver to be signed and I will have to compose such a legal form in order to test optimote with additional demographics.

You can see a working online version of the web app at this link: optimote web app

With the physical optimote sensor connected to the web app, you can load video media and program it graphically to react to a user's heart bio data.  There is a button that generates an csv spreadsheet containing the raw biodata, which can be analyzed and graphed to further evaluate and improve optimote.

Testing

The next step was to test out the device on small scale with a couple testers to ensure it is at least usable enough for further testing with many testers. 

After some fine tuning of the logic, namely the sensitivity of the actuation triggers, it seems to work decently.  

Calm mode noticeably puts media volume and tempo in sync with heart rate, with calming boosts when triggered. Thrill mode is especially fun with the use case I made custom media for. The thrill mode media plays an eerie scene looping with the famous Jaws movie piano theme playing in the background, becoming louder and faster as heart rate speeds up. When the user is sensed to be steadily decreasing in heart rate over X readings, it skips to a jump scare clip when they least expect it. Of course, this frightful media would not be suitable for a classroom, but the mode would work just the same for different, non-scary media. Focus mode definitely needs more data collection to improve the logic, but this data can be collected when larger scale testing is carried out.

So far it is a success!

Later, I plan to perform large scale user experience testing to determine just how effective and enjoyable this technology can become with more advanced features.

I am looking forward to the future of this technology and will upload a video or two demonstrating the tech progress thus far.

Prototyping

I first acquired the parts needed.

ITEM	QUANTITY	COST
VLC Desktop (Media Player App)	1	$0.00
Magnetic Breakaway USB Cable	1	$4.00
Casing (from 2 AAA battery holder)	1	$1.00
SPDT Slide Switch	1	$1.00
Optical Pulse Sensor	1	$4.00
Pro Micro Arduino Compatible Board	1	$5.00

After acquiring the components needed, the next step was to prototype the hardware and code that comprises the device.

Every evening over several days, I drafted the code to implement the previous algorithms. I did get stuck at one point and needed debugging assistance, but my friend Timm Carson worked with me for an hour and we fixed all the bugs. The main issue was a reason for nerdly laughter. - In the code, I had called out a non-existent pin on the Microcontroller.

This circuit is simply:

• MCU +5V to Sensor Vin
• MCU Ground to Sensor Ground
• MCU A0 to Sensor Signal
• Slide Switch Pole to D2
• Slide Switch Throw to Ground.

There are a few dependencies at this moment which are listed below.

• Pro Micro hardware is released under Creative Commons Share-alike 4.0 International.
• Pulse Sensor Amped hardware is licensed under the TAPR open source hardware license
• PulseSensorPlayground code library is licensed under the MIT License
• Arduino Environment is licensed under the GNU General Public License
• VLC Media Player is licensed under the GNU General Public License

I soldered together the device and composed a schematic. Shown below are photos of the prototype and schematic.

The footprint of the device itself is 1" x 1.5" x 0.35" so the case needed to have a slot cut for the switch to fit out. I used a dremel to cut the holes needed. It looks much better after being wrapped in tape, which adds the advantage of blocking light from the pulse sensor.

The next task will be testing in small scale till it is working well enough to schedule larger scale user testing.

Ideation

Now that clear needs were found, it was time to brainstorm ideas for media technology innovation to positively improve these points.

The question was: 

What device can be made to control media in tune with or intelligently in leading of each student’s bodily rhythm to improve calmness, focus, and entertainment?

I outlined a few ways an electronic device could read the psychological status of a student using aspects of certain existing sensor tech.

• brain wave sensor
• skin conductivity sensor
• heart beat sensor
• blood oxygen sensor
• breathing sensor
• motion sensor
• optical facial expression sensing
• sweat sensor

I then outlined a few possible ways a device could control a screen media player.

• IR remote control
• Direct computer control using hotkeys and automatic USB HID keystroke injection
• Bluetooth HID

Ultimately, I decided on using an optical heart rate sensor which looks for minuscule skin color fluctuation caused by the heart pumping blood. Since heart rate is innately tied to feelings and brain activity, this is the method I find most fitting.

For playing the media, my go-to way is the open-source VLC media player desktop app (not the limited version on the Windows store) because it has hotkey combos that work well for controls. This will work in conjunction with a board based on the atmega32u4 microcontroller chip because a computer can recognize it as a HID (human-interface-device) without difficult-to-find drivers. 

This is familiar because I have previously used these boards to make an automatic keyboard (USB Rubber Ducky) for my job where I had to bulk set up Chromebooks and needed an automated assistant to work alongside me. I simply plug the board into the computer and it did all the computer configuration, imitating a human typist with 2x to 3x my human speed and zero typing errors which freed me for specialized human work as opposed to these sorts of repetitive, time-constrained, and human-error-prone tasks.

The chip will read heart sensor data, process it through some clever algorithms, and output keystrokes to control the VLC media player app on each computer.

I sketched out a few ideal methods that the device might utilize to induce certain effects onto a user. I came up with three theoretical modes.

Needfinding

I came across a statistic cited by the CDC from the Kaiser Family Foundation while browsing the web: “Kids ages 8-18 now spend on average 7.5 hours per day on screen media." 

Upon reading that this statistic does not even include screen time spent at school, this interested me, and I wanted to dive into how this might affect people positively or negatively, particularly in the education and healthcare industries.

For the needfinding research I observed a group of school students in their environments naturally for 20 minutes daily over the course of a week. One benefit of having my workspace within visual range of a classroom was it became a great way to observe without being intrusive. My goal was to learn about their interactions with screen media through technology and how those interactions affected and related to their behavior.

For capturing notes, I used the field study chatbot which I had developed for conversational, streamlined data entry with spreadsheet data export. After color coding to sort by tags and connections, this became a way to visualize and understand the data.

Shown below is a snippet of some of the data collected through the chatbot during an observation session when students were watching educational videos on computers. Rows have been colored coded by tags to spot similarities and connections.

As an unsurprising but now justified insight, the observations seems to show how the unintentional behavior of few students correlated to sensory overload or boredom snowballed into inhibiting multiple students from learning and keeping engaged.

Distilled based on this data, three clear points of interest are:

• Entertainment – Students either have not enough or too much at different times.
• Focus – Students lose focus when either not entertained enough or are over stimulated by the technology.
• Calm – Some students become over stimulated and need de-escalation or prevention.

Being someone who can empathize with the sensory overload sensation from personal experience, I see a clear goal in developing a technology to treat these screen time problems and improve the screen time experience.