Project Logs

Collapse

• Tingle Research Paper Published

  Curt White • 05/30/2019 at 20:22 • 1 comment

  The Child Mind Institute Matter Lab has published a paper based on data gathered from the Tingle gesture recognition device (the primary prototype presented in this Hackaday project).

  Check it out:

  https://www.nature.com/articles/s41746-019-0092-2

  Thermal sensors improve wrist-worn position tracking

  Jake J. Son, Jon C. Clucas, Curt White, Anirudh Krishnakumar, Joshua T. Vogelstein, Michael P. Milham & Arno Klein 

  npj Digital Medicine 2, Article number: 15 (2019) 

• Building the Tingle

  Curt White • 10/19/2018 at 19:27 • 0 comments

  1: Unboxing and Dissembling the N68 Fitness Tracker

  This is the fitness tracker used as a foundation for current Tingle prototypes. It can be had for $30-40 from China ( Link1 , Link2 , Link3 , Link4 ) or $63 on Amazon Prime if you are in a hurry. The primary reason I chose the N68 it its cast metal enclosure and magnetic power adapter. The appearance of Tingle prototypes is extremely important. We will be asking parents of vulnerable children to put these devices on their child's wrist. A finished, off the shelf appearance is important to gaining their trust. In addition, these prototypes are handled by decision makers at CMI and potential funders, both of whom appreciate a polished product. The N68 as a color OLED screen, KX126 accelerometer, 8MB GigaDevice FLASH memory IC, vibration motor and simple LED/Photo-receptor heart rate sensor. 

  NOTE: As I'm documenting build steps I will be showing pictures of four devices being fabricated. Building these prototypes one at a time is very tedious. Also, multiple prototypes means you get to see multiple examples of each build step.

  The strap and glass cover are both removed. The cover is attached with glue. I use a hobby knife to cut around the edges of the cover, thereby loosening in.

  A small screw driver is used to pry up the cover.

  Here you can see the cover removed and placed upside down. The bottom section of the cover has a contact for the touch sensor affixed, and the top has an extension of the Bluetooth antenna. There is also a PCB antenna embedded in the circuit board itself so this isn't strictly necessary.

  Here are all four N68s with covers removed. 

  The electronics have been removed from the enclosure. The battery and heart rate (HR) sensor are lightly glued to the bottom of the enclosure but can easily be lifted out with a small prying tool.

  Here you can see the green LED for the HR sensor lit up. The HR sensor is not currently used in the Tingle prototype but this is just a matter of programming, these sensor is readily available for use. During a clinical study we hope to monitor HR during detected compulsive behavior events. The HR sensor might even offer additional filtering of false positive detections since elevated HR is correlated with the anxiety that underpins compulsive behaviors.

  N68 Fitness Tracker Components

  Closeup of N68 PCB ICs. External flash and HR sensor are not currently used in the Tingle but could be implements by simply writing additional code.

  ---

  2: Removing the OLED Display

  The Tingle does not use the N68's display. It uses up power and space which I would rather devote to sensors and sensor data processing. The display is easily removed with a soldering iron by gently pulling on the display as the soldering iron is played across the display contact area.

  With the display removed.

  Display removed from all four prototypes.

  ---

  3: Wiring Up the N68 PCB

  The N68 PCB will have to be connected to the sensor cap, an external programmer and power supply. Conveniently, there are factory test pads which I can solder wires to.

  Fluxing the PCB test pads I will be soldering wires to.

  Pre-soldering the test pads to make soldering easier +...

  Read more »

• Programming the Tingle

  Curt White • 10/19/2018 at 19:06 • 0 comments

  All Tingle related code is stored in this GitHub repository: https://github.com/curtpw/tingle-open

  Firmware, Mobile App, Web Interface and MLX90615 Thermal Sensor Programming

  The Tingle project consists of four applications: 

  1. Firmware running on the Tingle written in Arduino C for the Nordic nRF5x ArduinoCore (see programming instructions in wearable device hacking section of this project and the platform GitHub repository).
  2. Hybrid mobile app written in JavaScript using the Cordova hybrid app framework ( link to Android version on Google Play ).
  3. Web interface written in pure front-end JavaScript. This can run from a simple file server (or GitHub repository as a GitHub page). No server side code necessary ( link to live interface ).
  4. A small Arduino sketch to program the I2C addresses (2A, 2B, 2C, 2D) of the MLX90615 thermopile sensors.

  MLX90615 I2C Address Programming Setup

  This is my setup for programming MLX90615 thermal sensors. Any Arduino compatible microcontroller will work as long as it supports I2C (pretty much all of them do).

  N68 Fitness Tracker PCB Microscope GPIO Diagram

  Note: there are both red and blue versions of the N68 PCB. The only difference between them is the presence of a test pad for CS on the OLED display SPI bus. High resolution images available in the GitHub repository.

• Wearable Device Prototyping Platform

  Curt White • 10/19/2018 at 19:05 • 0 comments

  Open Source

  When I think open hardware I think Arduino and Raspberry Pi. When I think open software I think Linux, Python, JavaScript and countless libraries/modules thereof. These are tools and more importantly, community driven platforms. I use hacked fitness trackers to prototype wearable devices because nothing else works - nothing else is small, inexpensive and polished enough. They have become my go-to wearable device prototyping platform for both work and personal projects. This platform has the potential to be a successful community driven open source project - and the Hackaday Prize competition is where I decided to launch it. 
  

  I have had a lot of help along the way. Especially members of the smartwatch hacking slack group!  Hackaday is a good place to document projects, but GitHub is necessary for a full fledged open source project with multiple contributors. The primary GitHub repository for the "Hacking Wearables for Mental Health and More" Hackaday project is becoming exactly that. I have spent so much time building projects for Hackaday (I decided to build a significant project for each of the 5 challenges!) that I've been a little neglectful of the GitHub repository over the last couple months, but things will get humming again in no time. 

  44 stars and 7 forks is a great start. More of a "fledgling" open source project than a "full fledged" open source project but it is definitely legit, especially considering how niche it is. Personal first. The repositories for various tutorial examples and demonstration projects have gotten some GitHub love as well.

  Platform Demonstration Projects

  I created three demonstration sub-projects specifically to showcase hacked fitness trackers as a wearable device platform and entered them into the Hackaday Prize: Intraoral Respiration Monitor and Overdose Detector, Stomach Acid Powered Smartpill and EEG Headset. Two of those ended up as finalists despite being created as tutorials for this project. My hope is that the positive reaction these projects have garnered will add additional momentum to making the hacked fitness tracker prototyping platform a successful open source project.

  ---

  
  

  Stomach Acid Powered Smartpill

  Heard of using lemons, potatoes or salt water to create a simple battery? Stomach acid (dilute hydrochloric acid) can be used in the exact same way and is significantly more powerful. I use an array of zinc (anode) and copper strips (cathode) immersed in stomach acid (electrolyte) to turn my stomach into a galvanic cell battery. The stomach battery powers a hacked activity tracker small enough to swallow and reconfigured as a body monitoring 'smart pill'. The user's stomach is the smart pill's battery. By incorporating a $25 hacked activity tracker into existing research I am trying to make stomach acid powered smart pills inexpensive and open source. [LINK TO PROJECT]

  (Down camera feed is on an off frame screen so use your imagination when I refer to closeups of the smartpill)

  ---

  
  

  Intraoral Respiration Monitor and Overdose Detector

  A wireless full waveform respiration monitor worn entirely inside user's mouths (intra-oral) that streams data to a Web Bluetooth enabled web application. 

  The device measures the air pressure, humidity and skin temperature inside the user's airway. In short, BME280 air pressure sensor + MLX90615 thermopile thermometer + hacked miniature nRF51822 based activity tracker mounted on an ultra-thin custom dental retainer. 

  My primary goal is to detect opiate overdose. Depressed respiration is the best way to detect overdose and the only approved indicator for the administration of Narcan (Naloxone). Full waveform respiration data is surprisingly difficult to obtain. Spirometers, chest-straps and pulse oximetry (including photoplethysmogram) are relatively inaccurate - particularly when respiration becomes depressed. There are many...

  Read more »

• Adding Bioimpedance (ECG, EEG, EMG) to The Platform with ADS1292 ECG Activity Trackers

  Curt White • 08/27/2018 at 05:27 • 0 comments

  Their are a number of nRF52832 activity trackers which contain the Texas Instruments ADS1292 ECG front-end IC, for example the B20 (links 1 2 3 4 5 6 7 ), the CK12 (links 1 2 3 4 5 ), the G20 (links 1 2 3 4 5 ), and the B9 (links 1 2 3 4 5 ). Although the ADS1292 is marketed for ECG, it is part of a family of chips focused on an array of bioimpedence applications and there is no particular reason it can't be used for simple experimental EEG and EMG projects.

  I have provided an in depth tutorial on hacking the B20 activity tracker and applying FIR (Frequency Impulse Response) and FFT (Fast Fourier Transform) analysis to the devices ADS1292 data in this project: https://hackaday.io/project/160618-hacking-a-35-ecg-nrf52-fitness-tracker-into-eeg.

  I have added an Arduino Core variant file to the project GitHub repository for the B20 as well as code examples for EEG with FIR filters, EEG with FFT and ECG with Protocentral's Processing application.

  B20 ECG Fitness Tracker Specs:

  • Texas Instruments ADS1292 two channel (only one is used) ECG front end 
  • Nordic nRF52832 ARM Cortex M4 SoC/MCU
  • Bosch BMA253 Accelerometer
  • Gigadevice GD25Q16C 16MB Flash Memory IC
  • PPG Heart Rate Sensor
  • 0.95" 64*128 pixel monochrome SD1306 OLED display
  • Vibration motor
  • 80mAh LiPo battery and battery charger

  B20 nRF52832 MCU Pinout

  • P19: Button
  • P3: Vibration Motor
  • P5: SSD1306 OLED Display
  • P6: SSD1306 OLED Display
  • P7: SSD1306 OLED Display
  • P11: GD25Q16C 16MB Gigadevice Flash Memory
  • P12: GD25Q16C 16MB Gigadevice Flash Memory
  • P13: GD25Q16C 16MB Gigadevice Flash Memory
  • P14: GD25Q16C 16MB Gigadevice Flash Memory
  • P15: ADS1292 CS
  • P16: ADS1292 DIN
  • P17: ADS1292 SCK
  • P20: ADS1292 START
  • P22: ADS1292 RESET
  • P23: ADS1292 DOUT
  • P25: HR I2C SDA
  • P26: HR I2C SCL
  • P28: BMA253 Accelerometer CS
  • P29: BMA253 Accelerometer SDI
  • P30: BMA253 Accelerometer SDO
  • P31: BMA253 Accelerometer SCL

• Text Communication over Bluetooth Serial Example

  Curt White • 05/24/2018 at 07:52 • 0 comments

  I have created a simple font for the X9 OLED display and used it to demonstrate text communication over serial Bluetooth using my Android phone. The BLE_CHAT example will display input from the Nordic phone app on the X9 display. If "red", "blue", "green", "yellow", or "purple" is sent, the message will be displayed in the corresponding color and a custom reply will be sent from the X9 activity tracker to the phone app.

  I have also included a simple sketch demonstrating the super simple font.

  All files for examples can be found in the GitHub repository, including required libraries and application hex files which I compiled myself. If you have a hacked X9 you can immediately load my compiled application to quickly check out the demo. I am adding the actual example sketches to the hackaday.io project files as well.

  GitHub repository file directory:

  nRF5x-device-reverse-engineering --> X9-nrf52832-activity-tracker --> Firmware --> examples --> nrf52_X9Project_BLE_CHAT
  nRF5x-device-reverse-engineering --> X9-nrf52832-activity-tracker --> Firmware --> examples --> nrf52_X9Project_ALPHABET

  Nordic UART Bluetooth Serial app: https://play.google.com/store/apps/details?id=com.nordicsemi.nrfUARTv2

• Intraoral Respiration Monitor - A Computer Inside Your Mouth!

  Curt White • 04/23/2018 at 19:06 • 0 comments

  A wireless full waveform respiration monitor worn entirely inside user's mouth (intra-oral) that streams data to a Web Bluetooth enabled web application. We built this by hacking the tiniest ARM based activity tracker we could find! 

  The device measures the air pressure, humidity and skin temperature inside the user's airway. In short, BME280 air pressure sensor + MLX90615 thermopile thermometer + hacked miniature nRF51822 based activity tracker mounted on an ultra-thin custom dental retainer. 

  We also spun this off into a separate Hackaday project with more detail, check it out.

• The "Tingle" Gesture Recognition and Biofeedback Device For Compulsive Behaviors

  Curt White • 04/20/2018 at 03:31 • 0 comments

  The "Tingle" Gesture Recognition and Biofeedback Device is a project I'm working on at the Child Mind Institute. To put it very bluntly, it is designed to help kids stop compulsively tearing out their hair, a disorder called trichotillomania which is surprisingly prevalent. I built the prototype using the X9 activity tracker presented in this project.

  From the MATTER Lab website:

  As you are about to compulsively pull out your hair, bite your nails, or engage in some other body-focused repetitive behavior, you feel a tingle on your wrist. Then a notification is sent to an online dashboard. This information helps you to be mindful of your behavior as part of a behavior modification therapy, and helps your therapist monitor your progress. This is the rationale behind building the Tingle and applying for a patent. We have just run a pilot study and are preparing for a clinical trial.

  Prevalence of body-focused repetitive behaviors (BFRBs)

  Body-focused repetitive behaviors (BFRBs), symptoms of Obsessive Compulsive Disorder (OCD) and other conditions involving compulsions (e.g., Autism Spectrum Disorder) involve compulsively causing physical injury and/or damaging one’s physical appearance. These are among the most poorly understood symptoms; they are often misdiagnosed and undertreated. BFRBs include hitting oneself, biting, pulling out hair, skin picking and cutting, as well less severe but damaging behaviors such as nail biting, thumb sucking, and nose picking (Families & Health). These symptoms affect at least 5% of the population (Families & Health); hair pulling alone affects 1%, or about 3 million people in the US (Diefenbach, Reitman & Williamson 2002). BRFBs are highly comorbid. Studies have shown that as many as 70% of those with one BRFB will have another co-occurring BRFB (Conelea, Frank & Walther, 2017). While often impairing, affecting medical health and/or disfiguring, these symptoms are frequently reported but often not observed in clinical settings. This makes diagnosis, as well as treatment planning and monitoring, exceedingly difficult. To avoid pain and disfigurement, it is imperative to identify a reliable means to automatically identify and monitor BFRBs, especially outside the clinic setting. Clinicians need data on BFRB frequency and timing for the purposes of diagnosis, treatment planning and monitoring while patients need immediate, real-time feedback to make behavioral therapies more effective.

  The “Tingle” device

  To address this previously unmet clinical need, we have created a prototype for a wrist-worn device called the “Tingle” that can monitor and record BFRBs while also providing real-time (haptic) feedback (on the wrist) to the individual with BFRBs when they occur. The Tingle is the subject of U.S. Patent Application #15/816,706 filed January 26, 2018.

  Testing the accuracy of the Tingle device for detecting different simulated BRFBs.

  In a pilot study to establish the accuracy with which the Tingle can detect BFRBs in a controlled setting, we recruited 50 healthy, adult volunteers to wear the Tingle and repeatedly simulate 9 different behaviors (eating, smoking, thumb sucking, nail biting, nose picking, skin picking, and hair pulling from three locations). We just gathered the data and are analyzing the data now!

  Testing the clinical efficacy of feedback via the Tingle in therapy.

  This proof of concept leads to the critical next step of applying the Tingle to the clinical setting in which we will confirm the Tingle’s effectiveness in preparation for an FDA New Device Application and plan for commercialization. This will open the way for broad distribution in clinical practice as a diagnostic tool and to support the evaluation and implementation of pharmacological and behavior therapies for BFRBs.

  Conclusion

  The Tingle is well-positioned to address the tremendous unmet need in the care of individuals with BFRBs...

  Read more »

• Get More GPIO The Easy Way

  Curt White • 04/20/2018 at 00:28 • 0 comments

  If I couldn't add additional sensors to the X9 Activity Tracker it would be useless to me. To be perfectly honest, I don't even particularly like activity trackers. Unlike an Arduino, Teensy, ESP32 or ARM Cortex breakout board, hacked activity trackers don't provide convenient access to additional MCU GPIO for adding sensors etc. There is a simple solution: cannibalize GPIO from existing components.  I always remove the OLED for my own projects. I'd rather interact with devices using a mobile app or web application over Bluetooth. Bluetooth is the whole reason I find Nordic chips so appealing. Take a look at the below example of how I desolder the X9's OLED display to gain additional GPIO for sensors:

  The primary OLED GPIO (P14, P13, P12, P11) are really easy to access because there are nice big test pads to solder underneath the OLED ribbon cable. P15 is a little bit harder because you have to solder the place where the ribbon connected, but still no biggy. But what if you need more GPIO or want to keep the OLED? You can access P29 (heart rate detector photosensor) and P30 (touch sensor) by scraping off the solder mask from the appropriate traces and directly soldering a wire to the trace. You will want to isolate your connection to the MCU from supporting circuitry related to the original use of those GPIO. In order to do this, solder to the trace in a place where there is nothing connected between your solder point and the MCU (nRF52832), then cut the trace above your solder point.

  You can use this annotated, highly detailed composite microscope image of the X9 PCB (high resolution version in project files) and list of component pin connections to figure out traces you might want to solder into:

  • HR_LED_PIN            4
  • HR_DETECTOR       29
  • TOUCH_BUTTON    30
  • VIBRATE_PIN           8
  • BATTERY_PIN          28
  • OLED_CS                 15
  • OLED_RES               14
  • OLED_DC                13
  • OLED_SCL               12
  • OLED_SDA               11
  • OLED_LED_POW     16
  • OLED_IC_POW        17
  • KX126_CS                 24
  • KX126_SDI               19
  • KX126_SDO             20
  • KX126_SCL              18
  • KX126_INT               23
  • FLASH_CS                22
  • FLASH_SDI               19
  • FLASH_SDO             20
  • FLASH_SCL              18

• "Thermo" Position Tracking Wearable Device

  Curt White • 04/19/2018 at 23:29 • 0 comments

  The "Thermo" position tracking device is a controller for VR/AR or any other situation where tracking a user's hand in 3D coordinate space might be useful. The Oculus Rift and HTC Vive virtual reality headsets both have 6DoF (Six degrees of freedom) position tracking controllers. They also depend on a camera or some other external device to achieve 6DoF hand position tracking. At the Child Mind Institute MATTER Lab we applied machine learning techniques to vast amounts of thermal imaging data and leveraged the results using a device worn on the wrist with a fairly sparse sensor configuration. We are attempting to achieve the capabilities of the Oculus and Vive controllers without any external reference point. This device was prototyped using the X9 Pro activity tracker - the exact activity tracker presented in this project.

View all 10 project logs