While traveling alone and feeling in danger, one of the most crucial steps to prevent a potential crime is to let your emergency contacts know your whereabouts with a brief explanation of your situation. Although smartphones provide various features regarding location tracking and wireless communication, they might still be not suitable since reaching and utilizing a smartphone may be time-consuming and arduous in a time of crisis for people with mobility impairments. In light of recent developments in machine learning and IoT, there is a surge in devices enhancing smartphone features in the form of automated notifications and additional sensors, e.g. smartwatches and fitness wearables. Thus, in this project, I focused on developing an AIoT assistive device that improves smartphone features to inform emergency contacts automatically and instantly of the user's situation.

Approximately 15% of the population is affected by some kind of disability, and this number is rising exponentially due to aging populations and the spread of chronic diseases, according to WHO reports[2]. In this regard, budget-friendly and accessible AIoT assistive devices should be versatile and provide various features in a broader spectrum, considering people with disabilities and special needs.

After inspecting recent research papers on assistive devices, I noticed there are nearly no wearable appliances focusing on detecting personalized items to execute some predefined functions covertly, such as automated notifications, by utilizing smartphone features. Therefore, I decided to build a user-friendly and accessible assistive device to detect customized keychains (tokens) with object detection and inform emergency contacts of the user's situation automatically.

Since XIAO ESP32S3 Sense is an ultra-small-sized IoT development board providing a built-in OV2640 camera and a microSD card module on its expansion board, I decided to utilize XIAO ESP32S3 in this project. Thanks to the integrated modules on the expansion board, I was able to capture images and save them to the SD card as samples without requiring any additional procedures. Furthermore, XIAO ESP32S3 comes with integrated Wi-Fi/BLE connectivity and 8MB PSRAM. Therefore, I was also able to run my object detection model consecutively. Then, I connected the XIAO round display to XIAO ESP32S3 in order to notify the user of the current saved sample numbers on the SD card and the ongoing operation by showing assigned icons.

Since I wanted to capitalize on smartphone features (e.g., GPS, GPRS, BLE) to build a capable assistive device, I decided to develop an Android application from scratch with the MIT APP Inventor. As the user interface of the assistive device, the Android application can utilize the celluar network connection to transfer data packets to a web application via GPRS, obtain precise location data via GPS, and communicate with XIAO ESP32S3 via BLE so as to get model detection results and transmit commands for data collection.

After developing my Android application, I designed various keychains (tokens) denoting different emergencies and printed them with my 3D printer. Then, I utilized the Android application to transfer commands to XIAO ESP32S3 via BLE to capture images of these customized keychains so as to construct a notable data set.

After completing my data set, I built my object detection model with Edge Impulse to detect customized keychains (tokens) denoting different emergencies. I utilized Edge Impulse FOMO (Faster Objects, More Objects) algorithm to train my model, which is a novel machine learning algorithm that brings object detection to highly constrained devices. Since Edge Impulse is nearly compatible with all microcontrollers and development boards, I have not encountered any issues while uploading and running my model on XIAO ESP32S3. As labels, I utilized the names of emergency situations represented by customized keychains:

• Fine
• Danger
• Assist
• Stolen
• Call

After training...