Portable Sound Visualization AR Device

I have verified the operation of the microcontroller and microphone array and have produced an enclosure.

The enclosure has an additional AAA battery case for drive and a toggle switch for power.

I made only one board to connect the wires and secure the microcontroller board.

In the previous tests, we were able to obtain the sound image. Next, we will capture a camera image and try to combine it with the sound image, and output it to the display.

When combining the sound image and the camera image, it is important to align the field of view (FoV) of both images and bring the shooting axes as close together as possible. Ideally, the camera should be positioned at the center of the microphone array. 

However, in this case, we will place the two sensors as close together as possible and make adjustments. Since the exact center of the field of view may be slightly offset, it is recommended to tilt the placement slightly so that the center of the field of view intersects at the desired distance from the target.

Here, the camera and microphone array were tested on a polycarbonate board to which the camera and microphone array were fixed for testing.

When you combine the images obtained from the camera and the microphone array, it will look like the following. The sound image is enlarged to a suitable size and then cropped to align with the camera image. The code can be written very simply using MicroPython's OpenMV.

TV audio visualization example.

You can also visualize the music playing from the audio speakers.

Now, let's get back to the microphone array test. First, just for testing purposes, we'll connect the MaixBit and the microphone array using a breadboard and jumper wires.    

I wrote some test code that converts the sound information obtained by the microphone array into a heatmap of the sound source location and displays it on the display.

You can confirm that the location of the heatmap is moving according to the location of the chime.

It's Friday night, so I'll make some progress. I wanted to write the latest version of the firmware, but since I'm using a new PC, the USB serial communication device driver was not installed. Since MaixBit uses CH552, I need to install the driver on my PC following the instructions on this page.

https://wiki.sipeed.com/soft/maixpy/en/get_started/install_driver/bit.html

The method for rewriting the firmware is explained on this page.

https://wiki.sipeed.com/soft/maixpy/en/get_started/upgrade_maixpy_firmware.html

Also, while it is not directly related to this project, following the tutorial will allow you to easily try face detection using yolo with edge processing.

https://github.com/knorth55/maixbit_tutorials

If you have not experienced a demonstration of Edge AI using microcontrollers before, this is a great opportunity to try it out. I highly recommend giving it a go.

For the development environment of this device, we used MaixPy, which is a MicroPython-based platform. Before starting development, we checked the documentation page.

https://wiki.sipeed.com/soft/maixpy/en/index.html

The basic usage is similar to MicroPython, but it includes APIs specific to the K210 chip, such as FPIOA, KPU, and FFT. KPU is a very useful API for performing operations related to edge AI, while FFT is used for performing FFT calculations. Additionally, since OpenMV can be used, it is very easy to perform image processing. We also used OpenMV for this project.

Furthermore, APIs for controlling sensor modules developed by Sipeed  are also provided.

Before starting the program, let's do a little research on the hardware.

Official information about Maix Bit can be found here.

https://wiki.sipeed.com/soft/maixpy/en/develop_kit_board/maix_bit.html

https://dl.sipeed.com/shareURL/MAIX/HDK/Sipeed-Maix-Bit

The main specifications are as shown in the table.

GPIOs are arranged as follows.

A relatively large number of GPIOs are available, and communication protocols such as SPI, I2C, and UART can be easily implemented.
This is good news as wearable devices that convey sound direction through vibrations and other tactile feedback are also planned for the future.
Circuit diagrams and 3D data are also available, so development can be carried out quickly.

MaixBit can be developed using MaixPy, which is a MicroPython-based language. 

Maix IDE

The development environment is MaixIDE, which is used to connect to the PC via USBSerial. 

https://dl.sipeed.com/shareURL/MAIX/MaixPy/ide/v0.2.5

Firmware

The firmware is available on this website, and "maixpy_v0.6.2_84_g8fcd84a58_minimum.bin" version is used to avoid wasting RAM. 

https://dl.sipeed.com/shareURL/MAIX/MaixPy/release/master/maixpy_v0.6.2_84_g8fcd84a58

Kflash-gui

Kflash-gui was used to write the firmware. 

https://github.com/sipeed/kflash_gui/tree/master

Code created with MaixPy is saved as main.py on a microSD card and loaded automatically at startup. Alternatively, it can be saved and executed in Flash memory. 

uPyLoader

uPyLoader was used for reading and writing to Flash memory. 

https://github.com/BetaRavener/uPyLoader/releases

These tools are very useful when developing K210-based boards, so it is recommended to prepare them in advance.

We will connect MaixBit and the microphone array. The correspondence between the pins to be connected and the GPIO is shown in the following table.

We connected them with a jumper cable, and it looks like the picture below.

The microcontroller used in this project is also made by Sipeed. It is called the Maix Bit and was released a few years ago. It features a high-performance SoC called Kendryte K210, which can be used for edge computing.

AI devices evolve quickly, and boards from a few years ago may seem outdated, but I think the K210 is still a very capable device with plenty of potential for use. 

There are several types of development boards that use the K210, including those with many GPIO pins, smaller sizes, and those with an integrated enclosure, among others. The Maix Bit is a well-balanced type that has many GPIO pins and a compact size.

In addition to GPIO pins, the board is equipped with connectors for a camera and an LCD display. These camera and display will also be used in the device we are building this time. The display is a 320×240 pixel LCD, and the camera is an OV2640 type that uses an M12 lens mount, allowing the lens to be changed if necessary.
Although we will not be using it this time, the board also has one MEMS microphone.

When testing MaixBit, it is convenient to use a 400-hole breadboard as shown in the photo.

The microphone array used in this project is the 6+1 microphone array from Sipeed. It has a total of 7 MEMS microphones mounted at the vertices and center of a hexagon, with 3 microphones arranged along each of the 3 axes.

There are 10 pins on the back of the array for data transmission and reception.
In addition, there are 12 NeoPixel LEDs mounted around the perimeter of the PCB, which can be used to display the direction of the sound source using only the PCB.