FaceAssure: cloud based pan-tilt security camera

Design Updates

Our team made process in redesigning the camera security system in these key areas:

Image transmission. The camera’s Zentri AMW004 wifi module, is able to receive more image frames per second from the Microcontroller Unit (MCU), achieving up to 3-4 frames per second. This was done by increasing the UART baud rate 8x faster at 921600 Hz rather than 115200 Hz, and reducing the size of the data buffer.

Google Cloud server solution. The cloud server now receives all images and performs person recognition and tracking in real time for a single camera. Once the coordinates and size of the person are determined, a PID controller determines a new angle the pan-tilt base should now adopt and transmits the data via HTTP.

OpenCV facial recognition. The OpenCV library has been exploited to identify faces (frontal and profile), half body and full body to a reasonable degree of accuracy. Once a region of interest is identified, a CSRT tracker follows the person and updates the PID controller accordingly.

PWM servo actuators. The PWM servos now respond accordingly to the angles provided by the HTTP connection; this is achieved by updating the duty cycles for each servo.

Progress made on the project has been acceptable - we are on target to meet the project’s minimum viable product goals, including extending the system to incorporate multiple cameras. however we will not be able to explore our stretch goals of including an LCD screen.

Planning and Organization

Figure: Gantt Chart for the project

Item Desc.	Mfg. Part #	Unit Price	1000 Unit Price	Quantity	In Final Design?	Total Unit Price	Total Bulk Price
Mini Pan-Tilt Kit	1967	$18.95	$15.15	1	Yes	$62.34	$53.29
LCD screen	181	$9.95	$7.96	1	No
Omnivision camera module	2640	$9.44	$7.55	1	Yes
Silicon Labs Wifi Module	AMW004	$27.69	$25.63	1	Yes
Microchip Technology MCU	ATSAM4S8BA-AU	$5.26	$4.45	1	Yes
Custom PCB		$1.00	$0.50	1	Yes

Figure: Bill of Materials for the project

Results of Market Research

Our non-expert interviewees found a product that could recognize specific faces to be very useful in monitoring their private spaces, for example, alerting against front door package thieves or trespassers. On the topic of facial recognition, one interviewee noted that constant face detection alerts could get stale because of false alarms, and suggested using criminal face databases to reduce false alarms.

We interviewed Prof. Jack Tumblin for his expert input on the computer-vision aspects of the security camera that we are making. Most of his suggested surrounded our efforts to reduce latency on the camera, including using grayscale images rather and RGB for transmission to achieve compression. He also suggested using a more direct form of data transmission rather than UART, or bypassing the MCU directly and using a shared bus between the sensor and wifi chip to transmit images as quickly as possible.

In terms of image quality, Prof. Tumblin suggested reducing exposure time to reduce motion blur. To compensate, we would have to increase aperture size (if possible) and ISO settings.

Next Steps

By the end of the quarter, we will fine tune camera system to ensure that the face detection, tracking, and actuation work seamlessly. Amongst some measures, we will increase the UART baud rate to increase throughput, alter the server and CV configurations to improve latency, and tweak the PID controller to improve responsiveness to the detected person.

References

https://www.cnet.com/pictures/security-cameras-with-facial-recognition-tech-inside/

https://docs.opencv.org/3.3.0/d7/d8b/tutorial_py_face_detection.html

https://www.adafruit.com/product/1967

https://www.adafruit.com/product/181

https://usa.banggood.com/3pcs-Mini-OV2640-Camera-Module-CMOS-Image-Sensor-Module-for-Arduino-p-1452968.html

https://www.digikey.com/product-detail/en/silicon-labs/AMW004/1586-1012-ND/5269916

https://www.digikey.com/product-detail/en/microchip-technology/ATSAM4S8BA-AU/ATSAM4S8BA-AU-ND/3593645

Design Updates

Our team made significant process in reconfiguring the camera module into a web client in 4 key areas:

Image transmission. The camera’s Zentri AMW004 wifi module, as controlled by the Microcontroller Unit (MCU), is now capable of transmitting images to the host server on Google Cloud. This is done by sending HTTP POST requests with the appropriate headers for a image/jpeg Content-Type, and the Content-Length matching the size of the image file. At time of report, the Camera system is able to output about one frame of image output every second to the server in this manner.

Google Cloud server solution. The Google Cloud compute instance is the workhorse of the security camera system. Currently, a python Tornado web server accepts HTTP requests from any properly configured device on the internet that sends the appropriate headers. The contents of the request are stored as a jpeg device on the disk and a HTTP response indicates that the transaction was successful. A web interface allows a user to watch the image feed ‘live’ conditions, at a 200ms local refresh rate (the camera continues to be the bottleneck at one frame per second). Where appropriate, the response text will include data for x and y coordinate for the location of a detected face (see section on OpenCV).

PWM servo actuators.  The servos were successfully tested using the breakout boards. We were able to control the servo positions on the pan-tilt base using PWM and altering the value of the PWM duty cycle.

OpenCV facial recognition. The OpenCV library for python has been integrated into the web server. Currently, we are using a Haar Feature-based cascade classifier for face and eye detection. The classifier works reasonably well on the low pixel count of the images, but only when the face is directed squarely at the camera. It is not capable of distinguishing between faces of different individuals either.

The overall state of the project is healthy. So far, we are on target to meet the project’s minimum viable product goals. More time will be needed than expected to integrate the PWM servos into the current solution as this will require the wifi module to interface with the microcontroller seamlessly.

At the current moment, the stretch goals of incorporating and LCD screen and per-user facial recognition may not be feasible given the amount of development time and testing required to complete the features in the current iteration.

Planning and Organization

Figure: Gantt Chart for the project

Item Desc.	Mfg. Part #	Unit Price	1000 Unit Price	Quantity	In Final Design?	Total Unit Price	Total Bulk Price
Mini Pan-Tilt Kit	1967	$18.95	$15.15	1	Yes	$62.34	$53.29
LCD screen	181	$9.95	$7.96	1	No
Omnivision camera module	2640	$9.44	$7.55	1	Yes
Silicon Labs Wifi Module	AMW004	$27.69	$25.63	1	Yes
Microchip Technology MCU	ATSAM4S8BA-AU	$5.26	$4.45	1	Yes
Custom PCB		$1.00	$0.50	1	Yes

Figure: Bill of Materials for the project

Results of Market Research

Our non-expert interviewees found a product that could recognize specific faces to be very useful in monitoring their private spaces, for example, alerting against front door package thieves or trespassers. On the topic of facial recognition, one interviewee noted that constant face detection alerts could get stale because of false alarms, and suggested using criminal face databases to reduce false alarms.

We will be interviewing Prof. Jack Tumblin for his expert input on the computer-vision enabled security camera product that we are making.

Next Steps

By the next presentation, we will have the servos autonomously operating as faces are detected by the camera system and the cloud server. This will require steps to be taken to integrate PWM functionality into the codebase as well as the ability to parse HTTP responses. Should this fail, we will operate the servos either by manually commands from a web interface or on an automated timed loop, panning across the entire room.

We will aim to improve the OpenCV recognition capabilities of the Google Cloud server to recognize faces more reliably, at more extreme angles, and potentially differentiate between recognized (trained) and unrecognized faces. Should the servo be too noisy, we will implement a filter to attempt to reduce noise in the servo movements.

Finally, as a stretch goal we will attempt to increase the frame rate of the camera to about 5 frames per second to increase the usefulness of the closed loop servo control.

References

https://www.cnet.com/pictures/security-cameras-with-facial-recognition-tech-inside/

https://docs.opencv.org/3.3.0/d7/d8b/tutorial_py_face_detection.html

https://www.adafruit.com/product/1967

https://www.adafruit.com/product/181

https://usa.banggood.com/3pcs-Mini-OV2640-Camera-Module-CMOS-Image-Sensor-Module-for-Arduino-p-1452968.html

https://www.digikey.com/product-detail/en/silicon-labs/AMW004/1586-1012-ND/5269916

https://www.digikey.com/product-detail/en/microchip-technology/ATSAM4S8BA-AU/ATSAM4S8BA-AU-ND/3593645

Design Requirements

• Camera is configured as a web client
• Deploy and cloud server to maintain connection with camera modules perform image processing
• Use computer vision techniques to detect and classify faces seen in camera feeds
• Cameras track face movements using mounted servo system
• Alert is sent if the person’s face is not recognized

Design Description

Figure: Block diagram of proposed multi-camera system with central cloud server

Using a hub and spoke model will result in a high latency between sensors and actuators, because a TCP round-trip will be required to process each video frame.

The benefit to this is that the central server is capable of higher image processing loads. We will use a combination of Tornado, a lightweight python web server, and OpenCV, a powerful real-time computer vision library built on python, to our advantage on the cloud compute instance. This will allow the viewing and autonomous control of multiple video feeds..

Planning and Organization

	Action by	Start Date	End Date	Status
Define project goals	Nat	Apr 2, 2019	Apr 9, 2019	Complete
Research components	QK	Apr 2, 2019	Apr 9, 2019	Complete
Prepare order	Nat	Apr 2, 2019	Apr 9, 2019	Complete
Create new PCB layout	QK	Apr 9, 2019	Apr 11, 2019	Active
Setup Amazon AWS services	Nat	Apr 9, 2019	Apr 16, 2019	Complete
Camera client sends images to AWS	Nat	Apr 16, 2019	Apr 23, 2019	Active
Use breakout board to test LCD&Pan-tilt	QK	Apr 17, 2019	Apr 24, 2019	Upcoming
Put together breakout prototype	QK	Apr 18, 2019	Apr 25, 2019	Upcoming
AWS server processes images with CV	Nat	Apr 23, 2019	May 9, 2019	Upcoming
Make revisions and make embedded PCB	QK	Apr 25, 2019	May 2, 2019	Upcoming
Put together embedded prototype	QK	May 9, 2019	May 16, 2019	Upcoming
Make revisions and design revised PCB	QK	May 16, 2019	May 23, 2019	Upcoming
Pan-tilt base reacts to location of human face	Nat	May 9, 2019	May 30, 2019	Upcoming
Put together final prototype	All	May 30, 2019	Jun 12, 2019	Upcoming
Prepare final presentation	All	Jun 4, 2019	Jun 12, 2019	Upcoming

Figure: Gantt Chart for the project

Item Desc.	Mfg. Part #	Unit Price	1000 Unit Price	Quantity
Mini Pan-Tilt Kit	1967	18.95	15.15	1
LCD screen	181	9.95	7.96	1

Figure: Bill of Materials for the project

Splitting of work

Qiankai will largely focus on the hardware aspects of the design and fabrication process, including PCB design, servo motor control and final integration of different modules.

Nathaniel will design the central server functionality, modify the camera to accept wifi connections to the server and stream images, and implement computer vision techniques.

Results of Market Research

Based on home ownership data, about 40 million middle to upper income urban households own their properties globally, and many of these households face the real threat of petty household theft. To that end, they may find value in an autonomous visual security system that deters thieves while providing homeowners of critical alerts and valuable visual evidence of trespassers.

List of competitors

One major brand is SimpliSafe, which provides an array of household security camera products, entry sensors and motion sensors. At a price of US$500 for a complete solution, this may be prohibitive for even that most security-conscious of users. The products also do not include face detection and recognition

NestCam is another serious competitor that features built-in facial recognition for outdoor cameras, and is able to recognize ‘familiar persons’ with in-app alerts. Cloud storage backs up the video feeds as part of the product package. The camera retails for $349, a hefty price tag for a singular camera, or $600 for a pair.

Other brands: Arlo, Allianca, Blink, Nest Cam, Zmodo, Dlink, Netgear, Logitech

Results of Interviews

We conducted interviews with two potential users, but did not manage to interview an expert. Our interviewees found a product that could recognize specific faces to be very useful in monitoring their private spaces, for example, alerting against front door package thieves or trespassers. On the topic of facial recognition, one interviewee noted that constant face detection alerts could get stale because of false alarms, and suggested using criminal face databases to reduce false alarms

Next Steps

Through signal generator, we have found that duty cycle of square wave stands for position, and 1% duty cycle corresponds to 25 degree. Some parameters are included here, period of 20 ms, amplitude of 3.3 Vpp. Moreover, two axises of motors are controlled by two different channels, we will have to generate two relative signal to control the facing direction of pan-tilt kit. By the next presentation, we hope to provide a working demonstration of the servo motors through PWM signal generated by MCU. Our final goal is  to receive facial recognition information from webcam, and move motor to right position as we plan at the same time.

Another aim is to have a fully working camera module that streams images to the server at a reasonable rate, of at least one every two seconds. Should this not work, one contingency is to use lower resolution images or revamp the TCP protocol used to make it more efficient.

Finally, as a stretch goal we will attempt to demonstrate an initial prototype of the facial classifier that generically classifies faces seen on the camera’s image stream.

References

https://www.cnet.com/pictures/security-cameras-with-facial-recognition-tech-inside/