There are multiple ways to detect eyes in a video. This is a relatively new problem, so the standard technique has not been established. I have researched different methods, all of which have their own pros and cons. I hope to give a brief synopsis of each technique below.

Detecting Eye Blinks with Facial Landmarks

Eye blinks can be detected by referencing significant facial landmarks. Many software libraries can plot significant facial features within a given region of interest. Python’s dlib library uses Kazemi and Sullivan’s One Millisecond Face Alignment with an Ensemble of Regression Trees[3] to implement this feature.

The program uses a facial training set to understand where certain points exist on facial structures. The program then plots the same points on region of interests in other images, if they exists. The program uses priors to estimate the probable distance between keypoints [1].

The library outputs a 68 point plot on a given input image.

[1, Figure 1: Dlib Facial Landmark Plot]

For eye blinks we need to pay attention to points 37-46, the points that describe the eyes.

In Real Time Eye Blinking Using Facial Landmarks[2], Soukupová and Čech derive an equation that represents the Eye Aspect Ratio. The Eye Aspect Ratio is an estimate of the eye opening state.

Based on Figure 2, the eye aspect ratio can be defined by the below equation.

[2, Figure 2: Eye Facial Landmarks]

[2, Figure 3: Eye Aspect Ratio Equation]

“The Eye Aspect Ratio is a constant value when the eye is open, but rapidly falls to 0 when the eye is closed.” [1] Figure 4 show a person’s Eye Aspect Ratio over time. The person’s eyeblinks are obvious.

[2, Figure 4: Eye Aspect Ratio vs Time]

A program can determine if a person’s eyes are closed if the Eye Aspect Ratio falls below a certain threshold.

Clmtrackr is another facial landmark plotter. In previous logs, I had attempted extracting accurate and consistent eye position data from the plot. Because of this experience, I am eager to explore completely different blink detection algorithms and techniques.

Detecting Eye Blinks with Frame Differencing

Frame differencing is another blink detection technique. Essentially, a program compare subsequent video frames to determine if there was any movement in a select eye region.

In most programs, the first step is to detect detect any faces in a videofeed using a face detector, such as the Viola-Jones Face Detector. The face detector returns a bounding box of an area that contains a face, as seen in Figure 5.

[Figure 5: Face Detector Placing Bounding Box Over Face]

The program then analyses this region of interest for eyes using a similar detection tools. The program places a bounding box on any regions of interest.

[Figure 6: Eye Detector Placing Bounding Boxes Over Eyes]

The program then compares the difference the eye region of interests in subsequent frames.[5] Any pixels that are different can plotted  on a separate image. Figure 7 demonstrates a program using frame differencing to detect hand movement. A Binary Threshold and Gaussian Blur filter the images.

[Figure 7: Frame Differencing Program Detecting Hand Movement]

Detecting Eye Blinks with Pupil Detection

Using pupils to detect eye blinks has a similar process to frame differencing. Once again, the program starts off by placing a bounding box on any detected faces within a region of interest. The program then detects general eye regions within the face bounding box.

The program analyses the eye region of interest. The program runs a circle hough transform on this area. If the software can detect a circle roughly in the center of the region and that circle takes up roughly the right amount of area,  the software assumes that the circle is the pupil. If the pupil is detected, then the software can assume that the eye is open[4].

[Figure 8: Circle Hough Transform Detecting Pupil]

[Figure 9: Circle Hough Transform Detecting Pupil on User With No Glasses]

References

[1] A Rosebrock. (2017, Apr. 3). Facial landmarks with dlib, OpenCV, and Python [Online]. Available: https://www.pyimagesearch.com/2017/04/03/facial-landmarks-dlib-opencv-python/

[2] T. Soukupova and J. Cech. (2016, Feb. 3) Real-Time Eye Blink Detection using Facial Landmarks. Center for Machine Perception, Department of Cybernetics Faculty of Electrical Engineering, Czech Technical University in Prague. Prague, Czech Republic. [Electronic]. Availible: https://vision.fe.uni-lj.si/cvww2016/proceedings/papers/05.pdf

[3] V. Kazemi and J. Sullivan. (2014) One Millisecond Face Alignment with an Ensemble of Regression Trees. Royal Institute of Technology Computer Vision and Active Perception Lab. Stockholm, Sweden. [Electronic]. Availible: https://pdfs.semanticscholar.org/d78b/6a5b0dcaa81b1faea5fb0000045a62513567.pdf

[4] K. Toennies, F. Behrens, M. Aurnhammer. (2002, Dec. 10). Feasibility of Hough-Transform-based Iris Localisation for Real-Time-Application. Dept. of Computer Science, Otto-von-Guericke Universität. Magdeburg, Germany. [Electronic]. Available: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.69.3667&rep=rep1&type=pdf

[5] B. Raghavan. (2015). Real Time Blink Detection For Burst Capture. Stanford University. Standford, CA. [Electronic]. Available: https://web.stanford.edu/class/cs231m/projects/final-report-raghavan.pdf