Face tracking based on size alone is pretty bad.  It desperately needs a better recognition part.

There is another face detector using haar cascades.

opencv/samples/python/facedetect.py

These guys used a haar cascade with a dlib correlation function to match the most similar region in 2 frames.

https://www.guidodiepen.nl/2017/02/detecting-and-tracking-a-face-with-python-and-opencv/

These guys similarly went with the largest face.  

It was nowhere close & ran at 5fps instead of 7.8fps.  Obviously the DNN is the latest & greatest.

The Intel Movidius seems to be the only embedded GPU still produced.  Intel bought Movidius in 2016.  As is typical, they released a revised Compute Stick 2 in 2018 & didn't do anything since then but vest in peace.  It's bulky & expensive for what it is.  It takes some doing to port any vision model to it.