There is one line of code that has been a problem from the first day of the PiTrac project.  And although the processing associated with that line of code has improved over time, it’s still a single point of failure that limits the system’s reliability and accuracy.  The line of code is (currently) line 797 in ball_image_proc.cpp:

https://imgur.com/a/SDvgVAy

PiTrac uses this method to find circles in an image and specifically to produce a result set of x,y centers and radii.  HoughCircles() is part of the OpenCV computer vision toolkit.  The method is used in many places in PiTrac.  It is used to identify where the teed-up ball is positioned or to realize that PiTrac needs to wait for a ball to show up.  It is used to identify the strobed ball-in-flight images from Camera 2.  It is used to find the centers and radii of the ball images that will be chosen for spin analysis.  It’s really a workhorse.

Under ideal situations, calling HoughCircles() works pretty well.  And to be clear – there’s nothing wrong with OpenCV’s implementation of the underlying algorithm.  But especially with the overlapping ball images and relatively low resolution of the PiTrac cameras, we’ve struggled to make HoughCircles() work as well it can.  Small distortions of the ball images, such as around the edges of an image, can also create problems.  Even with the calibrated anti-distortion matrices that work to keep the image looking its best, some of the balls end up a little elliptical, and when we relax the circle detection, that can lead to mis-identification and/or errors identifying the exact centers and radii of the circles.  In addition, visual background noise can confuse the algorithm.

There is also a lot of wonky, brittle, hard-to-follow code all around this call.  And there are several currently-disabled remnants of earlier coding attempts that we’ve retained in the code in the hopes of someday getting those earlier ideas to work better.  Which makes the code even more confusing.  There’s also pre-processing such as performing Guassian (or other) blurring of the image (which at first seems like it would make things worse), image erosion, dilation, and so forth.  And there’s literally dozens of super-fussy parameters that help drive all of this.  Small changes in those parameters can break the circle detection process in weird and unpredictable ways.  In addition, some of the adaptive algorithms we've created to help HoughCircles cope with different lighting scenarios can slow the system down as the algorithm repeatedly calls HoughCircles to try to refine the best parameters.

An example of a currently-disabled approach is using HoughCircles to iteratively find the best searched-for-circle-radii (which can be enabled in the golf_sim_config.json file (the kUseBestCircleRefinement parameter, among others).  First, PiTrac calls HoughCircles in a rough manner with a wide range of possible ball radii to figure out what the real range is for the sizes of balls in an image.  And then  PiTrac calls the method a second time with that hopefully-much-smaller range of radii to look for.  

Problems have cropped up in this area so frequently, we even developed a separate “playground” app that allows the user to tweak the touchy parameters that go into HoughCircles in real time and see the results immediately.  That sometimes leads to finding better settings for the system.

We’ve also tried a few alternative circle detection strategies, such as elliptical detectors, arc extraction, and some other recent academic approaches.  But it still kinda sucks.

Anyway, if there’s anyone out there who wanted to take on a pretty complicated improvement assignment, working in the circle detection area of the code would be a great place to dig in.  We’re hoping someone out there could look at the types of images we’re dealing with (we’ve got lots of test images) and figure out a better (and hopefully simpler) way to do this.