PiTrac - The DIY Golf Launch Monitor

There is one line of code that has been a problem from the first day of the PiTrac project.  And although the processing associated with that line of code has improved over time, it’s still a single point of failure that limits the system’s reliability and accuracy.  The line of code is (currently) line 797 in ball_image_proc.cpp:

https://imgur.com/a/SDvgVAy

PiTrac uses this method to find circles in an image and specifically to produce a result set of x,y centers and radii.  HoughCircles() is part of the OpenCV computer vision toolkit.  The method is used in many places in PiTrac.  It is used to identify where the teed-up ball is positioned or to realize that PiTrac needs to wait for a ball to show up.  It is used to identify the strobed ball-in-flight images from Camera 2.  It is used to find the centers and radii of the ball images that will be chosen for spin analysis.  It’s really a workhorse.

Under ideal situations, calling HoughCircles() works pretty well.  And to be clear – there’s nothing wrong with OpenCV’s implementation of the underlying algorithm.  But especially with the overlapping ball images and relatively low resolution of the PiTrac cameras, we’ve struggled to make HoughCircles() work as well it can.  Small distortions of the ball images, such as around the edges of an image, can also create problems.  Even with the calibrated anti-distortion matrices that work to keep the image looking its best, some of the balls end up a little elliptical, and when we relax the circle detection, that can lead to mis-identification and/or errors identifying the exact centers and radii of the circles.  In addition, visual background noise can confuse the algorithm.

There is also a lot of wonky, brittle, hard-to-follow code all around this call.  And there are several currently-disabled remnants of earlier coding attempts that we’ve retained in the code in the hopes of someday getting those earlier ideas to work better.  Which makes the code even more confusing.  There’s also pre-processing such as performing Guassian (or other) blurring of the image (which at first seems like it would make things worse), image erosion, dilation, and so forth.  And there’s literally dozens of super-fussy parameters that help drive all of this.  Small changes in those parameters can break the circle detection process in weird and unpredictable ways.  In addition, some of the adaptive algorithms we've created to help HoughCircles cope with different lighting scenarios can slow the system down as the algorithm repeatedly calls HoughCircles to try to refine the best parameters.

An example of a currently-disabled approach is using HoughCircles to iteratively find the best searched-for-circle-radii (which can be enabled in the golf_sim_config.json file (the kUseBestCircleRefinement parameter, among others).  First, PiTrac calls HoughCircles in a rough manner with a wide range of possible ball radii to figure out what the real range is for the sizes of balls in an image.  And then  PiTrac calls the method a second time with that hopefully-much-smaller range of radii to look for.  

Problems have cropped up in this area so frequently, we even developed a separate “playground” app that allows the user to tweak the touchy parameters that go into HoughCircles in real time and see the results immediately.  That sometimes leads to finding better settings for the system.

We’ve also tried a few alternative circle detection strategies, such as elliptical detectors, arc extraction, and some other recent academic approaches.  But it still kinda sucks.

Anyway, if there’s anyone out there who wanted to take on a pretty complicated improvement assignment, working in the circle detection area of the code would be a great place to dig in.  We’re hoping someone out there could look at the types of images we’re dealing with (we’ve got lots of test images) and figure out a better (and hopefully simpler) way to do this.

So a great question on the Discord server was:

• “Could [the PiTrac cameras] be changed by having the #1 camera point forwards and the #2 camera [which is strobed] straight out? At least more than now so that the LM can be moved back (quite a bit). As it is now it doesn't take much of a miss to hit the LM.”

The following picture from the camera calibration instructions shows why it is relatively easy to hit PiTrac with the golf ball in the default camera configuration – because the Camera 1 looks “back” at the ball, which means PiTrac is further ahead of the ball.  And sometimes in harm’s way…

(Oh – and yes – I’ve hit the prototype PiTrac with the ball lots of times.  The poor thing has long suffered from my poor golf skills).

Summary Answer:

The really short answer is YES – we should be able to change the camera angles so that Camera 1 looks straight out (and down), and then Camera 2 looks to the left (in the picture above), basically down-range.  The angles and relative positions of the cameras in the system are presented as configuration values in the golf_sim_config.json file.  PiTrac is designed to be flexible (especially since it’s experimental).  If you switched the cameras around and calibrated them correctly and pushed the resulting position/angle information into the .json file, the system should still work.  And in this case, you would move the PiTrac back to be more or less even with where the teed-up ball is.  Although I’m sure I could hit some wild shot that will still nail the PiTrac right in the…ball-watching camera.

Even better would be if both cameras could look straight out, as we’ll discuss in a moment.  But we’re not quite there yet in terms of processing speed.

More Detailed Answer:

Let’s look at the current camera setup first.  With the Camera 2 (the strobed camera) facing straight out from the monitor as it is now, the camera is looking at the ball in flight basically perpendicularly (from the side).  In that case, we get images that look like:

In this configuration, the size of the ball is mostly constant, and should be affected only by the Z-axis position of the ball.  And the Z-Axis movement is going to determine the horizontal (side-to-side) launch angle, HLA.  By Z-axis, I mean straight out from the monitor, as suggested by the first picture, above.

But if Camera 2 is facing more down-range, closer to the angle of ball flight, the perspective changes, of course.  In that case, the same shot shown above will look more like this:

Here, the X-axis (left-right as the camera sees it) is more compressed, and the imaged size of the ball decreases with distance, because the ball is getting further from Camera 2 as it flies.  This alternate camera positioning (especially if taken to the extreme) creates a few challenges:

1. Especially at shallow VLA (vertical launch angles), there is more ball overlap because the camera is sort of looking at the ball from behind.  In an extreme case, the strobed ball imprints might all overlap and look like concentric circles as the ball flies downrange. 
2. The increased overlap may make it more difficult to find two good, clear, images to use in the spin-analysis, so spin accuracy could decrease.
3. Because of the compressed X-axis, HLA accuracy may decrease, because there are not as many pixels through which to measure HLA.  And at the (current) low resolution of the Pi GS camera, a single pixel error could mean mis-locating the ball by several millimeters or more.
4. The compressed X-axis also means that calculating the distance the ball moved between strobe pulses will likely be less accurate.  And that will affect the ball speed measurement, and that, of course, is an important metric for a launch monitor.

Positioning the Camera 2 so that it is looking straight out at the ball in flight was an attempt to maximize the accuracy of speed and VLA measurement, as well as the number of good, non-overlapped images for spin analysis.  Although it does so at the expensive of HLA accuracy, because the perceived radius of the ball (which is determined by the Z-axis position) is pretty touchy, and small errors in circle-detection can create larger errors in HLA determination.

However, it’s certainly possible that moving the cameras a little bit in the way @Merrygrin suggests won’t affect accuracy that much, but would still allow the system to be moved a little more from harm’s way.  We’ll have to experiment with that once we get things stable in the builder community, and hopefully some of our contributors will do the same.

A related question is why not just have both cameras point straight out?  This would make it really simple to switch between left- and right-handed golfers as well as allowing the safer positioning of the monitor.  There are a couple current problems with this:

1. Even with 500 FPS (2ms between images), the golf ball could (in a worst case, when it moves right after the last image is taken by the camera) move 20 centimeters downrange.  And with additional processing latency, it could be even further downrange by the time the Pi 1 system calculates the pixel differences between the most-recent frame and the prior frame and opens the Pi 2 camera shutter and starts sending strobe pulses.  By that time, the ball may have already moved out of the field of view.
2. By positioning Camera 2 right where the ball is, it effectively cuts the field of view in half.  The area to the left of the ball (from the camera’s perspective, for right-handed golfers) is wasted, because the ball only goes from the middle of the image to the right. 
3. For the same reason as for #2, above, this configuration effectively reduces the resolution of the camera in half for the pixels that can “see” the ball in flight.

That said, I’m optimistic we’ll get to a point where the cameras can be straight out some day.  And I’ve noticed that even now, if you set the ball too far back (to the right, looking at the PiTrac), the processing is fast enough on the Pi 5 that we’re actually sometimes getting errors at low ball speeds because many of the strobe flashes occur before the ball even gets within sight of the Camera 2.  For example: 

With all that said, we’d love to hear about people’s thoughts, ideas, musings, etc. on this subject!

Quite a few folks expressed an interest in hosting the (hopefully-growing) PiTrac community on Discord.  We'd started a GitHub Discussion forum, but have learned that Discord seems to be the preference.  

So... without further ado -- and without any experience or knowledge on how to run or moderate a Discord server -- here's an invite to use to get into the PiTrac server.  Please don't post anywhere super-public (is that really an issue?) so that we don't have to cull out a ton of bots.  But please join!

https://discord.gg/VfD7xraG  (updated Jan 22)

Now you can compile, run, debug and generally experiment with the majority of the PiTrac source code on a Windows or Mac using Visual Studio.  The Pi-specific (mostly camera-oriented) code automatically #define's itself out of the compile process when working on a non-unix operating system such as Windows.

This allows folks who would like to work on the code (or just walk through parts of it) to do so without making the investment in building a physical Pi-based PiTrac system.  Static images are used to simulate the images that the Pi cameras would otherwise take on the real Pi system  We've found this Visual Studio capability to be invaluable when debugging complicated image-processing problems.  Of course VS will run on the Pi operating system as well, but we  haven't found that development environment to be as productive.

The instructions are here.

Wow - we never expected anywhere near the number of clicks, views, shares and up-votes on Reddit!  Between r/golfsimulator an r/golf, the PiTrac project has received more than 140k views!

Please check out https://github.com/jamespilgrim/PiTrac/discussions.

We just pushed out a very early release of the PiTrac source code!  See  here.

If you'd like to try to build it, please see the Pi Setup document, which includes compilation instructions toward the end, preceded by a lot of prerequisites.  Please send any feedback and issues to pitrac.lm@gmail.com. 

The code drop is not quite ready for prime time, but we didn't want another weekend to go by without giving folks a chance to compile and see the source code.  We hope to have a little more finalized version out in the first half of January.

This version has no TruGolf connection ability, but we expect to include that next week.

Addendum:

If you're interested, you will need to make sure you have the latest "Bookworm" Pi O/S installed on the Pi you're building on.  Once you have the pre-requisites like OpenCV and libcamera and such, the build process should hopefully be a pretty straight-forward meson/ninja build.  Something like:

> mkdir ~/Dev

> cd ~/Dev

> git clone https://github.com/jamespilgrim/PiTrac.git

> export PITRAC_ROOT=/home/<your username>/Dev/PiTrac/Software/LMSourceCode

> cd $PITRAC_ROOT/ImageProcessing

> meson setup build

> ninja -C build

A fair question.  We're working on it!  A couple folks have asked - what's taking so long?  So, we figured we'd provide a couple examples of the stuff we are having to do now.  Which to be fair, probably should've been done a while ago.  But in any case...

PiTrac has interfaces to a couple of golf simulators as well as a multi-threaded socket-based framework to connect to future systems.  Which is great!  But those interfaces also introduced an issue as we're preparing to release the code.  Turns out some of the interfaces use 'secret' codes that are specific to each launch monitor vendor -- which now includes PiTrac.  Those codes have to be protected because of the NDA that allowed us to interface to those systems in the first place.

The decision has been to put the code around the key exchange in a separate object file or link library that will be distributed with the rest of the source code.  So far, so good.  But it turns out, the keys (and some related information) could still be discernable by a determined person by various de-compiling tools and other techniques.  Thus, we need to obfuscate (essentially encrypt) the data strings in the object file that will contain the proprietary information.  Not too hard - there's C++ packages for that, so ok.  But, then it turns out that the obfuscation code doesn't easily work in a multi-threaded environment (long story), and yes--of course--the simulator interface is multi-threaded.  So, we're working on that.  Of course, it will entail additional testing with the third-party golf simulators, which also takes time.

Anyway, that's one example of some of the work going on right now.  :/

Other work has been related to making it easier for the average--ish person to build PiTrac.  For example, we have been working to move to a different, standard version of the Gnu compiler that is currently being packaged with the Raspbian Pi Operating System (Version 12.2.0).  Until recently, we've been relying on C++20 or later versions of the compiler for all our projects, including PiTrac.  But we've recognized that having to compile your own compiler just to get a more modern development platform is not an easy task.  So, we've spent some time porting some our more-recent C++ code and libraries back to the version of the compiler that is currently distributed with the Pi O/S.  Which also involves more testing, of course.

Still, despite all of this, it's still a great project to work on!

PiTrac(*) - What is it?  And What is it Not?

Yes, of course - PiTrac is an open-source golf launch monitor that you can build yourself, that you never need to pay a subscription for, and–if you’re willing–that you can add your own features to.  But, it’s more than that, and we wanted to let folks know what it’s all about.

PiTrac is just a starting point of a DIY launch monitor to jump off from.  It’s a starting line, not a finish line.  We hope it can act as a seed that will grow further innovation.  It’s still early in its development.  We don’t even have left-handed golfer support yet. :/

PiTrac is a fun build journey.  By building one of these systems, you’ll push yourself through 3D printing, soldering, scripting, large(ish) software system build tools, linux utilities, web-based systems, interprocess communications, and maybe even some coding. PiTrac is a project that is sufficiently complicated and uses enough technologies that it can be a great learning platform.  If someone with few tech skills (but enthusiasm and a willingness to learn new things) took on building a PiTrac, they would come out the other end with a pretty good introduction to everything from 3D printing to Linux to building custom hardware.

PiTrac is something you can build without a full stable of equipment.  We’ve tried to design things so that, for example, you can 3D print the parts on the type of small-bed 3D printer you can find at many public libraries nowadays.  There’s only a single custom PCB part that you can generally have fabricated for a couple dollars and no surface mount chips or other things that need specialized equipment.  All of the third-party software relied upon by the system is free.

PiTrac is a photometric-based system.  We think that ball spin is pretty important from a ball-flight physics perspective.  Which is why we built a photometric system, not a radar-based system.  Some radar systems appear to work decently for spin, especially if you add little stickers to the ball, but we thought a photometric system would have the best potential for really accurate spin analysis.

PiTrac is an aggregation of several sub-projects.  Even if you’re not into golf simulation, we hope that at least some parts of the code will be helpful.  Maybe you just need a 3D model for a gimbal mount for a Raspberry Pi camera?  Maybe you’re a photographer into nichey high-speed droplet pictures and just want software that can trigger a flash and a camera shutter?  We hope there’s something here for you.

We also hope that the open-source nature of the system will promote a better understanding of the precision and accuracy of these types of simulated sports systems.  If a few other engineers can get interested enough to pick apart the current system’s shortcomings (there are many!:) and work to publish testing results, folks can get to know exactly how close the simulation is.  And hopefully work to improve it!  That sort of information isn’t really available in detail from current manufacturers.

Finally, we believe PiTrac is a tiny part of a quiet grassroots innovation movement of folks who want to build and control their own technology.  Even complicated tech!  Specifically, it’s a little push-back against a world where a lot of tech is available only from large organizations and where no one knows how the tech they rely on works, let alone how to build it themselves.  A gentle nudge against products that originally cost quite a lot to design and build–and were priced accordingly–but whose high retail prices have not kept up with the progression of technology.  A dream that building your own (possibly clunky) device is more satisfying than just buying that device, especially if it’s otherwise financially out of reach.  And we’re pretty sure that the growing number of hackerspaces, fablabs and makerspaces and other community-based digital fabrication workshops come from a similar thought-space.

What is PiTrac Not ?

It’s not a thing for sale.  We’re not selling anything except perhaps some circuit boards that folks can get themselves from any PCB manufacturer.  Who knows, though – maybe someone will base a business on this technology?

It’s not an outdoor system.  The infrared-strobe based foundation for PiTrac would make it pretty difficult to work outdoors when competing with the sun.  But that’s a tradeoff we were willing to make.

It’s certainly not a commercial-quality product.  All the things you need to do in order to product-ize a technology have yet to be done.  Half the features just baaaaaarely work. ;/  However, we’re hopeful that in time (especially with the help of more people in the open source community), PiTrac can become a stable, reliable easy-to-use launch monitor.

It’s not cheap, even by rich-country standards.  We hope to continually push the price down, but PiTrac still costs hundreds of dollars to build.  And that doesn’t include the price of tools you might need to purchase if you don’t already have them.

It’s not a golf simulator.  Users will likely still need to hook PiTrac up to a simulator like GsPro or E6 (both of whom do a great job with that and at price points that many people can afford).  But who knows?  Maybe an open source simulator is in the works somewhere?

It’s not simple to make or to use.  Making the system currently requires some decent technical chops, including working in Linux programming environments.

(*) Raspberry Pi is a trademark of Raspberry Pi Ltd.  The PiTrac project is not endorsed, sponsored by or associated with Raspberry Pi or Raspberry Pi products or services.

The public GitHub Repo is now up here:  https://github.com/jamespilgrim/PiTrac.   All of the designs for the 3D parts and the custom Connector Board PCB as well as most of the draft documentation are there.  Please consider starring it!

We never did hear back from the GitHub folks about what (if anything) was wrong with the original PiTrac-specific site. 

If you're looking to see what building a PiTrac is going to entail, you can start at the overview document here.  

Although we're still working to get the software finalized and pushed out, preliminary instructions on how to get the Pi computers ready to build that software is here.  

Please let us know any feedback you have, especially any mistakes you see!