What's New
1. Mixed Reality FPV Display
2. Track Builder
3. Updated GUI displays

I am super excited to get this update out, because this is the first time I have a proper mixed reality FPV display. If you remember from last time, I had 2 separate feeds for the gameview and the fpv livestream - but it was pretty difficult to fly this way and led to a lot of crashes (just watch the end of the last video lol). So for this update, I spent most of my time trying to unify those two video feeds by overlaying the overlay virtual objects onto the livestream. You should definitely watch the demo video so see the proper effect - I have to say it looks pretty cool in my extremely biased opinion :)

Another big new feature with this update was a new Track Builder tool to make it easy to actually design custom race tracks in your room. The main idea is to use the drone and it's position estimation system to place virtual 3D markers in your space, which you can then use as a reference to place your gates and other virtual objects. This made it much easier to place gates in the environment for me and especially helped me make sure I wasn't placing them in physically unreachable spaces (like partially through a wall/bed/table/etc).

Apart from that, I also added a few more GUI displays just to display some relevant information like battery charge and runtime remaining. These primarily were to help with some of the range anxiety feelings I wrote about in my previous update post.

Challenges

1. Getting the video feed into Unity    
  1. Also fixing the memory leak I introduced in my implementation
2. Overlaying the game objects onto the FPV feed
3. Figuring out how to build interesting race tracks

The first big challenge I faced this time was around getting the video feed into Unity and rendering it. If you remember, this is also where I got stuck last time (see my previous update the the details). The quick tl;dr though is that Unity doesn't support the MJPG video format (which how the video frames were being encoded by my video receiver). So instead of using Unity's WebCamTexture class to get the image data in, I decided to do the simplest thing I could think of which was to decode the video frames outside Unity and then pipe them in over the ROS network. Now, if I was trying to build an optimized low-latency, real-time system this would not be a great choice because I'm potentially introducing a whole slew of non-deterministic buffering delays and such...but for now my goal was just to get something workable. I'm also ignoring the fact that ROS1 doesn't support shared memory unless you're using nodelets (which I'm not), so that means I'm potentially using a ton of unnecessary RAM since every image gets decoded, stored into memory, then copied again into a buffer to be sent over the ROS network (and potentially copied back out when it's received if you're not careful). But, for the sake of getting something working, I decided to take on a large amount of technical debt just to see if it would even work. My implementation was then to setup a very simple video server node that connected to the webcam device, read each frame and sent it over the ROS network in a (compressed) image message. From Unity, I then setup a listener to watch for incoming image messages, and render them on a simple plane texture.

So how bad was this implementation? Well, it actually worked decently well, which surprised me. I was successfully able to access and render the image data in Unity. Additionally, even though I didn't do any actual latency testing this time, but it didn't seem to be introducing a significant amount of lag either. Unfortunately, what I did introduce was a memory leak. It took me a while to notice it, but the game was consuming an increasing amount of memory over time leading to my computer freezing if I was played long enough. Luckily, I was able to track it down to the way I was rendering the images: I was creating a new Unity material every frame, which would never get destroyed and release its memory). It was easy enough to fix by pre-initializing a single material and just updating it each frame instead of creating a new material each time.  So I'm chalking that up to my inexperience with Unity, but hey at least I figured it out. And my computer hasn't frozen since (also memory usage is constant over time now) ;)

Now that I had the image data in Unity, my next challenge was figuring out how to overlay it with the virtual game objects in a way that looked visually realistic. The image data coming into Unity was 2D whereas everything else in the game was 3D (like the drone, the camera, gates, etc...). I wanted the image data to be rendered behind everything else in the environment, so I needed a way to bring everything else into the foreground. I brainstormed a few approaches, several of which required directly manipulating pixel values either in Unity (through it's shader) or passing data out to a ROS node first. These were both pretty high-effort approaches, but after thinking about it (and getting some input from my cousin WIlliam, who's way better at Unity then I am), we realized a simpler approach was to render the image on a flat 2D plane fixed in front of the game camera far enough back that the environment objects would appear in front of the plane. In the real world this would be totally unrealistic (basically the equivalent of walking around with an enormous tv screen rigidly fixed to your head like 30ft away), but this worked pretty well, and this ended up being the implementation I went with. The result is what's shown in the demo video above.

What's Next

So with this update, I finally have a working proof-of-concept demonstrating a mixed reality drone racing game using both Unity and ROS. This is maybe the first really meaningful milestone I've hit so far, but this is just the beginning. There are a lot of directions I can take this, but I think the main things I want to focus on are:
1. Improving the position tracking
2. Improving the flight modes
3. 3D printing safety guards for the drone and camera
4. Open Source

One thing that is quite clear from watching the gameplay footage from demo video is that the virtual objects do not appear completely fixed in place like they should. For example, the gates sometimes move and shift around even when the drone is not moving, or don't always get closer when you move towards them. This is directly due to error in the drone's position estimation, which is computed by the drone's flight controller using measurements from a single lighthouse base station and the onboard IMU. The position estimate is primarily influenced by the base station measurements, but these require the drone to have a direct line of sight to the lighthouse and also to be within range of the lighthouse's laser sweeps. Without all these conditions being met, the drone only has it's IMU readings to rely on which are both noisy and subject to large amounts of drift over time. I have a couple approaches in mind that I'd like to explore for my next update. The simplest of these is to just set up a second base station (which is actually the recommended number of base stations to use). Depending on how well that works, I think a second approach would be use visual odometry to estimate motion from the camera feed and augment the drone's position estimate. Personally, this second approach sounds more fun, but we'll see how much a second base station improves things.

Beyond the position estimation, I also want to add more flight modes. For those of you familiar with FPV drones, right now this drone only operates in attitude/position hold mode meaning that it tries to level itself out in the absence of controller inputs. This is easier for me to not crash as I'm continue development (and also because I'm not a great FPV pilot), but it is a very conservative and limiting flight mode. Most experienced FPV pilots tend to start flying in what's called rate/acro mode quite early and basically every FPV pilot is flying in acro mode during a race or even when freestyling. So I definitely want to support this flight mode, because I want FPV pilots to be comfortable flying this drone too.

A related improvement to the flight modes is to design and 3D print a camera mount and some safety guards for the drone to provide some protection during during crashes. This should be pretty straightforward to do but will give me a lot more peace-of-mind when testing out new flight modes.

I also want to open source my project as well - this has been on my to do list for a while, but a lot of the work has been done in Unity, and I'm not a huge fan of the way Unity works with version control. But I'll figure something out and share it in my next update.
So that's it for now, thanks for reading and definitely let me know if you find any of this interesting or have questions/suggestions!