Dec 2022 - Ok it works but only sorta.  The Raspberry Pi 4 is giving us 8-10frames per second.  That isnt good enough for interaction.  The initial TouchDesigner patch (Cronin4392) works but alas is not good demo on which to build.   I still think the basic idea is good: a mostly open source platform to use rapidly evolving AI (ML/DL) Pose tools with powerful interactive generators (TouchDesigner, Unity, Unreal, etc). So what do we do to continue Pose2Art?

Here are some threads I plan to explore to make it better:

• Quick Overview Video

  Project documentation like this does a LOT better with a short introductory video. I am working on a better physical setup to shoot the video as well as climbing the learning cliff on a video editing tool
  

• Replace rPi4:
  The pi is cool and all but it does not have the processing power to handle the AI (neural net) processing at interactive rates. There are alternatives.  Google Coral? RockPi? PC w/older RTX board? I have an Nvidia Jetson Nano Dev Board.  It benchmarks better than rPi but is being retired by Nvidia in favor of newer faster boards.  However, I have one, and Qengineering has an OS build similar to the Pi one we used already. Code should port fairly easy. Lets give it a try.

• Alternative AI Processing:
  We already have 2 different AI engines running (TensorFlow Lite and MediaPipe).  There are others evolving for Pose Estimation and many other applications. Some track multiple people at the same time. Our modular approach (separate AI vs Rendering) makes it easier to switch, although our OSC messages only allow for one person.

• Interesting Interactive TD patches:
  The TD Samples for Particle Systems and Humanoid model animation are potential candidates. Most related TD examples build on Microsoft Kinect or Azure, although many rely on the Point Cloud depth sensor.  One interesting patch using skeleton data is Uisato's Theremin (YouTube Video,  toe on Patreon post) which uses only Wrist XY points.

• Revise OSC Messages: 
  

  Supporting alternative Pose tools requires some changes to the OSC messages we use.  It would be good to provide for multiple people and 3d data points.  Combining each landmark's xy(z)+confidence into a single message would reduce message traffic and length but perhaps increase parsing required in Rendering App.
  

  eg:  /pose##/landmark## x y z conf (4 float values in one msg)
  

• Source Independent Pose Data:
  

  There is no 'standard' for pose landmarks. Current 3 candidate tools have 3 different counts and anatomical assignments for landmarks.  It Would Be Nice If (IWBNI) our OSC messages and/or parser protected the Rendering app from the differences. An external table (CSV?) could provide a lookup between landmark ID number and landmark Name. An OSC message giving number of landmarks could be used to identify which mapping to use.  I have idea of how to build this in TD.
  

• Streaming Source Video:
  IWBNI the images used to extract Pose were available to Rendering. This would allow compositing with landmarks for (at least) testing/demo purposes. This requires a protocol to send individual images, preferrably one that is language (C++,Python) independent, and available or easily coded for different Rendering Apps.  I have seen examples using both UDP and TCP for sending compressed images (JPG). TCP is likely better given the size of image data.
• Alternative Rendering:

  TouchDesigner is cool, but not the only rendering tool available. Both Unity 3d and Unreal engines can take real time pose data to drive interactive animations.  This step should be done after at least the AltHardware and OSC Message revisions are done.
  

• Project Code/Style

  Clean Code makes for better reuse and/or education. The current C++ and Python capture/AI code are also inconsistent.  I recently learned some better organization styles for TouchDesigner.  Some cleanup should be addressed, perhaps during above revisions.

• Revise Details Section:

  The Details section of this hackaday project needs some cleanup too.

Meanwhile the 2022 winter holiday season is upon us, with other major projects starting in January 2023.  Hopefully I get to a few of the above steps before the new chaos decends upon me.