Quick project log to note Spring 2023 developments.  

After a bit delving into Google Coral (Log 6), I went back to the PC side for a bit to get some basic demos running in TouchDsigner (let alone Unity or Unreal).  The Pose2Art GitHub repo now has a bunch of prototypes, and an updated capture-osc gui tool.

• PosePCGui.py  : Tk ui that lets us select videos (looping) or  webcam as input, does capture and forwards OSC and NDI video out. Looped Video is very useful for feeding TD (etc) tools during development. I found several decent dance videos and cropped to short segments.

(Note: somewhere in between Feb and June OpenCV.CaptureVideo stopped working.  In June I got read errors that MSMF  layer could not access the webcam. It worked back in Feb??  simple CameraTest.py created and found need to use DSHOW flag - old direct show api)

A few TD networks have been created to explore options there.  For Example (in github repo):

• handDrawing.toe - move a light-ball on hand-tip
• handDrawing_wNDI.toe - move a light-ball on hand-tip, pass NDI video out
• landmarksAsSpheres.toe - puts a sphere at each landmark (incl face?)
• landmarksAsGeom_wNDI.toe - similar to sphere, also rcvs video via NDI
• handsAsEmitters.toe - particle emitters at hand tips (vid save to file)
• OSC_TubularTrim.toe - make flat/tube skeleton between landmarks
• osc_fluidHand.toe - beginnings of putting a fluid  emitter on hands

More experiments  being done in non-git test subfolder - digging into fluidSim, particles, alternative Skeleton-Pose tools that do 3d and-or crowds etc.  TD network layout has evolved through the development of those prototypes, with possible style developing that might  encapsulate in a tox module.

Also looking into parallel tool to do hand/gesture  recognition and share hand skeleton + Gestures via OSC.  There are a number of good (recent) examples of ML webcam for hands, and how to train your own Gestures layer over basic skeleton.  A Hand Tracking/Gesture camera might be used as a 2nd input, allowing webcam control of an installation. (have a kiosk with webcam above a good ?horizontal? background surface?)

Building the TD prototypes showed some needs for the smart cam side - passing video thru via NDI being big one.  Also created a pose_detector module that encapsulates the  basic flow, with options for different recognition engines - MediaPipe currently implemented,  AlphaPose stubbed (crowd recog?!)

Learned a bit more about ML models and will need to revise that part of the project to reflect the process, including pointers on training custom recognizers - particular to installation.