Hacked together a quick and dirty prototype using PocketSphinx and alsa.  You can see a quick video of it here (it goes without saying there will be profanity in the videos I post): 

For a first effort I'm pretty happy.  There are a ton of options for tuning & training the speech recognition so hopefully I can increase the accuracy.

While putting together the prototype I hit a few issues and dead ends, like the PS3 eye cam mic not playing well with pulseaudio, alsa's python bindings not working at all for some reason, and gstreamer looking way too complex to try using.  In the end I'm going to keep it simple and just use C++ with alsa and PocketSphinx.  Hoping to clean up the code into something presentable, put it on github, and keep iterating on it.