Got the connections and code done in about 6 hours. Turning off Bluetooth and understanding the wiring of the monoprice were the biggest challenges, especially since it took a while to find the wiring document online.
I had a straight RS232 cable but the monoprice needs a sort-of crossover cable so I'm using jumpers. Could probably hack away at the cable I have.
The comms is really basic 9600 8N1, so it could probably be shrunk a lot with an Arduino or maybe there's something with RS232-to-TTL built in.
There are different versions of the android app, one I used a while ago allows a popup that returns status.I thought an indicator of the current status would be useful, but you can just tap the button for the one you want if you're not sure (i.e. if the screen is black).
Other ideas:
Google Home integration
Have a single button turn the device power on, wait a bit, then switch HDMI to it.
Set a cron job to return it to the Fire TV at 3AM, since that's usually what we want to start with.
Switch to Xbox One when we're getting an incoming Skype call
Integrate with Cortana since the Xbox is sitting there listening anyway
Some audio playback cue when switching "OK, switching to Xbox One" in the Google voice would be neat