Finally finished the assembly and managed to cram everything into the case.

I had to do a few additional mods to try to mitigate some of the noise I was getting from the amplifier. 

After looking up the PAM 8403 ic I found out that there is a mute pin (active low), so after cutting the trace, soldering a wire and connecting it to a gpio pin, I have mute control on the pi.

I also threw together a quick stereo ground loop isolator with a couple small 1:1 transformers, some protoboard, and some audio jacks from an old sound card.

This helps quite a bit with noise, but the amplifier still seems to pick up some emi interference from the Pi, (you can hear when your ssh'd in and sending keystrokes)..

I also picked up a dedicated microphone amplifier, its much smaller then the original one I built and much more sensitive, so now you can pic up peoples voices from the other room.

Other then that everything seems to be running good.