Some quick researched showed that others have done this, so it's not impossible.

On the WRT54G v2 there are 2 USB ports available on the BCM4712, but they're just not used. The D+ and D- pins for both ports are terminated to ground via 4.7k resistors (marked RH19, RH20, RH21, RH22). These needed to be removed and replaced with 15k pull-down resistors.

There's no 5v source on the WRT54G, so I had to build my own using an LM7805 and a few Capacitors.

Once I was done with that, I had everything needed for a USB port. GND, 5V, D+ and D-

I attached a cheap USB Sound card, and now my WRT has a sound card!