It works! Nothing much to the design but some capacitor filtering and a specific resistor value. It takes 5 volts and shows up as a High-speed USB device:

usb 1-3.2.4: new high-speed USB device number 47 using xhci_hcd
usb 1-3.2.4: New USB device found, idVendor=1a40, idProduct=0101
usb 1-3.2.4: New USB device strings: Mfr=0, Product=1, SerialNumber=0
usb 1-3.2.4: Product: USB 2.0 Hub
hub 1-3.2.4:1.0: USB hub found
hub 1-3.2.4:1.0: 4 ports detected

So thats cool. I cheaped out on the decoupling capacitors for the crystal, as other designs I'd seen didn't have them. It worked without them, and they're trivial to add.  I could shrink this further by getting more aggressive with the pad layout, using one of those 2.5 x 2mm crystals, and 0603 passives.