Some things that cost me time and might save you some:

• A ~40 s wait for an IP address wasn't the firmware. Link came up at 3 s, DHCP answered at 39.5 s. The UniFi switch port wasn't in Edge mode, so STP held it for ~30 s. Edge mode: 5-6 s.
• Check your RCA footprint. My downloaded footprint treated the pins as unnumbered holes, and my hand-fixed pad assignment was wrong. Grounding one output silenced both channels. It also explained part of the hum.
• Host-test your protocol code, but know the tests share your assumptions. The S/PDIF tests were green and the decoder was wrong.
• When an ESP32 audio task misbehaves, look at the CPU first. Per-task run-time stats, logged only when an underrun happens, found more causes than any scope.

Rev2 of the PCB is drawn: a separate MIC5205 LDO with a ferrite bead for the DAC's analog supply, the DAC's soft-mute pin on GPIO38 with a pull-down so it stays silent through boot, a TSOP38438 IR receiver on GPIO4 so the box gets a dedicated remote of its own, the RCA footprints corrected, and the OLED reset line dropped (the 4-pin module doesn't have one).

Next up is the real test: the living room. After that, rev2 boards, a cleaner DAC supply, and a remote that isn't the TV's.