I found the problem regarding the signal noise!

The issue seems to have been caused by a broken clock crystal. After assembling the board I had already noticed that the clock for the OPL3 chips was not right. It measured 33 MHz instead of the 14.3 MHz that is required and the OPL3s would only produce hissing noises. For the time being I had written it off as a small problem for later, after I had fixed the horrible signal noise.

Though after two evenings of getting nowhere I decided to maybe tackle a more simple issue first, so I desoldered the little crystal and I replaced it with a through hole 14.3 MHz crystal that I also use tor the OPL3 Duo boards. I soldered it to the little pads on the board and powered it up again. At this point I though I had now broken the board with all my bodges, because my headphones were dead silent. No noises when the OPL Studio was booting, nothing when it was accessing the SD card, nothing when I was touching the screen. Obviously I started looking for what wire I had disconnected without noticing it. Everything was still connected. I went into the drum sequencer to create a little drum beat and lo and behold! There was crystal clear sound coming from the line out!

I can't quite explain what had gone wrong with the crystal that would cause all this noise, but what I'm thinking is that during soldering I put too much heat through the crystal with my hot air gun. This probably damaged the crystal causing its frequency to change to 33 MHz. Maybe signal noise was picked up by the crystal's GND causing some frequency modulation in the broken crystal that got picked up by the OPL3s and amplified. I don't know, I don't have a better explanation for what happened.

Long story short, the major noise issue has been fixed! There are some smaller problems that I want to address with the boards, such as the location of the power button so the board fits better in the enclosure. Also I'm not 100% happy yet with the amplifier, which will take some more tweaking.

I would've never thought that a broken crystal could cause this kind of problem and I'm happy to have caught it early on, in a small batch of prototype boards. It's a small setback toward the all in one OPL Studio!