After quite a bit of struggle I managed to get a GNU Radio source block going for the VST analyzer part. It's very basic at the moment, but it builds and runs.
I never really had a plan about how far to take this, so for the moment I just enjoy hating everything related, like conda, c++, pybind and a million other things that I have no (usable) knowledge about. I'm learning a lot and that's good.
Next I'll build another mini-backplane pcb to try 2 instruments side-by-side. Unfortunately the refurb dell motherboard I'm using doesn't support PCIe bifurcation so I'll need two separate miniSAS adapter cards, but I still have empty slots.
One of the nice things about this whole concept is that -by coincidence- there is no need to mess with a power supply / regulator circuitry for the PXIe instrument.
In general a module needs a +12V and a +3.3V rail and a +5V with negligible current draw. In case of the PXIe-5644R the spec sheet says the following:
Luckily the SATA power lines from the ATX power supply contain all three. I couldn't find what's the story behind most storage drives only using +5V and +12V to the extent that adapters from the old 5pin power connector are very common. I wonder if I am just not aware of disks using 3.3V or it was for a future use case in the standard that never become reality. Anyway, the connector has the 3.3V rail and the PSU will supply it so the only thing is to check current ratings. This happened to be more difficult than I though, and I found contradicting information on the interwebz. The old(-ish) SATA standard I have defines the per contact minimum rating at 1.5A. On the 15 pin SATA power connector there are 3 contacts for each positive voltage, and 2x 3 shared for ground. Since the 5V current draw is either zero or very small we can ignore that, thus both 3.3V and 12V have 3 contacts each in both directions giving a combined limit of 4.5A per rail. I put 3x SATA power inputs on the PCB, but only populated 2, for a maximum of 9A input per power rail over the connector contacts. (As a side note it is fortunate that 2 connectors is enough, because I spaced them too close on the board, not taking into account the protruding part of the cable connector. Since 2 is sufficient, I just left the middle one unpopulated.) The PXIe module has its own power conditioning and voltage regulation stage, but I still felt like I had to do something to compensate for the fact that the power is not delivered directly from a low impedance trace on the backplane, but through ~2 feet of wire, from a commercial psu. As a wholeheartedly half-a$$ed solution I put a 100uF tantalum polymer cap on each input voltage rail. I read something somewhere... I'm using 16 AWG wire, which is plenty thick and the MSI MPG A650GF psu can supply enough power for 2 PXIe-5644Rs.
Since I didn't want to connect any disks to the same SATA power lines, I decided to make new point to point cables for them
I'm very happy with how the mechanical integration came together. There are a couple of screw holes that don't line up with the PC case, but the cage is still held firm.
Tell me it doesn't look like it came out of a factory.
The difficult part was to figure out the signal assignment through the MiniSAS link.
MiniSAS HD is an SNIA spec, namely SFF-8643. Standard documents have tables detailing how PCIe can be routed over the cables and connectors of the interface. I bought a PCIe-MiniSAS HD adapter board on Amazon but that didn't follow the pinout in the SINA documents. These kind of adapter boards are used for storage devices, so I'm sure there's a reason why their signals assignment is that way, but I couldn't make sense of it so I had to reverse engineer based on the hardware.
Yet another difficulty was that on the adapter board the PCIe Reset signal was seemingly not connected. Maybe it's not needed for the storage devices or I just got two faulty units? I ended up populating one of the empty resistors with 470R. I overlapped photos from both sides of the card to trace back which PCIe line goes to which MiniSAS pin.
Marsupilami
Anyway, the connector has the 3.3V rail and the PSU will supply it so the only thing is to check current ratings. This happened to be more difficult than I though, and I found contradicting information on the interwebz. The old(-ish) SATA standard I have defines the per contact minimum rating at 1.5A. On the 15 pin SATA power connector there are 3 contacts for each positive voltage, and 2x 3 shared for ground. Since the 5V current draw is either zero or very small we can ignore that, thus both 3.3V and 12V have 3 contacts each in both directions giving a combined limit of 4.5A per rail.
I'm using 16 AWG wire, which is plenty thick and the MSI MPG A650GF psu can supply enough power for 2 PXIe-5644Rs.
The difficult part was to figure out the signal assignment through the MiniSAS link.
Yet another difficulty was that on the adapter board the PCIe Reset signal was seemingly not connected. Maybe it's not needed for the storage devices or I just got two faulty units? I ended up populating one of the empty resistors with 470R. 
