The desired hardware configuration is:
- Quad 68030 CPUs with 68882 coprocessors
- 4GB of RAM (LPDDR2?)
- RS232 console interface
- 10/100Mbit ethernet
- Intel 27512 EPROM
- MicroSD storage
The hope is to run Linux / BSD.
The biggest obstacles I see already are:
- My lack of electrical and digital design experience: I am reading a lot (see below) but I will likely need to cut my teeth by building a far more modest 68000 based computer. I'll also need to draw on community experience and support.
- Designing a memory controller: It will be necessary to use modern DRAM chips in order to get the high density needed to supply these 32-bit CPUs with a theoretical maximum 4GB of RAM. This will likely require implementing a specialized memory controller on a FPGA to handle the massive disparities in timing, in addition to driving the DRAM chips.
- Designing a bus arbiter: It appears multiprocessing features in the 68k family of CPUS were first introduced in the 68020. It's not entirely clear to me what kind of logic / circuitry is needed for bus arbitration in a 68030-based SMP. It's even less clear how multiple CPUs are initialized, and how they're initialized and managed by an operating system - there are likely hints in Intel's MultiProcessor Specification documents.
- 68030 SMP support in software: Linux / BSD run on 68030 systems but I doubt they support systems with multiple 68030 processors.
Very ambitious project, though it sounds like you are aware of it. I'd suggest comparing the 68030 and the 68040 in terms of bus arbitration. The '040 is more sophisticated , and might be easier for a multiprocessor design. Not to mention that having built in FPU will mean less processors fight over the owning the bus.
In particular, check out section 7.8.2.1 of Motorola M68040 User's Manual. The section, "DUAL M68040 FAIRNESS ARBITRATION", provides an state diagram and discussion on providing for fair bus arbitration between two 68040 processors. There a few other really interesting parts in that chapter (7.8/7.9). 7.8.2.2 is a similiar example, but with prioritizing one processor over another, and it even has a section showing how DMA fits into the picture. I'm not sure how much those section are '040 specific or not. If you want to stick with the '030 it'll likely be informative.
I'd suggest looking into a SPI host control of some sort. SD cards all can talk SPI, and there a few NIC chips like the ENC28J60 which will give you 10Mbit networking over SPI pretty easily. If you do the math, you'll be hard pressed to push more data over the network than 10Mbit, even if you assume you already have the data in RAM.
Good luck, project definitely looks interesting.