Here's a high level view of the overall system architecture:

The interesting things to note here are that both the VGA controller and the CPU are bus masters, however since the only device/bus that the VGA controller cares about is the SSRAM, the SSRAM bus is separate from the primary system bus. Since the CPU is the only writer to the memory, this also means that we can simplify the data paths. The bus arbiter and the SSRAM datapaths are shown to the left of the CPU, with the rest of the devices on the primary system bus on the left.

Note also that the VGA controller is also a slave on the CPU bus. This is because the CPU needs to be able to update the palette maps. More on that in the next update.