I will first focus on redesigning the CPU.  I don't think the design will change much from a high block level, but most of the chips will be different.  I had to make a couple compromises like I'm gating the clock to some of the 74LS374 which is a digital no-no.  The datapath now requires 17 chips - definitely more than you'd need for a simple CPU.

The state machine is just a shift-register.  Six clocks are required for every instruction.  Each instruction is 4 bytes - two bytes for a source address and two bytes for a destination address.  That's 4 fetches from memory, and then 1 to execute the read The reset of the decoding is a 74LS138 and 4 chips for the AND and NOT gates.  

I believe that brings us up to 22 chips for the CPU.  And that's not counting how many will be needed for the VGA.  Not great, but there's no microcode.  I like to use the project by roelh and his 1inch square CPU ( https://hackaday.io/project/161251-1-square-inch-ttl-cpu ) with 8 chips.  However, there's a giant flash ROM, so if you go by transistor count it's quite high compared to this one.  And the Gigatron combines the CPU and VGA into one, so that will definitely have less too.  I'm going to try to make the VGA circuit so that you could pop out this CPU and put whichever one you'd want in its place.