In the wild, you will find two approaches, best illustrated by the Motorola vs Intel debate.

• Motorola and their ilk map peripherals in the memory. Typically you end up with a single address decoding logic ("glue" chips) with a pretty wide variety of granularities.
• The Intel school have a dedicated IO space that uses a few dedicated instructions.

In the 70/80s, separate IO spaces would ease decoding at the cost of more IO pins on the CPU.

In the 90s/2k, well, memory has become black art then PCI arrived so the mess is much worse.

I design a dedicated "space" to separate differing resources because they have different requirements : latency, speed, bandwidth, granularity, protection/safety, ordering, restartability...

• Memory can be weakly ordered and optimised for bandwidth, it uses various cache levels and has a coarse granularity for protection. Usually, there is one main area of memory, maybe split among several homogeneous banks. You usually move cache-line-wide chunks of data in interleaved transactions...
• IO can have many uses, from controlling internal CPU resources such as TLB, protection settings, essential peripherals... to yexchanging data with other (more or less dependent) units such as other CPU or coprocessors... You need a clear and clean execution where access rights are immediately evaluated, with maybe some latency, but no speculative execution or risk to re-execute the instruction after a trap (for example) because this would mess with the environment.

I use IN and OUT instructions to access anything that is not related to data storage. This is more or less equivalent to Intel's MSR introduced with the Pentium, 25 years ago. Semaphores, synchronisation, interrupt management, debug, profiling... can only work with word-wide accesses and fine-grained rights. This allows capability-based (or whitelist, or object-based) rights management, for example each peripheral could be accessed only by a given thread ID. Of course this also greatly simplifies the memory system because you don't rely on certain properties, that are relegated to a dedicated channel.