Being frustrated from work and missing a challenge with an not too far away goal, I finally decided to tackle an idea I'm hatching since decades, namely to just go and build my own computer. I grew up with Atari ST and Commodore Amiga and such the 68K was my first choice. But then I thought, well, maybe I'll start a little more humble, with an 6809 system. Should be easy and cheap, a few chips, some ROM and RAM - and here you are. I am no professional in the matter, not at all, but a computer enthusiast since first contact with the Commodore PET series (the bigger ones) at school. Of course I professionally got into the PC world right away, but meanwhile, well, PCs... I can solder, and I played around a lot with Arduinos and such in the last years, so I felt fit enough to give it a go.
Sooo... I sticked myself to an internet terminal for a total of some 30 or more hours of research for details, example projects, data sheets, schematics, literature, hardware components, software possibilities and so on and so forth. After a few weeks with almost daily packets via ebay (unbelieveable, those Chinese), I had a generous stock of chips for experimenting with different designs.
Finding the appropriate design, the one which I thought would be the best for my idea, was, well, a real adventure. The point is the operating system. Hacking up a basic 6809 is indeed no big deal at all, after I had scribbled the first design of a very basic BASIC computer, I realised that it was already rather common, I read a lot of pages referencing to Grant Searle's design. But that's not exactly what I had imagined, I wanted to go a little bit bigger, to be able to run a real operating system on it. My idea is not to just get A computer running, but to stay as close as possible to the initial motorola family concept, to build the computer that Motorola DIDN'T build.
Now, 6809 and operating systems, that is a thing. Certainly not for the contemporary witness, but I remind you, those were pre-internet times... a good deal of digital archeology is essential... :-)
Finally I decided to go for the 6883/6847 - option for system design, as the result would be a machine quite close to a Dragon or a TRS CoCo - and such I think I'm having a good chance of getting an OS-9 ported and running on it. To be precise and honest, it's the slightly modified exemplary design from the data sheet of the 6883.
If this project is running well, I might tackle an even "purer" system, employing multiple 6829 and 6844 and try to get some UNIXoid OS running. (Pure megalomania :-) an 68k is SO much easier)
Sounds like you're decided about your operating system. However, if you ever revisit your decision I suggest you look at Cubix by Dave Dunsfield. It's written for the 6809 and, in my opinion, is a full featured OS, taking into account when it was created. In addition there's an emulator so you can try it without having a 6809 system. I run the emulator in DosBox on a Windows 10 system. I don't know that it's open source but it's, as of the date of this comment, freely downloadable from Dave's web site.