The backup process is a systemd timer service that stops the mon process, tarballs the mon data directory, and the restarts the mon process.  The tarballs are copied to an independent rpi for archival.

The OSDs are using 1.5 GiB/TB of ram, which comes to 6GB total for the two OSDs.  Using a 4G pi would make the OSDs unstable during heavy I/O or recovery.

I haven't noticed any appreciable wear on the mon SD cards.  The cluster spends most of it's time clean and idle, so there's not much activity most of the time.

Hi, I really like this little cluster. Could you elaborate on "the backup process" a little more please? Do you mean you're stopping the mon processes completely, backup the mon('s system) and restart? I'd stagger the backup points so not have them occur all at the same time. What folders exactly are you backing up? Or is it simply the whole system? And what is the green line up there with the mons?

Am I reading it right that the osds aren't even utilizing only around 1 gb/tb even while rebalancing? That should be comparable to rebuilding memory usage. So judging from that for 2x2 TB disk per pi you could've gone with 4 GB ram pi 4s without impacting performance too much after all.

On another note I read that the mons tend to write quite a bit of logs to their system disk and can wear out sd-cards and flash drives in a matter of months. Do you have any insight regarding the volume of logs your mons write?