Raspberry Pi Ceph Cluster

Per the inquiry by @Toby Archer here's the plot of CPU and disk temperature over the last 30 days.

Per the inquiry from @Aaron Covrig , here's the memory consumption over the past week:

The three series with nightly jumps are the mon instances, and the jumps are the mon services shutting down for the backup process.

The OSD rebalance is still running, but it did free up enough space that I was able to finish migrating my media files.  The total raw space currently in use is 66% of 54 TiB.

I added the (hopefully) last 10 drives for this year.  This brings the total raw capacity to 55 TiB and should be enough to finish migrating my data.

It looks like the PG autoscaler kicked in last night., and made the following changes:

• fs_ec_5_2: 64 pgs -> 256 pgs

The autobalancer should kick in later tonight or tomorrow to pull the OSDs back into alignment.

The cluster rebalance is finally complete.   The final distribution also appears to be tighter than before the rebalance.

The rebalance is still progressing.  It's particularly slower because I also changed the failure domain on my cephfs EC pool from OSD to HOST which increased the number of PGs that needed to be remapped.  Aside from that, the rate of progression appears fairly typical.  The two gaps in the plot are power outages caused by Hurricane Delta.

I added 4 more RPi 4's and 8 2TB drives to bring the overall disk count to 20.  The cluster rebalance looks like it's going to take a few days, but at least the I/O performance isn't too severely impacted.

I added a dashboard to my grafana server to show the telegraf data from the ceph cluster.  You can see the disk io and cpu load from the cluster rebalancing from 10 osds to 12 osds.