Raspberry Pi Cluster

It's been a while since I posted anything about my little cluster. Things have changed with it a great deal, so it's a good time to update you all!

Hardware-wise the last few weeks has seen some changes: The RBG Fan has gone, replaced by a USB powered fan, without RGB. I needed to be able to cool the cluster without the ominous glow.

A Node has been removed to allow for a small network switch to be mounted in the case itself, complete with some new short patch cables for between the nodes and the switch. It's much tidier now, and only requires a power cable and a single ethernet cable connecting to have it in action. 

All Nodes are now running official Pi PoE hats; the USB fan is powered by the top node.

Software-wise the node has temporarily been reduced to a Master and Single node maintaining my database instances; the third node is now a more general testing environment for running homeassistant, and the removed node is currently running an openmediavault test environment, complete with a Plex Server. It is possible to deploy containers on OMV, so I may look to perhaps expand on that and see what OMV can do.

The OMV test project is working towards me receiving an Argon Pi NAS enclosure in a few months, which will give me a four bay Pi NAS to handle all my storage (and will be done so with a RAID array for redundancy), so some of my databases may be migrated over to that and leave the cluster for webserving.

I love having this cluster, it's incredibly versatile. The ability to switch the OS by simply switching the USB drive is fantastic.

It's been a while since I've posted about this project, and just wanted to give a little update.

After pursuing K8s for a while and microk8s I couldn't quite get anything useful working for me on the cluster.

After switching my focus to k3s and the brilliant set of tools called k3s-up (created by the magnificent Alex Ellis) I have been able to quickly and easily get the four nodes of my cluster up and running.

Hardware changes - I've completed the transition to USB booting for all nodes, getting rid of the SD Cards.

This has not only improved performance, it has also added more storage and improved reliability.

I've also exchanged one of the cheaper PoE HATs for an official Pi PoE HAT. The lower profile fits the cluster case much better, and removes the need to connect it via jumper wires. I've got one more to replace in due course, but I'm leaving the top one as-is, as it fits fine on the top node and also powers the cluster case fan.

My next goals are to set up some useful instances on the cluster, such as Openfaas, a database instance and start packaging some of my own apps into containers for deployment on here such as an AI assisted smarthome App I'm working on, and also some AI driven image recognition software.

Hardware wise, I'd like to switch out one of the 2Gb Pi nodes for my Tinkerboard, explore Netbooting the nodes instead of USB booting, and finally picking up a second case and adding another four nodes to it. The ending result would be an 8 node cluster with two master nodes for HA, and 6 worker nodes.

Today I added a fourth node to my little cluster. I added a 4GB model Pi 4, which now gives me 2 4gb models and 2 2gb models. I have another 4gb model which was going to upgrade one of the 2gb ones, but that is going to be my new development Pi for other project and paired with a touchscreen. I've also increased my microSD cards to 32gb cards from the 16gb ones that were in there.

Having has limited success with the microk8s snap install I'm going to spend the weekend installing the standard k8s to them. Fingers crossed, good times lay ahead!

Today is a good day - we have a cluster!

Today I've installed Ubuntu server 20.04 to the three Raspberry Pis, separated the PoE hats from the Pi's with jumper cables to allow them to fit in the enclosure  (the top one fits fine, so no need to move that one), connected all three to my PoE switch, renaming them and reserving their IPs as I went.

Then I installed the microK8s snap to all of them, and joined the second and third nodes to the first, making it the master node.

I'm over the moon that I've finally got one up and running. Hopefully in the next few weeks I will be filling the 4th slot in this case with an 8gb Pi 4. This will make this little beaut a 16 core 16gb RAM box.

Next step is to get some .NET Core apps running.

Thanks to working from home I've been able to save some fuel funds. So, I treated myself to a new cluster case for my little cluster.

i love it!

Four bays with a large fan, but using the PoE hats makes them too tall to stack, so it only has two with hats at present. I'm sure mounting them alongside the Pi's and using jumper cables to the PoE pins will probably work just fine I reckon.

Today I took the plunge and started to dismantle my baby ha ha. The screen was great, but was just too big to keep for having it up and running at home.

I've got an old 8" Windows 8.1 tablet which can serve as a wireless terminal should I need to view the dashboards in a similarly screened device.

So, this is how the cluster looks for the time being; 4gb master sitting at the top, the two 2gb slaves top shelf and space to accommodate another two should I ever get the funds to pick up a few more.

It's much smaller, and should sit next to the switch behind our TV without too much fuss.

Next up I want to explore options with Software. K3s was going to be the approach, however I'm sorely tempted to run Ubuntu Server on them all and install the Kubernetes snap.

So, like the rest of the UK I'm stuck at home during this current outbreak. I'm able to continue to work from home, so the work I was doing will continue in as normal a capacity as possible.

The need to have this cluster up and running to demonstrate how Kubernetes can be used to greatly improve the deployment and running of our .NET Core apps and legacy PHP content is still very much there; if anything it's a perfect time to get it up and tested ready to showcase once we return to a more normal way of working.

That being said I don't have a desk on which to leave this running at home so some changes will be made to integrate it into my home network.

Firstly, it's too big - my home network currently resides behind our TV unit in our living room and space is already at a premium. To make it fit in without causing too many issues I'm going to separate the screen from the project and replace the current switch I'm using with the PoE switch from this project. I can then build a much smaller and discreet frame for the Pis themselves and leave them running headless and accessible remotely.

The screen will be built into in it's own frame to compliment my Totem Mini Lab that I use for quick prototyping, maintaining the form factor of the Lab and including an Arduino Mega and a pHAT Stack instead of breadboards.

My current home switch is an old 24 port Netgear switch.  Only 6 ports are being used, so the 10 port PoE switch will be able to handle the 6 devices plus the three Pis running in the cluster. I currently have a big old Cisco Access Point connected to my router and using a PoE injector to power it (it's a thirsty PoE+ device); I can use the last spare port on the switch (which is also PoE+) to remove some of the cable mess behing there.

Once the Cluster is up and running I can also remove one or two of the other devices that will no longer be needed for running projects on, and I'll still have a free port on the back of my router itself for quickly plugging in the odd thing for a test here and there.

Today is a good day - finally settled on my mounting orientation for the slaves, and finally got PoE working on my Pi4 boards.

The way I've opted to mount the three slaves on the back will allow for another three slaves to be mounted below the current three.

The Tinkerboard is not yet connected; I need a different PoE splitter with a smaller/angled micro usb end to fit as desired.

So far the master Pi (running Buster w/desktop interface for the touchscreen) has been successfully booted via PoE power, as have the two slaves running Buster Lite.

Tomorrow I'm going to liberate a long Ethernet cable from work so I can connect the cluster to my home network switch and begin configuring the boards (static IPs, changing passwords, hostnames etc) and then hopefully get k3s up and running and a test MVC application running.

I'm also looking for ways to make this portable; as in to run independently of whatever network it is connected to. I would like to take this to work to demonstrate Kubernetes and containerisation, but i would have issues just plugging in some devices with a few statically assigned IP addresses following the 192.168.x.x convention.

I wonder if adding a small router to act as a middle man would be the way to go?

Answers on a postcard 😂

Two 2Gb models and a 4Gb model mounted, complete with PoE boards.

Awaiting PoE splitter for the Tinkerboard.
May use the 4Gb as the master and to also drive the display.

Exciting! 

Next week I will make some Ethernet cables and install the software to the microSD cards. Booooyah!