CarbonCycle-
Putting the kit in an enclosure
10/19/2020 at 00:18 • 0 commentsI'm putting the core nodes and the console server in a stand-alone box. Additional nodes will have separate enclosures and power.
-
Seeking Prometheus
10/17/2020 at 07:03 • 0 commentsBuglists are useful.
For instance - next on my list was observability. That mean deploying Prometheus. Then I saw this:
Nothing to do for Prometheus #1576
Fortunately, I'm late to the conversation that yields:
As a workaround on 1.19 you can do as suggested in kubernetes/kube-state-metrics#1190 (comment) and modify https://github.com/prometheus-operator/kube-prometheus/blob/980e95de011319b88a3b9c0787a81dcdf338a898/manifests/kube-state-metrics-deployment.yaml#L26 from
image: quay.io/coreos/kube-state-metrics:v1.9.7
to
image: gcr.io/k8s-staging-kube-state-metrics/kube-state-metrics-arm64:v1.9.7
and everything comes up as expected.
Yay! ( I think. One finds out by giving it a go! )
-
The value-add in implementing serial consoles
10/17/2020 at 05:47 • 0 commentsUsually, I don't bother or I drag whatever kit I'm wrangling over to a display and keyboard. Turns out those micro-HDMI connectors are not awesome and vary considerably in physical dimension until persuaded to the male connector spec. And now there are 4 of them to contend with without disturbing the USB ports.
I decided my sanity requires more. Cycling power is not calming to my personal being. I'm fine with fixing broken, as long as I know just how that works. The console is quite special in Unix for this specific purpose. Just ask the kernel.
To make this work in a more "appliance" way - I am using a Pi3B to perform a very useful function - separate from the running cluster. This will provide the usual "lights out" function that defines a proper appliance.
1. Serial console - FT4232H Quad HS USB-UART/FIFO used to provide simple connectivity to four nodes over a single USB port. I'm using putty to create console logging files and have debug access when the network is unusable.
2. Fan control - The Pi 4B nodes need moving air and temperature monitoring. Not alot of air, but some. Drives the temps down an easy 20 degress C.
3. Physical sensor data collection. I'm planning on using Mycodo on this node for the PID cooling functions - Fan control and enclosure temp.
This is the way to go with multiple node deployments - you get a log of the boot process and all the messages for that eventual calamity will be captured in putty logs. The other useful thing is catching the cloud-init config. The root certs are presented on the console - if that matters to you. With a putty console, I can log that output. -
MSD devices
10/17/2020 at 02:12 • 0 commentsI've been building Raspian nodes with external storage for awhile. I've found the recent changes to the boot eeprom to work just fine to boot from USB. With Raspian. With Ubuntu - not so much. At this point, I consider it a bit of a rabbit hole. So I'm fine with booting initially from the MicroSD card. It doesn't get mounted - and that is perhaps a good thing as I can still debug a boot problem separately from the state of the MSD device.
The downside is that after kernel rebuild - that kernel is not installed automatically. So I'm a few revisions behind already. Resolving this issue with automation is currently deferred. Once I figure out a manual method that is reliable - I'll document that here after some testing.
Another note is the hardware used. The PCI-e to USB bridge boards I'm using use the RTL9210B-CG chip on a generic adapter. The good thing is: no heat issues. At all. The problem? The device has a firmware setting that turns off power ( IOERROR! ) after 10 minutes of no access. The upside? Microk8s hammers that disk regularly - so as long as Microk8s is running properly - the disk stays on.
I disable all that power stuff ( to no effect, apparently ) with:
- Disable sleep
sudo systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target- Re-Enable sleep
sudo systemctl unmask sleep.target suspend.target hibernate.target hybrid-sleep.target -
Core HA nodes are up.
10/17/2020 at 01:38 • 0 commentsFollowing the instructions here: Adding nodes
ubuntu@Node-1:~$ microk8s kubectl get all --all-namespaces -o=wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES ingress pod/nginx-ingress-microk8s-controller-4tb5s 1/1 Running 1 6h33m 192.168.1.40 node-3 <none> <none> kube-system pod/calico-node-ggdrf 1/1 Running 1 3d4h 192.168.1.40 node-3 <none> <none> ingress pod/nginx-ingress-microk8s-controller-kj8lt 1/1 Running 2 6h33m 192.168.1.41 node-2 <none> <none> kube-system pod/hostpath-provisioner-976f6d665-j7sgl 1/1 Running 1 3d19h 10.1.247.2 node-2 <none> <none> kube-system pod/calico-node-gh6jz 1/1 Running 1 3d20h 192.168.1.41 node-2 <none> <none> ingress pod/nginx-ingress-microk8s-controller-hkxqd 1/1 Running 1 6h33m 192.168.1.30 node-1 <none> <none> kube-system pod/calico-node-449m7 1/1 Running 1 3d20h 192.168.1.30 node-1 <none> <none> kube-system pod/calico-kube-controllers-847c8c99d-9nx7p 1/1 Running 1 3d20h 10.1.84.132 node-1 <none> <none> kube-system pod/coredns-86f78bb79c-ngbnx 1/1 Running 1 3d20h 10.1.84.131 node-1 <none> <none> NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR default service/kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 3d20h <none> kube-system service/kube-dns ClusterIP 10.152.183.10 <none> 53/UDP,53/TCP,9153/TCP 3d20h k8s-app=kube-dns NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE CONTAINERS IMAGES SELECTOR ingress daemonset.apps/nginx-ingress-microk8s-controller 3 3 3 3 3 <none> 6h33m nginx-ingress-microk8s quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.33.0 name=nginx-ingress-microk8s kube-system daemonset.apps/calico-node 3 3 3 3 3 kubernetes.io/os=linux 3d20h calico-node calico/node:v3.13.2 k8s-app=calico-node NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR kube-system deployment.apps/hostpath-provisioner 1/1 1 1 3d19h hostpath-provisioner cdkbot/hostpath-provisioner-arm64:1.0.0 k8s-app=hostpath-provisioner kube-system deployment.apps/calico-kube-controllers 1/1 1 1 3d20h calico-kube-controllers calico/kube-controllers:v3.13.2 k8s-app=calico-kube-controllers kube-system deployment.apps/coredns 1/1 1 1 3d20h coredns coredns/coredns:1.6.6 k8s-app=kube-dns NAMESPACE NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR kube-system replicaset.apps/hostpath-provisioner-976f6d665 1 1 1 3d19h hostpath-provisioner cdkbot/hostpath-provisioner-arm64:1.0.0 k8s-app=hostpath-provisioner,pod-template-hash=976f6d665 kube-system replicaset.apps/calico-kube-controllers-847c8c99d 1 1 1 3d20h calico-kube-controllers calico/kube-controllers:v3.13.2 k8s-app=calico-kube-controllers,pod-template-hash=847c8c99d kube-system replicaset.apps/coredns-86f78bb79c 1 1 1 3d20h coredns coredns/coredns:1.6.6 k8s-app=kube-dns,pod-template-hash=86f78bb79c ubuntu@Node-1:~I've enabled DNS, ingress and the (local) storage provider.