As any other girl or boy, I have been facing the same issue. What and how to deploy as Kubernetes edge platform. Somehow, after many sleepless nights in deep dark woods, after times of excitement followed by depression and despair, I had seen weak flickering light far far away. So my adventurous journey to perfect kube edge begins as yours.
Let’s establish our ground, by defining some expectations. They are sometimes a little bit contradictory, what else you can expect?
Requirements
- High availibility
- Easy to deploy/maintain
- Fast and archive storage
- Small footprint
- NIC teaming/bonding aka 2+ NICs
- Distributed storage = fast networking
- Cost-optimized solution (aka cheap)
- Operated by terraform
Hypervisor or bare metal Kubernetes
Difficult question to answer. The decision here depends on intended use and high availability requirements. Because we would like to have as few nodes as possible, we are looking for two or three-node configurations. Surprisingly there are not too many two-node options. On the other hand, when we realize, that we are looking for HCI (hyperconverged infrastructure) solution, it is not such a wonder. It is all about shared storage.
Storage
So lots of HA solutions omit the existence of this issue and expect, that you have reliable NFS, iSCSI, SMB storage. But you don’t you are on edge. You have only one option for Linux - DRBD. You will find a bunch of exmaples, how to get it going, spread across the internet. It is a mature technology, for example, SuSE uses it for reliable storage for SAP HANA. There is also derivative work HA-Lizard for xen-orchestra and lizardFS for general use.
To complete the list - vmware vSAN and Microsoft Storage Spaces Direct fulfill the two node requirement as well, but they are missing the pricepoint. Especialy Microsoft’s two-node solution is promising, but it misses the Terraform requirement. The networking part of IaC with Hyper-V is just pure pain (because it does not exist).
So with two nodes, you have DRBD = iSCSI. Nothing too fancy. I imagined the burden connected to the learning path and future issues originating from the fact, that even is just a bad choice for a number of cluster nodes and so some modern FS it will be.
When you need three nodes for storage, why not use baremetal Kubernetes directly on the nodes?
Kubernetes flavor
Yah - bare metal Kubernetes, three nodes here we go. Give me k3s for the edge and it is done, isn’t it?
It was painful procedure. The conclusion? Never read the documentation, no RTFM. Just deploy something, and issues will be solved by future generations.
Every tutorial, even advanced ones, deploys a single node and mentions that for production you need more. So you start counting, etcd cluster 3, master nodes 3, worker nodes 2. Then you find an example - one control node, two worker nodes and high availability is mentioned here. But you need to read the article carefully - it is for worker nodes, and you must have in the real scenario three nodes for controlplane. And if you would like to have highly available k3s - the magic of simple neat deployment is gone.
So let’s stop complaining - Microsoft has a recommendation for bare metal Kubernetes. I found it the hard way - so I tried k3s, kubeadm, Talos, Rancher, RKE on harvesterhci, etc. After all of that - yes microk8s is the right choice. You will more about deployment (virtual and physical) in the following articles in this series.
Storage - the saga continues
After all of that, we still did not choose the storage system for our persistent volumes. There are still a lot of viable options and we need to investigate them in much greater detail - Rook, Longhorn, OpenEBS, MinIO.