I wanted an inexpensive platform for doing some development work in a kubernetes environment. While deciding what was the best way to jump into that a few options came to mind :
- EKS - an Amazon managed solution for k8s
- Digital Oceaon - a k8s solution from Digital Ocean
- Oketeto - developer friendly environments for cloud native applications
- Build your own.
While you can guess where I wound up - I figure I’ll take a few minutes to explain why i didn’t take the other paths.
- EKS / Digital Ocean both seemed like a good starting point but it wasn’t exactly cheap and I could see costs spiraling if i made mistakes or didn’t shut it down between uses
- The Oketeto solution looked really awesome if I was interested in doing some toy applications without too much complexity.
- Building my own gave me ultimate flexibility in setting up, maintaining, and experimenting and there are a number of individuals out there showing how to do it on what is relatively in expensive hardware.
- 4 x Rasberry Pi’s
- 4 x Ethernet Cables
- 4 x microSD Cards
- 4 x Power Over Ethernet (POE) Hats
- 1 x Ethernet Switch (with POE)
- 1 x Rasberry Pi Case
There’s only a few real choices to make when starting out:
- How many nodes do I want in my cluster ? (you can always add more)
- How do I want to get power to those nodes ? (you can use POE hats or direct power)
- How do I want to store my rasberry pis ?
And thats about it. Keep in mind, if you go with a POE hat - you will need an ethernet switch capable of providing power. In general this path is a little more expensive, but it saves on cable management and makes for a much more mobile cluster as a result.
Assembly was very straight forward:
- mount the POE hat to the rasberry pi
- mount the rasberry pi into the case using provided instructions
- plug your ethernet cable into the ethernet switch and the rasberry pi board
After doing all that - some pretty lights should turn on and fans should spin up.
C4 Labs Case Cooling (Optional)
The C4 Labs case comes with external fans that usually would be powered off of the Rasberry Pi’s power pins. These pins are blocked by the POE hat (which does have its own fan). I’m a big fan of cooling, so I wanted to make sure we could pull extra air across the boards - so I took some USB cables I had lying around and directly wired them up to the cooling fans provided.
Installing the Operating System
This is pretty straight forward process that consists of a few stages : flashing the OS, configuring cloud OS for first boot, installing k8s (k3s). You’ll do this once for each node in the cluster.
user-data for first boot.
user-data file is a cloud-init spec file that describes how a computer should behave on boot -including initial setup of programs and configurations. If you’ve ever
had to take a computer from 0 to operational once, it should be clear why having a configuration file that handles this for you is good and provides efficiencies.
For now - here’s the
user-data file I’m using:
2# See cloud-init documentation for available options:
5hostname: bramblet-<d+> # update this with a unique hostname
10 - ubuntu: [root, sys]
13 - default
14 - name: bramblet
15 gecos: bramblet
16 sudo: ALL=(ALL) NOPASSWD:ALL
17 groups: sudo
18 ssh_import_id: None
19 lock_passwd: true
20 shell: /bin/bash
22 - <COPY PUBLIC KEY HERE>
Take the above code, edit it in the following ways:
- update the hostname to something unique, i suggest a monotonic increasing digit for each node in your cluster
- copy your public half an ssh key pair into the section
The major thing here is that on boot a computer with this
user-data file will allow someone with the paired private key to login at
bramblet@bramblet-<d+>. This allows us to run our linux distribution in a
headless state, not requiring a monitor/keyboard/mouse for initial operations.
Note:A patch of rasberry bushes is called a bramble. Hence the traditional name for a cluster of Rasberry Pi’s.
Flashing your microSD Card
Note:If you don’t want to run off your microSD card, but a USB drive instead follow the instructions here for USB boot steps.
- Go get the Rasberry Pi Image flasher for whatever computer you’re using from here.
- Boot it up, plugin your microSD card, and run the flashing program.
- Copy your edited
user-datafile to the microSD card after its flashed
- Plug it in to the Rasberry Pi module
- Start it up
- login to the rasberry pi with something like `ssh -i ~/.ssh/bramblet bramblet-@
- run the following commands :
1sudo apt upgrade -y
2sudo apt install -y docker.io
4sudo docker info # verify docker is setup and enabled
6sudo sed -i '$ s/$/ cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1 swapaccount=1/' /boot/firmware/cmdline.txt
k3sand setup the server node (only do this once):
1curl -sfL https://get.k3s.io | sh -
2sudo cat /var/lib/rancher/k3s/server/node-token
Save a copy of the node-token, k3s.yaml, and the ip-address for future use.
- Setup your worker nodes
4curl -sfL https://get.k3s.io | K3S_URL=https://$SERVER_NODE_IP:6443 K3S_TOKEN=$CLUSTER_TOKEN sh -
- verify that your cluster is running
1kubectl get nodes
You should now have a fully functional k8s cluster for use.
- Install kubectl on your development computer. A myriad of instructions are available here
- Copy the
k3.yamlfile from the server node to
~/.kube/configon your dev computer
- Run the following from your dev computer :
1kubectl create namespace kubernetes-dashboard
3kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.2.0/aio/deploy/recommended.yaml
- Navigate to the following address -