Rasberry Pi Bramble Build

Motivation

I wanted an inexpensive platform for doing some development work in a kubernetes environment. While deciding what was the best way to jump into that a few options came to mind :

  • EKS - an Amazon managed solution for k8s
  • Digital Oceaon - a k8s solution from Digital Ocean
  • Oketeto - developer friendly environments for cloud native applications
  • Build your own.

While you can guess where I wound up - I figure I’ll take a few minutes to explain why i didn’t take the other paths.

  • EKS / Digital Ocean both seemed like a good starting point but it wasn’t exactly cheap and I could see costs spiraling if i made mistakes or didn’t shut it down between uses
  • The Oketeto solution looked really awesome if I was interested in doing some toy applications without too much complexity.
  • Building my own gave me ultimate flexibility in setting up, maintaining, and experimenting and there are a number of individuals out there showing how to do it on what is relatively in expensive hardware.

Parts List

There’s only a few real choices to make when starting out:

  • How many nodes do I want in my cluster ? (you can always add more)
  • How do I want to get power to those nodes ? (you can use POE hats or direct power)
  • How do I want to store my rasberry pis ?

And thats about it. Keep in mind, if you go with a POE hat - you will need an ethernet switch capable of providing power. In general this path is a little more expensive, but it saves on cable management and makes for a much more mobile cluster as a result.

Core Build

Assembly

Assembly was very straight forward:

  • mount the POE hat to the rasberry pi
  • mount the rasberry pi into the case using provided instructions
  • plug your ethernet cable into the ethernet switch and the rasberry pi board

After doing all that - some pretty lights should turn on and fans should spin up.

C4 Labs Case Cooling (Optional)

The C4 Labs case comes with external fans that usually would be powered off of the Rasberry Pi’s power pins. These pins are blocked by the POE hat (which does have its own fan). I’m a big fan of cooling, so I wanted to make sure we could pull extra air across the boards - so I took some USB cables I had lying around and directly wired them up to the cooling fans provided.

Installing the Operating System

This is pretty straight forward process that consists of a few stages : flashing the OS, configuring cloud OS for first boot, installing k8s (k3s). You’ll do this once for each node in the cluster.

Setting up user-data for first boot.

The user-data file is a cloud-init spec file that describes how a computer should behave on boot -including initial setup of programs and configurations. If you’ve ever had to take a computer from 0 to operational once, it should be clear why having a configuration file that handles this for you is good and provides efficiencies.

For now - here’s the user-data file I’m using:

 1#cloud-config
 2# See cloud-init documentation for available options:
 3# https://cloudinit.readthedocs.io/
 4
 5hostname: bramblet-<d+>  # update this with a unique hostname
 6
 7ssh_pwauth: false
 8
 9groups:
10  - ubuntu: [root, sys]
11
12users:
13  - default
14  - name: bramblet 
15    gecos: bramblet
16    sudo: ALL=(ALL) NOPASSWD:ALL
17    groups: sudo
18    ssh_import_id: None
19    lock_passwd: true
20    shell: /bin/bash
21    ssh_authorized_keys:
22      - <COPY PUBLIC KEY HERE>

Take the above code, edit it in the following ways:

  • update the hostname to something unique, i suggest a monotonic increasing digit for each node in your cluster
  • copy your public half an ssh key pair into the section ssh_authorized_keys.

The major thing here is that on boot a computer with this user-data file will allow someone with the paired private key to login at bramblet@bramblet-<d+>. This allows us to run our linux distribution in a headless state, not requiring a monitor/keyboard/mouse for initial operations.

Note: A patch of rasberry bushes is called a bramble. Hence the traditional name for a cluster of Rasberry Pi’s.

Flashing your microSD Card

Note: If you don’t want to run off your microSD card, but a USB drive instead follow the instructions here for USB boot steps.

  • Go get the Rasberry Pi Image flasher for whatever computer you’re using from here.
  • Boot it up, plugin your microSD card, and run the flashing program.
  • Copy your edited user-data file to the microSD card after its flashed
  • Plug it in to the Rasberry Pi module
  • Start it up

Installing k3s

  • login to the rasberry pi with something like `ssh -i ~/.ssh/bramblet bramblet-@
  • run the following commands :
1sudo apt upgrade -y
2sudo apt install -y docker.io
3
4sudo docker info # verify docker is setup and enabled
5
6sudo sed -i '$ s/$/ cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1 swapaccount=1/' /boot/firmware/cmdline.txt
7sudo reboot
  • Install k3s and setup the server node (only do this once):
1curl -sfL https://get.k3s.io | sh -
2sudo cat /var/lib/rancher/k3s/server/node-token
3cat /etc/rancher/k3s/k3s.yaml

Save a copy of the node-token, k3s.yaml, and the ip-address for future use.

  • Setup your worker nodes
1SERVER_NODE_IP=<SERVER_NODE_IP>
2CLUSTER_TOKEN=<CLUSTER_TOKEN>
3
4curl -sfL https://get.k3s.io | K3S_URL=https://$SERVER_NODE_IP:6443 K3S_TOKEN=$CLUSTER_TOKEN sh -
  • verify that your cluster is running
1kubectl get nodes

You should now have a fully functional k8s cluster for use.

First Deployment

  • Install kubectl on your development computer. A myriad of instructions are available here
  • Copy the k3.yaml file from the server node to ~/.kube/config on your dev computer
  • Run the following from your dev computer :
1kubectl create namespace kubernetes-dashboard
2kubens kubernetes-dashboard
3kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.2.0/aio/deploy/recommended.yaml
4kubectl proxy
  • Navigate to the following address - http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/#/overview?namespace=_all