Longer Vision Technology

Github Blog

0%

A Cluster of Raspberry Pis (3) - k3s

Posted on Edited on In

In this blog, we're going to explore Kubernetes. However, there is a VERY FIRST question to be answered: what is the relationship between Kubernetes and Docker? Let's start our journey of today:

1 Kubernetes

Kubernetes (commonly stylized as k8s) is an open-source container-orchestration system for automating application deployment, scaling, and management. (cited from Wikipedia)

On Raspberry Pi, a lightweight variant of Kubernetes is normally preferred. A variety of choices are available:

Minikube vs. kind vs. k3s - What should I use? elaborates the differences among Minikube, kind and k3s. Its final table is cited as follows:

minikube kind k3s
runtime VM container native
supported architectures AMD64 AMD64 AMD64, ARMv7, ARM64
supported container runtimes Docker, CRI-O, containerd, gVisor Docker Docker, containerd
startup time: initial/following 5:19 / 3:15 2:48 / 1:06 0:15 / 0:15
memory requirements 2GB 8GB (Windows, MacOS) 512 MB
requires root? no no yes (rootless is experimental)
multi-cluster support yes yes no (can be achieved using containers)
multi-node support no yes yes
project page minikube kind k3s

Here in my case, I'm going to use k3s to manage and monitor the cluster. The following 2 blogs are strongly recommended from me. - Run Kubernetes on a Raspberry Pi with k3s - Kubernetes 1.18 broke “kubectl run”, here’s what to do about it

2. Preparation

Let's take a look at the IP info of ALL 4 Raspberry Pis. Let's take pi04 as the example this time. pi01, pi02, pi03 are having very similar IP info as pi04.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether b8:27:eb:c1:b8:76 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.247/24 brd 192.168.1.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet6 2001:569:7fb8:c600:ca9b:d758:2534:8a9d/64 scope global dynamic mngtmpaddr noprefixroute 
       valid_lft 7480sec preferred_lft 7180sec
    inet6 fe80::80b5:3a3e:defe:24aa/64 scope link 
       valid_lft forever preferred_lft forever
3: wlan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether b8:27:eb:94:ed:23 brd ff:ff:ff:ff:ff:ff
4: wlan1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether e8:4e:06:35:16:6d brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.245/24 brd 192.168.1.255 scope global dynamic noprefixroute wlan1
       valid_lft 86060sec preferred_lft 75260sec
    inet6 2001:569:7fb8:c600:5996:3c23:ef08:c367/64 scope global dynamic mngtmpaddr noprefixroute 
       valid_lft 7480sec preferred_lft 7180sec
    inet6 fe80::a0ff:56e3:77f5:9bf2/64 scope link 
       valid_lft forever preferred_lft forever

As mentioned in A Cluster of Raspberry Pis (1) - Configuration, pi04 is an old Raspberry Pi 3 Model B Rev 1.2 1GB, which is unfortunately with a broken Wifi interface wlan0. Therefore, I've got to insert a Wifi dongle in order to have Wifi wlan1 enabled.

3. k3s Installation and Configuration

3.1 k3s Installation on Master Node pi01

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
pi@pi01:~ $ curl -sfL https://get.k3s.io | sh -
[INFO]  Finding release for channel stable
[INFO]  Using v1.18.3+k3s1 as release
[INFO]  Downloading hash https://github.com/rancher/k3s/releases/download/v1.18.3+k3s1/sha256sum-arm.txt
[INFO]  Downloading binary https://github.com/rancher/k3s/releases/download/v1.18.3+k3s1/k3s-armhf
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s
pi@pi01:~ $ sudo kubectl get nodes
NAME   STATUS   ROLES    AGE     VERSION
pi01   Ready    master   2m42s   v1.18.3+k3s1

If we take a look at IP info, one additional flannel.1 interface is added as follows:

1
2
3
4
5
6
7
8
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default 
    link/ether 3e:53:91:ce:b9:46 brd ff:ff:ff:ff:ff:ff
    inet 10.42.0.0/32 brd 10.42.0.0 scope global flannel.1
       valid_lft forever preferred_lft forever
    inet 169.254.238.88/16 brd 169.254.255.255 scope global noprefixroute flannel.1
       valid_lft forever preferred_lft forever
    inet6 fe80::59c5:fbf9:aaf7:46fb/64 scope link 
       valid_lft forever preferred_lft forever

3.2 k3s Installation on Work Node pi01, pi02, pi03

Before moving forward, we need to write down node token on the master node, which will be used while the other work nodes join in the cluster.

1
2
pi@pi01:~ $ sudo cat /var/lib/rancher/k3s/server/node-token
K10cedbc6396ab68f6bee0d2df3eb005f0ff9ea17275aed2763b6bf07a06e83ce47::server:9a0d84f3bd8044b341c95f967badd5d3
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
pi@pi0X:~ $ curl -sfL http://get.k3s.io | K3S_URL=https://192.168.1.253:6443 \
> K3S_TOKEN=K10cedbc6396ab68f6bee0d2df3eb005f0ff9ea17275aed2763b6bf07a06e83ce47::server:9a0d84f3bd8044b341c95f967badd5d3 sh -
[INFO]  Finding release for channel stable
[INFO]  Using v1.18.3+k3s1 as release
[INFO]  Downloading hash https://github.com/rancher/k3s/releases/download/v1.18.3+k3s1/sha256sum-arm.txt
[INFO]  Skipping binary downloaded, installed k3s matches hash
[INFO]  Skipping /usr/local/bin/kubectl symlink to k3s, command exists in PATH at /usr/bin/kubectl
[INFO]  Skipping /usr/local/bin/crictl symlink to k3s, command exists in PATH at /usr/bin/crictl
[INFO]  Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-agent-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s-agent.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s-agent.service
[INFO]  systemd: Enabling k3s-agent unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s-agent.service → /etc/systemd/system/k3s-agent.service.
[INFO]  systemd: Starting k3s-agent

, where X=2, or 3, or 4.

3.3 Take a Look on pi01

1
2
3
4
5
6
pi@pi01:~ $ sudo kubectl get nodes
NAME   STATUS   ROLES    AGE     VERSION
pi02   Ready    <none>   5m7s    v1.18.3+k3s1
pi03   Ready    <none>   4m18s   v1.18.3+k3s1
pi01   Ready    master   20m     v1.18.3+k3s1
pi04   Ready    <none>   11s     v1.18.3+k3s1

3.4 Access Raspberry Pi Cluster from PC

We can further configure our PC to be able to access the Raspberry Pi Cluster. For details, please refer to Run Kubernetes on a Raspberry Pi with k3s. On my laptop, I can do:

1
2
3
4
5
6
 ✔  kubectl get nodes
NAME   STATUS   ROLES    AGE     VERSION
pi02   Ready    <none>   3h27m   v1.18.3+k3s1
pi03   Ready    <none>   3h26m   v1.18.3+k3s1
pi04   Ready    <none>   3h22m   v1.18.3+k3s1
pi01   Ready    master   3h42m   v1.18.3+k3s1

We can even specify the role name by the following command:

1
2
 ✔  kubectl label nodes pi0X kubernetes.io/role=worker
node/pi0X labeled

, where X=2, or 3, or 4.

Let's take a look at all nodes again:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
 64  ✔  kubectl get nodes
NAME   STATUS   ROLES    AGE    VERSION
pi04   Ready    worker   2d7h   v1.18.3+k3s1
pi03   Ready    worker   2d7h   v1.18.3+k3s1
pi01   Ready    master   2d7h   v1.18.3+k3s1
pi02   Ready    worker   2d7h   v1.18.3+k3s1
 66  ✘  kubectl get pods --all-namespaces
  ~ 
NAMESPACE              NAME                                         READY   STATUS             RESTARTS   AGE
kube-system            helm-install-traefik-m49zm                   0/1     Completed          0          2d7h
kube-system            svclb-traefik-djtgw                          2/2     Running            2          2d7h
kube-system            svclb-traefik-cn76f                          2/2     Running            2          2d7h
kube-system            local-path-provisioner-6d59f47c7-5gz67       1/1     Running            1          2d7h
kube-system            svclb-traefik-7bwtl                          2/2     Running            2          2d7h
kube-system            coredns-8655855d6-wg68z                      1/1     Running            1          2d7h
kube-system            traefik-758cd5fc85-vn7xg                     1/1     Running            1          2d7h
kube-system            metrics-server-7566d596c8-v2vgp              1/1     Running            1          2d7h
kube-system            svclb-traefik-r9x5v                          2/2     Running            4          2d7h
kubernetes-dashboard   dashboard-metrics-scraper-6b4884c9d5-ct6lh   1/1     Running            0          43h
kubernetes-dashboard   kubernetes-dashboard-7b544877d5-s4spx        0/1     CrashLoopBackOff   489        43h
 67  ✔  kubectl get pods
  ~ 
No resources found in default namespace.

4. Create Deployment

1
2
3
 73  ✔  kubectl create deployment nginx-sample --image=nginx
  ~ 
deployment.apps/nginx-sample created

After a while, nginx-sample will be successfully deployed.

1
2
3
4
5
6
7
8
9
10
11
12
 79  ✔  kubectl get deployments
  ~ 
NAME           READY   UP-TO-DATE   AVAILABLE   AGE
nginx-sample   1/1     1            1           12m
 80  ✔  kubectl get pods
  ~ 
NAME                           READY   STATUS    RESTARTS   AGE
nginx-sample-879f5cf56-kg6xf   1/1     Running   0          12m
 81  ✔  kubectl get pods -o wide
  ~ 
NAME                           READY   STATUS    RESTARTS   AGE   IP           NODE   NOMINATED NODE   READINESS GATES
nginx-sample-879f5cf56-kg6xf   1/1     Running   0          12m   10.42.0.15   pi01   <none>           <none>

Now, let's expose this service and take a look from the browser:

1
2
3
4
5
6
7
8
 82  ✔  kubectl expose deployment nginx-sample --type="NodePort" --port 80 
  ~ 
service/nginx-sample exposed
 83  ✔  kubectl get services 
  ~ 
NAME           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes     ClusterIP   10.43.0.1       <none>        443/TCP        96m
nginx-sample   NodePort    10.43.195.237   <none>        80:30734/TCP   72s