Figure-1
| PolarSPARC |
Practical Kubernetes on ODroid-N2 Cluster (ARM Edition)
| Bhaskar S | 12/15/2019 |
Overview
Kubernetes (or k8s for short) is an extensible open source container orchestration platform designed for managing containerized workloads and services at scale. It helps in automated deployment, scaling, and management of container centric application workloads across a cluster of nodes (bare-metal, virtual, or cloud) by orchestrating compute, network, and storage infrastructure on behalf of those user workloads.
The two main types of nodes in a Kubernetes cluster are:
Master :: the node that acts as the Control Plane for the cluster. It is responsible for all application workload deployment, scheduling, and placement decisions as well as detecting and managing changes to the state of deployed applications. It is comprised of a Key-Value Store, an API Server, a Scheduler, and a Controller Manager
Worker Node(s) :: node(s) that actually run the application containers. They are also on occasions referred to as Minion(s). The Master is also a node, but is not targeted for application deployment. It is comprised of an agent called kubelet, a network proxy called kube-proxy, and a Container Engine
The following Figure-1 illustrates the high-level architectural overview of Kubernetes:
The core components that make Kubernetes cluster are described as follows:
KV Store :: a highly reliable, distributed, and consistent key-value data store used for persisting and maintaining state information about the various components of the Kubernetes cluster. By default, Kubernetes uses etcd as the key-value store
API Server :: acts as the entry point for the Control Plane by exposing an API endpoint for all interactions with and within the Kubernetes cluster. It is through the API Server that requests are made for deployment, administration, management, and operation of container based applications. It uses the key-value store to persist and maintain state information about all the components of the Kubernetes cluster
Pod(s) :: it is the smallest unit of deployment in Kubernetes . One or more containers run inside it. Think of it as a logical host with shared network and storage. Application pods are scheduled to run on different worker nodes of the Kubernetes cluster based on the resource needs and application constraints. Every pod within the cluster gets its own unique ip-address. The application containers within a pod communicate with each other using localhost. Pod(s) are also the smallest unit of scaling in Kubernetes. In addition, Pod(s) are ephemeral - they can come and go at any time
Scheduler :: responsible for scheduling application pod(s) to run on the selected worker node(s) of the Kubernetes cluster based on the application resource requirements as well as application specific affinity constraints
Service :: provides a stable, logical networking endpoint for a group of pod(s) (based on a label related to an application pod) running on the wokrker node(s) of the Kubernetes cluster. They enable access to an application via service-discovery and spread the requests through simple load-balancing. To access an application, each service is assigned a cluster-wide internal ip-address:port
Controller Manager :: manages different types of controllers that are responsible for monitoring and detecting changes to the state of the Kubernetes cluster (via the API server) and ensuring that the cluster is moved to the desired state. The different types of controllers are:
Node Controller => responsible for monitoring and detecting the state & health (up or down) of the worker node(s) in the Kubernetes cluster
ReplicaSet => previously referred to as the Replication Controller and is responsible for maintaining the desired number of pod replicas in the cluster
Endpoints Controller => responsible for detecting and managing changes to the application service access endpoints (list of ip-address:port)
Plugin Network :: acts as the bridge (overlay network) that enables communication between the pod(s) running on different worker node(s) of the cluster. There are different implementations of this component by various 3rd-parties such as calico, flannel, weave-net, etc. They all need to adhere to a common specification called the Container Network Interface or CNI for short
kubelet :: an agent that runs on every worker node of the Kubernetes cluster. It is responsible for creating and starting an application pod on the worker node and making sure all the application containers are up and running within the pod. In addition, it is also responsible for reporting the state and health of the worker node, as well as all the running pods to the master via the API server
kube-proxy :: a network proxy that runs on each of the worker node (s) of the Kubernetes cluster and acts as an entry point for access to the various application service endpoints. It routes requests to the appropriate pod (s) in the cluster
Container Engine :: a container runtime that runs on each of the worker node(s) to manage the lifecycle of containers such as getting the images, starting and stopping containers, etc. The commonly used container engine is Docker
kubectl :: command line tool used for interfacing with the API Server. Used by administrators (or operators) for deployment and scaling of applications, as well as for management of the Kubernetes cluster
Installation and System Setup
The installation will be on a 5-node ODroid-N2 Cluster running Armbian Ubuntu Linux.
The following Figure-2 illustrates the 5-node ODroid-N2 cluster in operation:
For this tutorial, let us assume the 5-nodes in the cluster to have the following host names and ip addresses:
| Host name | IP Address |
|---|---|
| my-n2-1 | 192.168.1.51 |
| my-n2-2 | 192.168.1.52 |
| my-n2-3 | 192.168.1.53 |
| my-n2-4 | 192.168.1.54 |
| my-n2-5 | 192.168.1.55 |
Open a Terminal window and open a tab for each of the 5 nodes my-n2-1 thru my-n2-5. In each of the Terminal tabs, ssh into the corresponding node.
Each of the nodes my-n2-1 thru my-n2-5 need to have a unique identifier for the cluster to operate without any collisions. The unique node identifier is located in the file /etc/machine-id and we see all the nodes my-n2-1 thru my-n2-5 having the same value. This needs to be * FIXED*. On each of the nodes my-n2-1 thru my-n2-5, execute the following commands:
$ sudo rm -f /etc/machine-id
$ sudo dbus-uuidgen --ensure=/etc/machine-id
$ sudo rm /var/lib/dbus/machine-id
$ sudo dbus-uuidgen --ensure
$ sudo reboot now
Once again, in each of the Terminal tabs, ssh into the corresponding node.
Next, we need to setup the package repository for Docker. On each of the nodes my-n2-1 thru my-n2-5, execute the following commands:
$ sudo apt-get update
$ sudo apt-get install apt-transport-https ca-certificates curl software-properties-common -y
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
$ sudo apt-get update
$ sudo add-apt-repository "deb [arch=arm64] https://download.docker.com/linux/ubuntu xenial stable"
$ sudo apt-get update
For version 1.16 of Kubernetes (the version at the time of this article), the recommendeded Docker version is 18.09.
Ensure the version of Docker installed is *18.09*. Else will encounter the following error:
[ERROR SystemVerification]: unsupported docker version: 19.xx
We need to check for the latest package of Docker 18.09 in the repository. On any of the nodes (we will pick my-n2-1), execute the following command:
$ apt-cache madison docker-ce
The following would be a typical output:
docker-ce | 5:19.03.5~3-0~ubuntu-xenial | https://download.docker.com/linux/ubuntu xenial/stable arm64 Packages docker-ce | 5:19.03.4~3-0~ubuntu-xenial | https://download.docker.com/linux/ubuntu xenial/stable arm64 Packages docker-ce | 5:19.03.3~3-0~ubuntu-xenial | https://download.docker.com/linux/ubuntu xenial/stable arm64 Packages docker-ce | 5:19.03.2~3-0~ubuntu-xenial | https://download.docker.com/linux/ubuntu xenial/stable arm64 Packages docker-ce | 5:19.03.1~3-0~ubuntu-xenial | https://download.docker.com/linux/ubuntu xenial/stable arm64 Packages docker-ce | 5:19.03.0~3-0~ubuntu-xenial | https://download.docker.com/linux/ubuntu xenial/stable arm64 Packages docker-ce | 5:18.09.9~3-0~ubuntu-xenial | https://download.docker.com/linux/ubuntu xenial/stable arm64 Packages docker-ce | 5:18.09.8~3-0~ubuntu-xenial | https://download.docker.com/linux/ubuntu xenial/stable arm64 Packages docker-ce | 5:18.09.7~3-0~ubuntu-xenial | https://download.docker.com/linux/ubuntu xenial/stable arm64 Packages docker-ce | 5:18.09.6~3-0~ubuntu-xenial | https://download.docker.com/linux/ubuntu xenial/stable arm64 Packages docker-ce | 5:18.09.5~3-0~ubuntu-xenial | https://download.docker.com/linux/ubuntu xenial/stable arm64 Packages docker-ce | 5:18.09.4~3-0~ubuntu-xenial | https://download.docker.com/linux/ubuntu xenial/stable arm64 Packages docker-ce | 5:18.09.3~3-0~ubuntu-xenial | https://download.docker.com/linux/ubuntu xenial/stable arm64 Packages docker-ce | 5:18.09.2~3-0~ubuntu-xenial | https://download.docker.com/linux/ubuntu xenial/stable arm64 Packages docker-ce | 5:18.09.1~3-0~ubuntu-xenial | https://download.docker.com/linux/ubuntu xenial/stable arm64 Packages docker-ce | 5:18.09.0~3-0~ubuntu-xenial | https://download.docker.com/linux/ubuntu xenial/stable arm64 Packages docker-ce | 18.06.3~ce~3-0~ubuntu | https://download.docker.com/linux/ubuntu xenial/stable arm64 Packages docker-ce | 18.06.2~ce~3-0~ubuntu | https://download.docker.com/linux/ubuntu xenial/stable arm64 Packages docker-ce | 18.06.1~ce~3-0~ubuntu | https://download.docker.com/linux/ubuntu xenial/stable arm64 Packages docker-ce | 18.06.0~ce~3-0~ubuntu | https://download.docker.com/linux/ubuntu xenial/stable arm64 Packages docker-ce | 18.03.1~ce-0~ubuntu | https://download.docker.com/linux/ubuntu xenial/stable arm64 Packages docker-ce | 18.03.0~ce-0~ubuntu | https://download.docker.com/linux/ubuntu xenial/stable arm64 Packages docker-ce | 17.12.1~ce-0~ubuntu | https://download.docker.com/linux/ubuntu xenial/stable arm64 Packages docker-ce | 17.12.0~ce-0~ubuntu | https://download.docker.com/linux/ubuntu xenial/stable arm64 Packages docker-ce | 17.09.1~ce-0~ubuntu | https://download.docker.com/linux/ubuntu xenial/stable arm64 Packages docker-ce | 17.09.0~ce-0~ubuntu | https://download.docker.com/linux/ubuntu xenial/stable arm64 Packages
From the Output.1 above, we see the latest package for Docker 18.09 is 5:18.09.9~3-0~ubuntu-xenial.
Next, we need to install the choosen version of Docker. On each of the nodes my-n2-1 thru my-n2-5, execute the following command:
$ sudo apt-get install docker-ce=5:18.09.9~3-0~ubuntu-xenial -y
The following would be a typical output:
Reading package lists... Done Building dependency tree Reading state information... Done The following additional packages will be installed: aufs-tools cgroupfs-mount containerd.io docker-ce-cli git git-man liberror-perl pigz Suggested packages: git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui gitk gitweb git-cvs git-mediawiki git-svn The following NEW packages will be installed: aufs-tools cgroupfs-mount containerd.io docker-ce docker-ce-cli git git-man liberror-perl pigz 0 upgraded, 9 newly installed, 0 to remove and 0 not upgraded. Need to get 61.3 MB of archives. After this operation, 325 MB of additional disk space will be used. Get:1 https://download.docker.com/linux/ubuntu xenial/stable arm64 containerd.io arm64 1.2.10-3 [14.5 MB] Get:2 http://ports.ubuntu.com/ubuntu-ports bionic/universe arm64 pigz arm64 2.4-1 [47.8 kB] Get:3 http://ports.ubuntu.com/ubuntu-ports bionic/universe arm64 aufs-tools arm64 1:4.9+20170918-1ubuntu1 [101 kB] Get:4 http://ports.ubuntu.com/ubuntu-ports bionic/universe arm64 cgroupfs-mount all 1.4 [6320 B] Get:5 http://ports.ubuntu.com/ubuntu-ports bionic/main arm64 liberror-perl all 0.17025-1 [22.8 kB] Get:6 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main arm64 git-man all 1:2.17.1-1ubuntu0.4 [803 kB] Get:7 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main arm64 git arm64 1:2.17.1-1ubuntu0.4 [2941 kB] Get:8 https://download.docker.com/linux/ubuntu xenial/stable arm64 docker-ce-cli arm64 5:19.03.5~3-0~ubuntu-xenial [29.6 MB] Get:9 https://download.docker.com/linux/ubuntu xenial/stable arm64 docker-ce arm64 5:18.09.9~3-0~ubuntu-xenial [13.3 MB] Fetched 61.3 MB in 5s (11.6 MB/s) Selecting previously unselected package pigz. (Reading database ... 156190 files and directories currently installed.) Preparing to unpack .../0-pigz_2.4-1_arm64.deb ... Unpacking pigz (2.4-1) ... Selecting previously unselected package aufs-tools. Preparing to unpack .../1-aufs-tools_1%3a4.9+20170918-1ubuntu1_arm64.deb ... Unpacking aufs-tools (1:4.9+20170918-1ubuntu1) ... Selecting previously unselected package cgroupfs-mount. Preparing to unpack .../2-cgroupfs-mount_1.4_all.deb ... Unpacking cgroupfs-mount (1.4) ... Selecting previously unselected package containerd.io. Preparing to unpack .../3-containerd.io_1.2.10-3_arm64.deb ... Unpacking containerd.io (1.2.10-3) ... Selecting previously unselected package docker-ce-cli. Preparing to unpack .../4-docker-ce-cli_5%3a19.03.5~3-0~ubuntu-xenial_arm64.deb ... Unpacking docker-ce-cli (5:19.03.5~3-0~ubuntu-xenial) ... Selecting previously unselected package docker-ce. Preparing to unpack .../5-docker-ce_5%3a18.09.9~3-0~ubuntu-xenial_arm64.deb ... Unpacking docker-ce (5:18.09.9~3-0~ubuntu-xenial) ... Selecting previously unselected package liberror-perl. Preparing to unpack .../6-liberror-perl_0.17025-1_all.deb ... Unpacking liberror-perl (0.17025-1) ... Selecting previously unselected package git-man. Preparing to unpack .../7-git-man_1%3a2.17.1-1ubuntu0.4_all.deb ... Unpacking git-man (1:2.17.1-1ubuntu0.4) ... Selecting previously unselected package git. Preparing to unpack .../8-git_1%3a2.17.1-1ubuntu0.4_arm64.deb ... Unpacking git (1:2.17.1-1ubuntu0.4) ... Setting up aufs-tools (1:4.9+20170918-1ubuntu1) ... Setting up git-man (1:2.17.1-1ubuntu0.4) ... Setting up containerd.io (1.2.10-3) ... Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service → /lib/systemd/system/containerd.service. Setting up liberror-perl (0.17025-1) ... Setting up cgroupfs-mount (1.4) ... Setting up docker-ce-cli (5:19.03.5~3-0~ubuntu-xenial) ... Setting up pigz (2.4-1) ... Setting up git (1:2.17.1-1ubuntu0.4) ... Setting up docker-ce (5:18.09.9~3-0~ubuntu-xenial) ... update-alternatives: using /usr/bin/dockerd-ce to provide /usr/bin/dockerd (dockerd) in auto mode Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /lib/systemd/system/docker.service. Created symlink /etc/systemd/system/sockets.target.wants/docker.socket → /lib/systemd/system/docker.socket. Processing triggers for systemd (237-3ubuntu10.33) ... Processing triggers for man-db (2.8.3-2ubuntu0.1) ... Processing triggers for libc-bin (2.27-3ubuntu1) ...
Next, we need to ensure we are able to execute the Docker commands as the logged in user without the need for sudo. On each of the nodes my-n2-1 thru my-n2-5, execute the following commands:
$ sudo usermod -aG docker $USER
$ sudo reboot now
Once again, in each of the Terminal tabs, ssh into the corresponding node.
To verify the Docker installation, on each of the nodes my-n2-1 thru my-n2-5, execute the following command:
$ docker info
The following would be a typical output:
Client: Debug Mode: false Server: Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 0 Server Version: 18.09.9 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657 init version: fec3683 Security Options: seccomp Profile: default Kernel Version: 4.9.196-meson64 Operating System: Ubuntu 18.04.3 LTS OSType: linux Architecture: aarch64 CPUs: 6 Total Memory: 3.623GiB Name: my-n2-1 ID: QF32:QDZN:IQDM:34HX:NK3C:O3AP:Y6JZ:74DV:XXXL:KCBL:7K5D:36B4 Docker Root Dir: /var/lib/docker Debug Mode: false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false Product License: Community Engine
Next, we need to setup the package repository for Kubernetes. On each of the nodes my-n2-1 thru my-n2-5, execute the following commands:
$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
$ echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
$ sudo apt-get update
Next, we need to install Kubernetes. On each of the nodes my-n2-1 thru my-n2-5, execute the following command:
$ sudo apt-get install -y kubeadm
The following would be a typical output:
Reading package lists... Done Building dependency tree Reading state information... Done The following additional packages will be installed: conntrack cri-tools ebtables kubectl kubelet kubernetes-cni socat The following NEW packages will be installed: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 0 upgraded, 8 newly installed, 0 to remove and 1 not upgraded. Need to get 48.3 MB of archives. After this operation, 280 MB of additional disk space will be used. Get:2 http://ports.ubuntu.com/ubuntu-ports bionic/main arm64 conntrack arm64 1:1.4.4+snapshot20161117-6ubuntu2 [27.3 kB] Get:7 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main arm64 ebtables arm64 2.0.10.4-3.5ubuntu2.18.04.3 [74.2 kB] Get:8 http://ports.ubuntu.com/ubuntu-ports bionic/main arm64 socat arm64 1.7.3.2-2ubuntu2 [322 kB] Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main arm64 cri-tools arm64 1.13.0-00 [7965 kB] Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main arm64 kubernetes-cni arm64 0.7.5-00 [5808 kB] Get:4 https://packages.cloud.google.com/apt kubernetes-xenial/main arm64 kubelet arm64 1.16.3-00 [18.5 MB] Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main arm64 kubectl arm64 1.16.3-00 [8025 kB] Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main arm64 kubeadm arm64 1.16.3-00 [7652 kB] Fetched 48.3 MB in 5s (9383 kB/s) Selecting previously unselected package conntrack. (Reading database ... 157399 files and directories currently installed.) Preparing to unpack .../0-conntrack_1%3a1.4.4+snapshot20161117-6ubuntu2_arm64.deb ... Unpacking conntrack (1:1.4.4+snapshot20161117-6ubuntu2) ... Selecting previously unselected package cri-tools. Preparing to unpack .../1-cri-tools_1.13.0-00_arm64.deb ... Unpacking cri-tools (1.13.0-00) ... Selecting previously unselected package ebtables. Preparing to unpack .../2-ebtables_2.0.10.4-3.5ubuntu2.18.04.3_arm64.deb ... Unpacking ebtables (2.0.10.4-3.5ubuntu2.18.04.3) ... Selecting previously unselected package kubernetes-cni. Preparing to unpack .../3-kubernetes-cni_0.7.5-00_arm64.deb ... Unpacking kubernetes-cni (0.7.5-00) ... Selecting previously unselected package socat. Preparing to unpack .../4-socat_1.7.3.2-2ubuntu2_arm64.deb ... Unpacking socat (1.7.3.2-2ubuntu2) ... Selecting previously unselected package kubelet. Preparing to unpack .../5-kubelet_1.16.3-00_arm64.deb ... Unpacking kubelet (1.16.3-00) ... Selecting previously unselected package kubectl. Preparing to unpack .../6-kubectl_1.16.3-00_arm64.deb ... Unpacking kubectl (1.16.3-00) ... Selecting previously unselected package kubeadm. Preparing to unpack .../7-kubeadm_1.16.3-00_arm64.deb ... Unpacking kubeadm (1.16.3-00) ... Setting up conntrack (1:1.4.4+snapshot20161117-6ubuntu2) ... Setting up kubernetes-cni (0.7.5-00) ... Setting up cri-tools (1.13.0-00) ... Setting up socat (1.7.3.2-2ubuntu2) ... Setting up ebtables (2.0.10.4-3.5ubuntu2.18.04.3) ... Created symlink /etc/systemd/system/multi-user.target.wants/ebtables.service → /lib/systemd/system/ebtables.service. update-rc.d: warning: start and stop actions are no longer supported; falling back to defaults Setting up kubectl (1.16.3-00) ... Setting up kubelet (1.16.3-00) ... Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service → /lib/systemd/system/kubelet.service. Setting up kubeadm (1.16.3-00) ... Processing triggers for man-db (2.8.3-2ubuntu0.1) ... Processing triggers for systemd (237-3ubuntu10.33) ...
We need to reboot all the nodes. On each of the nodes my-n2-1 thru my-n2-5, execute the following command:
$ sudo reboot now
Once again, in each of the Terminal tabs, ssh into the corresponding node.
To verify the Kubernetes installation, on each of the nodes my-n2-1 thru my-n2-5, execute the following command:
$ kubeadm version
The following would be a typical output:
kubeadm version: &version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:20:25Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/arm64"}
Next, we need to ensure the packages for Docker and Kubernetes are not updated in the future by the software update process. On each of the nodes my-n2-1 thru my-n2-5, execute the following command:
$ sudo apt-mark hold kubelet kubeadm kubectl docker-ce
The following would be a typical output:
kubelet set on hold. kubeadm set on hold. kubectl set on hold. docker-ce set on hold.
By default, Docker uses cgroupfs as the cgroup driver. Kubernetes prefers systemd as the cgroup driver. We need to modify the Docker daemon configuration by specifying options in a JSON file called /etc/docker/daemon.json. On each of the nodes my-n2-1 thru my-n2-5, create the configuration file /etc/docker/daemon.json with the following contents:
Next, we need to restart the Docker daemon for the configuration to take effect. On each of the nodes my-n2-1 thru my-n2-5, execute the following commands:
$ sudo mkdir -p /etc/systemd/system/docker.service.d
$ sudo systemctl daemon-reload
$ sudo systemctl restart docker
Not using the systemd cgroup driver will cause the following error:
[preflight] Running pre-flight checks [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
To verify the Docker daemon started ok, on each of the nodes my-n2-1 thru my-n2-5, execute the following command:
$ journalctl -u docker
The following would be a typical output:
-- Logs begin at Sat 2019-12-14 21:14:19 EST, end at Sat 2019-12-14 21:49:26 EST. --
Dec 14 21:14:26 my-n2-1 systemd[1]: Starting Docker Application Container Engine...
Dec 14 21:14:27 my-n2-1 dockerd[3347]: time="2019-12-14T21:14:27.806496732-05:00" level=info msg="systemd-resolved is running, so using resolvconf: /run/systemd/res
Dec 14 21:14:27 my-n2-1 dockerd[3347]: time="2019-12-14T21:14:27.821800611-05:00" level=info msg="parsed scheme: \"unix\"" module=grpc
Dec 14 21:14:27 my-n2-1 dockerd[3347]: time="2019-12-14T21:14:27.822661404-05:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module
Dec 14 21:14:27 my-n2-1 dockerd[3347]: time="2019-12-14T21:14:27.824226106-05:00" level=info msg="parsed scheme: \"unix\"" module=grpc
Dec 14 21:14:27 my-n2-1 dockerd[3347]: time="2019-12-14T21:14:27.824838344-05:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module
Dec 14 21:14:27 my-n2-1 dockerd[3347]: time="2019-12-14T21:14:27.828116839-05:00" level=info msg="ccResolverWrapper: sending new addresses to cc: [{unix:///run/cont
Dec 14 21:14:27 my-n2-1 dockerd[3347]: time="2019-12-14T21:14:27.828945714-05:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Dec 14 21:14:27 my-n2-1 dockerd[3347]: time="2019-12-14T21:14:27.828101672-05:00" level=info msg="ccResolverWrapper: sending new addresses to cc: [{unix:///run/cont
Dec 14 21:14:27 my-n2-1 dockerd[3347]: time="2019-12-14T21:14:27.830093104-05:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Dec 14 21:14:27 my-n2-1 dockerd[3347]: time="2019-12-14T21:14:27.832076285-05:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0x400014e610, CONNECT
Dec 14 21:14:27 my-n2-1 dockerd[3347]: time="2019-12-14T21:14:27.844251802-05:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0x40001343a0, CONNECT
Dec 14 21:14:27 my-n2-1 dockerd[3347]: time="2019-12-14T21:14:27.846949059-05:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0x40001343a0, READY"
Dec 14 21:14:27 my-n2-1 dockerd[3347]: time="2019-12-14T21:14:27.851896887-05:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0x400014e610, READY"
Dec 14 21:14:27 my-n2-1 dockerd[3347]: time="2019-12-14T21:14:27.857097768-05:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
Dec 14 21:14:27 my-n2-1 dockerd[3347]: time="2019-12-14T21:14:27.886090322-05:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Dec 14 21:14:27 my-n2-1 dockerd[3347]: time="2019-12-14T21:14:27.893602818-05:00" level=info msg="Loading containers: start."
Dec 14 21:14:28 my-n2-1 dockerd[3347]: time="2019-12-14T21:14:28.821256841-05:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0
Dec 14 21:14:29 my-n2-1 dockerd[3347]: time="2019-12-14T21:14:29.134364234-05:00" level=info msg="Loading containers: done."
Dec 14 21:14:29 my-n2-1 dockerd[3347]: time="2019-12-14T21:14:29.374311397-05:00" level=info msg="Docker daemon" commit=039a7df graphdriver(s)=overlay2 version=18.0
Dec 14 21:14:29 my-n2-1 dockerd[3347]: time="2019-12-14T21:14:29.376444960-05:00" level=info msg="Daemon has completed initialization"
Dec 14 21:14:29 my-n2-1 systemd[1]: Started Docker Application Container Engine.
Dec 14 21:14:29 my-n2-1 dockerd[3347]: time="2019-12-14T21:14:29.444607195-05:00" level=info msg="API listen on /var/run/docker.sock"
Dec 14 21:49:11 my-n2-1 dockerd[3347]: time="2019-12-14T21:49:11.323542665-05:00" level=info msg="Processing signal 'terminated'"
Dec 14 21:49:11 my-n2-1 dockerd[3347]: time="2019-12-14T21:49:11.328379659-05:00" level=info msg="stopping event stream following graceful shutdown" error="" m
Dec 14 21:49:11 my-n2-1 systemd[1]: Stopping Docker Application Container Engine...
Dec 14 21:49:11 my-n2-1 systemd[1]: Stopped Docker Application Container Engine.
Dec 14 21:49:11 my-n2-1 systemd[1]: Starting Docker Application Container Engine...
Dec 14 21:49:11 my-n2-1 dockerd[9629]: time="2019-12-14T21:49:11.499488062-05:00" level=info msg="systemd-resolved is running, so using resolvconf: /run/systemd/res
Dec 14 21:49:11 my-n2-1 dockerd[9629]: time="2019-12-14T21:49:11.502141612-05:00" level=info msg="parsed scheme: \"unix\"" module=grpc
Dec 14 21:49:11 my-n2-1 dockerd[9629]: time="2019-12-14T21:49:11.502209240-05:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module
Dec 14 21:49:11 my-n2-1 dockerd[9629]: time="2019-12-14T21:49:11.502278577-05:00" level=info msg="parsed scheme: \"unix\"" module=grpc
Dec 14 21:49:11 my-n2-1 dockerd[9629]: time="2019-12-14T21:49:11.502295786-05:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module
Dec 14 21:49:11 my-n2-1 dockerd[9629]: time="2019-12-14T21:49:11.505887217-05:00" level=info msg="ccResolverWrapper: sending new addresses to cc: [{unix:///run/cont
Dec 14 21:49:11 my-n2-1 dockerd[9629]: time="2019-12-14T21:49:11.506035600-05:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Dec 14 21:49:11 my-n2-1 dockerd[9629]: time="2019-12-14T21:49:11.506181190-05:00" level=info msg="ccResolverWrapper: sending new addresses to cc: [{unix:///run/cont
Dec 14 21:49:11 my-n2-1 dockerd[9629]: time="2019-12-14T21:49:11.506446245-05:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Dec 14 21:49:11 my-n2-1 dockerd[9629]: time="2019-12-14T21:49:11.506671465-05:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0x40007a2230, CONNECT
Dec 14 21:49:11 my-n2-1 dockerd[9629]: time="2019-12-14T21:49:11.506255319-05:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0x40008b0710, CONNECT
Dec 14 21:49:11 my-n2-1 dockerd[9629]: time="2019-12-14T21:49:11.509814706-05:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0x40008b0710, READY"
Dec 14 21:49:11 my-n2-1 dockerd[9629]: time="2019-12-14T21:49:11.511738887-05:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0x40007a2230, READY"
Dec 14 21:49:11 my-n2-1 dockerd[9629]: time="2019-12-14T21:49:11.525913142-05:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Dec 14 21:49:11 my-n2-1 dockerd[9629]: time="2019-12-14T21:49:11.529808838-05:00" level=info msg="Loading containers: start."
Dec 14 21:49:12 my-n2-1 dockerd[9629]: time="2019-12-14T21:49:12.258591473-05:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0
Dec 14 21:49:12 my-n2-1 dockerd[9629]: time="2019-12-14T21:49:12.540886055-05:00" level=info msg="Loading containers: done."
Dec 14 21:49:12 my-n2-1 dockerd[9629]: time="2019-12-14T21:49:12.614462758-05:00" level=info msg="Docker daemon" commit=039a7df graphdriver(s)=overlay2 version=18.0
Dec 14 21:49:12 my-n2-1 dockerd[9629]: time="2019-12-14T21:49:12.614718313-05:00" level=info msg="Daemon has completed initialization"
Dec 14 21:49:12 my-n2-1 dockerd[9629]: time="2019-12-14T21:49:12.640530153-05:00" level=info msg="API listen on /var/run/docker.sock"
Dec 14 21:49:12 my-n2-1 systemd[1]: Started Docker Application Container Engine.
Next, we need to disable disk based swap. For that we need to perform two actions.
First action, on each of the nodes my-n2-1 thru my-n2-5, edit the file /etc/default/armbian-zram-config and change the line ENABLED=true to ENABLED=false.
Second action, on each of the nodes my-n2-1 thru my-n2-5, execute the following commands:
$ sudo systemctl disable armbian-zram-config
$ sudo reboot now
Once again, in each of the Terminal tabs, ssh into the corresponding node.
This completes the installation and system setup of the cluster nodes. Next stop - Kubernetes setup.
Kubernetes Setup
To get started, we will designate the node my-n2-1 as the master node and setup the control plane. To do that, execute the following command on my-n2-1:
$ sudo kubeadm init
The following would be a typical output:
[init] Using Kubernetes version: v1.16.3
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [my-n2-1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.1.51]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [my-n2-1 localhost] and IPs [192.168.1.51 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [my-n2-1 localhost] and IPs [192.168.1.51 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W1215 11:58:08.359442 4811 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W1215 11:58:08.366477 4811 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 25.513764 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.17" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node my-n2-1 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node my-n2-1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: zcp5a6.w03lcuhx068wvkqv
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.1.51:6443 --token zcp5a6.w03lcuhx068wvkqv \
--discovery-token-ca-cert-hash sha256:d2e38957f46a9eb089671924bca78ac4e02cdcc8db27e89677a014fe587b67c6
In order to use the kubectl command-line tool as a non-root user on the master node (my-n2-1), execute the following commands on my-n2-1:
$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
To list all the node(s) in Kubernetes cluster, execute the following command on the master node (my-n2-1):
$ kubectl get nodes
The following would be a typical output:
NAME STATUS ROLES AGE VERSION my-n2-1 NotReady master 2m37s v1.16.3
To verify the Kubernetes cluster started ok, execute the following command on the master node (my-n2-1):
$ kubectl get pods -n kube-system -o wide
The following would be a typical output:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES coredns-6955765f44-4gk4f 1/1 Running 0 40m 10.32.0.3 my-n2-1 <none> <none> coredns-6955765f44-wskl4 1/1 Running 0 40m 10.32.0.2 my-n2-1 <none> <none> etcd-my-n2-1 1/1 Running 0 40m 192.168.1.51 my-n2-1 <none> <none> kube-apiserver-my-n2-1 1/1 Running 0 40m 192.168.1.51 my-n2-1 <none> <none> kube-controller-manager-my-n2-1 1/1 Running 0 40m 192.168.1.51 my-n2-1 <none> <none> kube-proxy-tklp7 1/1 Running 0 40m 192.168.1.51 my-n2-1 <none> <none> kube-scheduler-my-n2-1 1/1 Running 0 40m 192.168.1.51 my-n2-1 <none> <none>
From the Output.10 above, we can see all the core components (api server, controller manager, etcd, and scheduler) are all up and running.
Now, we need to install an overlay Plugin Network for inter-pod communication. For our cluster, we will choose the weave-net implementation. To install the overlay network on the master node (my-n2-1), execute the following command:
$ kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
The following would be a typical output:
serviceaccount/weave-net created clusterrole.rbac.authorization.k8s.io/weave-net created clusterrolebinding.rbac.authorization.k8s.io/weave-net created role.rbac.authorization.k8s.io/weave-net created rolebinding.rbac.authorization.k8s.io/weave-net created daemonset.apps/weave-net created
To verify the Weave overlay network started ok, execute the following command on the master node (my-n2-1):
$ kubectl get pods -n kube-system -l name=weave-net -o wide
The following would be a typical output:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES weave-net-2sjh4 2/2 Running 0 10m 192.168.1.51 my-n2-1 <none> <none>
Additionally, to check the logs for the Weave overlay network, execute the following command on the master node (my-n2-1):
$ kubectl logs -n kube-system weave-net-ktjnv weave
The following would be a typical output:
INFO: 2019/12/08 17:07:12.422554 Command line options: map[conn-limit:200 datapath:datapath db-prefix:/weavedb/weave-net docker-api: expect-npc:true host-root:/host http-addr:127.0.0.1:6784 ipalloc-init:consensus=0 ipalloc-range:10.32.0.0/12 metrics-addr:0.0.0.0:6782 name:9a:59:d0:9a:83:f0 nickname:my-n2-1 no-dns:true port:6783]
INFO: 2019/12/08 17:07:12.422876 weave 2.6.0
INFO: 2019/12/08 17:07:12.780249 Bridge type is bridged_fastdp
INFO: 2019/12/08 17:07:12.780350 Communication between peers is unencrypted.
INFO: 2019/12/08 17:07:12.804023 Our name is 9a:59:d0:9a:83:f0(my-n2-1)
INFO: 2019/12/08 17:07:12.804267 Launch detected - using supplied peer list: []
INFO: 2019/12/08 17:07:12.844222 Unable to fetch ConfigMap kube-system/weave-net to infer unique cluster ID
INFO: 2019/12/08 17:07:12.844324 Checking for pre-existing addresses on weave bridge
INFO: 2019/12/08 17:07:12.853900 [allocator 9a:59:d0:9a:83:f0] No valid persisted data
INFO: 2019/12/08 17:07:12.866497 [allocator 9a:59:d0:9a:83:f0] Initialising via deferred consensus
INFO: 2019/12/08 17:07:12.866684 Sniffing traffic on datapath (via ODP)
INFO: 2019/12/08 17:07:12.872570 Listening for HTTP control messages on 127.0.0.1:6784
INFO: 2019/12/08 17:07:12.873074 Listening for metrics requests on 0.0.0.0:6782
INFO: 2019/12/08 17:07:13.540248 [kube-peers] Added myself to peer list &{[{9a:59:d0:9a:83:f0 my-n2-1}]}
DEBU: 2019/12/08 17:07:13.558983 [kube-peers] Nodes that have disappeared: map[]
INFO: 2019/12/08 17:07:13.661165 Assuming quorum size of 1
10.32.0.1
DEBU: 2019/12/08 17:07:13.911144 registering for updates for node delete events
For this tutorial, we designate that nodes my-n2-2 thru my-n2-5 to be the worker nodes of this Kubernetes cluster. From Output.8 above, we can determine the kubeadm join command to use on each worker node . For each of the nodes my-n2-2 thru my-n2-5 (in their respective Terminal tab), execute the following command:
$ sudo kubeadm join 192.168.1.51:6443 --token zcp5a6.w03lcuhx068wvkqv --discovery-token-ca-cert-hash sha256:d2e38957f46a9eb089671924bca78ac4e02cdcc8db27e89677a014fe587b67c6
The following would be a typical output:
[preflight] Running pre-flight checks [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' [kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.17" ConfigMap in the kube-system namespace [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
To list all the active nodes in this Kubernetes cluster, execute the following command on the master node (my-n2-1) (after waiting for about 30 secs ):
$ kubectl get nodes -o wide
The following would be a typical output:
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME my-n2-1 Ready master 51m v1.17.0 192.168.1.51 <none> Ubuntu 18.04.3 LTS 4.9.196-meson64 docker://18.9.9 my-n2-2 Ready <none> 2m58s v1.17.0 192.168.1.52 <none> Ubuntu 18.04.3 LTS 4.9.196-meson64 docker://18.9.9 my-n2-3 Ready <none> 2m38s v1.17.0 192.168.1.53 <none> Ubuntu 18.04.3 LTS 4.9.196-meson64 docker://18.9.9 my-n2-4 Ready <none> 2m35s v1.17.0 192.168.1.54 <none> Ubuntu 18.04.3 LTS 4.9.196-meson64 docker://18.9.9 my-n2-5 Ready <none> 2m21s v1.17.0 192.168.1.55 <none> Ubuntu 18.04.3 LTS 4.9.196-meson64 docker://18.9.9
Thats it !!! This completes all the necessary setup for this Kubernetes cluster.
Hands-on with Kubernetes
To list all the pod(s) running in Kubernetes cluster (including the system pods), execute the following command on the master node (my-n2-1):
$ kubectl get pods --all-namespaces -o wide
The following would be a typical output:
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system coredns-6955765f44-4gk4f 1/1 Running 0 52m 10.32.0.3 my-n2-1 <none> <none> kube-system coredns-6955765f44-wskl4 1/1 Running 0 52m 10.32.0.2 my-n2-1 <none> <none> kube-system etcd-my-n2-1 1/1 Running 0 52m 192.168.1.51 my-n2-1 <none> <none> kube-system kube-apiserver-my-n2-1 1/1 Running 0 52m 192.168.1.51 my-n2-1 <none> <none> kube-system kube-controller-manager-my-n2-1 1/1 Running 0 52m 192.168.1.51 my-n2-1 <none> <none> kube-system kube-proxy-9zxfj 1/1 Running 0 3m36s 192.168.1.55 my-n2-5 <none> <none> kube-system kube-proxy-c7mns 1/1 Running 0 3m53s 192.168.1.53 my-n2-3 <none> <none> kube-system kube-proxy-dv52p 1/1 Running 0 4m13s 192.168.1.52 my-n2-2 <none> <none> kube-system kube-proxy-mpwkb 1/1 Running 0 3m50s 192.168.1.54 my-n2-4 <none> <none> kube-system kube-proxy-tklp7 1/1 Running 0 52m 192.168.1.51 my-n2-1 <none> <none> kube-system kube-scheduler-my-n2-1 1/1 Running 0 52m 192.168.1.51 my-n2-1 <none> <none> kube-system weave-net-2sjh4 2/2 Running 0 21m 192.168.1.51 my-n2-1 <none> <none> kube-system weave-net-68lcd 2/2 Running 0 3m50s 192.168.1.54 my-n2-4 <none> <none> kube-system weave-net-7fh98 2/2 Running 1 4m13s 192.168.1.52 my-n2-2 <none> <none> kube-system weave-net-krdtz 2/2 Running 1 3m36s 192.168.1.55 my-n2-5 <none> <none> kube-system weave-net-ljm6k 2/2 Running 0 3m53s 192.168.1.53 my-n2-3 <none> <none>
As is evident from Output.16 above, we see an instance for API Server, etcd, Controller Manager, Scheduler, and Plugin Network (weave-net) all up and running.
To display detailed information about any pod (say the Controller Manager) in the Kubernetes cluster, execute the following command on the master node (my-n2-1):
$ kubectl describe pod kube-controller-manager-my-n2-1 -n kube-system
The following would be a typical output:
Name: kube-controller-manager-my-n2-1
Namespace: kube-system
Priority: 2000000000
Priority Class Name: system-cluster-critical
Node: my-n2-1/192.168.1.51
Start Time: Sun, 15 Dec 2019 11:58:39 -0500
Labels: component=kube-controller-manager
tier=control-plane
Annotations: kubernetes.io/config.hash: 536dc7132dfd0d2ca1d968c9ede1e024
kubernetes.io/config.mirror: 536dc7132dfd0d2ca1d968c9ede1e024
kubernetes.io/config.seen: 2019-12-15T11:58:35.86446527-05:00
kubernetes.io/config.source: file
Status: Running
IP: 192.168.1.51
IPs:
IP: 192.168.1.51
Controlled By: Node/my-n2-1
Containers:
kube-controller-manager:
Container ID: docker://63b0d105457f52849afa38d2e914b53e68b7e21786fc41cda322bb21bc5b86a4
Image: k8s.gcr.io/kube-controller-manager:v1.17.0
Image ID: docker-pullable://k8s.gcr.io/kube-controller-manager@sha256:0438efb5098a2ca634ea8c6b0d804742b733d0d13fd53cf62c73e32c659a3c39
Port: <none>
Host Port: <none>
Command:
kube-controller-manager
--authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
--authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
--bind-address=127.0.0.1
--client-ca-file=/etc/kubernetes/pki/ca.crt
--cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
--cluster-signing-key-file=/etc/kubernetes/pki/ca.key
--controllers=*,bootstrapsigner,tokencleaner
--kubeconfig=/etc/kubernetes/controller-manager.conf
--leader-elect=true
--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
--root-ca-file=/etc/kubernetes/pki/ca.crt
--service-account-private-key-file=/etc/kubernetes/pki/sa.key
--use-service-account-credentials=true
State: Running
Started: Sun, 15 Dec 2019 11:58:22 -0500
Ready: True
Restart Count: 0
Requests:
cpu: 200m
Liveness: http-get https://127.0.0.1:10257/healthz delay=15s timeout=15s period=10s #success=1 #failure=8
Environment: <none>
Mounts:
/etc/ca-certificates from etc-ca-certificates (ro)
/etc/kubernetes/controller-manager.conf from kubeconfig (ro)
/etc/kubernetes/pki from k8s-certs (ro)
/etc/ssl/certs from ca-certs (ro)
/usr/libexec/kubernetes/kubelet-plugins/volume/exec from flexvolume-dir (rw)
/usr/local/share/ca-certificates from usr-local-share-ca-certificates (ro)
/usr/share/ca-certificates from usr-share-ca-certificates (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
ca-certs:
Type: HostPath (bare host directory volume)
Path: /etc/ssl/certs
HostPathType: DirectoryOrCreate
etc-ca-certificates:
Type: HostPath (bare host directory volume)
Path: /etc/ca-certificates
HostPathType: DirectoryOrCreate
flexvolume-dir:
Type: HostPath (bare host directory volume)
Path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec
HostPathType: DirectoryOrCreate
k8s-certs:
Type: HostPath (bare host directory volume)
Path: /etc/kubernetes/pki
HostPathType: DirectoryOrCreate
kubeconfig:
Type: HostPath (bare host directory volume)
Path: /etc/kubernetes/controller-manager.conf
HostPathType: FileOrCreate
usr-local-share-ca-certificates:
Type: HostPath (bare host directory volume)
Path: /usr/local/share/ca-certificates
HostPathType: DirectoryOrCreate
usr-share-ca-certificates:
Type: HostPath (bare host directory volume)
Path: /usr/share/ca-certificates
HostPathType: DirectoryOrCreate
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: :NoExecute
Events: <none>
To list all the application pod(s) running in Kubernetes cluster, execute the following command on the master node (my-n2-1):
$ kubectl get pods
The following would be a typical output:
No resources found in default namespace.
To list all the service(s) running in Kubernetes cluster, execute the following command on the master node (my-n2-1):
$ kubectl get services
The following would be a typical output:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 64m
We will create a simple Python web application to display the host name as well as the ip-address when invoked via HTTP. The following are the contents of the simple Python web application stored under the /tmp directory on the master node (my-n2-1):
The following are the contents of the Dockerfile to create a Docker image for the the simple Python web application stored under the /tmp directory on the master node (my-n2-1):
To build a Docker image called py-web-echo with the tag v1.0, execute the following commands on the master node ( my-n2-1):
cd /tmp
docker build -t "py-web-echo:v1.0" .
The following would be a typical output:
Sending build context to Docker daemon 3.072kB Step 1/4 : FROM python:3.7.5-alpine3.9 3.7.5-alpine3.9: Pulling from library/python 0362ad1dd800: Pull complete 9b941924aae3: Pull complete fd7b3613915d: Pull complete 078d60b9b97e: Pull complete 7059e1dd9bc4: Pull complete Digest: sha256:064d9ce3e91a59535c528bc3c38888023791d9fc78ba9e5070f5064833f326ff Status: Downloaded newer image for python:3.7.5-alpine3.9 ---> 578ec6233872 Step 2/4 : RUN pip install flask ---> Running in d248e23dd161 Collecting flask Downloading https://files.pythonhosted.org/packages/9b/93/628509b8d5dc749656a9641f4caf13540e2cdec85276964ff8f43bbb1d3b/Flask-1.1.1-py2.py3-none-any.whl (94kB) Collecting Jinja2>=2.10.1 Downloading https://files.pythonhosted.org/packages/65/e0/eb35e762802015cab1ccee04e8a277b03f1d8e53da3ec3106882ec42558b/Jinja2-2.10.3-py2.py3-none-any.whl (125kB) Collecting Werkzeug>=0.15 Downloading https://files.pythonhosted.org/packages/ce/42/3aeda98f96e85fd26180534d36570e4d18108d62ae36f87694b476b83d6f/Werkzeug-0.16.0-py2.py3-none-any.whl (327kB) Collecting itsdangerous>=0.24 Downloading https://files.pythonhosted.org/packages/76/ae/44b03b253d6fade317f32c24d100b3b35c2239807046a4c953c7b89fa49e/itsdangerous-1.1.0-py2.py3-none-any.whl Collecting click>=5.1 Downloading https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl (81kB) Collecting MarkupSafe>=0.23 Downloading https://files.pythonhosted.org/packages/b9/2e/64db92e53b86efccfaea71321f597fa2e1b2bd3853d8ce658568f7a13094/MarkupSafe-1.1.1.tar.gz Building wheels for collected packages: MarkupSafe Building wheel for MarkupSafe (setup.py): started Building wheel for MarkupSafe (setup.py): finished with status 'done' Created wheel for MarkupSafe: filename=MarkupSafe-1.1.1-cp37-none-any.whl size=12629 sha256=8a200864ca113d03b4de2d951ae4a1d0806a3ff84128349770dfe3fb018a6458 Stored in directory: /root/.cache/pip/wheels/f2/aa/04/0edf07a1b8a5f5f1aed7580fffb69ce8972edc16a505916a77 Successfully built MarkupSafe Installing collected packages: MarkupSafe, Jinja2, Werkzeug, itsdangerous, click, flask Successfully installed Jinja2-2.10.3 MarkupSafe-1.1.1 Werkzeug-0.16.0 click-7.0 flask-1.1.1 itsdangerous-1.1.0 Removing intermediate container d248e23dd161 ---> 4ee40e66a655 Step 3/4 : ADD web-echo.py /web-echo.py ---> 31a0341bf9d7 Step 4/4 : CMD ["python", "/web-echo.py"] ---> Running in 1ee52ea10ad3 Removing intermediate container 1ee52ea10ad3 ---> 7cd037d24ef7 Successfully built 7cd037d24ef7 Successfully tagged py-web-echo:v1.0
To list all the Docker images on the master node (my-n2-1), execute the following command on the master node (my-n2-1):
$ docker images
The following would be a typical output:
REPOSITORY TAG IMAGE ID CREATED SIZE py-web-echo v1.0 7cd037d24ef7 3 minutes ago 119MB k8s.gcr.io/kube-proxy v1.17.0 ac19e9cffff5 7 days ago 114MB k8s.gcr.io/kube-apiserver v1.17.0 aca151bf3e90 7 days ago 166MB k8s.gcr.io/kube-controller-manager v1.17.0 7045158f92f8 7 days ago 156MB k8s.gcr.io/kube-scheduler v1.17.0 0d5c120f87f3 7 days ago 93.7MB python 3.7.5-alpine3.9 578ec6233872 4 weeks ago 109MB weaveworks/weave-npc 2.6.0 1c672c2f5870 5 weeks ago 36.6MB weaveworks/weave-kube 2.6.0 81393394d17d 5 weeks ago 111MB k8s.gcr.io/coredns 1.6.5 f96217e2532b 5 weeks ago 39.3MB k8s.gcr.io/etcd 3.4.3-0 ab707b0a0ea3 7 weeks ago 363MB k8s.gcr.io/pause 3.1 6cf7c80fe444 24 months ago 525kB
Note that we built the Docker image on the master node (my-n2-1). Since the pod(s) will be deployed on the worker node(s), we need to ensure the requisite docker images are present in the worker node(s).
For each of the worker nodes my-n2-2 thru my-n2-5 (in their respective Terminal tab), execute the following command:
$ docker pull python:3.7.5-alpine3.9
For each of the worker nodes my-n2-2 thru my-n2-5, execute the following command on the master node (my-n2-1):
$ docker save py-web-echo:v1.0 | bzip2 | ssh polarsparc@192.168.1.52 'bunzip2 | docker load'
$ docker save py-web-echo:v1.0 | bzip2 | ssh polarsparc@192.168.1.53 'bunzip2 | docker load'
$ docker save py-web-echo:v1.0 | bzip2 | ssh polarsparc@192.168.1.54 'bunzip2 | docker load'
$ docker save py-web-echo:v1.0 | bzip2 | ssh polarsparc@192.168.1.55 'bunzip2 | docker load'
Not having the Docker images in the worker node(s) will cause the pod(s) to be stuck in the ContainerCreating status
In Kubernetes, a pod is what encapsulates Docker container(s). To deploy our web application Docker image py-web-echo:v1.0 in our Kubernetes cluster, we need a pod manifest file in YAML format .
The following are the contents of the pod manifest file called web-echo-pod.yaml stored under the /tmp directory on the master node (my-n2-1):
The following section explains the elements of the web-echo-pod.yaml manifest file:
apiVersion :: specifies the version of the API (v1 in this example)
kind :: specifies the type of Kubernetes object to deploy (Pod in this example)
metadata :: associates a name ( web-echo-pod in this example) with the type of Kubernetes object. Also, allows one to tag some labels, which are simple key-value pairs, with the Kubernetes object. In this example, we have one label with the key app that has a value of web-echo
spec :: specifies what is in the pod. In this example, we want to deploy the Docker image py-web-echo:v1.0 which is exposed via the network port 8888
imagePullPolicy :: indicates to Kubernetes not to pull the container image
To deploy the pod to our Kubernetes cluster, execute the following command on the master node (my-n2-1):
$ kubectl apply -f /tmp/web-echo-pod.yaml
The following would be a typical output:
pod/web-echo-pod created
To list all the application pod(s) running in Kubernetes cluster, execute the following command on the master node (my-n2-1):
$ kubectl get pods -o wide
The following would be a typical output:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES web-echo-pod 1/1 Running 0 2m26s 10.44.0.1 my-n2-2 <none> <none>
From Output.23, we see that our application pod have been deployed on the node my-n2-2 of our Kubernetes cluster.
To display detailed information about the deployed application pod web-echo-pod, execute the following command on the master node (my-n2-1):
$ kubectl describe pods web-echo-pod
The following would be a typical output:
Name: web-echo-pod
Namespace: default
Priority: 0
Node: my-n2-2/192.168.1.52
Start Time: Sun, 15 Dec 2019 14:58:21 -0500
Labels: app=web-echo
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"labels":{"app":"web-echo"},"name":"web-echo-pod","namespace":"default"},"spe...
Status: Running
IP: 10.44.0.1
IPs:
IP: 10.44.0.1
Containers:
web-echo:
Container ID: docker://0af2c99fd074b5ee3c0b9876eb9ad44ca446400c2190b4af6fa1a18543bff723
Image: py-web-echo:v1.0
Image ID: docker://sha256:7cd037d24ef7c842ffe005cfcb548a802fc13661c08c8bb4635c365f77e5a3aa
Port: 8888/TCP
Host Port: 0/TCP
State: Running
Started: Sun, 15 Dec 2019 14:58:23 -0500
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-tvl5x (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
default-token-tvl5x:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-tvl5x
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 7m39s default-scheduler Successfully assigned default/web-echo-pod to my-n2-2
Normal Pulled 7m38s kubelet, my-n2-2 Container image "py-web-echo:v1.0" already present on machine
Normal Created 7m38s kubelet, my-n2-2 Created container web-echo
Normal Started 7m37s kubelet, my-n2-2 Started container web-echo
From the Output.23 (as well as Output.24) above, we see the ip-address of the deployed web application to be 10.44.0.1.
To test the deployed web application using the curl command, execute the following command on any of the nodes my-n2-1 through my-n2-5:
$ curl http://10.44.0.1:8888
The following would be a typical output:
Hello from container -> web-echo-pod [10.44.0.1]
To display the logs of the deployed web application web-echo-pod, execute the following command on the master node (my-n2-1):
$ kubectl logs web-echo-pod
The following would be a typical output:
* Serving Flask app "web-echo" (lazy loading) * Environment: production WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Debug mode: off * Running on http://0.0.0.0:8888/ (Press CTRL+C to quit) 10.32.0.1 - - [15/Dec/2019 20:11:33] "GET / HTTP/1.1" 200 - 10.36.0.0 - - [15/Dec/2019 20:11:58] "GET / HTTP/1.1" 200 -
To delete the deployed web application web-echo-pod, execute the following command on the master node (my-n2-1):
$ kubectl delete pod web-echo-pod
The following would be a typical output:
pod "web-echo-pod" deleted
It is *NOT* that common to deploy a single Pod. It is more common to deploy a higher level Kubernetes object called a ReplicaSet . A ReplicaSet defines how many replicas of a Pod need to be deployed and maintained in the Kubernetes cluster.
The following are the contents of the ReplicaSet manifest file called web-echo-rs.yaml stored under the /tmp directory on the master node (my-n2-1):
The following section explains some of the elements of the web-echo-rs.yaml manifest file:
apiVersion :: specifies the version of the API (apps/v1 in this example)
replicas :: indicates the desired instances of the Pod to be running in the Kubernetes cluster
selector :: identifies and selects a group of Kubernetes objects with the same key-value label (key app and value web-echo in this example)
template :: is the embedded specification for a Pod
To deploy the ReplicaSet to our Kubernetes cluster, execute the following command on the master node (my-n2-1):
$ kubectl apply -f /tmp/web-echo-rs.yaml
The following would be a typical output:
replicaset.apps/web-echo-rs created
To list all the deployed ReplicaSet(s) running in Kubernetes cluster, execute the following command on the master node (my-n2-1 ):
$ kubectl get replicasets -o wide
The following would be a typical output:
NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR web-echo-rs 3 3 3 7m web-echo py-web-echo:v1.0 app=web-echo
To display detailed information about the deployed ReplicaSet named web-echo-rs, execute the following command on the master node ( my-n2-1):
$ kubectl describe replicasets web-echo-rs
The following would be a typical output:
Name: web-echo-rs
Namespace: default
Selector: app=web-echo
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"apps/v1","kind":"ReplicaSet","metadata":{"annotations":{},"name":"web-echo-rs","namespace":"default"},"spec":{"replicas":3,...
Replicas: 3 current / 3 desired
Pods Status: 3 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
Labels: app=web-echo
Containers:
web-echo:
Image: py-web-echo:v1.0
Port: 8888/TCP
Host Port: 0/TCP
Environment: <none>
Mounts: <none>
Volumes: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 14m replicaset-controller Created pod: web-echo-rs-xn94l
Normal SuccessfulCreate 14m replicaset-controller Created pod: web-echo-rs-9x9b9
Normal SuccessfulCreate 14m replicaset-controller Created pod: web-echo-rs-tbd49
To list all the application pod(s) running in Kubernetes cluster, execute the following command on the master node (my-n2-1):
$ kubectl get pods -o wide
The following would be a typical output:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES web-echo-rs-9x9b9 1/1 Running 0 63s 10.42.0.1 my-n2-4 <none> <none> web-echo-rs-tbd49 1/1 Running 0 63s 10.44.0.1 my-n2-2 <none> <none> web-echo-rs-xn94l 1/1 Running 0 63s 10.36.0.1 my-n2-3 <none> <none>
From Output.31, we see that our application pod(s) have been deployed on the 3 nodes my-n2-2, my-n2-3, and my-n2-4 with unique ip-addresses of 10.44.0.1, 10.36.0.1, and 10.42.0.1 respectively.
As indicated early on, application pod(s) are ephemeral. They can come up and go at any time. This means their ip-address(es) can change any time. We need a higher level abstraction that provides a stable ip-address for other application pod(s) to use. This is where a Service object comes in handy. It provides a single stable ip-address for other applications to use and distributes the load across the different backend application pod(s) it is fronting.
There are 3 types of Service(s) in Kubernetes:
ClusterIP :: exposes the Service on an ip-address that is internal to the Kubernetes cluster. This means the Service is accessible from *ONLY* within the Kubernetes cluster. This is the default type
NodePort :: exposes the Service on each worker node's ip-address at a high port in the range 30000 to 32767. Applications external to the Kubernetes cluster are be able to access the Service at the worker node's ip-address and the assigned node port
LoadBalancer :: exposes the Service externally using a cloud providers Load Balancer such as AWS, Azure, or Google Cloud
The following are the contents of the ClusterIP based Service manifest file called web-echo-svc-cip.yaml stored under the /tmp directory on the master node (my-n2-1):
To deploy the Service to our Kubernetes cluster, execute the following command on the master node (my-n2-1):
$ kubectl apply -f /tmp/web-echo-svc-cip.yaml
The following would be a typical output:
service/web-echo-svc created
To list all the Service(s) running in Kubernetes cluster, execute the following command on the master node (my-n2-1):
$ kubectl get services -o wide
The following would be a typical output:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 9h <none> web-echo-svc ClusterIP 10.96.238.16 <none> 8888/TCP 105s app=web-echo
From the Output.33 above, we see the application web-echo can be accessed from anywhere in the cluster via the ip-address 10.96.238.16 and port 8888.
To test the deployed Service endpoint using the curl command, execute the following command 5 times on any of the nodes my-n2-1 through my-n2-5:
$ curl http://10.96.238.16:8888
The following would be a typical output:
Hello from container -> web-echo-rs-xn94l [10.36.0.1] Hello from container -> web-echo-rs-9x9b9 [10.42.0.1] Hello from container -> web-echo-rs-tbd49 [10.44.0.1] Hello from container -> web-echo-rs-9x9b9 [10.42.0.1] Hello from container -> web-echo-rs-tbd49 [10.44.0.1]
To display detailed information about the Service endpoint labeled web-echo-svc, execute the following command on the master node ( my-n2-1):
$ kubectl describe service web-echo-svc
The following would be a typical output:
Name: web-echo-svc
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"web-echo-svc","namespace":"default"},"spec":{"ports":[{"name":"ht...
Selector: app=web-echo
Type: ClusterIP
IP: 10.96.238.16
Port: http 8888/TCP
TargetPort: 8888/TCP
Endpoints: 10.36.0.1:8888,10.42.0.1:8888,10.44.0.1:8888
Session Affinity: None
Events: <none>
To delete the deployed web-echo-svc object, execute the following command on the master node (my-n2-1):
$ kubectl delete service web-echo-svc
The following would be a typical output:
service "web-echo-svc" deleted
The following are the contents of the NodePort based Service manifest file called web-echo-svc-nop.yaml stored under the /tmp directory on the master node (my-n2-1):
To deploy the Service to our Kubernetes cluster, execute the following command on the master node (my-n2-1):
$ kubectl apply -f /tmp/web-echo-svc-nop.yaml
The following would be a typical output:
service/web-echo-svc created
To list all the Service(s) running in Kubernetes cluster, execute the following command on the master node (my-n2-1):
$ kubectl get services -o wide
The following would be a typical output:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 9h <none> web-echo-svc NodePort 10.96.144.75 <none> 8888:32546/TCP 38m app=web-echo
To display detailed information about the Service endpoint labeled web-echo-svc, execute the following command on the master node ( my-n2-1):
$ kubectl describe service web-echo-svc
The following would be a typical output:
Name: web-echo-svc
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"web-echo-svc","namespace":"default"},"spec":{"ports":[{"name":"ht...
Selector: app=web-echo
Type: NodePort
IP: 10.96.144.75
Port: http 8888/TCP
TargetPort: 8888/TCP
NodePort: http 32546/TCP
Endpoints: 10.36.0.1:8888,10.42.0.1:8888,10.44.0.1:8888
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
From the Output.39 above, we see the deployed Service node port is 32546.
Open a browser and access the url http://192.168.1.53:32546. The following illustration in Figure-3 below would be a typical browser display:
BINGO - works as expected !!!
With this, we conclude the basic exercises we performed on our Kubernetes cluster.
References