This blog post follows Kelsey Hightower’sKubernetes The Hard Way, I highly recommend go through his repo. I’m writing this blog post to keep it as reference for me and share with other people whoever want to try it. So, feel free to correct me if any mistakes and ping me for any queries. This series divided into 3 parts and all configuration/scripts are available in my github repo. Well that has been said, let’s start building the cluster.
Below is my laptop configuration. Make sure you have enough resources in your laptop.(or depends on resources, you can reduce nodes in cluster, etc.)
Acer Predator Helios 200
Intel Core i5 8th Gen
First let’s talk about the cluster in Kubernetes The Hard Way which has 3 controller nodes, 3 worker nodes and a load balancer on GCP. I want to deploy cluster with multiple masters, but I was afraid it is too much for my laptop. So, I reduced to 2 controller nodes, 2 worker nodes (or VMs in my case) and replaced GCP load balancer with nginx docker container as a load balancer, the clusters looks like below.
Installation of packages
*NOTE: The following components will be installed on host machine(laptop)
Because we want to run nginx load balancer container on host
Install cfssl and cfssljson binaries
In offical “Kubernetes The Hard Way”, cluster network configuration done via gcloud and obviously we are not going to use it. We have to choose subnets manually for our cluster nodes,CIDR for pods and K8s services. So, here is what I come with
Linux Bridge & NAT
As you can see in above diagram, we are going to use linux bridge to connect our VMs and nginx container. Also we need to do NATing for our VMs in order to access Internet.
In order to launch docker container(nginx load balancer container) on different linux bridge(other than default docker0), we need to create docker network and specify that network while launching the container. Below command creates docker network with br0 as bridge
Create workspace directory
We can save all configuration and generate certificates in this directory
2. Provisioning Compute Resources
Specify cluster info(hostname, IP and user to login) in controllers.txt and workers.txt files respectively like in below. In the same way add those VM IPs in /etc/hosts file like below. These files are useful to automate things like copy files to nodes or generating certificates for these nodes, etc. You will see in a moment.
Below are the IPs, hostname and username for the nodes that I choose
In previous section, we installed kvm hypervisor and now lets spin up 4 VMs and specify bridge name under network section like in below screenshot.(I used “Virtual Machine Manger” GUI to launch VMs)
*I’m not covering OS installation in VM. You can easly find it on Internet.
*NOTE: While installing OS, please select static IP and specify IPs according to their node names
*TIP: Install OS in VM and clone VM 3 time
Once the OS installation is completed, check the connectivity between the host-VM and VM-VM and you should able to ssh both host-to-VM and VM-to-VM. For handy, you can copy ssh keys, so that don’t have to enter password every time.
3. Provisioning a CA and Generating TLS Certificates
It is a good/recommended practice to setup encrypted communication between the components of K8s. In this section we will create public key certificates and private keys for below components using CloudFlare’s PKI toolkit as we downloaded earlier.(Know more about PKI)
K8s uses node authorization which is a special-purpose authorization mode that specifically authorizes API requests made by kubelets
In order to be authorized by the Node Authorizer, Kubelets must use a credential that identifies them as being in the system:nodes group, with a username of system:node:<nodeName>. Let’s create a certificate and private key for each worker nodes (In my case n1 and n2)
Controller Manager Client Certificate
Kube Proxy Client Certificate
Scheduler Client Certificate
Kubernetes API Server Certificate
kube-api server certificate’s hostname should include following things
All controller’s hostname
All controller’s IP
Load balancer’s hostname
Load balancer’s IP
Kubernetes’s service(Both ‘service name’ and IP which are 10.32.0.1 and kubernetes.default)
Service Account Key Pair
Service account key pair certificate is used to sign service account tokens
Copy Certificates to Nodes
4. Generating kubeconfig Files for Authentication
kubeconfig are used for authentication between the kubernetes components and users-to-kubernetes. kubeconfig consists of mainly 3 things
api-server's IP and its certificate which encodes in `base64`
User related info like who are authenticating ,their cerificate and key or service account token
Holds Cluster's and User's reference. If you have multiple clusters and users, this `context` becomes handy
In this section, we are going to generate kubeconfig for below components
Generating kubelet kubeconfig
The user in kubeconfig should be system:node:<Worker_name> which should match Kubelet hostname that we specified while generating kubelet client certificate. This will ensure Kubelets are properly authorized by the Kubernetes Node Authorizer.
Generate kube-proxy kubeconfig
Generate kube-controller-manager kubeconfig
Generate kube-scheduler kubeconfig
Generate admin kubeconfig
Copy kubeconfig files to nodes
5. Generating the Data Encryption Config and Key
Kubernetes stores different types of data including cluster state, application configurations, and secrets. Kubernetes supports the ability to encrypt cluster data at rest.In this section we will generate an encryption key and an encryption config suitable for encrypting Kubernetes Secrets.
The Encrypted Key
The Encryption Config File
Copy to Controller Nodes
Till now we have done following things
Provisioned compute resources
Generated kubeconfig files
Copied certificate files and kubeconfigs to nodes
In the next post, we will bootstrap controller nodes