Introduction Hello my dear fellow humans, hope you are having a great day. Today’s guide is on how to recover from a disaster for Strimzi Kafka with Velero. First of all, what is Strmzi Kafka? https://strimzi.io Strimzi provides a way to run an Apache Kafka cluster on Kubernetes in various deployment configurations. Back in a while, I worked on Strimzi Kafka deployment on Openshift, very easy to set up and manage production-level Kafka cluster on Kubernetes, I have to give credit to the Strimzi project team, did a great job on documentation, support on Github discussions and active developments. ...

👉 This writeup is part of “Elasticsearch Deploy Docs” series. Important Elasticsearch Configuration 📄 Official Docs Mainly 3 configuration files elasticsearch.yml - Elasticsearch config jvm.options - Elasticsearch JVM settings config log4j2.properties - Elasticsearch logging config Environment Variables export the ES_PATH_CONF etc/default/elasticsearch (Sourced environment variables from. Recommended) Settings 📄 Official Docs Before going to production, it is recommended go through be below elasticsearch configs. Refer sample_config directory for configuration Configuration Description Configuration Reference Path settings Log and data config Refer here Cluster name Cluster name Refer here Node name Node name Refer here Network host IP address that elasticsearch bind on Refer here Discovery settings Cluster discovery and initial master config Refer here Heap size JVM heap memory configuration Recommended heap size should be half of system memory. Make sure min and max heap memory same value. Refer here Heap dump path Heap dump location path config Default config is sufficient. Refer here GC logging Garbage collection logging configuration Default config is sufficient. Refer here Temp directory Configure private temporary directory that Elasticsearch uses is excluded from periodic cleanup Important System Configuration 📄 Offical Docs ...

👉 This writeup is part of “Elasticsearch Deploy Docs” series Install 📄 Office Docs Hardware Requirement 📄 Offical Docs Resource Minimum Recommended Memory 16 GB 64 GB CPU 8 Cores 16 Disk Depends Depends JDK Installation Pick JVM compatibility version with elasticsearch from here Install OpenJDK from here Download and install JDK 11 (Another guide here) $ apt-get install openjdk-11-jdk -y $ java -version openjdk version "11.0.6" 2020-01-14 OpenJDK Runtime Environment (build 11.0.6+10-post-Ubuntu-1ubuntu118.04.1) OpenJDK 64-Bit Server VM (build 11.0.6+10-post-Ubuntu-1ubuntu118.04.1, mixed mode, sharing) Elasticsearch Installation Download latest elasticsearch from here (As of today the latest version is 7.6.2) Recommended to download/install package via .dep or PPA which postscripts creates user, groups and adds under systemd Install via apt-get from here ...

👉 This writeup is part of “Elasticsearch Deploy Docs” series Rolling Upgrade Elasticsearch 📄 Official Docs ⚠️ A rolling upgrade allows an Elasticsearch cluster to be upgraded one node at a time so upgrading does not interrupt service As of now, the current latest version of elasticsearch is v7.7.1. Below procedure is for rolling upgrade from 7.6.2=>7.7.1. 1. Divide the cluster into 2 groups Example node names carbon-x a. Non master-eligible nodes carbon-2 carbon-3 b. Master-eligible nodes carbon-1 Upgrade order (Important!) ...

👉 This writeup is part of “Elasticsearch Deploy Docs” series Kibana Installation 📄 Office docs ❗ The elasticsearch should be up and running before you start kibana installation procedure Install via apt-get from here As of today the kibana version is 7.6.2 $ wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - $ sudo apt-get install apt-transport-https $ echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list $ sudo apt-get update && sudo apt-get install kibana Configure Kibana Refer kibana.yml configuration file in current directory and modify accordingly ...

👉 This writeup is part of “Elasticsearch Deploy Docs” series Elastalert 📄 Official docs 📂 Official project Repo Elastalert is developed by Yelp written in python, queries docs in elasticsearch and send alerts depends on the rules. Since Elastalert is not part of Elasticsearch plugin, we can install it where ever we want to. Installation ❗ Refer elasticsearch-deploy-notes/elastalert for example config $ sudo apt-get install python3-pip $ sudo pip3 install elastalert $ sudo pip3 install -U PyYAML $ mkdir -p /opt/elastalert/rules ## Copy alert rules yaml files and config file to /opt/elastalert and /opt/elastalert/rules accordingly from this repo Recommended to create index in elasticsearch for elastalert to store metadata $ elastalert-create-index Elastic Version: 7.7.0 Reading Elastic 6 index mappings: Reading index mapping 'es_mappings/6/silence.json' Reading index mapping 'es_mappings/6/elastalert_status.json' Reading index mapping 'es_mappings/6/elastalert.json' Reading index mapping 'es_mappings/6/past_elastalert.json' Reading index mapping 'es_mappings/6/elastalert_error.json' New index elastalert_status created Done! Test rules in case if it is needed $ elastalert-test-rule --config /opt/elastalert/config.yaml /opt/elastalert/rules/heartbeat_checks.yml Postfix Gmail SMTP In oder to use Gmail as SMTP, you need to enable 2-Factor authentication and generate app password ...

Introduction Hello guys, today I came up with an interesting write-up, that is how to set up backup and restore with Velero on Kubernetes. A year back I worked on Strimzi Kafka, a deployment solution for deploying production-level Kafka on Kubernetes. Strimzi Kafka uses persistance volume(PV) as a disk which is a managed disk from a cloud provider(e.g. Azure, AWS, etc), but I couldn’t find a proper backup solution in order to configure PV backup and restore. Sure, you can configure these managed disk backups from Terraform or manually in cloud provider portals. But tools like Velero, backup PV from kubernetes side which is more visible and easy to manage which is what you will see in a moment. ...

It has been 2 years since I wrote a new post. Due to busy work, moving to a new city, new jobs and getting married, I wasn’t able to keep up with writing blog posts. Finally, I’m back now, I have been thinking of changing blog themes for a long time. I spent some time exploring Jekyll themes and tried to modify them according to my requirements. As you can see here GitHub issue. ...

Looks like my blog posts are like Sherlock TV Show episodes, posting once in a while…anyways I’m back now. As you might know, GitHub recently launched GitHub Actions where people can automate workflows like build, test, and deploy code from GitHub. I have started reading docs a little bit, I have to admit, setting up workflows on GitHub Actions is not that hard. I can directly start creating a workflow from available workflow templates in the “Actions” tab. Then I saw the workflow template for “Jekyll Site CI/CD”, which builds the Jekyll site. I got some feeling that I automate my workflow of building and deploying my blog (which currently you are reading) on GitHub Pages. So, Let’s see how I setup CI/CD ...

Hallo alle zusammen, after a long time I’m writing this blog and I come with an interesting and long post I know what you are thinking, I steal Kelsey Hightower’s Kubernetes The Hard Way tutorial, but hey!, I did some research and try to fit K8s cluster(Multi-Master!) in a laptop with Docker as ‘CRI’ and Flannel as ‘CNI’. This blog post follows Kelsey Hightower’s Kubernetes The Hard Way, I highly recommend go through his repo. I’m writing this blog post to keep it as a reference for me and share it with other people who want to try it. So, feel free to correct me if there are any mistakes and ping me for any queries. This series is divided into 3 parts and all configuration/scripts are available in my github repo. Well, that has been said, let’s start building the cluster. ...