$ apt-get install openjdk-11-jdk -y
$ java -version
openjdk version "11.0.6" 2020-01-14
OpenJDK Runtime Environment (build 11.0.6+10-post-Ubuntu-1ubuntu118.04.1)
OpenJDK 64-Bit Server VM (build 11.0.6+10-post-Ubuntu-1ubuntu118.04.1, mixed mode, sharing)

$ wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
$ sudo apt-get install apt-transport-https
$ echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
$ sudo apt-get update && sudo apt-get install elasticsearch

$ mkdir /var/data/elasticsearch
$ chown -R elasticsearch:elasticsearch /var/data/elasticsearch

$ sudo swapoff -a

## *** Perminent Config ***
## Comment out any lines that contain the word 'swap' in '/etc/fstab'

$ sudo sysctl -w vm.max_map_count=262144

## *** Perminent Config ***
## update the vm.max_map_count setting in /etc/sysctl.conf

$ sudo su
# ulimit -n 65535

## *** Perminent Config ***
## echo 'elasticsearch  -  nofile  65535'| sudo tee /etc/security/limits.conf
## Ubuntu ignores the limits.conf file for processes started by init.d. To enable the limits.conf file, edit /etc/pam.d/su and uncomment the following line
## # session    required   pam_limits.so

$ sudo systemctl edit elasticsearch
## Above command opens editor to override the config. Add below setting in it
[Service]
LimitMEMLOCK=infinity

$ sudo su
# ulimit -u 4096

## *** Perminent Config ***
## echo 'elasticsearch  -  nproc  4096'| sudo tee /etc/security/limits.conf
## Ubuntu ignores the limits.conf file for processes started by init.d. To enable the limits.conf file, edit /etc/pam.d/su and uncomment the following line
## # session    required   pam_limits.so

...
-Xms15g
-Xmx15g
...

$ sudo systemctl enable elasticsearch
$ sudo systemctl start elasticsearch

$ curl -XGET 10.29.103.18:9200
{
  "name" : "carbon-1",
  "cluster_name" : "carbon",
  "cluster_uuid" : "zpoOdANFSXujgNMplMz1fQ",
  "version" : {
    "number" : "7.6.2",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "ef48eb35cf30adf4db14086e8aabd07ef6fb113f",
    "build_date" : "2020-03-26T06:34:37.794943Z",
    "build_snapshot" : false,
    "lucene_version" : "8.4.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

$ curl -XGET 10.29.103.18:9200/_cat/nodes
10.29.103.18 7 74  5 0.07 0.91 1.95 dilm * carbon-1
10.29.103.10 2 87 25 0.08 1.08 1.68 dil  - carbon-3
10.29.103.16 5 76 25 0.18 1.13 1.75 dil  - carbon-2

$ cd /usr/share/elasticsearch/bin
$ ./elasticsearch-certutil --help
Simplifies certificate creation for use with the Elastic Stack

Commands
--------
csr - generate certificate signing requests
cert - generate X.509 certificates and keys
ca - generate a new local certificate authority
http - generate a new certificate (or certificate request) for the Elasticsearch HTTP interface

Non-option arguments:
command

Option             Description
------             -----------
-E <KeyValuePair>  Configure a setting
-h, --help         Show help
-s, --silent       Show minimal output
-v, --verbose      Show verbose output

./elasticsearch-certutil cert
...
Please enter the desired output file [elastic-certificates.p12]:/etc/elasticsearch/elastic-certificates.p12
...

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/elastic-certificates.p12

$ sudo systemctl restart elasticsearch

$ cd /usr/share/elasticsearch
$ sudo bin/elasticsearch-setup-passwords auto
Initiating the setup of passwords for reserved users elastic,apm_system,kibana,kibana_system,logstash_system,beats_system,remote_monitoring_user.
The passwords will be randomly generated and printed to the console.
Please confirm that you would like to continue [y/N]y

Changed password for user apm_system
PASSWORD apm_system = xxxxxxxxxxxxxx

Changed password for user kibana
PASSWORD kibana = xxxxxxxxxxxxxx

Changed password for user logstash_system
PASSWORD logstash_system = xxxxxxxxxxxxxx

Changed password for user beats_system
PASSWORD beats_system = xxxxxxxxxxxxxx

Changed password for user remote_monitoring_user
PASSWORD remote_monitoring_user = xxxxxxxxxxxxxx

Changed password for user elastic
PASSWORD elastic = xxxxxxxxxxxxxx

$ cd /usr/share/elasticsearch
$ sudo bin/elasticsearch-certutil http

## Elasticsearch HTTP Certificate Utility

The 'http' command guides you through the process of generating certificates
for use on the HTTP (Rest) interface for Elasticsearch.

This tool will ask you a number of questions in order to generate the right
set of files for your needs.

## Do you wish to generate a Certificate Signing Request (CSR)?

A CSR is used when you want your certificate to be created by an existing
Certificate Authority (CA) that you do not control (that is, you don't have
access to the keys for that CA).

If you are in a corporate environment with a central security team, then you
may have an existing Corporate CA that can generate your certificate for you.
Infrastructure within your organisation may already be configured to trust this
CA, so it may be easier for clients to connect to Elasticsearch if you use a
CSR and send that request to the team that controls your CA.

If you choose not to generate a CSR, this tool will generate a new certificate
for you. That certificate will be signed by a CA under your control. This is a
quick and easy way to secure your cluster with TLS, but you will need to
configure all your clients to trust that custom CA.
Generate a CSR? [y/N]n

## Do you have an existing Certificate Authority (CA) key-pair that you wish to use to sign your certificate?

If you have an existing CA certificate and key, then you can use that CA to
sign your new http certificate. This allows you to use the same CA across
multiple Elasticsearch clusters which can make it easier to configure clients,
and may be easier for you to manage.

If you do not have an existing CA, one will be generated for you.

Use an existing CA? [y/N]n
A new Certificate Authority will be generated for you

## CA Generation Options

The generated certificate authority will have the following configuration values.
These values have been selected based on secure defaults.
You should not need to change these values unless you have specific requirements.

Subject DN: CN=Elasticsearch HTTP CA
Validity: 5y
Key Size: 2048

Do you wish to change any of these options? [y/N]n

## CA password

We recommend that you protect your CA private key with a strong password.
If your key does not have a password (or the password can be easily guessed)
then anyone who gets a copy of the key file will be able to generate new certificates
and impersonate your Elasticsearch cluster.

IT IS IMPORTANT THAT YOU REMEMBER THIS PASSWORD AND KEEP IT SECURE

CA password:  [<ENTER> for none]

## How long should your certificates be valid?

Every certificate has an expiry date. When the expiry date is reached clients
will stop trusting your certificate and TLS connections will fail.

Best practice suggests that you should either:
(a) set this to a short duration (90 - 120 days) and have automatic processes
to generate a new certificate before the old one expires, or
(b) set it to a longer duration (3 - 5 years) and then perform a manual update
a few months before it expires.

You may enter the validity period in years (e.g. 3Y), months (e.g. 18M), or days (e.g. 90D)

For how long should your certificate be valid? [5y] 5Y

## Do you wish to generate one certificate per node?

If you have multiple nodes in your cluster, then you may choose to generate a
separate certificate for each of these nodes. Each certificate will have its
own private key, and will be issued for a specific hostname or IP address.

Alternatively, you may wish to generate a single certificate that is valid
You entered the following hostnames.

 - elk-staging.mydomain.com

Is this correct [Y/n]y

## Which IP addresses will be used to connect to your nodes?

If your clients will ever connect to your nodes by numeric IP address, then you
can list these as valid IP "Subject Alternative Name" (SAN) fields in your
certificate.

If you do not have fixed IP addresses, or not wish to support direct IP access
to your cluster then you can just press <ENTER> to skip this step.
across all the hostnames or addresses in your cluster.

If all of your nodes will be accessed through a single domain
(e.g. node01.es.example.com, node02.es.example.com, etc) then you may find it
simpler to generate one certificate with a wildcard hostname (*.es.example.com)
and use that across all of your nodes.

However, if you do not have a common domain name, and you expect to add
additional nodes to your cluster in the future, then you should generate a
certificate per node so that you can more easily generate new certificates when
you provision new nodes.

Generate a certificate per node? [y/N]n

## Which hostnames will be used to connect to your nodes?

These hostnames will be added as "DNS" names in the "Subject Alternative Name"
(SAN) field in your certificate.

You should list every hostname and variant that people will use to connect to
your cluster over http.
Do not list IP addresses here, you will be asked to enter them later.

If you wish to use a wildcard certificate (for example *.es.example.com) you
can enter that here.

Enter all the hostnames that you need, one per line.
When you are done, press <ENTER> once more to move on to the next step.

elk-staging.mydomain.com
Enter all the IP addresses that you need, one per line.
When you are done, press <ENTER> once more to move on to the next step.

85.xx.xx.xx
85.xx.xx.xx
85.xx.xx.xx

You entered the following IP addresses.

 - 85.xx.xx.xx
 - 85.xx.xx.xx
 - 85.xx.xx.xx

Is this correct [Y/n]y

## Other certificate options
The generated certificate will have the following additional configuration
values. These values have been selected based on a combination of the
information you have provided above and secure defaults. You should not need to
change these values unless you have specific requirements.

Key Name: elk.mydomain.com
Subject DN: CN=elk, DC=mydomain, DC=org
Key Size: 2048

Do you wish to change any of these options? [y/N]n

## What password do you want for your private key(s)?

Your private key(s) will be stored in a PKCS#12 keystore file named "http.p12".
This type of keystore is always password protected, but it is possible to use a
blank password.

If you wish to use a blank password, simply press <enter> at the prompt below.
Provide a password for the "http.p12" file:  [<ENTER> for none]

## Where should we save the generated files?

A number of files will be generated including your private key(s),
public certificate(s), and sample configuration options for Elastic Stack products.

These files will be included in a single zip archive.

What filename should be used for the output zip file? [/usr/share/elasticsearch/elasticsearch-ssl-http.zip]
./elasticsearch-certutil http  6.38s user 0.47s system 6% cpu 1:38.49 total

$ unzip elasticsearch-ssl-http.zip
$ tree .
.
├── ca
├── README.txt
└── ca.p12
├── elasticsearch
├── README.txt
├── http.p12
└── sample-elasticsearch.yml
└── kibana
    ├── README.txt
    ├── elasticsearch-ca.pem
    └── sample-kibana.yml

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: "/etc/elasticsearch/http_ssl/http.p12"

elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/ca/elasticsearch-ca.pem" ]

output.elasticsearch:
  hosts: ['https://elk-staging.mydomain.com:9201']
  username: admin
  password: 123123123
  ssl:
    certificate_authorities: ["/etc/elasticsearch-ca.pem"]

setup.dashboards:
  enabled: true

setup.kibana:
  host: "https://elk-staging.mydomain.com:5691"
  ssl:
    certificate_authorities: ["/etc/letsencryptauthorityx3.pem"]

## Go elasticseatch bin directory
$ cd /usr/share/elasticsearch/
## Install Job Scheduler plugin
$ sudo bin/elasticsearch-plugin install https://d3g5vo6xdbdb9a.cloudfront.net/downloads/elasticsearch-plugins/opendistro-job-scheduler/opendistro-job-scheduler-1.8.0.0.zip
## Install Alerting plugin
$ sudo bin/elasticsearch-plugin install https://d3g5vo6xdbdb9a.cloudfront.net/downloads/elasticsearch-plugins/opendistro-alerting/opendistro_alerting-1.8.0.0.zip
## Restart Elasticsearch
$ sudo systemctl restart elasticsearch

PUT _cluster/settings
{
  "persistent": {
    "xpack.monitoring.collection.enabled": true
  }
}