Elasticsearch cluster can be set up on AWS ec2 instance as shown below.

Setup

The following configuration details are for Ubuntu distro.

  1. Install Java.

     sudo add-apt-repository ppa:webupd8team/java
 sudo apt-get update
 sudo apt-get install oracle-java7-installer

  2. Install Elasticsearch

     wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
 echo "deb http://packages.elastic.co/elasticsearch/1.6/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-1.6.list
 sudo apt-get update
 sudo apt-get install elasticsearch

  3. Setup EBS.

     # Mounting EBS to /mnt  
 sudo mkfs.ext4 /dev/xvdf
 sudo mkdir /mnt
 sudo mount /dev/xvdf /mnt

 # Configure fstab
 # Add following line in /etc/fstab document
 # /dev/xvdf       /mnt    auto    defaults,nobootwait,comment=cloudconfig 0       2
 echo "/dev/xvdf       /mnt    auto    defaults,nobootwait,comment=cloudconfig 0       2" | sudo tee -a /etc/fstab

  4. Create folder for Elasticsearch in EBS.

     # create a folder "data" in /mnt
 sudo mkdir -p /mnt/data
 # change the ownership to elasticsearch user
 sudo chown elasticsearch:elasticsearch /mnt/data

  5. Configure Elasticsearch.

     # Edit Elasticsearch config file /etc/elasticsearch/elasticsearch.yml
 # Cluster name should be same across all nodes.
 cluster.name: es-cluster
 # Set data path to correct folder in attached drive.
 path.data: /mnt/data
 # Disable Multicast Discovery. Amazon doesn't allow multicast. So, we discover cluster with Unicast
 discovery.zen.ping.multicast.enabled: false
 # Config at least two IP in all machines. These two machines will be pinged by other machines in the cluster for discovery
 discovery.zen.ping.unicast.hosts: ["machine1", "machine2"]  

 # Automatically start during bootup
 sudo update-rc.d elasticsearch defaults 95 10

  6. Optionally setup Head Plugin to view status of Elasticsearch.

     # Install Elastic-Head Plugin
 sudo /usr/share/elasticsearch/bin/plugin -install mobz/elasticsearch-head

Now, open http://instance-ip:9200/_plugin/head/ to view the Elasticsearch status

Repeat same procedure in all nodes to manually set up Elasticsearch cluster.

Notes

  • Create EC2 instances for each node required in the cluster, in the same availability zone.
  • Let all the nodes be of same EC2 instance type.
  • Better to create all instances within same security group.
  • Security Group of the machines must allow ICMP packets between the machines.
  • SSD EBS storage is attached for better performance.
  • Http REST API access is relatively faster than native Java based client.
  • Better to do round robin access to nodes to avoid overloading a single node for all queries from clients.