Setup single node elasticsearch, logstash and Kibana4 in aws ec2


System Settings

1.To change hostname 

  Edit host name file
  sudo nano /etc/hostname

  Add these contents in hostname
  ec2-54-XXX-XXX-XX.compute-1.amazonaws.com


2. To change hosts file 
     To edit hosts file
     sudo nano /etc/hosts
     Add ip address and hostname in hosts file
      10.200.1.X ec2-XX-XX_XX.compute-1.amazonaws.com

Java installation

1. To download java 

To download 1.7 use this
   wget -c --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http %3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/7u75-b13/jdk-7u75-linux-x64.tar.gz" –output-document="jdk-7u75-linux-x64.tar.gz"

2. Extract jdk in jvm folder  

1. Create folder jvm in /usr/lib/ path
     sudo mkdir jvm
2. Move jdk file to jvm and extract it
     sudo mv jdk-7u75-linux-x64.tar.gz /usr/lib/jvm

3.Set java home and alternatives 

   To set java home
      1. Open /etc/bash.bashrc add these contents in bash.bashrc file
              export JAVA_HOME=/usr/lib/jvm/jdk1.7.0_75
              export PATH=$PATH:$JAVA_HOME/bin



Run bash.bashrc 
    source /etc/bash.bashrc
Set java alternatives
 sudo update-alternatives --install "/usr/bin/java" "java" "/usr/lib/jvm/jdk1.7.0_75 /bin/java" 1
 sudo update-alternatives --install "/usr/bin/javac" "javac" "/usr/lib/jvm/jdk1.7.0_75/bin/javac" 1

Download & install elasticsearch and logstash

Download Logstash 1.4.2 and elasticsearch 1.3.2 use this link

Install logstash and elasticsearch 

  sudo dpkg -i elasticsearch-XX.deb
  sudo dpkg -i logstash-1-XX.deb

Install aws plugin

In aws need to install aws plugin for to run elasticsearch
To download aws plugin use this link   

Install plugin 

   cd /usr/share/elasticsearch
For elasticsearch 1.3.2
   sudo bin/plugin install elasticsearch/elasticsearch-cloud-aws/2.3.0


For elasticsearch 1.4.4
    sudo bin/plugin install elasticsearch/elasticsearch-cloud-aws/2.4.1
Install marvel
Marvel is used for monitor cluster
    cd /usr/share/elasticsearch
    sudo bin/plugin -i elasticsearch/marvel/latest

Install Head

sudo bin/plugin -i --install mobz/elasticsearch-head


Note :- plugin install not working then manually download plugin zip and
bin/plugin --url file:///path/to/plugin --install plugin-name

Proxy setting  
 bin/plugin -DproxyHost=host_name -DproxyPort=port_number --install mobz/elasticsearch-head
 bin/plugin -DproxyHost=10.200.1.3 -DproxyPort=3128 --install mobz/elasticsearch-head

Install Kibana 4 

 To download kibana 4
Extract and Rename kibana
 sudo mv kibana-4.0.1-linux-x64 kibna-4

To run kibna
 Run bin/kibana on unix, or bin\kibana.bat on Windows.
 Visit http://localhost:5601

To change settings in elasticsearch

To edit elasticsearch.yml file 
   sudo nano /etc/elasticserch/elasticsearch.yml
Add these contents in elasticsearch.yml file to run elasticsearch in aws

  cluster.name: elasticsearch_production
  cloud.aws.access_key: XXXXXXXXXXXXXXXXXXXXXXX
  cloud.aws.secret_key: XXXXXXXXXXXXXXXXXXXXXXXX
  plugin.mandatory: cloud-aws
  discovery.type: "ec2"
  discovery.ec2.groups: "XXXXXXXX"
  discovery.ec2.cloud.aws.region: "XXXXXXX"
  discovery.zen.ping.multicast.enabled: false
  #discovery.zen.ping.unicast.hosts: ["XXXXXXXX.compute-1.amazonaws.com"]
  network.publish_host: XXXXXXXXXXX


Then restart elasticsearch

To verify elasticsearch is working

Check cluster status

  curl -XGET http://localhost:9200/_cluster/health?pretty=true

Check index names

 sudo curl -XGET 'localhost:9200/_cat/indices?v' 

To create delete all indexes

   sudo curl -XDELETE 'localhost:9200/*'

Use marvel to check status

   http://5XXXX-XX1:9200/_plugin/marvel/

To start elasticsearch as a service

sudo update-rc.d elasticsearch defaults 95 10
sudo /etc/init.d/elasticsearch start

Bulk api

To upload json data directly into elasticsearch
  curl -XPOST 'localhost:9200/bank/account/_bulk?pretty' --data-binary @accounts.json
  curl 'localhost:9200/_cat/indices?v'



Directory
structure of elasticsearch and logstash

 Elasticsearch



    


  • 
Binaries and stuff:
 /usr/share/elasticsearch 
 
    

    • 



  • 
Plugin manager:
 /usr/share/elasticsearch/bin/plugin 
 
    

    • 



  • 
Configuration:
 /etc/elasticsearch/elasticsearch.yml 
 
    

    • 



  • Data: /var/lib/elasticsearch/<cluster-name>
  
    


    • 




Logstash


    

  • 
Binaries and stuff: /opt/logstash
  
    

    • 



  • 
Configuration:
 /etc/logstash/conf.d 
 
    

    • 



  • Logs: /var/log/logstash 
 
    


    • 



 








































Comments











Post a Comment



















Popular posts from this blog









Proxy setting in java





















  3 properties can set in http,https and ftp proxy handler   Http proxy in java   1. http.proxyHost: the host name of the proxy server  2. http.proxyPort: the port number, the default value being 80.  3. http.nonProxyHosts:a list of hosts that should be reached directly, bypassing the proxy.   Add http proxy when run main class   1. $ java -Dhttp.proxyHost=webcache.example.com GetURL  2. $ java -Dhttp.proxyHost=webcache.example.com -Dhttp.proxyPort=8080 -Dhttp.nonProxyHosts=”localhost|host.example.com” GetURL   Add https proxy when run main class  1. $ java -Dhttps.proxyHost=webcache.example.com GetURL  2. $ java -Dhttps.proxyHost=webcache.example.com -Dhttps.proxyPort=8080 -Dhttp.nonProxyHosts=”localhost|host.example.com” GetURL   Add http proxy in java code   //Set the http proxy to webcache.example.com:808 0   System.setProperty("http.proxyHost", "webcache.example.com");  System.setProperty("http.proxyPort", "8080");  System.setProperty("...








Read more










Kibana 4 Installation and Run as a service in ubuntu





















      Download and install kibana4   To download kibana 4   wget https://download.elasticsearch.org/kibana/kibana/kibana-4.0.1-linux-x64.tar.gz     Extract and Rename kibana   tar -xvzf kibana-4.0.1-linux-x64.tar.gz   sudo mv  kibana-4.0.1-linux-x64  kibna-4     To run kibana   *Run bin/kibana on unix, or bin\kibana.bat on Windows.    *Visit http://localhost:5601   To set kibana 4 as a service       1. Create  a file in /etc/init.d/kibana   2. Add kibana.txt  contents into  file   3 .          Give permissions to       sudo chmod 755 /etc/init.d/kibana   4. Update kibna in update-rc.d       sudo update-rc.d kibana   5. To start kibana as a service      sudo service kibana start               6. Disable autostart for a tomcat service         update-rc.d kibana disable      Kibana.txt    #!/bin/sh  #Change kibana path as per your config  KIBANA_EXEC="/opt/kibana/bin/kibana"  LOG_FILE="/var/log/kibana/log"  PID_FILE="/var/run...








Read more










Using logstash to import csv json files into elasticsearch





















  Using logstash to import csv files into elasticsearch    test.csv file contains these data's  01/01/2012 12:01:00 AM,18900 TIMES AV,SAN LORENZO,CA,94541,290,VANDALISM ($400 OR MORE),ACSO,12000048,"(37.67705445, -122.11298064)"  01/01/2012 12:01:00 AM,21000 SHERMAN DR,CASTRO VALLEY,CA,94552,250,FORGERY,ACSO,13006643,"(37.71437982, -122.02231781)"  01/01/2012 12:03:00 AM,A ST / MISSION BL,HAYWARD,CA,94541,999,REPORT NUMBER WAS CANCELLED,ACSO,11021284,"(37.66655037, -122.10000611)"  01/01/2012 12:03:00 AM,A ST / MISSION BL,HAYWARD,CA,94541,999,REPORT NUMBER WAS CANCELLED,ACSO,12000010,"(37.66655037, -122.10000611)"  01/01/2012 12:03:00 AM,A ST / MISSION BL,HAYWARD,CA,94541,90D,DUI ALCOHOL/DRUGS,ACSO,11021283,"(37.66655037, -122.10000611)"   Logstash conf file   input{   file {     path => "/home/ubuntu/test.csv"     start_position => beginning   }  }   filter {      csv {      separator => ","     ...








Read more