Setup elasticsearch behind nginx server


Setup elasticsearch behind nginx server

Install nginx server
  1. sudo apt-get update
  2. sudo  apt-get install nginx

If already nginx exists in the system, check if it’s latest version.
To check the version
guidanz-devaraj@guidanzlocal:~$ nginx -v
nginx version: nginx/1.1.19

To start , stop, check status of  the server
 guidanz-devaraj@guidanzlocal:~$ sudo service nginx status
* nginx is not running
guidanz-devaraj@guidanzlocal:~$ sudo service nginx start
Starting nginx: nginx.
guidanz-devaraj@guidanzlocal:~$ sudo service nginx status
* nginx is running
guidanz-devaraj@guidanzlocal:~$ sudo service nginx stop
Stopping nginx: nginx.
guidanz-devaraj@guidanzlocal:~$ sudo service nginx status
* nginx is not running


To confirm the nginx is working or not. Check localhost:80 is working in the browser.
It will reply as Welcome to nginx!

Setup elasticsearch

Before setup the nginx server install elasticsearch. https://www.elastic.co/guide/en/elasticsearch/reference/current/setup.html

In latest elasticsearch versions the bind address of the elasticsearch is localhost.
To change the settings edit config/reporting.yml page. If need change the network host name as you system ip.



# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 10.9.14.4  

So after starting the elasticsearch in console, ip is showing like this.

[2016-04-11 12:34:51,855][INFO ][http] [Mondo] publish_address {10.9.14.4:9200}, bound_addresses {10.9.14.4:9200}

Nginx setup

Nginx is makes elasticsearch url should be routable and hide the original url.
To make this let’s take system domain name as dev.elasticsearch.com.

So now we are gonna to route guidanzlocal.com to localhost:9200 and restrict the users to access localhost:9200 url.

To achieve this.

First create a new file in /etc/nginx/sites-enabled/elasticsearch and paste this content.

server {
    listen 80;
    server_name dev.elasticsearch.com;
    location / {
       rewrite ^/(.*) /$1 break;
       proxy_ignore_client_abort on;
       proxy_pass http://localhost:9200;
       proxy_redirect http://localhost:9200 http://dev.elasticsearch.com/;
       proxy_set_header  X-Real-IP  $remote_addr;
       proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header  Host $http_host;
    }
}




Explanation about nginx code.

  • listen parameter for which server listening on particular port. In the server name defined the host name of the system.

  • Location is used for redirect page to different url, for an example if url need to redirect to home page when the url ends with /home. By default slash is the redirect page.

  • Proxy pass parameter tells what url is proxied.

  • Proxy redirect parameter defines coming url can be redirect to which url.


Create a symbolic link

Run this command to create symbolic link

sudo ln /etc/nginx/sites-enabled/elasticsearch /etc/nginx/sites-available/elasticsearch

After add a symbolic link then reload in nginx using this command.

sudo service nginx reload

So when access dev.elasticsearch.com
curl dev.elasticsearch.com

its redirect page to elasticsearch url page like this.

{
  "name" : "Shinchuko Lotus",
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "2.2.0",
    "build_hash" : "8ff36d139e16f8720f2947ef62c8167a888992fe",
    "build_timestamp" : "2016-01-27T13:32:39Z",
    "build_snapshot" : false,
    "lucene_version" : "5.4.1"
  },
  "tagline" : "You Know, for Search"
}

Adding basic http authentication in elasticsearch
To create a basic authentication we need to create a htpasswd which is key value pair of username and encrypted password.

To do this we need some libraries need to install which apache-utils. use this command to install apache utils.

    sudo apt-get update
    sudo apt-get install apache2 apache2-utils


User name and password

To create a username and password run this command

sudo htpasswd -c /etc/nginx/.htpasswd username

For example
sudo htpasswd -c /etc/nginx/.htpasswd devaraj
New password:
Re-type new password:
Adding password for user devaraj

In /etc/nginx .htpasswd file created with key

when print the file

/etc/nginx$ cat .htpasswd
devaraj:$apr1$e/qHcmjR$Ww1pAdZ3LloAHCPguGXcu/

So username and password is created. Add these file in elasticsearch file to restrict elasticsearch with username and password.

server {
    listen 80;
    server_name dev.elasticsearch.com;
    location / {
       rewrite ^/(.*) /$1 break;
       proxy_ignore_client_abort on;
       proxy_pass http://localhost:9200;
       proxy_redirect http://localhost:9200 https://dev.elasticsearch.com/;
       proxy_set_header  X-Real-IP  $remote_addr;
       proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header  Host $http_host;
       auth_basic "Elasticsearch Authentication";
       auth_basic_user_file /etc/nginx/.htpasswd;
}
}


Add these content in elasticsearch file and reload the nginx.

sudo nano /etc/nginx/sites-enabled/elasticsearch
sudo service nginx reload

After reloading the page and then try to access page in browser it will ask for username and password.

curl 10.9.14.4

401 Authorization Required

401 Authorization Required

nginx/1.1.19

curl -u devaraj 10.9.14.4
Enter host password for user 'devaraj':
{
 "name" : "Shinchuko Lotus",
 "cluster_name" : "elasticsearch",
 "version" : {
    "number" : "2.2.0",
    "build_hash" : "8ff36d139e16f8720f2947ef62c8167a888992fe",
    "build_timestamp" : "2016-01-27T13:32:39Z",
    "build_snapshot" : false,
    "lucene_version" : "5.4.1"
 },
 "tagline" : "You Know, for Search"
}



So http authentication is added for nginx elasticsearch.


HTTPS and SSL connection nginx

To generate elasticsearch certificate

sudo openssl genrsa -des3 -out elastic.key 1024
Generating RSA private key, 1024 bit long modulus
...............................++++++
.......++++++
e is 65537 (0x10001)
Enter pass phrase for elastic.key:
Verifying - Enter pass phrase for elastic.key:


sudo openssl rsa -in elastic.key -out elastic.key

sudo openssl x509 -req -days 3650 -in elastic.csr -signkey elastic.key -out elastic.crt

After change the server configurations slightly and replace elasticsearch config

server {
    listen 443;
    ssl on;
    ssl_certificate /etc/nginx/ssl/elastic.crt;
    ssl_certificate_key /etc/elasticsearch/ssl/elastic.key;
    server_name dev.elasticsearch.com;
    location / {
       rewrite ^/(.*) /$1 break;
       proxy_ignore_client_abort on;
       proxy_pass http://localhost:9200;
       proxy_redirect http://localhost:9200 https://dev.elasticsearch.com/;
       proxy_set_header  X-Real-IP  $remote_addr;
       proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header  Host $http_host;
       auth_basic "Elasticsearch Authentication";
       auth_basic_user_file /etc/nginx/.htpasswd;
}
}

server{
    listen 80;
    server_name es.domain.com;
    return 301 https://$host$request_uri;
}

After reload the nginx and then dev.elasticsearch.com is redirected to https secured server.











Comments

Post a Comment

Popular posts from this blog

Proxy setting in java

3 properties can set in http,https and ftp proxy handler Http proxy in java 1. http.proxyHost: the host name of the proxy server 2. http.proxyPort: the port number, the default value being 80. 3. http.nonProxyHosts:a list of hosts that should be reached directly, bypassing the proxy. Add http proxy when run main class 1. $ java -Dhttp.proxyHost=webcache.example.com GetURL 2. $ java -Dhttp.proxyHost=webcache.example.com -Dhttp.proxyPort=8080 -Dhttp.nonProxyHosts=”localhost|host.example.com” GetURL Add https proxy when run main class 1. $ java -Dhttps.proxyHost=webcache.example.com GetURL 2. $ java -Dhttps.proxyHost=webcache.example.com -Dhttps.proxyPort=8080 -Dhttp.nonProxyHosts=”localhost|host.example.com” GetURL Add http proxy in java code //Set the http proxy to webcache.example.com:808 0 System.setProperty("http.proxyHost", "webcache.example.com"); System.setProperty("http.proxyPort", "8080"); System.setProperty("...
Read more

Using logstash to import csv json files into elasticsearch

Using logstash to import csv files into elasticsearch test.csv file contains these data's 01/01/2012 12:01:00 AM,18900 TIMES AV,SAN LORENZO,CA,94541,290,VANDALISM ($400 OR MORE),ACSO,12000048,"(37.67705445, -122.11298064)" 01/01/2012 12:01:00 AM,21000 SHERMAN DR,CASTRO VALLEY,CA,94552,250,FORGERY,ACSO,13006643,"(37.71437982, -122.02231781)" 01/01/2012 12:03:00 AM,A ST / MISSION BL,HAYWARD,CA,94541,999,REPORT NUMBER WAS CANCELLED,ACSO,11021284,"(37.66655037, -122.10000611)" 01/01/2012 12:03:00 AM,A ST / MISSION BL,HAYWARD,CA,94541,999,REPORT NUMBER WAS CANCELLED,ACSO,12000010,"(37.66655037, -122.10000611)" 01/01/2012 12:03:00 AM,A ST / MISSION BL,HAYWARD,CA,94541,90D,DUI ALCOHOL/DRUGS,ACSO,11021283,"(37.66655037, -122.10000611)" Logstash conf file  input{  file {    path => "/home/ubuntu/test.csv"    start_position => beginning  } } filter {     csv {     separator => ","    ...
Read more

Kibana 4 Installation and Run as a service in ubuntu

Download and install kibana4 To download kibana 4 wget https://download.elasticsearch.org/kibana/kibana/kibana-4.0.1-linux-x64.tar.gz Extract and Rename kibana tar -xvzf kibana-4.0.1-linux-x64.tar.gz sudo mv kibana-4.0.1-linux-x64 kibna-4 To run kibana *Run bin/kibana on unix, or bin\kibana.bat on Windows. *Visit http://localhost:5601 To set kibana 4 as a service 1. Create a file in /etc/init.d/kibana 2. Add kibana.txt  contents into  file 3 . Give permissions to    sudo chmod 755 /etc/init.d/kibana 4. Update kibna in update-rc.d     sudo update-rc.d kibana 5. To start kibana as a service     sudo service kibana start 6. Disable autostart for a tomcat service     update-rc.d kibana disable Kibana.txt #!/bin/sh #Change kibana path as per your config KIBANA_EXEC="/opt/kibana/bin/kibana" LOG_FILE="/var/log/kibana/log" PID_FILE="/var/run...
Read more