Download Entire site using wget


To download a entire website using this wget command


wget --mirror -p --convert-links -P ./LOCAL-DIR WEBSITE-URL
  • –mirror : turn on options suitable for mirroring.
  • -p : download all files that are necessary to properly display a given HTML page.
  • --convert-links : after the download, convert the links in document for local viewing.
  • -P ./LOCAL-DIR : save all the files and directories to the specified directory.
  • WEBSITE-URL : Url of the site to be downloaded
     

Comments

Post a Comment

Popular posts from this blog

Proxy setting in java

3 properties can set in http,https and ftp proxy handler Http proxy in java 1. http.proxyHost: the host name of the proxy server 2. http.proxyPort: the port number, the default value being 80. 3. http.nonProxyHosts:a list of hosts that should be reached directly, bypassing the proxy. Add http proxy when run main class 1. $ java -Dhttp.proxyHost=webcache.example.com GetURL 2. $ java -Dhttp.proxyHost=webcache.example.com -Dhttp.proxyPort=8080 -Dhttp.nonProxyHosts=”localhost|host.example.com” GetURL Add https proxy when run main class 1. $ java -Dhttps.proxyHost=webcache.example.com GetURL 2. $ java -Dhttps.proxyHost=webcache.example.com -Dhttps.proxyPort=8080 -Dhttp.nonProxyHosts=”localhost|host.example.com” GetURL Add http proxy in java code //Set the http proxy to webcache.example.com:808 0 System.setProperty("http.proxyHost", "webcache.example.com"); System.setProperty("http.proxyPort", "8080"); System.setProperty("...
Read more

Using logstash to import csv json files into elasticsearch

Using logstash to import csv files into elasticsearch test.csv file contains these data's 01/01/2012 12:01:00 AM,18900 TIMES AV,SAN LORENZO,CA,94541,290,VANDALISM ($400 OR MORE),ACSO,12000048,"(37.67705445, -122.11298064)" 01/01/2012 12:01:00 AM,21000 SHERMAN DR,CASTRO VALLEY,CA,94552,250,FORGERY,ACSO,13006643,"(37.71437982, -122.02231781)" 01/01/2012 12:03:00 AM,A ST / MISSION BL,HAYWARD,CA,94541,999,REPORT NUMBER WAS CANCELLED,ACSO,11021284,"(37.66655037, -122.10000611)" 01/01/2012 12:03:00 AM,A ST / MISSION BL,HAYWARD,CA,94541,999,REPORT NUMBER WAS CANCELLED,ACSO,12000010,"(37.66655037, -122.10000611)" 01/01/2012 12:03:00 AM,A ST / MISSION BL,HAYWARD,CA,94541,90D,DUI ALCOHOL/DRUGS,ACSO,11021283,"(37.66655037, -122.10000611)" Logstash conf file  input{  file {    path => "/home/ubuntu/test.csv"    start_position => beginning  } } filter {     csv {     separator => ","    ...
Read more

Kibana 4 Installation and Run as a service in ubuntu

Download and install kibana4 To download kibana 4 wget https://download.elasticsearch.org/kibana/kibana/kibana-4.0.1-linux-x64.tar.gz Extract and Rename kibana tar -xvzf kibana-4.0.1-linux-x64.tar.gz sudo mv kibana-4.0.1-linux-x64 kibna-4 To run kibana *Run bin/kibana on unix, or bin\kibana.bat on Windows. *Visit http://localhost:5601 To set kibana 4 as a service 1. Create a file in /etc/init.d/kibana 2. Add kibana.txt  contents into  file 3 . Give permissions to    sudo chmod 755 /etc/init.d/kibana 4. Update kibna in update-rc.d     sudo update-rc.d kibana 5. To start kibana as a service     sudo service kibana start 6. Disable autostart for a tomcat service     update-rc.d kibana disable Kibana.txt #!/bin/sh #Change kibana path as per your config KIBANA_EXEC="/opt/kibana/bin/kibana" LOG_FILE="/var/log/kibana/log" PID_FILE="/var/run...
Read more