Part-7 : Install Sqoop (version 1.4.x)

By | September 18, 2017

In this part we will discuss how to install SQOOP on HDP platform. I assume that you have followed instructions from Part-1 to 6 on how to install HDP and Hive on a single node cluster.

Get latest version of “sqoop-*-bin.tar.gz” file link from below official SQOOP site,
http://sqoop.apache.org

Go to Downloads directory and download latest tar.gz file for sqoop.

Ensure that you download version that matches with your installed hadoop major version. If your hadoop version is 2.x.x then download sqoop version that ends with 2.x.x. For example if your hadoop version is 2.8.1 then download sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz. Please note that sqoop version can be different like 1.x.x.

[cc lang=”bash”]
cd ~/Downloads
wget http://apache.claz.org/sqoop/1.4.6/sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz[/cc]

Extract downloaded file,
[cc lang=”bash”]tar -zxvf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz
[/cc]

Create directory for sqoop,
[cc lang=”bash”]mkdir /usr/lib/sqoop[/cc]

Move extracted sqoop folder to /usr/lib/sqoop
[cc lang=”bash”]mv sqoop-1.4.6.bin__hadoop-2.0.4-alpha /usr/lib/sqoop/[/cc]

Set environment variables
[cc lang=”bash”]vi ~/.bashrc[/cc]

Add below lines
[cc lang=”bash”]export SQOOP_HOME=/usr/lib/sqoop/ sqoop-1.4.6.bin__hadoop-2.0.4-alpha
export PATH=$PATH:$SQOOP_HOME/bin[/cc]

Reload environment variables,
[cc lang=”bash”]source ~/.bashrc[/cc]

If you are planning to import data from MySQL database then you need to copy related jar file in Sqoop installation directory.

[cc lang=”bash”]
cp /usr/share/java/mysql-connector-java.jar $SQOOP_HOME/lib
[/cc]

Sqoop installation is complete. Check Sqoop version with below command
[cc lang=”bash”]
$sqoop version
17/09/21 19:15:40 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
Sqoop 1.4.6
git commit id c0c5a81723759fa575844a0a1eae8f510fa32c25
Compiled by root on Mon Apr 27 14:38:36 CST 2015
[/cc]

Let’s try few sqoop commands. To import just one table from mysql database, use below command. This assumes you have a MySQL server running on localhost and it has “employees_db” database with table name “employees”. Also ensure you have “employees” database created in Hive. i.e. run “[cc lang=”bash”]hive>create database employees;”[/cc] on hive prompt.

Sqoop command,
[cc lang=”bash”]sqoop import \
–connect jdbc:mysql://localhost/employees_db \
–username root \
–password mysqlpassword \
–table employees –m 1 \
–hive-import \
–hive-overwrite \
–hive-database employees \
–target-dir /user/hive/warehouse/employees/[/cc]

To import all tables from mysql database
[cc lang=”bash”]sqoop import-all-tables \
–connect jdbc:mysql://localhost/employees_db \
–username root \
–password mysqlpassword \
–hive-import \
–hive-overwrite \
–hive-database employees \
–m 1 \[/cc]

Leave a Reply

Your email address will not be published. Required fields are marked *