Part-3 : Install Sqoop (version 1.4.x)

In this part we will discuss how to install SQOOP on HDP platform. I assume that you have followed instructions from Part-1 and Part-2 on how to install HDP and Hive on a single node cluster.

Get latest version of “sqoop-*-bin.tar.gz” file link from below official SQOOP site,

Go to Downloads directory and download latest tar.gz file for sqoop.

Ensure that you download version that matches with your installed hadoop major version. If your hadoop version is 2.x.x then download sqoop version that ends with 2.x.x. For example if your hadoop version is 2.8.1 then download sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz. Please note that sqoop version can be different like 1.x.x.

cd ~/Downloads

Extract downloaded file,

tar -zxvf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz

Create directory for sqoop,

mkdir /usr/lib/sqoop

Move extracted sqoop folder to /usr/lib/sqoop

mv sqoop-1.4.6.bin__hadoop-2.0.4-alpha /usr/lib/sqoop/

Set environment variables

vi ~/.bashrc

Add below lines

export SQOOP_HOME=/usr/lib/sqoop/ sqoop-1.4.6.bin__hadoop-2.0.4-alpha

Reload environment variables,

source ~/.bashrc

If you are planning to import data from MySQL database then you need to copy related jar file in Sqoop installation directory.

cp /usr/share/java/mysql-connector-java.jar $SQOOP_HOME/lib

Sqoop installation is complete. Check Sqoop version with below command

$sqoop version
17/09/21 19:15:40 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
Sqoop 1.4.6
git commit id c0c5a81723759fa575844a0a1eae8f510fa32c25
Compiled by root on Mon Apr 27 14:38:36 CST 2015

Let’s try few sqoop commands. To import just one table from mysql database, use below command. This assumes you have a MySQL server running on localhost and it has “employees_db” database with table name “employees”. Also ensure you have “employees” database created in Hive. i.e. run ”

hive>create database employees;"

on hive prompt.

Sqoop command,

sqoop import \
--connect jdbc:mysql://localhost/employees_db \
--username root \
--password mysqlpassword \
--table employees --m 1 \
--hive-import \
--hive-overwrite \
--hive-database employees \
--target-dir /user/hive/warehouse/employees/

To import all tables from mysql database

sqoop import-all-tables \
--connect jdbc:mysql://localhost/employees_db \
--username root \
--password mysqlpassword \
--hive-import \
--hive-overwrite \
--hive-database employees \
--m 1 \