sqoop

Sqoop Job 1. create a sqoop job that import a MySQL table to HDFS 1 2 3 4 5 6 7 8 9 10 11 sqoop job \ --create sqoop_import_products \ -- import \ --connect "jdbc:mysql://quickstart:3306/retail_db" \ --username retail_dba \ --password cloudera \ --table products \ --target-dir "/cca175/ps27/products/" \ --fields-terminated-by "|" \ --lines-terminated-by "\n" \ -m 5 NOTE: there is a blank before the import keyword in -- import. --import (no blank before import) will not work!...

Previously we talked about sqoop import, now in this article, we will continue on the journey with sqoop export. Sqoop Export Common Scenarios 1. export a Hive table to MySQL 1 2 3 4 5 6 7 sqoop export \ --connect "jdbc:mysql://quickstart:3306/retail_db" \ --username retail_dba \ --password cloudera \ --table products_export \ --export-dir "/user/hive/warehouse/cca175/ps19/products_export" \ --batch NOTE: The --table and --export-dir variables are required for sqoop export. These specify the table to populate in the database, and the directory in HDFS that contains the source data....

Sqoop Import Common Scenarios 1. import a table to HDFS 1 2 3 4 5 6 sqoop import \ --connect "jdbc:mysql://quickstart:3306/retail_db" \ --username retail_dba \ --password cloudera \ --table products \ -m 1 NOTE: --connect, --username, --password are database connection variables for connecting to target RDBMS. In this case, we use MySQL as our data source. --table is the specific table that we want to import to HDFS -m 1, or --num-mappers 1 is the parameters of map tasks to use to perform....

In the following articles, I will go through some common scenarios when using Sqoop in real world. NOTE: All the following problem scenarios are based on Cloudera QuickStart VM v5.8 and all the solutions can be reproduced using the aforementioned environment. Sqoop Eval Sqoop Eval allows users to execute user-defined queries against respective database servers and preview the results in console. 1 2 3 4 5 sqoop eval \ --connect "jdbc:mysql://quickstart:3306/retail_db" \ --username retail_dba \ --password cloudera \ --query "select * from retail_db....

Introduction Sqoop is a tool designed to transfer data between Hadoop and relational database servers. Sqoop ships with a help tool. To display a list of all available tools, type the follow command: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 $ sqoop help usage: sqoop COMMAND [ARGS] Available commands: codegen Generate code to interact with database records create-hive-table Import a table definition into Hive eval Evaluate a SQL statement and display the results export Export an HDFS directory to a database table help List available commands import Import a table from a database to HDFS import-all-tables Import tables from a database to HDFS list-databases List available databases on a server list-tables List available tables in a database version Display version information See 'sqoop help COMMAND' for information on a specific command....