Stats & exploratory analysis Numpy. Pandas. SciPy. Statsmodels. Patsy. Emcee. Machine Learning Scikit-learn. TensorFlow. Keras. Theano. Lifelines. Data Visualization Matplotlib. Plotly. Seaborn. Geomap. Folium. Networkx. Basemap. Web Scraping BeautifulSoup. ScraPy. Requests. Natural Language Processing NLTK. Gensim. TextBlob. Utility when-changed. tenacity .
Sqoop Job 1. create a sqoop job that import a MySQL table to HDFS 1 2 3 4 5 6 7 8 9 10 11 sqoop job \ --create sqoop_import_products \ -- import \ --connect "jdbc:mysql://quickstart:3306/retail_db" \ --username retail_dba \ --password cloudera \ --table products \ --target-dir "/cca175/ps27/products/" \ --fields-terminated-by "|" \ --lines-terminated-by "\n" \ -m 5 NOTE: there is a blank before the import keyword in -- import. --import (no blank before import) will not work!...
Previously we talked about sqoop import, now in this article, we will continue on the journey with sqoop export. Sqoop Export Common Scenarios 1. export a Hive table to MySQL 1 2 3 4 5 6 7 sqoop export \ --connect "jdbc:mysql://quickstart:3306/retail_db" \ --username retail_dba \ --password cloudera \ --table products_export \ --export-dir "/user/hive/warehouse/cca175/ps19/products_export" \ --batch NOTE: The --table and --export-dir variables are required for sqoop export. These specify the table to populate in the database, and the directory in HDFS that contains the source data....
Sqoop Import Common Scenarios 1. import a table to HDFS 1 2 3 4 5 6 sqoop import \ --connect "jdbc:mysql://quickstart:3306/retail_db" \ --username retail_dba \ --password cloudera \ --table products \ -m 1 NOTE: --connect, --username, --password are database connection variables for connecting to target RDBMS. In this case, we use MySQL as our data source. --table is the specific table that we want to import to HDFS -m 1, or --num-mappers 1 is the parameters of map tasks to use to perform....
In the following articles, I will go through some common scenarios when using Sqoop in real world. NOTE: All the following problem scenarios are based on Cloudera QuickStart VM v5.8 and all the solutions can be reproduced using the aforementioned environment. Sqoop Eval Sqoop Eval allows users to execute user-defined queries against respective database servers and preview the results in console. 1 2 3 4 5 sqoop eval \ --connect "jdbc:mysql://quickstart:3306/retail_db" \ --username retail_dba \ --password cloudera \ --query "select * from retail_db....