Posts

Show Replication Factor of a File 1 hadoop fs -stat %r <YOUR_FILE_BLOCK> Update Replication Factor of a File 1 hadoop fs -setrep -R -w 3 <YOUR_FILE_BLOCK>

A B C D E F G H I J K L M N O One-hot encoding One-hot encoding is a way to represent categorical variables as numerical data, so that it can be used in machine learning algorithm. It involves creating a new binary column for each unique category in the categorical feature. For example, if a categorical feature has three categories, A, B, and C. Then three new columns, one for each category would be created....

Try not to use SELECT * and provide specific fields Bad Example 1 SELECT * FROM user; Good Example 1 SELECT id, username, tel FROM user; Reason Specify fields will save computational resources, and reduce network expense. It’s possible using pre-built table index to reduce table returns, and improve query efficiency. Avoid using OR in the where clause to connect conditions Bad Example 1 2 SELECT id, username, tel FROM user WHERE id=1 OR salary=500; Good Example Use UNION ALL to concatenate results 1 2 3 4 5 SELECT id, username, tel FROM user WHERE id=1 UNION ALL SELECT id, username, tel FROM user WHERE salary=500; Use two separate SQL 1 2 SELECT id, username, tel FROM user WHERE id=1 1 2 SELECT id, username, tel FROM user WHERE salary=500; Reason The use of OR may invalidate the table index, and thus will do full table scan....

Reference Lydia Hallie: CS Visualized: Useful Git Commands

Query Execution Order Most people would write their SQL queries starting from SELECT part, because it’s more intuitive and close to our natural language. But actually that’s not the way that SQL queries been executed in query engine. Below is the execution order of a SQL query: FROM, JOIN. Tables are joined to get the base data. WHERE. The base data is filtered. GROUP BY. The filtered based data is grouped....