Posts

Spark Job optimisation 1 spark-submit --py-files ./rs_commons_util.zip --executor-cores 4 --num-executors 4 ./main.py Reference How We Optimise Apache Spark Jobs Apache Spark: Config Cheatsheet What I Learned From Processing Big Data With Apache Spark Cloudera: How-to: Tune Your Apache Spark Jobs (Part 1) Cloudera: How-to: Tune Your Apache Spark Jobs (Part 2) Hortonworks: Spark num-executors setting Best Practices Writing Production-Grade PySpark Jobs Github: ekampf/PySpark-Boilerplate Github: snowplow/spark-example-project

DataFrame Create a DataFrame Get DataFrame Column Headers list(df) Reference RealPython.com: Python Pandas: Tricks & Features You May Not Know TowardDataScience.com: 23 great Pandas codes for Data Scientists Analyticsvidhya.com: 12 Useful Pandas Techniques in Python for Data Manipulation

Sometimes, I need to keep my Windows PC awake for a certain period of time. I found a simple PowerShell script that can help me with this. You can create a PowerShell Script named tt.ps1 with the following code: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 # Description: Time Travel with PowerShell # Date: 2024-05-10 $Signature = @' [DllImport("user32....

Reference 28 Jupyter Notebook tips, tricks, and shortcuts Nazif Berat: Boosting Your Jupyter Notebook Productivity

According to the AWS Document , Redshift provides PG_TABLE_DEF table that contains all the table information, and you can query that table like 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 SELECT DISTINCT schemaname, tablename, column, type, encoding, distkey, sortkey, notnull FROM PG_TABLE_DEF WHERE 1=1 AND schemaname in ('<SCHEMA_NAME>') ORDER BY schemaname ASC, tablename ASC ; However, this only returns all the information in the public schema, I assume there maybe some permission problem with my account....