Posts

example setup 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 ######### BASIC ######### # brew /usr/bin/ruby -e "$(curl -fsSL https://raw....

Reference Pants Official document Getting started with Pants

Situation Recently we need to life and shift some datasets from AWS Redshift to AWS Aurora in daily basis. Intuitively I was thinking this progress should be very straightforward, because both Redshift and Aurora are nothing but Postgres variants, and we could utilise all the Postgres toolings (e.g., pg_dump, pg_restore, COPY etc) to transfer the data. But in reality, nothing is hard until you start to implement and write the actual code to do the work....

Data Manipulation Dedup DataFrame Sometimes we want to drop all the duplicated data in our DataFrame, and we can use the drop_duplicates() function. For Example: 1 2 3 4 5 6 7 8 9 10 11 12 df = pd.DataFrame({ 'brand': ['Yum Yum', 'Yum Yum', 'Indomie', 'Indomie', 'Indomie'], 'style': ['cup', 'cup', 'cup', 'pack', 'pack'], 'rating': [4, 4, 3.5, 15, 5] }) df brand style rating 0 Yum Yum cup 4.0 1 Yum Yum cup 4....

Table Creation Add essential field checking rules when creating table 1 2 3 4 5 6 7 8 9 10 CREATE TABLE IF NOT EXISTS time ( start_time TIMESTAMP CONSTRAINT time_pk PRIMARY KEY, hour INT NOT NULL CHECK (hour >= 0), day INT NOT NULL CHECK (day >= 0), week INT NOT NULL CHECK (week >= 0), month INT NOT NULL CHECK (month >= 0), year INT NOT NULL CHECK (year >= 0), weekday VARCHAR NOT NULL ); Column Referencing If you already knew there are some foreign key referencing acrossing different tables, you can specify that when creating your table....