When you logon a Linux server, there must be couples of things that you have to do everyday, e.g. change to your project directory, check disk space, or check the server load. Of course you can create some shortcuts and alias in your .bashrc file, but you still need to manual execute your shortcut command and I am too lazy to even execute that command. Well you can solve this problem by modifying your profile file....
What is Airflow? Airflow is a platform to programmatically author, schedule and monitor your workflows and pipelines. What are the benefits for using Airflow? Programmatically author workflow In Airflow, you can define your workflow programmatically with Python scripts and that would put you in a very good position by leveraging all the convenience and sweet that Python provide. This is a huge improvement if you experienced with Oozie or other GUI-typed (or even without a GUI) scheduling tools....
S3 -> COPY -> Redshift Staging Database -> Redshift Database Reference Data Engineering in S3 and Redshift with Python Amazon redshift: bulk insert vs COPYing from s3
Uploading local files to AWS S3 with boto3 is quite straight forward. You can install the AWS python SDK boto3 via 1 pip install boto3 Before any implementation, please make sure you have enough permission to interactive with S3. In order to upload file to S3, you can do something like the the following 1 2 3 4 import boto3 s3res = boto3.resource("s3", region="us-east-1") s3.meta.client.upload_file("<LOCAL_FILE_PATH>", "<YOUR_BUCKET>", "<YOUR_KEY>") For example, you can use the above snippet like...