AWS Storage Comparison

Block Storage vs Object Storage Block Storage: block level operations are possible one block changed, e.g piece of the file, that contains the changed data can be updated in block level Object Storage: Entire file must be removed and new file needs to be put there S3 vs EBS vs EFS S3: S3 to be used for WORM operations, e.g. Write once Read many times Scalable, size not be planned not suitable for hosting OS or Database EBS:...

November 3, 2019 · 1 min · 147 words · Eric

Common AWS S3 CLI command examples  [draft]

Reference TheGeekStuff: 28 Essential AWS S3 CLI Command Examples to Manage Buckets and Objects

February 2, 2019 · 1 min · 14 words · Eric

Redshift Pipelines  [draft]

Reference Mark Smallcombe: 15 Examples of Data Pipelines Built with Amazon Redshift

July 25, 2018 · 1 min · 12 words · Eric

Redshift Data Ingestion from S3  [draft]

S3 -> COPY -> Redshift Staging Database -> Redshift Database Reference Data Engineering in S3 and Redshift with Python Amazon redshift: bulk insert vs COPYing from s3

May 25, 2018 · 1 min · 27 words · Eric

Upload Files to S3 Bucket

Uploading local files to AWS S3 with boto3 is quite straight forward. You can install the AWS python SDK boto3 via 1 pip install boto3 Before any implementation, please make sure you have enough permission to interactive with S3. In order to upload file to S3, you can do something like the the following 1 2 3 4 import boto3 s3res = boto3.resource("s3", region="us-east-1") s3.meta.client.upload_file("<LOCAL_FILE_PATH>", "<YOUR_BUCKET>", "<YOUR_KEY>") For example, you can use the above snippet like...

May 23, 2018 · 1 min · 191 words · Eric