Block Storage vs Object Storage Block Storage: block level operations are possible one block changed, e.g piece of the file, that contains the changed data can be updated in block level Object Storage: Entire file must be removed and new file needs to be put there S3 vs EBS vs EFS S3: S3 to be used for WORM operations, e.g. Write once Read many times Scalable, size not be planned not suitable for hosting OS or Database EBS:...
Reference TheGeekStuff: 28 Essential AWS S3 CLI Command Examples to Manage Buckets and Objects
Reference Mark Smallcombe: 15 Examples of Data Pipelines Built with Amazon Redshift
S3 -> COPY -> Redshift Staging Database -> Redshift Database Reference Data Engineering in S3 and Redshift with Python Amazon redshift: bulk insert vs COPYing from s3
Uploading local files to AWS S3 with boto3 is quite straight forward. You can install the AWS python SDK boto3 via 1 pip install boto3 Before any implementation, please make sure you have enough permission to interactive with S3. In order to upload file to S3, you can do something like the the following 1 2 3 4 import boto3 s3res = boto3.resource("s3", region="us-east-1") s3.meta.client.upload_file("<LOCAL_FILE_PATH>", "<YOUR_BUCKET>", "<YOUR_KEY>") For example, you can use the above snippet like...