https://eric.nz/tags/aws/ Recent content in aws on Eric's Note https://eric.nz/%3Clink%20or%20path%20of%20image%20for%20opengraph,%20twitter-cards%3E https://eric.nz/%3Clink%20or%20path%20of%20image%20for%20opengraph,%20twitter-cards%3E Hugo -- gohugo.io Sat, 27 Nov 2021 20:45:21 +0000 https://eric.nz/posts/aws-cost-optimisation/ Sat, 27 Nov 2021 20:45:21 +0000 https://eric.nz/posts/aws-cost-optimisation/ Reference AWS Workshop: Cost Optimisation https://eric.nz/posts/sys-design-data-lake-on-aws/ Tue, 10 Nov 2020 10:16:29 +0000 https://eric.nz/posts/sys-design-data-lake-on-aws/ Reference https://aws.amazon.com/blogs/architecture/lets-architect-modern-data-architectures/ https://garystafford.medium.com/building-a-simple-data-lake-on-aws-df21ca092e32 https://medium.com/pythonistas/complete-guide-to-aws-data-lake-4cc85259deb0 https://eric.nz/posts/aws-cert-sysops-prep/ Sat, 08 Aug 2020 12:40:14 +0000 https://eric.nz/posts/aws-cert-sysops-prep/ Billing and Cost Management User-Defined Cost Allocation Tags User-defined tags are tags that you define, create and apply to resources.After you have create and applied them, you can activate them on the Billing and Cost Management console for cost allocation tracking. The detailed steps are: Log in to the AWS Management Console of the new account Use the Tag Editor to create the new user-defined tags Use the Cost Allocation Tag manager in the payer account to mark the tags as cost allocation tags Certificate Manager (ACM) When you request a public certificate, AWS Certificate Manager (ACM) generates a public/private key pair https://eric.nz/posts/aws-cloud-practitioner/ Fri, 17 Jan 2020 14:00:34 +0000 https://eric.nz/posts/aws-cloud-practitioner/ General Information Introduction to AWS Certified Cloud Practitioner Examination time is around 90 minutes. Examination score is between 100-1000, and the minimum passing score is 700. Exam Content Outline Domain 1 - Cloud Concepts (26%) 1.1 Define the AWS Cloud and it’s value proposition 1.2 Identify aspects of AWS Cloud economics 1.3 List the different cloud architecture design principles Domain 2 - Security and Compliance (25%) 2.1 Define the AWS Cloud shared responsibility model 2. https://eric.nz/posts/aws-athena-with-datagrip/ Wed, 11 Dec 2019 08:36:00 +0000 https://eric.nz/posts/aws-athena-with-datagrip/ Instructions Step 0 - Check Local AWS CLI Config Before you start to configure DataGrip, please ensure your AWS CLI config is EXACTLY like following screenshot IAM Role Profile needs to store in the credentials You can refer to this official AWS document: Configuration basics - AWS command-Line Interface Step 1 - Install DataGrip For Mac User Option 1: If you have brew installed on your workstation, you could execute the following command in your favourite terminal 1 brew install --cask datagrip Option 2: Download the executable at https://www. https://eric.nz/posts/aws-storage-comparison/ Sun, 03 Nov 2019 15:13:43 +0000 https://eric.nz/posts/aws-storage-comparison/ Block Storage vs Object Storage Block Storage: block level operations are possible one block changed, e.g piece of the file, that contains the changed data can be updated in block level Object Storage: Entire file must be removed and new file needs to be put there S3 vs EBS vs EFS S3: S3 to be used for WORM operations, e.g. Write once Read many times Scalable, size not be planned not suitable for hosting OS or Database EBS: https://eric.nz/posts/aws-cli-s3-command/ Sat, 02 Feb 2019 19:34:49 +0000 https://eric.nz/posts/aws-cli-s3-command/ Reference TheGeekStuff: 28 Essential AWS S3 CLI Command Examples to Manage Buckets and Objects https://eric.nz/posts/aws-cli-request-time-too-skewed-error/ Sat, 02 Feb 2019 19:34:49 +0000 https://eric.nz/posts/aws-cli-request-time-too-skewed-error/ This morning I was about to kickoff some typical jobs with AWS CLI on an Linux EC2 instance. The AWS CLI command always raised the error, 1 2 3 [user@host ~]$ aws s3 ls An error occurred (RequestTimeTooSkewed) when calling the ListBuckets operation: The difference between the request time and the current time is too large. even with the simplest list object command the error were still there!!! By querying Stack Overflow, some people had the same problem with S3 upload, and suggested to compare the local time with Amazon server-side time. https://eric.nz/posts/aws-emr-get-cluster-info-with-powershell/ Wed, 21 Nov 2018 03:11:48 +0000 https://eric.nz/posts/aws-emr-get-cluster-info-with-powershell/ In order to get information from an existing EMR cluster, we can use 1 PS S:\ Get-EMRCluster -ClusterId $ClusterId The command will then return a system object in Amazon.ElasticMapReduce.Model.Cluster type. The Cluster object provides the following attributes that maybe useful MasterPublicDnsName. The DNS name of the master node. NormalizedInstanceHours. An approximation of the cost of the cluster. ReleaseLabel. The release label of Amazon EMR. Status. The current status details about the cluster. https://eric.nz/posts/aws-emr-jobflow-error/ Tue, 30 Oct 2018 00:00:22 +0000 https://eric.nz/posts/aws-emr-jobflow-error/ I came across this error this morning with EMR and Spark steps. 1 An error occurred (ValidationException) when calling the RunJobFlow operation: 1 validation error detected: Value '[ <YOUR-SPARK-JOB> ]' at 'steps.45.member.hadoopJarStep.args' failed to satisfy constraint: Member must satisfy constraint: [Member must have length less than or equal to 10280, Member must have length greater than or equal to 0, Member must satisfy regular expression pattern: [\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*] Or 1 botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the RunJobFlow operation: Size of step parameter length exceeded the maximum allowed. https://eric.nz/posts/aws-redshift-table-info/ Wed, 29 Aug 2018 21:57:48 +0000 https://eric.nz/posts/aws-redshift-table-info/ According to the AWS Document , Redshift provides PG_TABLE_DEF table that contains all the table information, and you can query that table like 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 SELECT DISTINCT schemaname, tablename, column, type, encoding, distkey, sortkey, notnull FROM PG_TABLE_DEF WHERE 1=1 AND schemaname in ('<SCHEMA_NAME>') ORDER BY schemaname ASC, tablename ASC ; However, this only returns all the information in the public schema, I assume there maybe some permission problem with my account. https://eric.nz/posts/aws-redshift-pipeline/ Wed, 25 Jul 2018 06:01:42 +0000 https://eric.nz/posts/aws-redshift-pipeline/ Reference Mark Smallcombe: 15 Examples of Data Pipelines Built with Amazon Redshift https://eric.nz/posts/aws-emr-best-practice/ Tue, 03 Jul 2018 21:45:00 +0000 https://eric.nz/posts/aws-emr-best-practice/ Reference AWS Big Data Blog: Best practices for resizing and automatic scaling in Amazon EMR https://eric.nz/posts/aws-redshift-data-ingestion-from-s3/ Fri, 25 May 2018 06:01:42 +0000 https://eric.nz/posts/aws-redshift-data-ingestion-from-s3/ S3 -> COPY -> Redshift Staging Database -> Redshift Database Reference Data Engineering in S3 and Redshift with Python Amazon redshift: bulk insert vs COPYing from s3 https://eric.nz/posts/aws-upload-files-to-s3-with-boto3/ Wed, 23 May 2018 01:16:06 +0000 https://eric.nz/posts/aws-upload-files-to-s3-with-boto3/ Uploading local files to AWS S3 with boto3 is quite straight forward. You can install the AWS python SDK boto3 via 1 pip install boto3 Before any implementation, please make sure you have enough permission to interactive with S3. In order to upload file to S3, you can do something like the the following 1 2 3 4 import boto3 s3res = boto3.resource("s3", region="us-east-1") s3.meta.client.upload_file("<LOCAL_FILE_PATH>", "<YOUR_BUCKET>", "<YOUR_KEY>") For example, you can use the above snippet like https://eric.nz/posts/aws-redshift-query-execution-plan/ Wed, 09 May 2018 03:56:59 +0000 https://eric.nz/posts/aws-redshift-query-execution-plan/ https://eric.nz/posts/aws-summit-2017/ Thu, 01 Jun 2017 01:38:00 +0000 https://eric.nz/posts/aws-summit-2017/ TL;DR There are not many valuable gifts so no need to go early Sponsor’s keynote are all like product promoting session Do check the speaker’s background before choosing any session If you just want to know technical information, try to avoid product people’s talk No need for laptop or notebook, smart phones can handle everything If you want to get more gifts, you need to talk with more sponsors If you find the keynote is boring, don’t waste your time Keynotes Modern Data Architecture on AWS / SQL Strikes Back! https://eric.nz/posts/aws-concept/ Sun, 07 May 2017 04:49:08 +0000 https://eric.nz/posts/aws-concept/ CloudWatch CloudWatch’s Free Tier metric update frequency is 5 minutes In the Detailed monitoring data available for your EBS volumes, provisioned IOPS volumes automatically send 1 minute metrics to CloudWatch. EBS EC2 ec2-revoke RevokeSecurityGroupIngress means remove one or more rules from a security group. The value you specify in the revoke request must be existing rule’s value for the rule to be removed. ec2-create-group CreateSecurityGroup means create a security group for use with your account. https://eric.nz/posts/aws-unable-to-ssh-ec2/ Tue, 13 Dec 2016 19:26:50 +0000 https://eric.nz/posts/aws-unable-to-ssh-ec2/ Yesterday when I tried to ssh to remote EMR cluster, it raised this problem 1 2 3 4 5 6 7 8 9 10 11 12 13 14 [ec2-user@ds-uat-airflow1 ~]$ ssh -i ~/YOUR_SECRET.pem hadoop@ip-YOUR_IP_ADDRESS.us-west-2.compute.internal @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed.