Block Storage vs Object Storage Block Storage: block level operations are possible one block changed, e.g piece of the file, that contains the changed data can be updated in block level Object Storage: Entire file must be removed and new file needs to be put there S3 vs EBS vs EFS S3: S3 to be used for WORM operations, e.g. Write once Read many times Scalable, size not be planned not suitable for hosting OS or Database EBS:...
Reference TheGeekStuff: 28 Essential AWS S3 CLI Command Examples to Manage Buckets and Objects
This morning I was about to kickoff some typical jobs with AWS CLI on an Linux EC2 instance. The AWS CLI command always raised the error, 1 2 3 [user@host ~]$ aws s3 ls An error occurred (RequestTimeTooSkewed) when calling the ListBuckets operation: The difference between the request time and the current time is too large. even with the simplest list object command the error were still there!!! By querying Stack Overflow, some people had the same problem with S3 upload, and suggested to compare the local time with Amazon server-side time....
In order to get information from an existing EMR cluster, we can use 1 PS S:\ Get-EMRCluster -ClusterId $ClusterId The command will then return a system object in Amazon.ElasticMapReduce.Model.Cluster type. The Cluster object provides the following attributes that maybe useful MasterPublicDnsName. The DNS name of the master node. NormalizedInstanceHours. An approximation of the cost of the cluster. ReleaseLabel. The release label of Amazon EMR. Status. The current status details about the cluster....
I came across this error this morning with EMR and Spark steps. 1 An error occurred (ValidationException) when calling the RunJobFlow operation: 1 validation error detected: Value '[ <YOUR-SPARK-JOB> ]' at 'steps.45.member.hadoopJarStep.args' failed to satisfy constraint: Member must satisfy constraint: [Member must have length less than or equal to 10280, Member must have length greater than or equal to 0, Member must satisfy regular expression pattern: [\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*] Or 1 botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the RunJobFlow operation: Size of step parameter length exceeded the maximum allowed....