Posts

Performance Metrics Scalability Ability of a system to grow and manage increased traffic. Increased volume of data or requests. Our goal is we want to achieve this growth without a lose in performance. Bad system design could result in a bottleneck on the number of users or traffic our application can handle, or could result in exponentially increasing cost to server a small increased traffic. Reliability Probability a system will fail during a period of time....

General System Design Tips Always to keep your design for your application as simple as possible. Only change things once that’s required! Distributed System A Distributed System is basically a group of computer, aka Cluster, working together and the goal is that all the complexity should be hidden completely from the users. From user perspective, they just interact with a single computer. Behind the scenes when you think about something like Google, Amazon, Facebook, they have multiple data center with tons of servers to manage....

CAP Theorem CAP Theorem stands for Consistency, Availability, and Partition Tolerant. Reference CAP Theorem

Yesterday when I tried to ssh to remote EMR cluster, it raised this problem 1 2 3 4 5 6 7 8 9 10 11 12 13 14 [ec2-user@ds-uat-airflow1 ~]$ ssh -i ~/YOUR_SECRET.pem hadoop@ip-YOUR_IP_ADDRESS.us-west-2.compute.internal @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed....

Prune images 1 docker image prune -af or 1 docker rm $(docker ps -q -a -f "status=exited")