https://eric.nz/tags/concept/ Recent content in concept on Eric's Note https://eric.nz/%3Clink%20or%20path%20of%20image%20for%20opengraph,%20twitter-cards%3E https://eric.nz/%3Clink%20or%20path%20of%20image%20for%20opengraph,%20twitter-cards%3E Hugo -- gohugo.io Wed, 13 May 2020 21:38:38 +0000 https://eric.nz/posts/snowflake-concept/ Wed, 13 May 2020 21:38:38 +0000 https://eric.nz/posts/snowflake-concept/ Basic Concept Snowflake is a new model cloud-based enterprise-level data warehouse. Architecture Snowflake’s architecture is a hybrid of traditional shared-disk database architectures and shared-nothing databases architectures. Similar to shared-disk architectures, Snowflake uses a central data repository for persisted data that is accessible from all compute nodes in the data warehouse. But similar to shared-nothing architectures, Snowflake processes queries using MPP (massively parallel processing) compute clusters where each node in the cluster stores a portion of the entire dataset locally. https://eric.nz/posts/airflow-concept/ Fri, 01 Jun 2018 23:31:29 +0000 https://eric.nz/posts/airflow-concept/ What is Airflow? Airflow is a platform to programmatically author, schedule and monitor your workflows and pipelines. What are the benefits for using Airflow? Programmatically author workflow In Airflow, you can define your workflow programmatically with Python scripts and that would put you in a very good position by leveraging all the convenience and sweet that Python provide. This is a huge improvement if you experienced with Oozie or other GUI-typed (or even without a GUI) scheduling tools. https://eric.nz/posts/sys-design-core-concept/ Tue, 31 Oct 2017 20:19:41 +0000 https://eric.nz/posts/sys-design-core-concept/ Vertical Scaling vs Horizontal Scaling Vertical Scaling scale-up, add more CPU/Memory/Disk on the same server. Horizontal Scaling scale-out, add more servers. Load Balancer (LB) A Load Balancer is a device/service that sits between the user and the server group, and act as an invisible facilitator, ensuring that all resource servers are used equally. Load Balancing can optimise the resource time and avoid unevenly overloading some compute nodes while other compute nodes are left idle. https://eric.nz/posts/aws-concept/ Sun, 07 May 2017 04:49:08 +0000 https://eric.nz/posts/aws-concept/ CloudWatch CloudWatch’s Free Tier metric update frequency is 5 minutes In the Detailed monitoring data available for your EBS volumes, provisioned IOPS volumes automatically send 1 minute metrics to CloudWatch. EBS EC2 ec2-revoke RevokeSecurityGroupIngress means remove one or more rules from a security group. The value you specify in the revoke request must be existing rule’s value for the rule to be removed. ec2-create-group CreateSecurityGroup means create a security group for use with your account. https://eric.nz/posts/ml-glossary/ Tue, 20 Aug 2013 19:37:02 +0000 https://eric.nz/posts/ml-glossary/ A B C D E F G H I J K L M N O One-hot encoding One-hot encoding is a way to represent categorical variables as numerical data, so that it can be used in machine learning algorithm. It involves creating a new binary column for each unique category in the categorical feature. For example, if a categorical feature has three categories, A, B, and C. Then three new columns, one for each category would be created. https://eric.nz/posts/sql-advanced-concepts/ Thu, 14 Mar 2013 15:36:15 +0000 https://eric.nz/posts/sql-advanced-concepts/ Query Execution Order Most people would write their SQL queries starting from SELECT part, because it’s more intuitive and close to our natural language. But actually that’s not the way that SQL queries been executed in query engine. Below is the execution order of a SQL query: FROM, JOIN. Tables are joined to get the base data. WHERE. The base data is filtered. GROUP BY. The filtered based data is grouped. https://eric.nz/posts/sql-basic-concepts/ Thu, 14 Feb 2013 15:36:15 +0000 https://eric.nz/posts/sql-basic-concepts/ Join Full Join Inner Join Left Join Right Join Aggregation Function COUNT(). MAX(). MIN(). SUM(). AVG(). The GROUP BY statement groups rows that have the same values into summary rows, like ‘find the number of customers in each country’. The GROUP BY statement is often used with above aggregation functions to group the result-set by one or more columns.