Posts

Reference Mark Smallcombe: 15 Examples of Data Pipelines Built with Amazon Redshift

Converting to Parquet https://stackoverflow.com/questions/45043554/how-to-read-a-list-of-parquet-files-from-s3-as-a-pandas-dataframe-using-pyarrow Reference Twitter: Dremel Made Simple with Parquet Databricks: The Parquet Format and Performance Optimization Opportunities Boudewijn Braams

Reference AWS Big Data Blog: Best practices for resizing and automatic scaling in Amazon EMR

kubectl create is so called Imperative Management. This approach will tell Kubernetes API what you want to create, replace or delete, not how you want your Kubernetes cluster world to look like. kubectl apply is part of Declarative Management approach, where changes that you may have applied to a live object (i.e. through scale) are maintained even if you apply other changes to the object. Both approaches are valid ways to work in production....

Problem Definition One typical Airflow usage scenario is to continuously execute some workflow with regular base, and the output data of last iteration will be the input data for the next iteration. One way we can do that is to keep your output data as a local file or store that into database table, and read and update those data in every iteration. However, with those solutions you need to manual handle database connections and that is not convenient sometime....