Data Engineering

Reference 28 Jupyter Notebook tips, tricks, and shortcuts Nazif Berat: Boosting Your Jupyter Notebook Productivity

Converting to Parquet https://stackoverflow.com/questions/45043554/how-to-read-a-list-of-parquet-files-from-s3-as-a-pandas-dataframe-using-pyarrow Reference Twitter: Dremel Made Simple with Parquet Databricks: The Parquet Format and Performance Optimization Opportunities Boudewijn Braams

Problem Definition One typical Airflow usage scenario is to continuously execute some workflow with regular base, and the output data of last iteration will be the input data for the next iteration. One way we can do that is to keep your output data as a local file or store that into database table, and read and update those data in every iteration. However, with those solutions you need to manual handle database connections and that is not convenient sometime....

What is Airflow? Airflow is a platform to programmatically author, schedule and monitor your workflows and pipelines. What are the benefits for using Airflow? Programmatically author workflow In Airflow, you can define your workflow programmatically with Python scripts and that would put you in a very good position by leveraging all the convenience and sweet that Python provide. This is a huge improvement if you experienced with Oozie or other GUI-typed (or even without a GUI) scheduling tools....

Inmon Framework (Enterprise Data Warehouse) A centralized data repository that Kimball Framework (Dimensional Data Warehouse) Reference Data Warehouse Architecture Comparison: Kimball and Inmon dbt: Kimball in the Context of Morden Data Warehouse