Data Warehouse

Not in 3NF (Big Data)

  • The data should be de-normalised to 2NF
  • This means you get data redundancy
  • This means you need more storage
  • But you can get at the data more quickly
  • The purposes of a DW is to provide aggregate data which is in suitable format for decision making.

Typical Data Warehouse Architecture

data warehouse arch

Staging / Integration Layer

  • Create data in particular day
  • Ensure all data comes from the same format

Data Warehouse / Data Vault

  • All Data are having the same format
  • Everything in 2NF

Data Marts

  • Different department or team can have their dedicated Data Mart, with their own business logic applied.
  • Provide information more quickly
  • One user does data manipulation in his Data Mart will not affect other user in their Data Mart

ETL and Data Marts

  • Extraction, Transformation and Loading
    • Extraction. Get the Data
    • Transformation. make it useful.
    • Loading. Save it to the warehouse.
  • Data Marts (Sub-sets of the Data Warehouse)
    • Don’t mess with my data.
    • Keep it simple for the user.
    • Small problems are easier to solve