https://eric.nz/categories/databricks/ Recent content in Databricks on Eric's Note https://eric.nz/%3Clink%20or%20path%20of%20image%20for%20opengraph,%20twitter-cards%3E https://eric.nz/%3Clink%20or%20path%20of%20image%20for%20opengraph,%20twitter-cards%3E Hugo -- gohugo.io Sun, 20 Aug 2023 11:41:57 +0000 https://eric.nz/posts/dbx-cert-ml-ass-prep/ Sun, 20 Aug 2023 11:41:57 +0000 https://eric.nz/posts/dbx-cert-ml-ass-prep/ https://eric.nz/posts/dbx-cert-de-pro-prep/ Wed, 26 Apr 2023 10:26:32 +0000 https://eric.nz/posts/dbx-cert-de-pro-prep/ 01 - Modeling Data Management Solutions Bronze Ingestion Patterns Ingestion Patterns Singleplex: One-to-One mapping of source datasets to bronze tables. Multiplex: Many-to-One mapping, i.e. many datasets are mapped to one bronze table. Singleplex is the traditional ingestion model where each data source or topic is ingested separately. Singleplex usually works well for batch processing. However, for streaming processing of large datasets, if you have many streaming jobs, one per topic, you will hit the maximum limit of concurrent jobs in your workspace. https://eric.nz/posts/dbx-cert-de-pro-guide/ Tue, 25 Apr 2023 20:09:52 +0000 https://eric.nz/posts/dbx-cert-de-pro-guide/ General Databricks Certified Data Engineer Professional: link Time allotted to complex exam is 2 hours (120 minutes) Exam fee $200 USD Number of questions 60 Question type: multiple choice questions Passing scores is at least 70% on the overall exam Code Example data manipulation code will be in SQL when possible Structured Streaming code will be in Python Runtime version is DBR 10.4 LTS Practice Exam: link Target Audience Data Engineer, >= 2yoe Advanced, practitioner certification Assess candidates at a level equivalent to two or more years with data engineering with Databricks Expectation Understanding of the Databricks platform and developer tools Ability to build optimised and cleaned data processing pipelines using the Spark and Delta Lake APIs Ability to model data into a Lakehouse using knowledge of general data modeling concepts Ability to make data pipelines secure, reliable, monitored, and tested before deployment Out of Scope The following is not expected of a Professional-level data engineer: https://eric.nz/posts/dbx-best-practice/ Thu, 16 Mar 2023 10:26:32 +0000 https://eric.nz/posts/dbx-best-practice/ Table Partitioning Table Cloning Table Overwriting Performance Optimisation OPTIMIZE VACUUM Delta Live Table Create a Silver Table from a Bronze Table Data Ingestion Multiple Data Source Single Data Source Data Quality Quality Enforcement with CONSTRAINT General GLOBAL TEMP VIEW CREATE TABLE USING Auto Loader VS COPY INTO SCD Type 2 Table Change Data Capture (CDC) feed Change Data Feed (CDF) Dynamic View Column level control Row level control Structured Streaming Streaming query to handle late-arriving data Streaming query to calculate business-level aggregation for each non-overlapping five-minute interval. https://eric.nz/posts/dbx-delta-lake-whitepaper/ Sun, 12 Mar 2023 14:28:12 +0000 https://eric.nz/posts/dbx-delta-lake-whitepaper/ Delta Lake builds upon standard data formats. Delta Lake table gets stored on storage in one or more data files in Apache Parquet format, along with transaction logs in JSON format. Reference Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics Michael Armburst (@databricks) Ali Ghodsi (@databricks, @uc berkeley) Reynold Xin (@databricks) Matei Zaharia (@databricks, @stanford) Michael Armburst: Boston Spark Meetup @ Wayfair / Delta Lake: Open Source Reliability and Quality for Data Lakes Delta Lake Inside YouTube: Understanding Delta File Logs - The Heart of the Delta Lake https://eric.nz/posts/dbx-reference/ Wed, 08 Mar 2023 15:56:40 +0000 https://eric.nz/posts/dbx-reference/ Documentation Azure Databricks documentation, link Databricks documentation, link Data objects in Databricks, link SQL Language Reference, link Databricks Alphabetical list of built-in functions, link Data Engineering With Databricks, GitHub Tools Databricks Utilities, link Unzip dbc files, GitHub Tutorials Data Engineering With Databricks, GitHub Databricks Training tutorials Databricks: Databricks Training, YouTube Databricks on the AWS Cloud Quick Start Reference Deployment dbt: Configure Databricks for dbt Cloud https://eric.nz/posts/dbx-cert-de-ass-prep/ Tue, 07 Mar 2023 15:20:27 +0000 https://eric.nz/posts/dbx-cert-de-ass-prep/ 00 - General Databricks documentation, link Data Engineering With Databricks, GitHub 01 - Databricks Workspace and Services Databricks Architecture and Services Databricks Control Plane Web Application Databricks SQL Databricks Machine Learning Databricks Data Science and Engineering Repos / Notebooks Job Scheduling Cluster Management Cluster Cluster are made up of one or more virtual machine (VMs) instances Driver node. Coordinate activities of executors, aka master node in EMR. Executor node. Run tasks composing a Spark job, aka run node in EMR. https://eric.nz/posts/dbx-cert-de-ass-guide/ Mon, 20 Feb 2023 06:28:47 +0000 https://eric.nz/posts/dbx-cert-de-ass-guide/ General Databricks Certified Data Engineer Associate: link Time allotted to complex exam is 1.5 hours (90 minutes) Exam fee $200 USD Number of questions 45 Passing scores is at least 70% on the overall exam Code Example data manipulation code will be in SQL when possible Structured Streaming code will be in Python Runtime version is DBR 10.4 LTS Practice Exam: link Expectation Databricks Lakehouse Platform (24%). Understand how to use and the benefits of using the Databricks Lakehouse Platform and its tools. https://eric.nz/posts/dbx-lakehouse-fundamentals/ Wed, 08 Feb 2023 13:51:23 +0000 https://eric.nz/posts/dbx-lakehouse-fundamentals/ What is a Data Lakehouse? History of Data Warehouse Pros Business Intelligence (BI) Analytics Structured & Clean Data Predefined Schemas Cons Not support semi or unstructured data Inflexible schemas Struggled with volume and velocity upticks Long processing time History of Data Lake Pros Flexible data storage Structured, semi-structured, and unstructured data Steaming support Cost efficient in the cloud Support for AI and Machine Learning Cons No transactional support Poor data reliability Data Lake are not supportive of transactional data, and cannot enforce data quality Primarily due to multiple data types Slow analysis performance Because large volume of data, the performance of analysis is slower the timeliness of decision-making results has never manifested Data governance concerns Governance over the data in a data lake creates challenges with security, and privacy enforcement due to the unstructured nature of the contents of a data lake Data Warehouse still needed Problems with Complex Data Environment Data Lake didn’t fully replaced Data Warehouse for reliable BI insights, Business has implemented complex systems to have Data Lake, Data Warehouse, and additional systems to handle streaming data, machine learning and artificial intelligence requirements. https://eric.nz/posts/dbx-learning-path/ Wed, 08 Feb 2023 13:34:08 +0000 https://eric.nz/posts/dbx-learning-path/ Learning Path what is Databricks ( link ) what is Databricks Lakehouse ( link ) what are the ACID guarantees on Databricks ( link ) What is the medallion Lakehouse architecture ( link ) Databricks Architecture ( link ) Launching a Databricks all-purpose compute cluster ( link ) Creating a Databricks notebook ( link ) Executing notebook cells to process, query, and preview data ( link ) Create, run, and manage Databricks Jobs ( link ) Configuring incremental data ingestion to Delta Lake ( link ) Scheduling a notebook as a Databricks job ( link ) Databricks SQL ( link ) https://eric.nz/posts/dbx-glossary/ Wed, 01 Feb 2023 15:53:31 +0000 https://eric.nz/posts/dbx-glossary/ A ACL Access Control List (ACL). Auto Compaction Auto Compaction is part of the Auto Optimise feature in Databricks. It checks after an individual write, if files can further be compacted, if yes it runs an OPTIMISE job with 128MB file sizes instead of 1GB file size used in the standard OPTIMISE. Auto Compaction use 128MB for compacting files, while OPTIMISE command use 1GB. Auto Loader Auto Loader monitors a source location, in which files accumulate, to identify and ingest only new arriving files with each command run.