Azure Data Engineer Associate  [draft]

Apache Spark Cache data into the memory of local executor Spark Pool Clusters View Active Spark Streams Data Shuffling Azure Cosmos DB NoSQL Database to support API models at planet scale with low latency performance Schema Representation Azure Databricks Key Components Capacity Limits Workspace Security Network & Network Security Azure Databricks Cluster Types Azure Data Factory Azure Data Factory Core Concepts Global parameters Interact with Data Factory using Programming Languages Linked Service Diagnostic Logs Integration Runtime Azure Data Lake Gen2 (ADLS) Staging Area Processing Analytical Data workloads for best Performance Azure Event Hubs Options for an input to Azure Stream Analytics job with low latency and high throughput Azure Storage Storage Account Name Storage Analytics Storage RESTful API Access data in Storage Account with REST API Log every operation in Azure Storage Account activity in real-time Azure Synapse Analytics What is Azure Synapse Analytics?...

October 12, 2024 · 19 min · 4020 words · Eric

Fabric Analytics Engineers Associate  [draft]

Fabric Terms Capacity Lakehouse Managed Table Lakehouse Shortcut Lakehouse View Power BI DAX Formatter Power BI Model Explorer Power BI Tabular Editor (Best Practice Analyser) Tabular Editor Tenant Subscription Developer can deploy in Dev Test Workspace and view Prod Workspace Fabric Configuration Enable read-write access to dataset stored in Fabric via XMLA Endpoint Create and Publish custom Direct Lake semantic model using External Tools Lakehouse Handle frequent schema change in Copy Activity Convert Lakehouse files from CSV to Delta format for Power BI Reporting SQL Analytics Endpoint Performance Optimisation - Lots of useless logs, lots of small files Query S3 data with SQL Endpoint Prevent new tables been added to Lakehouse automatically Query data from multiple tables in different Lakehouses without copying data Data Factory Lookup Activity Pass values across multiple steps in Data Factory Pipeline Power BI Query Folding Incremental Refresh XMLA Endpoint Power BI Mode Display all rows including duplicates in PowerBI visual Distinct Count vs Unique Count in PowerQuery Calculate Duplicated Rows in PowerQuery Power BI File Formats Reduce Query Execution time for visuals on all pages on top of large dataset Incremental Refresh fails in Power BI report Semantic Model Enable Scale-out for a Semantic Model Analyse frequently used columns in memory Performance Optimisation Prevent implicit measures Identify Surrogate key column for SCD Type 2 table in Semantic Model Ensure user always use Direct Lake mode with semantic model PySpark Large dataset join a small dataset Necessary fields for Slow Change Dimension (SCD) Type2 Filver vs KeepFilters Table Maintenance Operations Miscellaneous Get maximum value of a column with PowerQuery Microsoft Fabric Capacity Metrics App Dynamic Management View (DMV) Fabric Terms Capacity Lakehouse Managed Table A managed table is a physical table that stores data in a Lakehouse....

October 3, 2024 · 13 min · 2683 words · Eric

Azure Fundamentals of Generative AI

Generative AI describes a category of capabilities within AI that create original content. GenAI applications are powered by Language Models, which are specialised type of machine learning models that you can use to perform natural language processing (NLP) tasks, including: determining sentiment or otherwise classifying natural language text. summarising text. comparing multiple text sources for semantic similarity. generating new natural language. Consider the following ways you can improve the response a copilot provides:...

July 31, 2024 · 1 min · 123 words · Eric

Azure Fundamentals (AZ-900) - Part I

Describe Cloud Computing Introduction to Cloud Computing Describe the Shared Responsibility Model Define Cloud Model Describe the Consumption-Based Model Describe the Benefits of Using Cloud Services Describe the Benefits of High Availability & Scalability in the Cloud Describe the Benefits of Reliability & Predictability in the Cloud Describe the Benefits of Security & Governance in the Cloud Describe the Benefits of Manageability in the Cloud Describe Cloud Service Types Describe Infrastructure as a Service (IaaS) Describe Platform as a Service (PaaS) Describe Software as a Service (SaaS) Reference Describe Cloud Computing Introduction to Cloud Computing What is Cloud Computing?...

July 26, 2024 · 6 min · 1192 words · Eric

Azure Fundamentals (AZ-900) - Part II

Describe the Core Architectural Components of Azure What is Microsoft Azure? Get started with Azure Accounts Describe Azure Physical Infrastructure Describe Azure Management Infrastructure Describe Azure Compute and Networking Services Describe Azure Virtual Machines Describe Azure Virtual Desktop Describe Azure Containers Describe Azure Functions Describe Application Host Options Describe Virtual Networking Describe Azure Virtual Private Networks Describe Azure ExpressRoute Describe Azure DNS Describe Azure Storage Services Describe Azure Storage Accounts Describe Azure Storage Redundancy Describe Azure Storage services Identify Azure data migration options Identify Azure file movement options Describe Azure Identity, Access and Security Describe Azure directory services Describe Azure authentication methods Describe Azure external identities Describe Azure conditional access Describe Azure Role-Based Access Control (RBAC) Describe zero trust model Describe defense-in-depth Describe Microsoft Defender for Cloud Reference Describe the Core Architectural Components of Azure What is Microsoft Azure?...

July 24, 2024 · 13 min · 2623 words · Eric