Posts

Right way to explore tables in BigQuery This LinkedIn Post by Yingjun Wu ( link ) had show cased a ridiculous case of BigQuery pricing model. BigQuery charges based on the amount of data referenced, not by the amount of data processed. This means that even you used a LIMIT keyword to limit the number of rows returned, BigQuery will still charge you the amount of data query referenced. The right way to explore tables in BigQuery is to use TABLESAMPLE clause to sample the data....

Creating and Maintaining dbt documentation Debugging data modelling errors Developing dbt models Materialisation Implementing and Maintaining External Dependencies Implementing dbt tests Indirect Selection Leveraging the dbt state Managing data pipelines Understanding dbt models governance restrict-access configuration Core Concepts Common dbt commands Hooks Materialisation Semantic Layer Snapshot How to create snapshot Valid configurations in dbt_project.yml dbt Tests Why do we need test? What makes a good test? Test on One Database Object Troubleshooting Table missing primary key constraint during execution Optimise SQL models with {{ref()}} function Best Practices Structure a dbt project Reference Creating and Maintaining dbt documentation Debugging data modelling errors Developing dbt models Materialisation view incremental table ephemeral If the materialized option is not specified, the default dbt materialisation is view....

Authentication & Authorisation Let Pre-authenticated user to access application via Entra ID Privileged Identity Management (PIM) Enable Principle of Least Possible Privilege, Just-in-Time access, and periodic review access Monitoring and Alerting Receive alerts based on conditions met in the application Logging Correlate Azure Resources Usage and Performance data Miscellaneous Azure Bicep Authentication & Authorisation Let Pre-authenticated user to access application via Entra ID Entra ID Enterprise Application. This would be most suitable solution to meet the requirement of pre-authenticating users using their Entra ID accounts....

Azure Machine Learning HyperDrive Overview How HyperDrive Works HyperDrive Search Algorithms Model Interpretability Ensure version control and tracking all experiments for a team Increase Model Training Speed Effectively Manage Computational Resources Consume a large dataset stored in Azure Data Lake for Machine Learning model training Ensure Model does not exhibit bias based on feature Azure Machine Learning Designer Overview Missing required Python packages in Azure Machine Learning Designer Azure Monitor Application Insights Model Evaluation Methods Confusion Matrix Recall Precision Accuracy F1-Score ROC AUC Miscellaneous Model Deployment within Constrained Environment One-Hot Encoding Initial run with Minimal Resources Azure Blob Storage vs Azure Data Lake Storage Azure Machine Learning HyperDrive Overview HyperDrive in Azure Machine Learning is a tool used for hyperparameter tuning....

Azure OpenAI Terms Authentication with Azure OpenAI APIs Connect to app based on OpenAI Resource Connect to OpenAI app via OpenAI SDK Prevent Hate Speech OpenAI RESTful APIs OpenAI Model Settings Token Types OpenAI Cost Model Common OpenAI SDK methods Improve the quality of responses from OpenAI-backed Chatbot with minimal efforts Improve the accuracy of Chatbot response for similar questions Create OpenAI based marketing press release app Azure AI Search Glossary Stages for AI Enrichment Types of projections and Usage Key Types Have compromised key and want to prevent unauthorised access to Search Endpoint Have throttled request for AI Search Implement autocompletion with Cognitive Search Azure AI Services Improve reading experience for dyslexia Different types of content display in Chatbot Azure Cognitive Services Overview Provision Cognitive Services with Azure CLI Deploy a containerised Cognitive Services service for Sentiment Analysis Configure your App to connect to Cognitive Services service with minimal administrative effort Ensure Cognitive Services based Chatbot recognise your company’s products and codename Content Safety Optimise content filter configurations for objectionable content Monitor Online Activity Protected Material Detection Speech Types of Speech Services Optimise the text-to-speech sound quality for poor network conditions Enable Speech Capability for a Chatbot Document Intelligence What is Document Intelligence Translate files from German to French with custom glossary Deploy Document Intelligence resource in an ARM template Extract Information from Scanned document with minimal development efforts Pre-built models in Document Intelligence Threshold of Confidence score in Document Intelligence Vision Analyse image with Vision SDK Compute Vision API vs Custom Vision API Computer Vision vs Document Intelligence Custom Vision Overview Steps to build and train a custom classification model Face Increase the likelihood to detect faces in blurred images and sideways faces Translate Common Translate API Endpoints Service Endpoint for Translate Language Minimise similar utterances from input Use container on the host computer from Language Understanding (LUIS) Ensure only specific Azure processes can access Language Service Common Packages and its usage scenarios Entity Extraction Key Phrase Extraction Video Indexer Best way to index a 20GB Video file with Azure Video Indexer Miscellaneous Only specific resources to access Azure AI Services Identify unusual values in Time Series Detect when a user ask questions via Video and Audio Azure OpenAI Terms Frequency penalty parameter....