Azure Machine Learning

HyperDrive Overview

  • HyperDrive in Azure Machine Learning is a tool used for hyperparameter tuning.
  • It automates the process of finding the best set of hyperparameters for a machine learning model, by running multiple training experiments in parallel with different combinations of hyperparameters.

How HyperDrive Works

  1. Create a HyperDrive configuration.
    1. You specify the hyperparameters to tune, the ranges for these hyperparameters, the number of concurrent runs, and the search algorithm to use.
  2. Launch multiple training runs.
    1. HyperDrive will run multiple training jobs in parallel, each with a different combination of hyperparameters from defined search space.
  3. Evaluate performance.
    1. HyperDrive evaluates each run based on a metrics (e.g. accuracy, loss), and selects the best hyperparameter configuration that leads to the optimal performance of the model.
  4. Early termination.
    1. To save time and resources, HyperDrive can use early termination policies.
    2. If a run performs poorly early on, HyperDrive can terminate that run before it completes to focus on more promising hyperparameter sets.

HyperDrive Search Algorithms

  1. Random Search.
    1. Samples hyperparameters randomly from the search space.
    2. It can efficiently explore the search space and doesn’t need to check all the possible values unlike grid search.
  2. Grid Search.
    1. Exhaustively searches through a specified hyperparameter grid.
    2. It can be computationally expensive and time-consuming for large datasets.
  3. Bayesian Optimisation.
    1. Tries to intelligently select hyperparameters based on past performance of previous runs, reducing the number of trials needed to find the optimal values.
    2. Bayesian Sampling balances exploration and trends to find optimal solutions more efficiently than grid or random sampling.
    3. It can be computationally expensive for large datasets.
  4. Bandit Policy.
    1. This is to used to terminate poorly performing runs early based on a specified metric.
    2. Bandit policy enhances computational efficiency by terminating runs that are not promising.

Model Interpretability

  1. Set the primary metric to log_loss and enable model explanation.
  2. You can also use Azure’s Model Interpretability Toolkit to get global and local feature importance information, that can help understand how the model is making predictions.

Ensure version control and tracking all experiments for a team

  1. Use Azure Machine Learning Experiment Tracking capabilities, allowing data scientists to keep track of different runs, compare metrics, and manage models in a centralised way, facilitating efficient collaboration and project management.

Increase Model Training Speed

  1. Create an Azure Machine Learning training cluster as the compute target, and use the parallel run steps in the pipeline to train models concurrently.
  2. Azure Machine Learning training clusters are designed to run machine learning workloads, and they can scale to provide more compute resources as needed.
  3. The ParallelRunStep in Azure Machine Learning pipelines allows for concurrent execution of tasks, which is ideal for running multiple model training tasks in paralle.

Effectively Manage Computational Resources

  1. Use Azure Machine Learning Compute Clusters with min_node set to 0, and max_nodes defined as per the computation demand.
  2. Azure Machine Learning compute clusters allow automatic scaling based on the workload.
  3. Setting min_nodes to 0 means no charges are incurred during periods of inactivity.

Consume a large dataset stored in Azure Data Lake for Machine Learning model training

  1. Register the data asset in Azure Machine Learning, then use the Dataset object in the Python script to consume the data.

Ensure Model does not exhibit bias based on feature

  1. Use Automated ML with fairness constraints.
    1. Automated ML in Azure can include fairness constrains to ensure model do not exhibit bias against certain groups.

Azure Machine Learning Designer

Overview

  1. Azure Machine Learning Designer offers a visual interface for building, training, and deploying machine learning models, making it an ideal choice for constructing the required pipeline.
  2. It allows managing each step of the process, from data preprocessing and feature selection to model training and evaluation in a visual way without writing code.

Missing required Python packages in Azure Machine Learning Designer

  1. Create a custom Docker Image with the required Python packages installed, and use the Python Script in Docker module.
  2. The Python Script in Docker module allows you to run Python scripts inside a Docker container, which can be customised with any required packages.

Azure Monitor

Application Insights

  • Application Insights is an Azure service that helps you monitor the performance and usage of your application.
  • Application Insights can be enabled for Azure ML real-time endpoints to monitor the performance, detect anomalies, and get detailed telemetry on the web service’s usage and performance.

Model Evaluation Methods

Confusion Matrix

  • True Positive (TP). The prediction is positive, reality is positive.
  • True Negative (TN). The prediction is negative, reality is negative.
  • False Positive (FP). The prediction is positive, reality is negative.
  • False Negative (FN). The prediction is negative, reality is positive.
Actual\PredictedPositiveNegative
PositiveTrue Positive (TP)False Negative (FN)
NegativeFalse Positive (FP)True Negative (TN)

Recall

$$ Recall = \frac{TP}{TP + FN} $$

  • Recall (召回率) measures the ration of true positives to the sum of true positives and false negatives.
  • Recall is also known as Sensitivity or True Positive Rate.
  • Recall is focusing on how many of the actual positive cases were correctly identified by the model.
  • Recall is important when the cost of false negatives is very high.

Precision

$$ Precision = \frac{TP}{TP + FP} $$

  • Precision (精确率) measures the ratio of true positives to the sum of true positives and false positives.
  • Precision is focusing on how many of the predicted positive cases were actually positive.

Accuracy

$$ Accuracy = \frac{TP + TN}{TP + TN + FP + FN} $$

  • Accuracy (准确率) measures the ratio of the sum of true positives and true negatives to the total number of cases.

F1-Score

$$ F1-Score = 2 \times \frac{Precision \times Recall}{Precision + Recall} $$

  • F1-Score is the harmonic mean of precision and recall.
  • When Precision and Recall are both high, the F1-Score is high.
  • When either Precision or Recall is low, the F1-Score will be low, that means the model has some serious issues.
  • F1-Score equals to 1 means the model is perfect, and 0 means the model is terrible.
  • F1-Score is a good metric to use when the classes are imbalanced.

ROC

  • Receiver Operating Characteristic Curve (ROC) is calculated based on True Positive Rate (Recall) and False Positive Rate (FPR).

$$ TPR = \frac{TP}{TP + FN} $$

$$ FPR = \frac{FP}{FP + TN} $$

  • ROC curve shows how well the model distinguishes between positive and negative classes by varying the classification threshold.
  • The better the model, the closer the ROC curve is to the top-left corner of the plot.

AUC

  • Area Under the Curve (AUC) is the area under the ROC curve, which is a single scalar value summarising the performance of the model across all possible classification thresholds.
  • AUC = 1 presents a perfect model that perfectly distinguishes between positive and negative classes.
  • AUC = 0.5 indicates that the model performs no better than random guessing.
  • AUC < 0.5 suggest the model performs worse than random guessing. it could be classifying everything in the opposite direction.

Usage

  • Imbalanced datasets.
    • AUC-ROC is particularly useful when dealing with imbalanced datasets.
    • In such cases, accuracy alone may be misleading because the AUC and ROC curve provide a more comprehensive picture of the model’s performance across various threshold.
  • Binary classification problems.
    • AUC and ROC curves are mostly used to evaluate binary classifiers.

Miscellaneous

Model Deployment within Constrained Environment

  • Azure IoT Edge allows you to deploy complex event processing, machine learning, image recognition, and other high-value AI without writing it in-house.
  • Models can be deployed an run locally on Edge devices, which makes it an ideal choice for environments with restricted internet access.

One-Hot Encoding

  1. One-Hot Encoding is a technique used to convert categorical data into a numerical format.

Initial run with Minimal Resources

  1. You can use compute_target = 'local' to run the initial training experiment with minimal resources.
  2. Specifying the compute target as local will run the experiment on the local machine, which is useful for debugging and testing purposes, ensuring that it works as intended before deploying to a more resource-intensive cloud environment.

Azure Blob Storage vs Azure Data Lake Storage

  1. Azure Blob Storage
    1. is a general-purpose object storage solution for a wide variety of storage scenarios.
    2. Ideal for applications that need to store and retrieve large amounts of unstructured data, such as website content, backup files, archives, and streaming data.
  2. Azure Data Lake Storage (ADLS)
    1. Specifically optimised for big data analytics and large-scale data workloads.
    2. Commonly used for data lakes where organisations store vast amounts of raw data, including structured, semi-structured, and unstructured data, to perform analytics and machine learning, and real-time reporting.
    3. ADLS is generally a better choice when you’re dealing with large volumes of data that require real-time or near-real-time processing and analytics.