Azure OpenAI

Terms

  1. Frequency penalty parameter.
    1. It adjusts the likelihood of repeated phrases in the generated text.
  2. Abuse Monitoring.
    1. It helps to detect and report abuse after it occurs.
  3. Temperature parameter.
    1. It controls the randomness of the model’s response.

Authentication with Azure OpenAI APIs

  1. API Key is a common method for authenticating application when accessing APIs, including Azure OpenAI services.
  2. API Keys are straightforward to use and provide the necessary authentication to ensure that only authorised applications can access the specific deployments optimised for the intended workloads.

Connect to app based on OpenAI Resource

  1. You will need a deployment endpoint because each deployment of GPT-3 model in Azure OpenAI provides a unique endpoint URL that application can use to access specific deployment.
  2. By using the deployment endpoint, each app can connect to the specific deployment optimised for its intended workload.
  3. This ensures that the apps access the correct model deployment.

Connect to OpenAI app via OpenAI SDK

  1. OpenAI SDK requires the following three pieces of information to connect to the app: Deployment name, Endpoint, and Key.
  2. Deployment name.
    1. This specifies which deployment of the model you want to use.
    2. In Azure OpenAI, you might have multiple deployments of the same model, optimised for different tasks.
  3. Endpoint.
    1. This is the URL where your model is hosted.
    2. It directs the API requests to the correct Azure service instance.
  4. Key.
    1. This is the authentication key that allows your web app to access the Azure OpenAI service securely.
    2. It ensures that only authorised applications can use the service.

Prevent Hate Speech

  1. A Content Filter is designed to prevent the model from generating inappropriate or harmful outputs, such as hate speech.

OpenAI RESTful APIs

  1. openai.ChatCompletion.create() is used to create a chat completion request with the OpenAI model, which is suitable for generating responses to user questions in a conversational format.
  2. openai.Embedding.create() is used to create embeddings for text, which are typically used for tasks like similarity search, clustering, and other NLP tasks.
  3. openai.image.create() is used for generating images based on text prompts.

OpenAI Model Settings

NameDescriptionRangeExample
TemperatureControls the randomness of the model’s responses. Higher values increase creativity; lower values make out more deterministic0 to 10.2 for factual answers; 0.8 for creative writing
Top Probability (Top_p)Controls nucleus sampling by limiting the model to only the top percentage of probable tokens0 to 10.9 to sample from the top 90% of likely tokens; 0.5 for more focused outputs.
Max tokensLimits the number of tokens in the model’s response1 to model specific limit100 tokens for concise responses; 500 tokens for longer, detailed outputs
Frequency penaltyPenalise repeated words/tokens in the response, reducing redundancy-2 to 21.5 to avoid repetition in response.
Presence PenaltyPenalise the appearance of existing topics, encouraging the model to talk about new subjects.-2 to 21 to encourage new topics in a conversation
Stop SequenceSpecifies a toke nor sequence of tokens where the model should stop generating further tokensAny string or token sequence“\n” to stop after a new line, or “End of Response” to end after a customer phrase
Logit biasAllow you to adjust the likelihood of specific tokens appearing in the response by modifying their probability-100 to 100Set bias to -50 for no to reduce the likelihood of negative responses
Best_ofGenerates multiple completions server-side and returns the best one1 or more (usually 1-5)3 to generate 3 completions and choose the best response
EchoWhether the API should return the prompt along with the responseTrue/FalseSet True to include the input prompt in the output, useful for debugging or logging

Token Types

  1. Prompt token are the tokens you used to send the request to the model.
  2. Completion token are the tokens used by the model to generate a response.

OpenAI Cost Model

  1. The subscription will be charged based on the token used in the session, which includes both the prompt tokens and the completion tokens.
  2. For example, you use 50 prompt tokens and the model generate 100 completion tokens, you will be billed for a total of 150 tokens for that API call.

Common OpenAI SDK methods

  1. openai_client.GetCompletions() is used to get a completion response from the OpenAI model for a given prompt.
  2. openai_client.GetEmbeddings() is used to obtain vector representations of text inputs, which are useful for tasks like semantic search and clustering, not for generating natural language responses.
  3. openai_client.GetImageGenerations() is used for generating images from text prompts.

Improve the quality of responses from OpenAI-backed Chatbot with minimal efforts

  1. Provide grounding content.
    1. Grounding content helps to provide the model with specific context, facts, or knowledge it can use to generate more accurate and relevant responses.
    2. It can be done without modifying the model itself, thereby minimising development efforts.
  2. Add sample request/response pairs.
    1. Providing the model with examples of good interactions can help guide it towards generating higher-quality responses.
    2. This approach also does not require modifying or retraining the model, thus minimising development efforts.
  3. Other options to improve the quality, but will require more efforts
    1. Fine-tune the model requires additional effort to gather data, train the model, and validate the results.
    2. Retrain the language model by using your own data. It is a resource-intensive process that involves significant development efforts and computational resources.
    3. Train a custom large language model. A custom LLM from scratch requires extensive resources, data and development effort.

Improve the accuracy of Chatbot response for similar questions

  1. Add alternative phrasing to the question and answer (QA) pair.
    1. By adding variations of the question, you can help the QA Maker service understand that different phrasings of the same question should return the same answer.
  2. Retrain the model.
    1. After adding alternative phrasing, you need to retrain the model to incorporate the new data.
  3. Republish the model.
    1. Once the model is retrained, it needs to be republished so that the updated version is available for chatbot to use.

Create OpenAI based marketing press release app

  1. Create a deployment that use the GPT-3.5 Turbo model.
  2. Apply the Marketing Writing Assistant system message template.
  3. Deploy the solution to a new web app.

Glossary

  1. Document Cracking.
    1. Document Cracking is the process of opening files and extracting content.
    2. Text-based content can be extracted from files on a service, rows in a table, or other forms.
    3. If you add a skillset and image skills, document cracking can also extract images and queue them for image processing.
  2. Indexer.
    1. An Indexer in Azure AI Search is a crawler that extracts
  3. Skill.
  4. Skillset.
  5. Projection.
    1. Projections define the physical tables, objects, and files in a knowledge store that accept content from an Azure AI Search Enrichment pipeline.

Stages for AI Enrichment

AI Enrichment refers to integration with Azure AI Services to process content that isn’t searchable in its raw form. Through enrichment, analysis and inference are used to create searchable content and structure where none previously existed.

  1. Import.
    1. Import is the first step.
    2. The indexer connects to a data source and pulls content into the search service.
    3. Azure Blob Storage is the most common data resource used in AI Enrichment scenarios, but any supported data source can provide content.
  2. Enrich & Index.
    1. Enrichment starts when the indexer “crack document”, and extract images and text.
  3. Explore.
    1. Explore is the last step.
    2. Output is always a search index that you can query from a client app.
    3. Output can optionally be a knowledge store consisting of blobs and available in Azure Storage that are accessed through data exploration tools or downstream processes.
    4. If you’re creating a knowledge store, projections determine the data path for enriched content.

Types of projections and Usage

ProjectionStorageUsage
TablesAzure Table StorageUsed for data that’s best represented as tabular format, or whenever you need granular representations of your data (e.g. data frame). This is good for Power BI for data analysis.
ObjectsAzure Blob StorageUsed when you need the full JSON representation of your data nd enrichments in one JSON document.
FilesAzure Blob StorageUsed when you need to save normalised, binary image files.

Key Types

TypePermission LevelMaximumHow created
Admin KeyFull access (read-write) for all content operations2primary key & secondary key in the portal, are generated when service is created and can be individually re-generated on demand
Query KeyRead-only50One query key is generated with the service. More can be created on demand by admin

Have compromised key and want to prevent unauthorised access to Search Endpoint

  1. Add a new query key
    1. A fresh key is created to replace the compromised one.
    2. Azure AI Search allows you to create up to 50 query keys, specifically designed to give read-only access to the search index.
  2. Change the app to use the new key
  3. Delete the compromised key
  1. Throttling typically occurs due to resource limitations or capacity issues.
  2. To reduce the likelihood of search query requests being throttled, you should consider solutions, such as increasing the number of replicas, or scaling the service to a higher tier.
  3. Other wrong options
    1. Adding index would help with organising and querying different types of data, but would not reduce the likelihood of throttling caused by high query volume.
    2. Enable Customer-Managed Key (CMS) address data security and encryption at the rest, but like above it would not reduce the likelihood of throttling.
  1. Make API queries to the autocomplete endpoint and include suggesterName in the body.
    1. This specify which suggester to use for autocompletion.
    2. This is essential for retrieving relevant suggestions based on user input, making the autocomplete functionality effective.
  2. Add a suggester that has three product name fields as source fields.
    1. A suggester is needed to define the fields from which autocomplete suggestions will be generated.
    2. By including all relevant product name fields, the system can provide comprehensive suggestions that account for different ways products might be named or referred to.
  3. Set the analyser property for the three product name variants.
    1. It ensures that the text is processed correctly, considering the specific language or custom processing required.
    2. This helps in generating accurate autocomplete suggestions by applying the correct text analysis to the product name fields.
  4. Other wrong options
    1. SearchAnalyser is used for search query processing, rather than generating autocomplete suggestions.

Azure AI Services

Improve reading experience for dyslexia

  1. Use Azure AI Immersive Reader to improve the reading experience for users with reduced comprehension and learning differences.
  2. It provides features like text-to-speech, word highlighting, and other tools to enhance reading comprehension.

Different types of content display in Chatbot

  1. Carousel.
    1. Carousel is used to display multiple attachments in a horizontal layout.
  2. Adaptive card.
    1. Adaptive card is used to display richly formatted content, which is appropriate for presenting detailed information about products or services.
  3. List.
    1. List is used to display a vertical list.

Azure Cognitive Services

Overview

  1. Azure AI Service helps developers and organisations rapidly create intelligent, cutting-edge, market-ready, and responsible applications with out-of-the-box and prebuilt and customizable APIs and models.
  2. Azure AI Service allows you to use multiple cognitive services, including both Speech and Language APISs through a single endpoint and credentials.

Provision Cognitive Services with Azure CLI

  1. az cognitiveservices account create is the command to create new Azure Cognitive Services. It has several parameters
    1. --kind to specify the type of cognitive service you want to use, e.g. Face, TextAnalytics
    2. --location is to specify the Azure region to deploy the service.
    3. --api-properties is to specify the properties of the cognitive service, such as the pricing tier and the number of units.
    4. --encryption is to specify the encryption settings for this resource in JSON format, allowing you to configure customer-managed keys for data protection. For encryption, you will use Microsoft Key Vault to achieve that, like the following
1
2
3
4
5
6
7
8
9

az cognitiveservices account create -n myresource -g myResourceGroup --assign-identity --kind TextAnalytics --sku S -l WestEurope --yes
    --encryption '{
      "keySource": "Microsoft.KeyVault",
      "keyVaultProperties": {
        "keyName": "KeyName",
        "keyVersion": "secretVersion",
        "keyVaultUri": "https://issue23056kv.vault.azure.net/"
      }
    }'

Deploy a containerised Cognitive Services service for Sentiment Analysis

1
2
3
4
5

docker run --rm -it -p 5000:5000 --memory 8g --cpus 1\
  mcr.microsoft.com/azure-cognitive-services/sentiment:{IMAGE_TAG} \
  Eula=accept \
  Billing=contoso.cognitiveservices.azure.com \
  ApiKey={API_Key}

Configure your App to connect to Cognitive Services service with minimal administrative effort

  1. Use Endpoint URI and Subscription Key because Cognitive Services typically uses the endpoint URI and subscription key to authenticate and access the service.
  2. This method is straightforward and minimise the administrative effort as it involves only using the keys provided when the Cognitive Services resource is created.

Ensure Cognitive Services based Chatbot recognise your company’s products and codename

  1. Use Named Entity Recognition (NER) to recognise and categorise specific entities, such as your company’s products and codenames.
  2. This minimise development efforts as NER is designed to identify and extract specific entities from text, without the need for manual coding or training.

Content Safety

Optimise content filter configurations for objectionable content

  1. Safety Metaprompt feature allows you to run tests on how the content moderation filters behave with specific sample inputs, helping you optimise the configurations for objectionable content.
  2. This feature is designed for testing and adjusting safety prompts to ensure they align with the content filtering requirements.

Monitor Online Activity

  1. Monitor online activity is used for real-time monitoring and analysing live content streams, not designed for running tests on sample questions to optimise content filter configuration.

Protected Material Detection

  1. Protected material detection is focused on identifying sensitive content such as Personal Identifiable Information (PII), copyright violations, and other legally protected materials.

Speech

Types of Speech Services

  1. SpeechRecogniser is designed to convert spoken language into text.
    1. This service can listen to customer calls and transcribe their spoken queries into text, which can be used to query the database for order details.
  2. SpeechSynthesiser is designed to convert text to speech.
    1. This service is necessary for providing spoken response to customer, but not converting customer calls into text queries.
  3. TranslationRecogniser is used for translating spoken language into another language.
  4. VoiceProfileClient is used for speaker identification and verification.

Optimise the text-to-speech sound quality for poor network conditions

  1. Effort attribute of the voice element allows you to specify audio effects to optimise the quality of synthesised speech for specific environment, such as car interiors.
  2. Using the effect attribute, you can apply pre-defined audio effects like eq_car to enhance the clarity and compensate for distortions caused by environmental factors.

Enable Speech Capability for a Chatbot

  1. Enable WebSockets for the chatbot app.
    1. WebSockets are required to establish real-time, bidirectional communication between the client and the server.
  2. Create a Speech Service.
    1. Speech Service provides the necessary capabilities to convert speech to text, text to speech, and handle other speech-related functionalities required for enabling speech in the chatbot.
  3. Register a Direct Line Speech Channel.
    1. Direct Line Speech channel allows the bot to integrate with the Speech service, and handle voice interactions.

Document Intelligence

What is Document Intelligence

  1. Document Intelligence is specifically designed to extract structured and unstructured text from documents, including PDF files, scanned receipts.
  2. This service uses Optical Character Recognition (OCR) and advanced AI and machine learning models to accurately extract text, tables, and other data from various document formats, making it the ideal solution for extracting text.

Translate files from German to French with custom glossary

  1. Define a document translation specification that has a French target.
    1. source language: German
    2. target language: French
    3. other necessary translation settings, to preserve the original document format.
  2. Upload a glossary file to the container for German files.
    1. Glossary file helps ensure that specific terms and translated accurately according to your custom definitions.
    2. Uploading the glossary to the German container ensures that it is applied during the translation process to maintain consistency in terminology.
  3. Perform an asynchronous translation by using the document translation specification.
    1. Asynchronous translation allows the service to translate the files without blocking other operations.

Deploy Document Intelligence resource in an ARM template

  1. The resource type you need to choose is Microsoft.CognitiveServices/accounts. This is because Document Intelligence is part of the Cognitive Services suite.
  2. Document Intelligence is formally known as Form Recognizer, is the correct kind value to use for deploying a Document Intelligence resource.

Extract Information from Scanned document with minimal development efforts

  1. Document Intelligence provides a pre-built model specifically designed for extracting common field from invoices, such as shipping address, billing address, customer id, amount due, total tax, and subtotal.
  2. Using pre-built model minimise development effort as it is already trained to recognise and extract these specific fields.

Pre-built models in Document Intelligence

  1. prebuilt-read model is designed to extract both printed and handwritten text from documents such as PDFs.
  2. prebuilt-document model is used for extracting key-value pairs and tables from documents.
  3. prebuilt-contract model is specialised for contracts and legal documents, focusing on extracting structured data like key-value pairs, and is not tailored for recognising handwritten text.

Threshold of Confidence score in Document Intelligence

  1. 0.75 is commonly used in AI Document analysis tasks to strike a balance between precision and recall.
  2. A confidence level of 0.75 means that model is 75% confident that the detected content is handwritten, which is generally reliable for this type of tasks.
  3. It minimise false positives while ensuring accurate detection of handwritten content.

Vision

Analyse image with Vision SDK

  1. You can use the analyze_image() function from the Azure Vision service to analyse an image, and extract information such as objects, text, descriptions, tags and more.
  2. You can specify the visuals features that you want from the VisualFeatureTypes emum, such as VisualFeatureTypes.Description to get a description of the image, or VisualFeatureTypes.Tags to get tags that describe the image.

Compute Vision API vs Custom Vision API

  1. Compute Vision API can be used to extract text, and other information from images, including PDFs.
  2. Custom Vision API is used to build and train custom image classification models, allowing you to specify your own labels and train models to detect them.

Computer Vision vs Document Intelligence

  1. Compute Vision is generally used for analysing and extracting text from images, identifying objects, and generating descriptions.
    1. It is not specifically designed for extracting detailed invoice data like invoice items, sales amount, and customer details.
  2. Document Intelligence is specifically designed to analyse and extract structured data from documents, including invoices.
    1. It can accurately extract invoice items, sales amount, and customer details, minimizing development efforts and providing reliable results.

Custom Vision

Overview

  1. Azure AI Custom Vision is an image recognition service that lets you build, deploy, and improve your own image identifier models.
  2. An image identifier applies labels to images according to their visual characters.
  3. Each label represents a classification or object.
  4. Custom Vision allows you to specify your own labels and train custom models to detect them.

Steps to build and train a custom classification model

  1. From the Custom Vision portal, open the project
  2. Upload sample images of the new products.
  3. Label the sample images.
  4. Retrain the model.
  5. Publish the model.

Face

Increase the likelihood to detect faces in blurred images and sideways faces

  1. Use the Detection Model to detect faces. The Detection Model is designed to detect faces in images.
  2. Default detection model is detection_01, which is not optimised for small, side-view or blurry faces.
  3. There are other options like detection_02 and detection_03 that improve accuracy on small, side-view and blurry faces.

Translate

Common Translate API Endpoints

API EndpointDescription
/translate?to=enTranslate text from one language to English
/detectDetect the language of a text sample

Service Endpoint for Translate

Service EndpointRegion
Global (recommended): api.cognitive.microsofttranslator.comClosest available data center
Americas: api-nam.cognitive.microsofttranslator.comEast US2, West US2
Asia Pacific: api-apc.cognitive.microsofttranslator.comJapan East, Southeast Asia
Europe (except Switzerland): api-eur.cognitive.microsofttranslator.comFrance Central, West Europe
Switzerland: see docsSwitzerland North

Language

Minimise similar utterances from input

  1. Pattern.any is used to identify entities that can have multiple forms, such as “John F. Kennedy” or “JFK”.
  2. Pattern.any allows the Language Understanding model to recognise both full name and their corresponding code as same entity, without needing to list every possible variation.
  3. This approach minimise the number of utterances required and ensures consistent entity recognition.

Use container on the host computer from Language Understanding (LUIS)

  1. Export package fro container from LUIS portal or LUIS APIs
  2. Move package file into the required input directory on the host computer.
    1. Do no rename, alter, overwrite, or decompress the LUIS package file.
  3. Run the container with docker run command, with the required mount and billing settings.
  4. Querying the container’s prediction endpoint

Ensure only specific Azure processes can access Language Service

  1. You will need to set Virtual Network Rules because they allow you to restrict access to the Azure AI Language service to specific Azure resources within a virtual network.
  2. By configuring Virtual Network rules, you can ensure that only designated processes and services within your Azure infrastructure can access the Language service.
  3. This approach minimise administrative effort by leveraging Azure’s built-in network security capabilities.
  4. Other wrong options
    1. IPsec rules are typically used for securing network traffic over IP networks. It requires more complex configuration and management compared to Virtual Network rules.
    2. Azure Application Gateway is a web traffic load balancer that enables you to manage traffic to your web application. It is not specifically designed for restricting access to Azure services, and need additional configuration.
    3. Virtual Network Gateway provides a connection between your on-premise network and your Azure virtual network. While it can be part of a secure network, it is not the most direct method.

Common Packages and its usage scenarios

  1. azure-ai-language-conversations this package is designed for working with Azure AI services that handle
    1. language understanding.
    2. intent recognition.
  2. azure-cognitiveservices-language-textanalytics is focused on text analytics task such as
    1. sentiment analytics
    2. key phrase extraction
    3. entity recognition.
  3. azure-mgmt-cognitiveservices is used for managing Azure Cognitive Services resources, such as
    1. creating and configuring Cognitive Services
  4. azure-cognitiveservices-speech is sued for integrating
    1. speech recognition and synthesis capabilities.

Entity Extraction

  1. You can use the recognise_entities() function to recognise all the entities within a sentence.
  2. This function from Azure Text Analytics service, is deigned to identify named entities in the text, such as locations, organisations, and people.
  3. For example, with the input text of “Our tour of Paris included a visit to the Eiffel Tower”, the function would return “Paris” and “Eiffel Tower” as entities.

Key Phrase Extraction

  1. You can use extract_key_phrase() function from Azure Text Analytics service.
    1. For example, with the input text of “Our tour of Paris included a visit to the Eiffel Tower”, the function would return “tour”, “Paris”, “visit”, and “Eiffel Tower” as key phrases.
  2. extract_key_phrase() function doesn’t provide confidence score of the output.
    1. If you want confidence score, you can use recognise_entities() function.

Video Indexer

Best way to index a 20GB Video file with Azure Video Indexer

  1. From OneDrive, create a download link, and then copy the link to the Azure AI Video Indexer website.
  2. This allows you to directly use the link for indexing the video without the need to download and re-upload the file.
  3. By creating the download link, you ensure that Azure AI Video Indexer can access the file directly from OneDrive.

Miscellaneous

Only specific resources to access Azure AI Services

  1. In virtual network, enable a service endpoint for Azure AI Services.
    1. Service Endpoint allows secure direct connectivity to Azure services over an optimised route within Azure backbone network.
    2. This helps in preventing external access to AI Service while ensuring the resource is accessible with the virtual network.
  2. In AI Service, modify the virtual network settings.
    1. This involves configuration the service to limit access to specific virtual networks and subnets, effectively preventing unauthorised external access.
    2. This configuration ensures that only resources within the specified virtual network can access AI service.

Identify unusual values in Time Series

  1. Azure AI Anomaly Detector is specially designed to detect anomalies in time series data
  2. It used advanced machine learning models to identify unexpected patterns or behaviours in data, which is essential for predictive maintenance.
  3. It can help you to do
    1. Univariate Anomaly Detection, i.e. detect anomalies over one variable, like revenue, cost, etc.
    2. Multivariate Anomaly Detection, i.e. detect anomalies over multiple variables with correlations, which are usually gathered from equipment or other complex systems.

Detect when a user ask questions via Video and Audio

  1. Speech-to-text