Overview

The blog post delves into the exciting world of virtual assistant solutions, specifically focusing on creating a custom copilot using Azure AI. Copilots are virtual assistants hosted in the cloud, powered by large language models (LLMs) as chat engines. These copilots can be customized with business data, custom prompts, and integrated with third-party services and plugins. While Microsoft offers built-in copilots like Copilot for M365, Copilot in Bing, and GitHub Copilot, we can also build our own copilots leveraging the same infrastructure that Microsoft’s Copilots are based on.

LLM Models:

Generative AI models – commonly known as Large Language Models (LLMs) – are very powerful at generating different types of content, like grammatically correct text, art creatives, and code, with just a short textual input called a prompt. However, when using them in real-world production scenarios, they have limitations. Specifically:

• They can answer questions related only to the data they were trained on.
• They do not know facts that happened after their date of training.
• They do not have access to data protected by firewalls.

Empowering Generative AI Models with RAG

The Retrieval Augmented Generation (RAG) pattern overcomes these limitations. It provides the LLM with relevant and fresh data to answer user questions by injecting information through the prompt. Here’s how it works:

1. The user question is converted into a query to search the data store for relevant information (usually private data as enterprise data).
2. Matching results from the data store are combined with the user prompt and system message (including instructions for the model) and injected into the model prompt.

LLMOps and the Development Life Cycle      

With the rise of Large Language Models (LLMs), the field of Natural Language Processing (NLP) has undergone a paradigm shift. Unlike classical machine learning, where models are trained on custom data using specific statistical algorithms, modern AI applications augment pre-trained LLMs with custom data and prompts to achieve non-deterministic outcomes. This shift impacts the entire development life cycle, leading to the emergence of LLMOps (MLOps applied to LLMs).

The copilot application development life cycle consists of four stages:

1. Initialization: Define the business use case and design the solution.
2. Experimentation: Build and test the solution with a small dataset.
3. Evaluation and Refinement: Assess the solution with a larger dataset, considering metrics like groundedness, coherence, and relevance.
4. Production: Deploy and monitor the final application.

This iterative process allows us to continuously improve the copilot solution during both evaluation and production stages. 

Summary of Solutions to Create Your Own Copilot

There are several ways to create your own Copilot using the Microsoft ecosystem. This document will explore two options:

using the LLM (Long Language Model) and

using the RAG (Reinforcement and Actor-Critic) model.

In case you want to build your First Copilot using the LLM model, you can refer to the following link: Create your own Copilot that uses your own data with an Azure OpenAI Service Model. This post will provide you with step-by-step instructions on how to create a Copilot using the LLM model.

Alternatively, if you want to use the RAG model to create your Copilot, you can refer to the following link: Build Your Personal RAG Chatbot on a PDF document: Langchain, ChromaDB on GPT 3.5

Beyond the Basics: Advanced Serverless AI Chatbots Using RAG and LangChain.js

This post will walk you through the process of creating a Copilot using the RAG model.

In the section below, we will explore various solutions to build your Copilot. Let’s dive in:

Refer this link for all the examples: Develop apps that use Azure AI services | Microsoft Learn

1.       End-to-end solutions (Enterprise Chat using RAG)

This article shows you how to deploy and run the Enterprise chat app sample for Python. This sample implements a chat app using Python, Azure OpenAI Service, and Retrieval Augmented Generation (RAG) in Azure AI Search to get answers about employee benefits at a fictitious company. The app is seeded with PDF files including the employee handbook, a benefits document and a list of company roles and expectations.

Architectural overview

A simple architecture of the chat app is shown in the following diagram:

Key components of the architecture include:

• A web application to host the interactive chat experience.
• An Azure AI Search resource to get answers from your own data.
• An Azure OpenAI Service to provide:
  • Keywords to enhance the search over your own data.
  • Answers from the OpenAI model.
  • Embeddings from the ada model

Get started with the Python enterprise chat sample using RAG – Python on Azure | Microsoft Learn

2.       Build a chat app with Azure OpenAI in Python

This repository includes a simple Python Quart app that streams responses from ChatGPT to an HTML/JS frontend using JSON Lines over a ReadableStream.

The repository is designed for use with Docker containers, both for local development and deployment, and includes infrastructure files for deployment to Azure Container Apps. 

Architecture overview

A sample chat app that uses Python and the Quart framework to stream chat completions from the OpenAI SDK. Deployable to Azure Container Apps.

3.       Build a LangChain with Azure OpenAI in Python

LangChain with Azure OpenAI and ChatGPT (Python v2 Function)

This sample shows how to take a human prompt as HTTP Get or Post input, calculates the completions using chains of human input and templates. This is a starting point that can be used for more sophisticated chains.

4.       Build a ChatGPT Plugin with Azure Container Apps in Python

This is a quickstart for sample for creating ChatGPT Plugin using GitHub Codespaces, VS Code, and Azure. The sample includes templates to deploy the plugin to Azure Container Apps using the Azure Developer CLI. To gain access to ChatGPT plugins, join waitlist here!

5.       Summarize Text using Azure AI Language with Azure Functions

https://github.com/Azure-Samples/function-python-ai-textsummarize

This sample shows how to take text documents as a input via BlobTrigger, does Text Summarization processing using the AI Congnitive Language service, and then outputs to another text document using BlobOutput binding.

6.       Azure AI Python Template Gallery

7.       Smart load balancing with Azure Container Apps

Scale Azure OpenAI for Python chat using RAG with Azure Container Apps

This solution demonstrate how to add load balancing to your application to extend the chat app beyond the Azure OpenAI token and model quota limits. This approach uses Azure Container Apps to create three Azure OpenAI endpoints, as well as a primary container to direct incoming traffic to one of the three endpoints.

Architecture for load balancing Azure OpenAI with Azure Container Apps

Because the Azure OpenAI resource has specific token and model quota limits, a chat app using a single Azure OpenAI resource is prone to have conversation failures due to those limits.

To use the chat app without hitting those limits, use a load balanced solution with Azure Container Apps. This solution seamlessly exposes a single endpoint from Azure Container Apps to your chat app server.

The Azure Container app sits in front of a set of Azure OpenAI resources. The Container app solves two scenarios: normal and throttled. During a normal scenario where token and model quota is available, the Azure OpenAI resource returns a 200 back through the Container App and App Server.

When a resource is in a throttled scenario such as due to quota limits, the Azure Container app can retry a different Azure OpenAI resource immediately to fullful the original chat app request.

8.       Smart load balancing with Azure API Management

This solution demonstrates how to add enterprise-grade load balancing to your application to extend the chat app beyond the Azure OpenAI token and model quota limits. This approach uses Azure API Management to intelligently direct traffic between three Azure OpenAI resources.

Architecture for load balancing Azure OpenAI with Azure API Management

Because the Azure OpenAI resource has specific token and model quota limits, a chat app using a single Azure OpenAI resource is prone to have conversation failures due to those limits.

To use the chat app without hitting those limits, use a load balanced solution with Azure API Management. This solution seamlessly exposes a single endpoint from Azure API Management to your chat app server.

The Azure API Management resource, as an API layer, sits in front of a set of Azure OpenAI resources. The API layer applies to two scenarios: normal and throttled. During a normal scenario where token and model quota is available, the Azure OpenAI resource returns a 200 back through the API layer and backend app server.

When a resource is throttled due to quota limits, the API layer can retry a different Azure OpenAI resource immediately to fulfill the original chat app request.

9.       Evaluate your chat app

This Solution shows you how to evaluate a chat app’s answers against a set of correct or ideal answers (known as ground truth). Whenever you change your chat application in a way which affects the answers, run an evaluation to compare the changes. This demo application offers tools you can use today to make it easier to run evaluations.

Architectural overview

Key components of the architecture include:

• Azure-hosted chat app: The chat app runs in Azure App Service. The chat app conforms to the chat protocol, which allows the evaluations app to run against any chat app that conforms to the protocol.
• Azure AI Search: The chat app uses Azure AI Search to store the data from your own documents.
• Sample questions generator: Can generate a number of questions for each document along with the ground truth answer. The more questions, the longer the evaluation.
• Evaluator runs sample questions and prompts against the chat app and returns the results.
• Review tool allows you to review the results of the evaluations.
• Diff tool allows you to compare the answers between evaluations.

10.    Load test your chat app with Locust

This solution provides the process to perform load testing on a Python chat application using the RAG pattern with Locust, a popular open-source load testing tool. The primary objective of load testing is to ensure that the expected load on your chat application does not exceed the current Azure OpenAI Transactions Per Minute (TPM) quota. By simulating user behavior under heavy load, you can identify potential bottlenecks and scalability issues in your application. This process is crucial for ensuring that your chat application remains responsive and reliable, even when faced with a high volume of user requests.

Conclusion:

The reference url used in this blog is the Microsoft link Develop apps that use Azure AI services | Microsoft Learn

I found the following link to be incredibly useful in gaining insights into various scenarios for building your GenAI solution. The link, provided by Microsoft, offers a comprehensive guide for developers looking to leverage Azure AI services.

By exploring this resource, developers can gain access to valuable documentation, samples, and templates for developing intelligent applications. Whether you’re working with Python, .NET, or any other programming language, these resources provide a seamless development experience, allowing you to harness the power of Azure AI services efficiently.

I recommended going through post to understand these scenarios, stay tunned as I explore more I will be publishing more posts on building your own custom copilot

In case you are a new reader and would like to explore my recent post on GenAI, please refer to the post below, like it, and provide a comment.
 
Create your own Copilot:
https://lnkd.in/eVRkr2Sv
 
Create Copilot with Prompt Flow:
https://lnkd.in/eGchqU3D
 
Open AI:
https://lnkd.in/eZzUdQAi
 
Build a Custom Copilot using Microsoft Phi-3 SLM:
https://lnkd.in/eC6tmK5V
 
Build your personal RAG Chatbot:
https://lnkd.in/eHMYqMB7

Build Serverless AI chatbots using RAG and LangChain.js:
https://lnkd.in/ecSTtmze

References:

Develop apps that use Azure AI services | Microsoft Learn

Building your own copilot – yes, but how? Prompt Flow vs Custom Solutions (microsoft.com)

A pro-code guide to build your custom copilot faster (microsoft.com)

Check all the samples: Develop apps that use Azure AI services | Microsoft Learn