Machine learning operations (MLOps) practices in Azure Machine Learning for the purpose of managing the lifecycle of your models. Applying MLOps practices can improve the quality and consistency of your machine learning solutions.

Overview:

Azure Machine Learning (Azure ML) offers a robust framework for managing the lifecycle of machine learning models, known as MLOps. This framework is built on DevOps principles, aiming to streamline workflows and enhance the efficiency of machine learning operations. The key components of Azure ML’s model management and deployment include:

• Reproducible Machine Learning Pipelines: Azure ML allows the creation of machine learning pipelines that are repeatable and reusable. These pipelines can include steps for data preparation, feature extraction, hyperparameter tuning, and model evaluation.
• Reusable Software Environments: The platform supports the creation of consistent software environments that can be used across training and deployment phases, ensuring that models run reliably in different stages.
• Model Registration, Packaging, and Deployment: Azure ML provides the capability to register, package, and deploy models from anywhere, along with tracking metadata and governance data for the entire machine learning lifecycle.
• Monitoring and Alerts: The service includes features for monitoring machine learning applications for operational and machine learning-related issues, comparing model inputs between training and inference, and setting up alerts for events in the machine learning lifecycle.
• Automation of the Machine Learning Lifecycle: Azure ML integrates with Azure pipelines to automate the end-to-end machine learning lifecycle, allowing for frequent testing and updating of models alongside other applications and services.

Set up MLOps with Azure DevOps

Set up MLOps with Azure DevOps – Azure Machine Learning | Microsoft Learn

Azure Machine Learning allows you to integrate with Azure DevOps pipeline to automate the machine learning lifecycle. Some of the operations you can automate are:

• Deployment of Azure Machine Learning infrastructure
• Data preparation (extract, transform, load operations)
• Training machine learning models with on-demand scale-out and scale-up
• Deployment of machine learning models as public or private web services
• Monitoring deployed machine learning models (such as for performance analysis)

We will be covering the Lab details/demo in another post. Before that lets understand  recommended Azure architecture for MLOps and AzureMLOps (v2) solution accelerator.

Azure architecture for MLOps and AzureMLOps (v2) solution accelerator

This section describes three Azure architectures for machine learning operations. They all have end-to-end continuous integration (CI), continuous delivery (CD), and retraining pipelines. The architectures are for these AI applications:

• Classical machine learning
• Computer vision (CV)
• Natural language processing (NLP)

The architectures are the product of the MLOps v2 project. They incorporate the best practices that the solution architects discovered in the process of creating multiple machine learning solutions. The result is deployable, repeatable, and maintainable patterns as described here.

All of the architectures use the Azure Machine Learning service.

For an implementation with sample deployment templates for MLOps v2, see Azure MLOps (v2) solution accelerator on GitHub.

Potential use cases

• Classical machine learning: Time-Series forecasting, regression, and classification on tabular structured data are the most common use cases in this category. Examples are:
  • Binary and multi-label classification
  • Linear, polynomial, ridge, lasso, quantile, and Bayesian regression
  • ARIMA, autoregressive (AR), SARIMA, VAR, SES, LSTM
• CV: The MLOps framework presented here focuses mostly on the CV use cases of segmentation and image classification.
• NLP: This MLOps framework can implement any of those use cases, and others not listed:
  • Named entity recognition
  • Text classification
  • Text generation
  • Sentiment analysis
  • Translation
  • Question answering
  • Summarization
  • Sentence detection
  • Language detection
  • Part-of-speech tagging

Simulations, deep reinforcement learning, and other forms of AI aren’t covered by this article.

Architecture

The MLOps v2 architectural pattern is made up of four main modular elements that represent these phases of the MLOps lifecycle:

• Data estate
• Administration and setup
• Model development (inner loop)
• Model deployment (outer loop)

These elements, the relationships between them, and the personas typically associated with them are common for all MLOps v2 scenario architectures. There can be variations in the details of each, depending on the scenario.

The base architecture for MLOps v2 for Machine Learning is the classical machine learning scenario on tabular data. The CV and NLP architectures build on and modify this base architecture.

Current architectures

The architectures currently covered by MLOps v2 are as below.

• Classical machine learning architecture
• Machine Learning CV architecture
• Machine Learning NLP architecture

1. Classical Machine Learning Architecture: This model is designed for time-series forecasting, regression, and classification on structured tabular data. 

It includes use cases like binary and multi-label classification, various regression models (linear, polynomial, ridge, lasso, quantile, Bayesian), and time-series analysis (ARIMA, autoregressive, SARIMA, VAR, SES, LSTM).

2. Machine Learning CV (Computer Vision) Architecture: The focus here is on segmentation and image classification within the realm of computer vision. 

This architecture supports the MLOps framework for CV applications.

3. Machine Learning NLP (Natural Language Processing) Architecture: This architecture can implement a range of NLP use cases such as named entity recognition, text classification, text generation, sentiment analysis, translation, question answering, summarization, sentence detection, language detection, and part-of-speech tagging.

All three architectures are built upon the Azure Machine Learning service and are part of the MLOps v2 project, which aims to provide deployable, repeatable, and maintainable patterns for machine learning solutions.

They share a common MLOps v2 architectural pattern consisting of four main modular elements:

Data Estate, Administration and Setup, Model Development (inner loop), and Model Deployment (outer loop).

These elements represent the phases of the MLOps lifecycle and are common across all scenario architectures, with variations depending on the specific use case

Classical machine learning architecture

Summary of Classical Machine Learning Architecture Workflow

1. Data Estate:
   • Managed by data engineers.
   • Includes data sources and targets for data science projects.
   • Azure data platforms are used based on best practices and customer use cases.
2. Administration and Setup:
   • First step in MLOps v2 deployment.
   • Tasks include creating source code repositories, Machine Learning workspaces, datasets, compute resources, and defining roles and access controls.
   • Involves the infrastructure team, data engineers, machine learning engineers, and data scientists.
3. Model Development (Inner Loop):
   • Iterative data science workflow within a secure Machine Learning workspace.
   • Workflow includes data ingestion, exploratory data analysis, experimentation, model development, evaluation, and registration of a candidate model.
   • Data scientists and machine learning engineers are the key personas.
4. Machine Learning Registries:
   • Models that are candidates for production are registered.
   • CI pipelines are triggered for model promotion to the deployment phase.
   • Machine learning engineers are the primary personas.
5. Model Deployment (Outer Loop):
   • Consists of staging and testing, production deployment, and monitoring.
   • CD pipelines manage model promotion through production and potential retraining.
   • Machine learning engineers are the main personas involved.
6. Staging and Test:
   • Includes retraining, testing on production data, performance testing, data quality checks, and responsible AI checks.
   • Conducted in secure Machine Learning workspaces.
7. Production Deployment:
   • After passing staging and test, models are promoted to production with gated approval.
   • Deployment options include managed batch endpoints or online endpoints with Azure Arc.
8. Monitoring:
   • Collection of metrics for model, data, and infrastructure performance.
   • Monitoring includes checks for model and data drift, performance on new data, and infrastructure issues.
9. Data and Model Monitoring (Events and Actions):
   • Automated triggers and notifications based on metric thresholds or schedules.
   • Actions include automated retraining or loopback to model development for investigation.
10. Infrastructure Monitoring (Events and Actions):
    • Triggers based on infrastructure concerns like endpoint response lag or compute capacity.
    • Actions may involve reconfiguring compute and network resources.

This workflow outlines a comprehensive approach to managing the lifecycle of machine learning models, from development to deployment and monitoring, ensuring a robust and scalable MLOps practice.

Machine Learning CV architecture

Summary of the Machine Learning CV Architecture Workflow:

1. Data Estate: Identifies potential data sources and targets, with Azure Blob Storage and Azure Data Lake Storage as recommended sources for images in CV scenarios.
2. Administration and Setup: Involves resource and role management, with an additional step for CV: creating image labeling and annotation projects.
3. Model Development (Inner Loop): An iterative process within a secure workspace, with image labeling and annotation as a key component for CV models.
4. Machine Learning Registries: Models ready for production are registered, triggering CI pipelines for promotion to the deployment phase.
5. Model Deployment (Outer Loop): Involves staging, testing, and monitoring before promoting the model through CD pipelines for production deployment.
6. Staging and Test: Includes performance tests and AI bias checks. Retraining on production data is optional; the focus is on using production data for model development.
7. Production Deployment: Models that pass the test phase are promoted to production with managed endpoints or Kubernetes deployment using Azure Arc.
8. Monitoring: Collects metrics to monitor model, data, and infrastructure performance, including checks for model performance on new images.
9. Data and Model Monitoring: In CV scenarios, poor model performance on new images leads to human-in-the-loop review and annotation, potentially returning to the development loop for updates.
10. Infrastructure Monitoring: Automated triggers and notifications address infrastructure issues, looping back to the setup and administration phase for potential reconfiguration.

Machine Learning NLP architecture

Summary of the Machine Learning NLP Architecture Workflow:

1. Data Estate: Focuses on identifying data sources and targets, with best practices indicated by a green check mark.
2. Administration and Setup: Involves managing resources and roles, with an additional step for NLP: creating labeling and annotation projects.
3. Model Development (Inner Loop): An iterative process within a secure workspace, emphasizing annotators, tokenization, normalization, and embeddings for text data.
4. Machine Learning Registries: Developed models are registered, triggering CI pipelines for promotion to the deployment phase.
5. Model Deployment (Outer Loop): Includes pre-production staging, testing, and monitoring, with CD pipelines managing the promotion of the model for production deployment.
6. Staging and Test: Varies with customer practices, typically involving retraining, testing, performance checks, and responsible AI checks in secure workspaces.
7. Production Deployment: Models that pass the test phase are promoted to production with managed endpoints or Kubernetes deployment using Azure Arc.
8. Monitoring: Allows for the collection and action on changes in model, data, and infrastructure performance, including model and data drift and AI issues.
9. Data and Model Monitoring: In NLP scenarios, poor model performance on new text leads to human-in-the-loop review and annotation, with potential updates in the development loop.
10. Infrastructure Monitoring: Automated triggers and notifications address infrastructure issues, looping back to the setup and administration phase for reconfiguration.

Components

• Machine Learning: A cloud service for training, scoring, deploying, and managing machine learning models at scale.
• Azure Pipelines: This build and test system is based on Azure DevOps and is used for the build and release pipelines. Azure Pipelines splits these pipelines into logical steps called tasks.
• GitHub: A code hosting platform for version control, collaboration, and CI/CD workflows.
• Azure Arc: A platform for managing Azure and on-premises resources by using Azure Resource Manager. The resources can include virtual machines, Kubernetes clusters, and databases.
• Kubernetes: An open-source system for automating deployment, scaling, and management of containerized applications.
• Azure Data Lake: A Hadoop-compatible file system. It has an integrated hierarchical namespace and the massive scale and economy of Blob Storage.
• Azure Synapse Analytics: A limitless analytics service that brings together data integration, enterprise data warehousing, and big data analytics.
• Azure Event Hubs. A service that ingests data streams generated by client applications. It then ingests and stores streaming data, preserving the sequence of events received. Consumers can connect to the hub endpoints to retrieve messages for processing. Here we are taking advantage of the integration with Data Lake Storage.

Conclusion

Embracing MLOps within Azure ML accelerates model experimentation and deployment while ensuring quality and traceability. It empowers teams to focus on model innovation rather than infrastructure, with tools for automating and monitoring the end-to-end machine learning lifecycle. This approach is pivotal for businesses aiming to streamline their machine learning operations and maintain a competitive edge in the market.

Azure Machine Learning’s approach to model management and deployment is comprehensive, offering tools and practices that align with the evolving needs of machine learning operations. By leveraging MLOps principles, Azure ML ensures faster development and deployment of models, quality assurance, and end-to-end lineage tracking, which are crucial for maintaining the integrity and efficiency of machine learning solutions. This makes Azure ML a valuable asset for any organization looking to streamline their machine learning workflows and achieve consistent results.

References:

MLOps: Machine learning model management – Azure Machine Learning | Microsoft Learn

Set up MLOps with Azure DevOps – Azure Machine Learning | Microsoft Learn

End-to-end machine learning operations (MLOps) with Azure Machine Learning – Training | Microsoft Learn

Azure DevOps for CI/CD – Azure Machine Learning | Microsoft Learn

What is the Azure Machine Learning designer(v2)? – Azure Machine Learning | Microsoft Learn

Azure/mlops-v2: Azure MLOps (v2) solution accelerators. Enterprise ready templates to deploy your machine learning models on the Azure Platform. (github.com)