Azure iPaaS – Enterprise Integration Architecture and Scenarios

Rajeev Singh

3 years ago

There are various integration scenarios and for each of these scenarios Azure iPaaS platform has various Architecture defined along with best practices. We will cover these scenarios and its architecture in details.

There are various integration scenarios supported by Azure iPaaS and the corresponding architectures and best practices. Whether it’s application integration, data integration, or B2B integration, Azure iPaaS offers a comprehensive platform to streamline business processes and enable seamless data exchange.

By following the recommended architectures and best practices, organizations can leverage the full potential of Azure iPaaS and achieve their integration goals effectively.

We will now look at these scenarios in more detail and its architecture.

1. Basic Enterprise Integration to Orchestrate calls to Enterprise backend systems.

Use case:

This architecture is used for integration scenarios where the workflow is triggered by synchronous calls to back-end services.

This architecture uses two of those services: Logic Apps to orchestrate workflows, and API Management to create catalogs of APIs.

The backend systems can include software-as-a-service (SaaS) systems, Azure services, and existing web services in your enterprise.

Workflow details:

Integration Services is a collection of services that you can use to integrate applications, data, and processes for your enterprise. This architecture uses two of those services: Logic Apps to orchestrate workflows, and API Management to create catalogs of APIs.

In this architecture, composite APIs are built by importing logic apps as APIs.

You can also import existing web services by importing OpenAPI (Swagger) specifications or importing SOAP APIs from WSDL specifications.

The API gateway helps to decouple front-end clients from the back end.

Backend systems. The right-hand side of the diagram shows the various backend systems that the enterprise has deployed or relies on. These systems might include SaaS systems, other Azure services, or web services that expose REST or SOAP endpoints.

Azure Logic Apps. In this architecture, logic apps are triggered by HTTP requests. You can also nest workflows for more complex orchestration.

Azure API Management. API Management consists of two related components:

API gateway. The API gateway accepts HTTP calls and routes them to the backend.
- Developer portal. Each instance of Azure API Management provides access to a developer portal.

Azure DNS. Azure DNS provides name resolution by using the Azure infrastructure.

By hosting your domains in Azure, you can manage your DNS records by using the same credentials, APIs, tools, and billing that you use for your other Azure services.

Azure Active Directory (Azure AD). Use Azure AD to authenticate clients that call the API gateway. Azure AD supports the OpenID Connect (OIDC) protocol.

Recommendations

These considerations implement the pillars of the Azure Well-Architected Framework, which is a set of guiding tenets that you can use to improve the quality of a workload.

Pillar	Description
Reliability	The ability of a system to recover from failures and continue to function.
Security	Protecting applications and data from threats.
Cost optimization	Managing costs to maximize the value delivered.
Operational excellence	Operations processes that keep a system running in production.
Performance efficiency	The ability of a system to adapt to changes in load.

Recommendation using Azure Well-Architected Framework.

Microsoft Well Architected Framework	Recommendations
Reliability	Review the SLA for each service, APIM, Logic Apps. Regularly back up APIM configurations.
Security	Restrict access for calling Logic Apps endpoints to only the IP address of API Management. To make sure users have appropriate access levels, use Azure role-based access control (Azure RBAC). Secure public API endpoints in API Management by using OAuth or OpenID Connect. Connect to back-end services from API Management by using mutual certificates. Enforce HTTPS on the API Management APIs.Store secrets, Logic Apps: store secrets in Key Vault, APIM: manage secrets by using objects (named values or properties)
Cost optimization	Use the Azure pricing calculator to estimate costs. API Management: You’re charged for all API Management instances when they’re running. If you have scaled up and don’t need that level of performance all the time, manually scale down or configure autoscaling. Logic Apps Logic Apps uses a serverless model. Billing is calculated based on action and connector execution.
Operational excellence	DevOps: Create separate resource groups for production, development, and test environments. Separate resource groups make it easier to manage deployments, delete test deployments, and assign access rights. Deployment: Use Azure Resource Manager templates to deploy the Azure resources, follow the infrastructure as Code (IaC) Process.Staging: Consider staging your workloads, which means deploying to various stages and running validations at each stage before moving on to the next one. Workload isolation: Put API Management and any individual logic apps in their own separate Resource Manager templates. By using separate templates, you can store the resources in source control systems. Versions: Each time you change a logic app’s configuration or deploy an update through a Resource Manager template, Azure keeps a copy of that version and keeps all versions that have a run history. You can use these versions to track historical changes or promote a version as the logic app’s current configuration.
Performance efficiency	APIM: To increase the scalability of API Management, add caching policies where appropriate. To analyze the usage for your service, select Capacity Metric on the Metrics menu and then scale up or scale down as appropriate. Consider traffic patterns when scaling. Customers with more volatile traffic patterns need more capacity. The consistent capacity that’s greater than 66% might indicate a need to scale up. The consistent capacity that’s under 20% might indicate an opportunity to scale down. Before you enable the load in production, always load-test your API Management service with a representative load. Logic Apps: The Logic Apps serverless model means administrators don’t have to plan for service scalability. The service automatically scales to meet demand. Note: For greater reliability and scalability, use message queues and events to decouple the backend systems.

2. Enterprise Integration using Hybrid Connections

Use case:

In case your organization needs to connect to an on-prem LOB and doesn’t want to go with the Network solution path, Azure Hybrid Connection is the solution.

Workflow details:

Hybrid Connections requires a relay agent to be deployed where it can reach both the desired endpoint as well as to Azure.

The relay agent, Hybrid Connection Manager (HCM), calls out to Azure Relay over port 443.

From the web app site, the App Service infrastructure also connects to Azure Relay on your application’s behalf.

Through the joined connections, your app is able to access the desired endpoint.

The connection uses TLS 1.2 for security and shared access signature (SAS) keys for authentication and authorization.

3. Data integration with Logic Apps and SQL Server

Use case:

This solution is used to automate data integration tasks that you perform in response to API calls, especially for on-premises databases.

Workflow details:

This solution uses Azure Logic Apps to integrate cloud data into on-premises data storage.

API Management accepts API calls in the form of HTTP requests.
API Management securely routes the HTTP requests to Logic Apps.
Each HTTP request triggers a run in Logic Apps:
1. Logic Apps uses secured template parameters to retrieve database credentials from Azure Key Vault.
1. Logic Apps uses Transport Layer Security (TLS) to send the database credentials and a database statement to the on-premises data gateway.
The on-premises data gateway connects to a SQL Server database to run the statement.
SQL Server stores the data and makes it available to apps that user’s access.
Azure Monitor collects information on Logic Apps events and performance.

Below components are used:

Azure API Management creates consistent, modern API gateways for back-end services.

Azure Logic Apps automates workflows by connecting apps and data across clouds

An on-premises data gateway acts as a bridge that connects on-premises data with cloud services like Logic Apps.

Azure Key Vault stores and controls access to secrets such as tokens, passwords, and API keys.

SQL Server provides a solution for storing and querying structured and unstructured data

Azure Monitor collects data on environments and Azure resources. This information is helpful for maintaining availability and performance

Alternatives:

There are a few alternatives to the above solution.

Use fully managed Azure database service.
For complex automation tasks, consider Azure Functions instead of Logic Apps.
For simple integration, consider Power Automate.

Recommendations:

Microsoft Well Architected Framework	Recommendations
Reliability	On-prem Gateway: For high availability, add the on-premises gateway to a cluster instead of installing a standalone gateway. Logic Apps: Automatically scales to meet demand. But have a few limitations for on-prem Logic Apps, Power Apps, and Power Automate support both read and write operations through the gateway: The gateway has a 2-MB payload limit for write operations. The gateway has a 2-MB request limit and an 8-MB compressed data response limit for read operations. URL for the GET request has a 2048-character limit.
Security	On-premises Gateway: The on-premises data gateway uses credential encryption and user authentication to protect data during transfers between on-premises and Azure systems. API Management: API Management helps to ensure that only authorized clients call your logic app. Since API Management is the only client that should call your logic app, consider restricting your app’s inbound IP addresses. You can also use one of these authorization schemes to limit access to your logic app: Shared access signatures (SAS). Azure Active Directory Open Authentication (Azure AD OAuth). Consider using Azure role-based access control (Azure RBAC) to only permit specific users or groups to manage, edit, and view your logic apps.
Cost optimization	Use API Management Basic, Standard, or Premium based on Logic Apps action executions and connector executions. Use the correct SQL Server database Editions Explore the below strategies for minimizing Logic Apps costs. Run SQL statements in batches. Create stored procedures to organize database results in an efficient way. Specify precise trigger conditions for workflows. Turn off logic apps that don’t have to run constantly.

4. On-premises data gateway for Azure Logic Apps

Use case:

There could be scenarios where you need to trigger the Logic Apps workflow using any front end, e.g., Azure Spring Apps or .Net Web Apps.

Typical uses for this architecture include:

Cloud-based Azure Logic Apps workflows that require data from on-premises software as part of their run.
Extending the capabilities of existing on-premises software by triggering Logic Apps workflows in the cloud.

Workflow details:

The architecture consists of the following:

Azure Spring Apps. Spring Apps provides a managed service that’s designed and optimized specifically for Spring microservices that are written in Java.

Azure Logic Apps. Logic apps are automated workflows that are provided as a scalable cloud service for common enterprise orchestration tasks. Logic apps include connectors for many popular cloud services, on-premises products, or other Software-as-a-Service applications. The Logic Apps workflow includes the following features:

Trigger that fires whenever a new Azure Queue storage message is received.

Action to parse the JavaScript Object Notation (JSON) body of the queue message.

Action to query SQL Server for relevant data.

Action to compose a response from the collected data.

Action to persist the composed response to a list on the SharePoint Server.

On-premises data gateway. An on-premises data gateway is bridge software that connects on-premises data to cloud services. The gateway typically installs on a dedicated on-premises virtual machine.

SQL Server. This is an installation of SQL Server.

SharePoint Server. This is an installation of SharePoint Server.

Recommendations:

Microsoft Well Architected Framework	Recommendations
Reliability	Avoid single point of failure issues for an on-premises data gateway by installing the software on multiple on-premises virtual machines. Consider High availability architecture for SharePoint Server SQL Server AlwaysOn
Security	While it’s possible to expose your on-premises servers to the public internet, it’s preferable to use an on-premises data gateway. This gateway creates a secure read/write connection between your on-premises data sources and Azure.
Cost optimization	While it’s possible to expose your on-premises servers to the public internet, it’s preferable to use an on-premises data gateway. This gateway creates a secure read/write connection between your on-premises data sources and Azure.
Operational excellence	The corresponding Azure resource for an on-premises data gateway should only be created after the corresponding software is installed on an on-premises virtual machine. Consider storing workflow configuration as a JSON template within an Azure Resource Manager template to automate deployment.
Performance efficiency	Consider the capacity constraints of the on-premises servers.

5. Enterprise Integration using message broker and events.

Use case:

There could be scenarios where you use API management and Logic Apps for Basic Integration and also need greater scalability and reliability. In this case, you can leverage message brokers and events to decouple services.

Workflow details:

Earlier we saw “Basic Enterprise Integration” which uses Logic Apps to orchestrate workflow directly with backend systems and APIM to create catalogs of APIs.

This new version of the architecture adds two components that help make the system more reliable and scalable:

Azure Service Bus. Service Bus is a secure, reliable message broker.

Azure Event Grid. Event Grid is an event routing service. It uses a publish/subscribe (pub/sub) eventing model.

Why use Asynchronous communication (using a message broker) over making direct synchronous calls?

Provides load-leveling to handle bursts in workloads, using the Queue-Based Load Leveling pattern.
Provides for broadcasting of messages to multiple consumers using the Publisher-Subscriber pattern.
Reliably tracks the progress of long-running workflows that involve multiple steps or multiple applications.
Helps to decouple applications.
Integrates with existing message-based systems.
Allows work to be queued when a backend system isn’t available.

Why use Event Grid?

Event Grid enables the various components in the system to react to events as they happen, rather than relying on polling or scheduled tasks.

As with a message queue and topics, it helps decouple applications and services. An application or service can publish events, and any interested subscribers will be notified. New subscribers can be added without updating the sender.

Many Azure services support sending events to Event Grid.

For example, a logic app can listen for an event when new files are added to a blob store. This pattern enables reactive workflows, where uploading a file or putting a message on a queue kicks off a series of processes. The processes might be executed in parallel or in a specific sequence.

Recommendations:

The recommendations described in Basic Enterprise Integration apply to this architecture.

Service Bus:

When you create a logic app to consume Service Bus messages, we recommend using the proxied push model with Event Grid integration. It’s often more cost-efficient, because the logic app doesn’t need to poll Service Bus.
Use PeekLock for accessing a group of messages. When you use PeekLock, the logic app can perform steps to validate each message before completing or abandoning the message. This approach protects against accidental message loss.

Event Grid

When an Event Grid trigger fires, it means at least one event happened. For example, when a logic app gets an Event Grid trigger for a Service Bus message, it should assume that several messages might be available to process.

Microsoft Well Architected Framework	Recommendations
Reliability	Azure AD: Azure AD is a globally distributed, highly available SaaS platform. API Management: API Management can be deployed in several highly available configurations, according to business requirements and cost tolerance. Logic Apps: Geo-redundant storage is available for Logic Apps on the Consumption plan tier. Event Grid: Event Grid resource definitions for topics, system topics, domains, and event subscriptions and event data are automatically replicated across three availability zones (when available) in the region. When there’s a failure in one of the availability zones, Event Grid resources automatically fail over to another availability zone without any human intervention. Service Bus: Service Bus Premium supports Geo-disaster recovery and Availability Zones. Replication is available for Service Bus Standard.
Security	Service Bus: To secure Service Bus, use Azure Active Directory (Azure AD) authentication paired with managed identities. Use Azure RBAC to grant permissions to a security principal, which may be a user, a group, or an application service principal (a managed identity in this case).Where Azure AD isn’t available, you can use shared access signature (SAS). You can grant a user access to Service Bus resources with specific rights by using SAS authentication. Event Grid: The Event Grid service secures event delivery through a validation code. If you use Logic Apps to consume the event, validation is automatically performed. Network security should be considered throughout the design. Service Bus Premium can be bound to a virtual network (VNet) subnet service endpoint, securing the namespace to only accept traffic from authorized virtual networks. Additionally, use private endpoints to lock down your VNet traffic to private traffic over Private Link. Logic Apps Standard and Premium Logic Apps can be configured to accept inbound traffic through private endpoints and to send outbound traffic through VNet integration. API Management provides several options to secure access to your API Management instance and APIs using an Azure virtual network.
Cost optimization	API Management: Auto Scaling Logic Apps: Logic Apps uses a serverless model. Billing is calculated based on action and connector execution. Service Bus queues, topics and subscriptions Service Bus queues and subscriptions support both proxied push and pull models for delivering messages. In the pull model, every polling request is metered as an action. Even with long polling at 30 secs (default), cost can be high. Unless you need real-time delivery of messages, consider using the proxied push model. Service Bus queues are included in all tiers (Basic, standard, and premium tiers). While Service Bus topics and subscriptions are available in standard and premium tiers. Event Grid Event Grid uses a serverless model. Billing is calculated based on the number of operations (event executions). Operations include ingress of events to Domains or Topics, advanced matches, delivery attempts, and management calls. Usage of up to 100,000 operations is free of charge.
Operational excellence	Use DevOps approach for Basic Enterprise Integration. Automating recovery operations: With automation in mind, you can combine Azure Log Monitoring with Azure Automation to automate the failover of your Service Bus resources.
Performance efficiency	To achieve higher scalability, the Service Bus Premium tier can scale out the number of messaging units

6. Publish internal APIs to external users.

Use case:

There could be a scenario where an organization needs to consolidate multiple APIs internally using Azure API Management deployed inside a Virtual Network.

Synchronize customer address information internally after the customer makes a change.
Attract developers to your platform by exposing unique data assets.

In this scenario, an organization hosts multiple APIs using Azure Application Service Environment (ILB ASE), and they want to consolidate these APIs internally by using Azure API Management (APIM) deployed inside a Virtual Network. The internal API Management instance could also be exposed to external users to allow for utilization of the full potential of the APIs. This external exposure could be achieved using Azure Application Gateway forwarding requests to the internal API Management service, which in turn consumes the APIs deployed in the ASE.

Workflow details:

The data flows as follows:

Developers check in code to a GitHub repository that’s connected to a CI/CD pipeline agent that’s installed on an Azure VM.
The agent pushes the build to the API application that’s hosted on ILB ASE.
Azure API Management consumes the preceding APIs via HOST headers that are specified in API Management policy.
API Management uses the App Service Environment’s DNS name for all the APIs.
Application Gateway exposes API Management’s developer and API portal.
Azure Private DNS is used to route the traffic internally between ASE, API Management, and Application Gateway.
External users utilize the exposed developer portal to consume the APIs via Application Gateway’s public IP.

Components used:

Azure Virtual Network enables Azure resources to securely communicate with each other, the internet, and on-premises networks.

Azure Private DNS allows domain names to be resolved in a virtual network without needing to add a custom DNS solution.

Azure API Management helps organizations publish APIs to external, partner, and internal developers to use their data and services.

Application Gateway is a web traffic load balancer that helps you to manage traffic to your web applications.

Internal Load Balancer App Service Environment is an Azure App Service feature that provides a fully isolated and dedicated environment for securely running App Service apps at high scale.

Azure DevOps is a service for managing your development lifecycle and includes features for planning and project management, code management, build, and release.

Application Insights is an extensible Application Performance Management (APM) service for web developers on multiple platforms.

Azure Cosmos DB is Microsoft’s globally distributed, multi-model database service.

Recommendations:

Microsoft Well Architected Framework	Recommendations
Reliability	Availability: Deploy Azure API Management service as a Multi-Region deployment for higher availability and also to reduce latencies. This feature is only available in Premium mode. The API Management service in this specific scenario consumes APIs from App Service Environments. You can also use APIM for APIs hosted on the internal on-premises infrastructure. Resiliency: Though this example scenario talks more about configuration, the APIs hosted on the App Service Environments should be resilient enough to handle errors in the requests, which are eventually managed by the API Management service and Application Gateway. Consider Retry and Circuit breaker patterns in the API design
Security	This scenario is hosted completely on an internal network, API Management and ASE are already deployed on secured infrastructure (Azure VNet). You can integrate Application Gateways with Microsoft Defender for Cloud to provide a seamless way to prevent, detect, and respond to threats to the environment.
Cost optimization	API Management is offered in four tiers: developer, basic, standard, and premium. Configure based on your requirement, App Service Environments and Application Gateway.
Performance efficiency	You can scale out API Management instances depending upon a number of factors, like number and rate of concurrent connections, the kind and number of configured policies, request and response sizes, and back-end latencies on the APIs. Scaling out instance options are available in Basic, Standard, and Premium Tiers, but are bound by an upper scale limit in Basic and Standard tiers. App Service Environments are designed for scale with limits based on the pricing tier. You can configure the apps hosted under the App Service Environments to scale out (number of instances) or scale up (instance size) depending upon the requirements of the application. Azure Application Gateway auto scaling is available as a part of the Zone redundant SKU in all global Azure regions. See the public preview feature regarding App gateway Auto scaling.

7. B2B enterprise integration workflows with Azure Logic Apps and Enterprise Integration Pack

Use case:

There could be scenarios for business-to-business (B2B) solutions and seamless communication between organizations, you can build automated, scalable, enterprise integration workflows by using Azure Logic Apps with the Enterprise Integration Pack (EIP).

If you’re familiar with Microsoft BizTalk Server, the EIP follows similar concepts and makes B2B capabilities easy to use. However, one major difference is that the EIP is architecturally based on integration accounts. These accounts are cloud-based containers in Azure that simplify how you store, manage, and use B2B artifacts for B2B communication, including trading partners, agreements, maps, schemas, certificates, and so on.

With these artifacts, you can build B2B workflows and integration solutions that include cloud services, such as Azure, Microsoft, and other software-as-service (SaaS) apps, on-premises systems, and custom apps by using Azure Logic Apps and choosing from hundreds of connectors.

Workflow details:

The following diagram shows the high-level steps to start building B2B logic app workflows:

Conclusion

We covered six scenarios for integration. These scenarios will provide a comprehensive understanding of the integration process and how it can be applied in various situations.

By understanding these scenarios, you will be able to effectively integrate different systems and processes, ensuring smooth and efficient operations within your organization.

In the upcoming post, we will delve deeper into each integration scenario and provide step-by-step instructions on how to get started with them.