Enterprise integration with message broker and events

This example architecture is built on the Basic enterprise integration architecture and extends it to show how to integrate enterprise backend systems, using message brokers and events to decouple services for greater scalability and reliability. Ensure that you are familiar with that design and the components used in the basic integration architecture to prepare for this article as it provides foundational information on the core components that will not be reproduced here.

The backend systems referenced in this design may include software as a service (SaaS) systems, Azure services, and existing web services in your enterprise.

Download a Visio file of this architecture.

Architecture

The architecture shown here builds on a simpler architecture that is shown in Basic enterprise integration. That architecture uses Logic Apps to orchestrate workflows directly with backend systems and API Management to create catalogs of APIs.

This version of the architecture adds two components that help make the system more reliable and scalable:

Asynchronous communication using a message broker provides a number of advantages over making direct, synchronous calls to backend services:

• Provides load-leveling to handle bursts in workloads, using the Queue-Based Load Leveling pattern.
• Provides for broadcasting of messages to multiple consumers using Publisher-Subscriber pattern.
• Reliably tracks the progress of long-running workflows that involve multiple steps or multiple applications.
• Helps to decouple applications.
• Integrates with existing message-based systems.
• Allows work to be queued when a backend system is not available.

Event Grid enables the various components in the system to react to events as they happen, rather than relying on polling or scheduled tasks. As with a message queue and topics, it helps decouple applications and services. An application or service can publish events, and any interested subscribers will be notified. New subscribers can be added without updating the sender.

Many Azure services support sending events to Event Grid. For example, a logic app can listen for an event when new files are added to a blob store. This pattern enables reactive workflows, where uploading a file or putting a message on a queue kicks off a series of processes. The processes might be executed in parallel or in a specific sequence.

Recommendations

The recommendations described in Basic enterprise integration apply to this architecture.

Service Bus

Service Bus has two delivery modes, pull or proxied push. In the pull model, the receiver continuously polls for new messages. Polling can be inefficient, especially if you have many queues that each receive a few messages, or if there a lot of time between messages. In the proxied push model, Service Bus sends an event through Event Grid when there are new messages. The receiver subscribes to the event. When the event is triggered, the receiver pulls the next batch of messages from Service Bus.

When you create a logic app to consume Service Bus messages, we recommend using the proxied push model with Event Grid integration. It's often more cost efficient, because the logic app doesn't need to poll Service Bus. For more information, see Azure Service Bus to Event Grid integration overview. Currently, Service Bus Premium tier is required for Event Grid notifications.

Use PeekLock for accessing a group of messages. When you use PeekLock, the logic app can perform steps to validate each message before completing or abandoning the message. This approach protects against accidental message loss.

Event Grid

When an Event Grid trigger fires, it means at least one event happened. For example, when a logic app gets an Event Grid triggers for a Service Bus message, it should assume that several messages might be available to process.

Considerations

These considerations implement the pillars of the Azure Well-Architected Framework, which is a set of guiding tenets that can be used to improve the quality of a workload. For more information, see Microsoft Azure Well-Architected Framework.

Reliability

• Azure AD: Azure AD is a globally distributed, highly available SaaS platform. Please refer to the SLA for guaranteed availability details.
• API Management: API Management can be deployed in a variety of highly available configurations, according to business requirements and cost tolerance. Refer to the Ensure API Management availability and reliability for a full review of options. Please also refer to the SLA for guaranteed availability details.
• Logic Apps: Geo-redundant storage is available for Logic Apps on the Consumption plan tier. For information on designing a business continuity and disaster recovery solution, refer to the guidance. Please also refer to the SLA for guaranteed availability details.
• Event Grid: Event Grid resource definitions for topics, system topics, domains, and event subscriptions and event data are automatically replicated across three availability zones (when available) in the region. When there's a failure in one of the availability zones, Event Grid resources automatically failover to another availability zone without any human intervention. Please refer to the Geo-disaster recovery across regions for guidance on designing a disaster recovery solution for failing over to another region. Please also refer to the SLA for guaranteed availability details.
• Service Bus: Service Bus Premium supports Geo-disaster recovery and Availability Zones. Replication is available for Service Bus Standard. Please also refer to the SLA for guaranteed availability details.

Security

To secure Service Bus, use Azure Active Directory (Azure AD) authentication paired with managed identities. Azure AD integration for Service Bus resources provides Azure role-based access control (RBAC) for fine-grained control over a client’s access to resources. You can use Azure RBAC to grant permissions to a security principal, which may be a user, a group, or an application service principal (a managed identity in this case).

Where Azure AD isn't available, you can use shared access signature (SAS). You can grant a user access to Service Bus resources with specific rights by using SAS authentication.

If you need to expose a Service Bus queue or topic as an HTTP endpoint, for example, to post new messages, use API Management to secure the queue by fronting the endpoint. You can then secure the endpoint with certificates or OAuth authentication as appropriate. The easiest way to secure an endpoint is using a logic app with an HTTP request/response trigger as an intermediary.

The Event Grid service secures event delivery through a validation code. If you use Logic Apps to consume the event, validation is automatically performed. For more information, see Event Grid security and authentication.

Network security

Network security should be considered throughout the design.

Cost Optimization

In general, use the Azure pricing calculator to estimate costs. Here are some other considerations.

API Management

You are charged for all API Management instances when they are running. If you have scaled up and don't need that level of performance all the time, manually scale down or configure autoscaling.

For light usage workloads, consider the consumption tier which is a low-cost, serverless option. The consumption tier is billed per API call, whereas the other tiers are billed per hour.

Logic Apps

Logic Apps uses a serverless model. Billing is calculated based on action and connector execution. For more information, see Logic Apps pricing.

Service Bus queues, topics and subscriptions

Service Bus queues and subscriptions support both proxied push and pull models for delivering messages. In the pull model, every polling request is metered as an action. Even with long polling at 30 secs (default), cost can be high. Unless you need real-time delivery of messages, consider using the proxied push model.

Service Bus queues are included in all tiers (Basic, standard, and premium tiers). While Service Bus topics and subscriptions are available in standard and premium tiers. For more information, see Azure Service Bus pricing.

Event Grid

Event Grid uses a serverless model. Billing is calculated based on the number of operations (event executions). Operations include ingress of events to Domains or Topics, advanced matches, delivery attempts, and management calls. Usage of up to 100,000 operations is free of charge.

For more information, see Event Grid pricing.

For more information, see the cost section in Microsoft Azure Well-Architected Framework.

Operational Excellence

The Basic Enterprise Integration reference architecture provides guidance on DevOps patterns, which align to the Well-Architected Framework's Operational Excellence pillar.

Automating recovery operations as much as possible is an integral component of Operational Excellence. With this in mind, you can combine Azure Log Monitoring with Azure Automation to automate the failover of your Service Bus resources. Refer to the diagram in failover flow documentation for an example of automation logic to initiate a failover.

Performance efficiency

To achieve higher scalability, the Service Bus Premium tier can scale out the number of messaging units. Refer to the Service Bus Premium and Standard messaging tiers documentation for a review of the Premium tier benefits and the autoscaling feature documentation to learn about configuring the autoscaling of messaging units.

Additional recommendations for Service Bus can be found at Best practices for performance improvements by using Service Bus Messaging.

Next steps