Basic OpenAI end-to-end chat reference architecture
For teams evaluating Azure OpenAI chat patterns, the main architectural lesson is separation of concerns: the web UI, orchestration layer, search retrieval, and model endpoint are distinct services with distinct failure modes. That means the design is easy to reason about for a proof of concept, but each hop adds latency, identity handling, and operational surface area that must be planned before any production rollout.
The retrieval path is the technical hinge of the system. Prompt flow turns the application into a RAG pipeline by extracting intent, querying Azure AI Search, and packaging grounding data for the model call. That makes data quality, index freshness, and query relevance as important as model selection. In practice, teams need to treat the search schema and prompt format as coupled artifacts, because changes to one can alter answer quality even when the model stays the same.
Operationally, the biggest trade-off in this basic version is simplicity versus resilience. Single-region hosting, non-zonal App Service, basic search tiering, and unmanaged scaling keep the POC small, but they also concentrate risk in the UI, orchestration compute, and search layer. The deployment model is therefore useful for learning flow packaging and endpoint invocation, not for benchmarking availability or throughput expectations.
Security is intentionally partial here, which is a useful signal for architecture review. The design leans on identity, managed services, and content filtering, but it leaves network perimeter hardening and production-grade secret handling for a later phase. For architects, the practical takeaway is that moving from demo to enterprise chat means reworking authentication paths, adding private networking, and formalizing telemetry, scale, and release controls around the flow itself.
- A user issues an HTTPS request to the App Service’s default domain on azurewebsites.net. This domain automatically points to the App Service’s built-in public IP. The transport layer security (TLS) connection is established from the client directly to app service. The certificate is managed completely by Azure.
- Easy Auth, a feature of Azure App Service, ensures that the user accessing the site is authenticated with Microsoft Entra ID.
- The client application code deployed to App Service handles the request and presents the user a chat UI. The chat UI code connects to APIs also hosted in that same App Service instance. The API code connects to an Azure Machine Learning managed online endpoint to handle user interactions.
- The managed online endpoint routes the request to an Azure Machine Learning compute instance where the Azure Machine Learning prompt flow orchestration logic is deployed.
- The Azure Machine Learning prompt flow orchestration code begins executing. Among other things, the logic extracts the user’s query from the request.
- The orchestration logic connects to Azure AI Search to fetch grounding data for the query. The grounding data is added to the prompt that will be sent to Azure OpenAI in the next step.
- The orchestration logic connects to Azure OpenAI and sends the prompt that includes the relevant grounding data.
- The information about original request to App Service and the call to the managed online endpoint are logged in Application Insights, using the same Log Analytics workspace that Azure OpenAI telemetry flows to.
Machine Learning prompt flow
- The user enters a prompt in a custom chat user interface (UI).
- The interface’s API code sends that text to prompt flow.
- Prompt flow extracts the user intent, either a question or directive, from the prompt.
- Optionally, prompt flow determines the data store(s) that hold data that’s relevant to the user prompt.
- Prompt flow queries the relevant data stores.
- Prompt flow sends the intent, the relevant grounding data, and any history provided in the prompt to the language model.
- Prompt flow returns the result so that it can be displayed on the UI.
Components
Many of the components of this architecture are the same as the resources in the basic App Service web application architecture because the chat UI is based on that architecture. The components highlighted in this section focus on the components used to build and orchestrate chat flows, data services, and the services that expose the language models.- Machine Learning is a managed cloud service that you can use to train, deploy, and manage machine learning models. This architecture uses several other features of Machine Learning that are used to develop and deploy executable flows for AI applications that are powered by language models:
- Machine Learning prompt flow is a development tool that you can use to build, evaluate, and deploy flows that link user prompts, actions through Python code, and calls to language learning models. Prompt flow is used in this architecture as the layer that orchestrates flows between the prompt, different data stores, and the language model. Machine Learning can directly host two types of prompt flow runtimes.
- Automatic runtime: A serverless compute option that manages the lifecycle and performance characteristics of the compute and allows flow-driven customization of the environment. This architecture uses the automatic runtime for simplicity.
- Compute instance runtime: An always-on compute option in which the workload team must select the performance characteristics. This runtime offers more customization and control of the environment.
- Managed online endpoints let you deploy a flow for real-time inferencing. In this architecture, they’re used as a PaaS endpoint for the chat UI to invoke the prompt flows hosted by the Machine Learning automatic runtime.
- Machine Learning prompt flow is a development tool that you can use to build, evaluate, and deploy flows that link user prompts, actions through Python code, and calls to language learning models. Prompt flow is used in this architecture as the layer that orchestrates flows between the prompt, different data stores, and the language model. Machine Learning can directly host two types of prompt flow runtimes.
- Storage is used to persist the prompt flow source files for prompt flow development.
- Container Registry lets you build, store, and manage container images and artifacts in a private registry for all types of container deployments. In this architecture, flows are packaged as container images and stored in Container Registry.
- Azure OpenAI is a fully managed service that provides REST API access to Azure OpenAI’s language models, including the GPT-4, GPT-3.5-Turbo, and embeddings set of models. In this architecture, in addition to model access, it’s used to add common enterprise features such as managed identity support, and content filtering.
- Azure AI Search is a cloud search service that supports full-text search, semantic search, vector search, and hybrid search. AI Search is included in the architecture because it’s a common service used in the flows behind chat applications. AI Search can be used to retrieve and index data that’s relevant for user queries. The prompt flow implements the RAG Retrieval Augmented Generation pattern to extract the appropriate query from the prompt, query AI Search, and use the results as grounding data for the Azure OpenAI model.
Recommendations and considerations
- The App Service Plan is configured for the
Standardtier, which doesn’t have Azure availability zone support. The App Service becomes unavailable in the event of any issue with the instance, the rack, or the datacenter hosting the instance. As you move toward production, follow guidance in the reliability section of the baseline highly available zone-redundant web application. - Autoscaling for the client user interface isn’t enabled in this basic architecture. To prevent reliability issues due to lack of available compute resources, you’d need to overprovision to always run with enough compute to handle max concurrent capacity.
- Azure Machine Learning compute doesn’t offer support for availability zones. The orchestrator becomes unavailable in the event of any issue with the instance, the rack, or the datacenter hosting the instance. See the zonal redundancy for flow deployments in the baseline architecture to learn how to deploy the orchestration logic to infrastructure that supports availability zones.
- Azure OpenAI isn’t implemented in a highly available configuration. To learn how to implement Azure OpenAI in a reliable manner, see Azure OpenAI – reliability in the baseline architecture.
- Azure AI Search is configured for the
Basictier, which doesn’t have Azure availability zone support. To achieve zonal redundancy, deploy AI Search with the Standard pricing tier or higher in a region that supports availability zones, and deploy three or more replicas. - Autoscaling isn’t implemented for the Machine Learning compute. See machine learning reliability guidance in the baseline architecture.
Content filtering and abuse monitoring
Identity and access management
- Create separate managed identities for the following Machine Learning resources, where applicable:
- Workspaces for flow authoring and management
- Compute instances for testing flows
- Online endpoints in the deployed flow if the flow is deployed to a managed online endpoint
- Implement identity-access controls for the chat UI by using Microsoft Entra ID
Machine Learning role-based access roles
| Managed identity | Scope | Role assignments |
|---|---|---|
| Workspace managed identity | Resource group | Contributor |
| Workspace managed identity | Workspace Storage Account | Storage Blob Data Contributor |
| Workspace managed identity | Workspace Storage Account | Storage File Data Privileged Contributor |
| Workspace managed identity | Workspace Key Vault | Key Vault Administrator |
| Workspace managed identity | Workspace Container Registry | AcrPush |
| Online endpoint managed identity | Azure OpenAI | Cognitive Services OpenAI User |
| Online endpoint managed identity | Workspace Container Registry | AcrPull |
| Online endpoint managed identity | Workspace Storage Account | Storage Blob Data Reader |
| Online endpoint managed identity | Machine Learning workspace | AzureML Workspace Connection Secrets Reader |
| Compute instance managed identity | Workspace Container Registry | AcrPull |
| Compute instance managed identity | Workspace Storage Account | Storage Blob Data Reader |
- This architecture assumes that there are limited calls to Azure OpenAI. For this reason, we suggest you use pay-as-you-go pricing and not provisioned throughput. As you move toward a production solution, follow the Azure OpenAI cost optimization guidance in the baseline architecture.
- The App Service plan is configured for the Standard pricing tier on a single instance, which doesn’t offer protection from an Availability Zone outage. The baseline App Service architecture recommends you use Standard or Premium plans with three or more worker instances for high availability which will impact your cost.
- Scaling isn’t configured for the managed online endpoint managed compute. For production deployments, you should configure auto scaling. Further, the baseline end-to-end chat architecture recommends deploying to Azure App Service in a zonal redundant configuration. Both of these architectural changes affect your cost when moving to production.
- Azure AI Search is configured for the Basic pricing tier with no added replicas. This topology could not withstand an Azure availability zone failure. The baseline end-to-end chat architecture recommends you deploy with the Standard pricing tier or higher and deploy three or more replicas, which impact your cost as you move toward production.
- There are no cost governance or containment controls in place in this architecture. Make sure you guard against ungoverned processes or usage that could incur high costs for pay-as-you-go services like Azure OpenAI.
Machine Learning – built-in prompt flow runtimes
requirements.txt file and flow.dag.yaml configuration. The automatic runtime is low maintenance, ephemeral, and application-driven.
Diagnostics are configured for all services. All services but Machine Learning and App Service are configured to capture all logs. The Machine Learning diagnostics are configured to capture the audit logs that are all resource logs that record customer interactions with data or the settings of the service. App Service is configured to capture AppServiceHTTPLogs, AppServiceConsoleLogs, AppServiceAppLogs, and AppServicePlatformLogs. During the proof of concept phase, it’s important to get an understanding of what logs and metrics are available to be captured. When you move to production, you should eliminate log sources that are not adding value and are adding noise and cost to your workload’s log sink.
Language model operations
- This architecture uses the consumption or pay-as-you-go model for most components. The consumption model is best-effort and might be subject to noisy neighbor or other stressors on the platform. As you move toward production, you should determine whether your application requires provisioned throughput which ensures reserved processing capacity for your Azure OpenAI model deployments. Reserved capacity provides predictable performance and throughput for your models.
- The Azure Machine Learning online endpoint doesn’t have automatic scaling implemented so you’d need to provision a SKU and instance quantity that can handle peak load. The service, as configured, doesn’t dynamically scale in to efficiently keep supply aligned with demand. As you move toward production, follow the guidance about how to autoscale an online endpoint.
Enjoyed this article? Sign up for our newsletter to receive regular insights and stay connected.
