Improve Amazon Bedrock Observability with Amazon CloudWatch AppSignals
For teams shipping GenAI on AWS, the key architectural shift is treating Bedrock calls as first-class application dependencies rather than opaque API calls. That matters because control-plane actions such as agent, knowledge base, datasource, and guardrail configuration behave differently from runtime inference requests. Separating those paths in observability helps engineers understand whether a failure is coming from setup, orchestration, or model execution.
Application Signals adds value where standard service metrics stop: cross-service correlation. In practice, the hard problem is not seeing that latency increased, but proving which hop introduced it and whether the bottleneck is the application code, the selected model, or the request path through retrieval and agent logic. Trace linkage and dependency views shorten that investigation, which is especially important when LLM usage is embedded inside distributed workflows.
The operational trade-off is richer telemetry versus higher discipline around what you emit and monitor. Token counts, prompt/response metadata, and model identifiers can improve cost attribution and tuning, but they also push teams to define clearer SLOs for quality, latency, and throughput. Without that framing, observability can become a dashboard of interesting but unactionable data.
Model comparison is another practical use case, but it should be handled like an engineering decision, not a one-time benchmark. Different models can behave differently under real workload mix, so performance comparisons need to be revisited as prompts, context windows, and retrieval patterns evolve. For platform teams, the main win is having the telemetry needed to make those choices with evidence instead of guesswork.
- Prompt metrics like token usage, costs, and model IDs for individual transactions and operations, apart from service-level aggregations.
- Output quality factors including potential toxicity, harm, truncation due to length limits, and failures from exceeding token allowances.
- Performance visibility for advanced LLM use cases like agent-based interactions, custom knowledge stores with Retrieval-Augmented Generation (RAG), etc.
- Performance visibility to compare LLMs to choose the best model based on price, performance and tuning.
Walkthrough
In this post, weโll present how you can leverage the enhanced telemetry for Amazon Bedrock in Amazon CloudWatch Application Signals to automatically assess the performance of the LLMs in Amazon Bedrock in the context of their applications, reducing the Mean Time to Identify (MTTI) with improved visibility and debugging capabilities. You can drill down to the performance of individual models using the golden signals of Application Performance Monitoring (APM), correlate service operation performance to the LLMs, and debug anomalies using traces for Bedrock.Prerequisites
- An AWS Account with access to the appropriate models enabled.
- A sample Generative AI application using Amazon Bedrock. For the purposes of this blog post, follow the instructions from GitHub to deploy the sample app. If using this sample application, follow the Amazon Bedrock models access documentation to enable access to the
amazon.titan-text-express-v1and theanthropic.claude-3-sonnet-20240229-v1:0models if not already enabled. - If not already enabled, follow the instructions to Enable Application Signals in your account.
Service Map
Navigate to the Amazon CloudWatch console and choose Service Map under the Application Signals section in the left navigation pane. As shown in Figure 1, you will now see two new types of nodes for Amazon Bedrock.
Bedrock node
This represents all Bedrock Control Plane Requests including model configuration and deployment requests. Depending on your application, you may notice more than one Bedrock node as a dependency. Each Bedrock node groups together calls to configure a type of resource, where the resource could be one of the following:- Agentย โ Agents for Amazon Bedrock offers you the ability to build and configure autonomous agents in your
application. Agents orchestrate interactions between foundation models (FMs), data sources, software applications, and user conversations. - Knowledge base โ Allows you to integrate proprietary information into your generative-AI applications.
- Datasource โ The source of information to the knowledge base. For example, Amazon S3, Confluence, etc.
- Guardrail โ Enables you to implement safeguards for your generative AI applications based on your use cases and responsible AI policies. You can create multiple guardrails tailored to different use cases and apply them across multiple foundation models (FM), providing a consistent user experience and standardizing safety and privacy controls across generative AI applications. You can use guardrails with text-based user inputs and model responses.
KnowledgeBase type Bedrock resource with id FIFFBKOOWC. To find additional details, you can navigate through the dependency as highlighted below for the BedrockRuntime dependency.
BedrockRuntime node
This node represents all data plane requests to Amazon Bedrock including model inference requests. Depending on your application, you may notice more than oneBedrockRuntime node as a dependency, where each node will represent the interactions with a model. In the given example, the code makes a request to both the amazon.titan-text-express-v1 and the anthropic.claude-3-sonnet-20240229-v1:0 models and hence Figure 1 above shows 2 instances of BedrockRuntime.
By selecting the BedrockRuntime node, as seen in Figure 3 below, you will get details on the model in use, a summary of service performance and golden signals including number of requests, average latency, error and fault rates. Adding to the summary, this section also includes the top 3 paths by fault rate, latency and error rate with easy to navigate links. This top-level summary can be useful to easily pinpoint which request path including Amazon Bedrock is causing increased latency or errors. Since we have only one path in our application, letโs open the customers-service-java link to understand more about the latency of the request.
customer-service-java and Amazon Bedrock. From the image, you can see that it is a dependency introduced by the InvokeModel request and the dependency on both the models in use. To dive deeper into the metrics, select a specific dependency metric data point, which provides the details of the correlated traces.
InvokeModel request to the Anthropic Claude model is the biggest contributor to the latency or processing time.
prompt_token_countโ The number of tokens in the prompt also called input token count (inputTextTokenCounton Figure 6)generation_token_countโ The number of tokens in the generated text also called output token count (outputTokenCounton Figure 6)
Cleanup
Refer to the cleanup instructions in the README appropriate to the deployment method chosen.Conclusion
In this post we explained how you can use Amazon CloudWatch Application Signals to monitor generative AI applications built using Amazon Bedrock enabling you to choose the optimal LLM for your use case, while delivering a delightful user experience. The metadata and automatic instrumentation enable you to thoughtfully choose and configure appropriate SLOs to deliver the best experience while optimizing for cost and throughput. Application Signals support for Amazon Bedrock is currently available in the Java and Python AWS SDK for requests to all versions of the Amazon Titan, Anthropic Claude and Meta Llama LLMs. Application Signals is generally available in 28 commercial AWS Regions, except CA West (Calgary) Region, AWS GovCloud (US) Regions and China Regions. For pricing, see Original Postricing/" shape="rect">Amazon CloudWatch pricing. Try Application Signals with the AWS One Observability Workshop sample application. To get started, see the documentation to enable Amazon CloudWatch Application Signals for Amazon EKS, Amazon EC2, native Kubernetes and custom instrumentation for other platforms.About the authors:
Rodrigue Koffi
Rodrigue is a Specialist Solutions Architect at Amazon Web Services for Observability. He is passionate about observability, distributed systems, and machine learning. He has a strong DevOps and software development background and loves programming with Go. Find him on LinkedIn at /grkoffi.Jyothi Madanlal
Jyothi Madanlal is a seasoned IT professional with over 20+ years of experience. Currently, as a Solution Architect at AWS, she helps New Zealand Software/SaaS companies leverage cutting-edge technology to solve complex business challenges. Jyothi is passionate about finding efficient, sustainable and scalable solutions that drive impactful results. In her spare time, she can be found hiking with her dog, Pepper.Enjoyed this article? Sign up for our newsletter to receive regular insights and stay connected.
