Gartner identified Applied Observability as a Top strategic Technology Trend for 2023. I penned down a few ideas “on the back of an envelope” as the co-author of the trend with Frances Karamouzis – who was the original brains behind it.

As organizations launch new digital products and services, the need for fine-grained visibility into product usage is important not only for operational support but also for customer success, product management and sales teams. Building observable systems helps both business and IT stakeholders understand what features customers care about the most, what they struggle with and how they can improve the service.

Observability also offers a new approach to managing business change within organizations – Instead of monitoring business applications and processes, which is fundamentally reactive, observability relies on instrumenting processes with the necessary data and control mechanisms to enable proactive and preemptive actions. Applied Observability combines telemetry data obtained across multiple systems to make data-driven business and IT decisions. IT leaders must go beyond using observability for system reliability and build an observable digital business.

Examples include:

• Customer adoption of new and existing features – this can point to reasons for customer churn
• Demographics of active vs inactive users
• Trend analysis of customer satisfaction and its correlation with service levels and performance
• Ability to provide granular insight into business systems and workflows since observability is “built-in”

Context

Observability provides a measure of how well the internal state of a system can be inferred from its external output. Applied observability extends this principle and applies granular visibility across multiple technology domains such as applications, infrastructure, data, network, and security as well as business processes to provide valuable business insight. This ability to harness observability data enables increased innovation, improved resilience and enhanced customer adoption, engagement, and experience.

Historically, organizations found it difficult to either plan or learn from past failures due to insufficient data. However, we have an exact opposite problem today – it is not the lack but an overabundance of data that make it difficult to extract the right signals from noisy data. Applied observability is about aggregating, correlating, and analyzing observability data across multiple technology tiers and domains and making that data available for both business and technical roles to inform manual and autonomous decisions.

Extend Observability Benefits to Multiple Roles

Applied observability involves harnessing the value of observability data to meet the needs of different roles across business and IT. For example, application teams responsible for customer experience will benefit from customer observability data. Similarly, I&O teams responsible for managing service levels will need IT observability data such as events, logs, metrics, and traces. Likewise, finance departments managing operational expenses may want to know the reason behind increased cloud spend and have enriched data to make informed decisions.

Observability data is only as useful as its ability to convert knowledge of the health and performance of individual systems to overall business health and performance. Connecting individual parts using telemetry across all layers of the business and IT systems topology is key to realizing the full benefits of observability. The core observability domains that underpin this trend include:

• Data observability
• Infrastructure observability including networks, endpoints, compute, storage, communication devices etc.
• Application observability including services, APIs, databases, open-source, and 3^rd party dependencies
• Security observability
• Business decision making related to product decisions

Applied observability extends the use of observability to address business intelligence issues. For example, tracking customer adoption and sustained use of a newly released feature requires continuous observability into user experience and usage patterns. The ability to obtain answers to exploratory questions using observability data helps to create new knowledge and bridge knowledge gaps for both engineering and business stakeholders in the organization.

Applied observability makes observability a shared responsibility and a shared practice within the organization.

Points of Value and Points of Failure

Observability data in and of itself has no value unless it augments decision making at different “points of value”. Points of value can be viewed as the critical junctions in business workflows that affect service levels, user experience and business decisions. Therefore, it is crucial to ensure that observability data is easy to use across teams – those that are driving the change and others impacted by the change.

Applying Observability helps discover discrepancies between how we think systems behave vs how they actually behave.

The increased complexity of distributed system architectures creates the need for end-to-end observability to ensure that systems are reliable and resilient. Distributed systems are characterized by multiple points of failure, complex web of interactions and the inability to predict the impact of component failures on the system (as a whole). As consumers use digital services as the first and in many cases the only touchpoint to meet their needs, it dramatically increases the need for digital resilience. Optimizing resilience metrics such as MTTR and RTO requires accurate observability data for troubleshooting and root cause analysis.

Recommendations

Introduce observability as a core ingredient of application design in addition to practices such as observability-driven development. Much like security, integrate observability into the complete software development lifecycle.

Improve the user experience of digital products and services by instrumenting business workflows and applying observability to every layer in the technology stack to discover, learn and improve the usage of IT services.

Upskill and equip teams with capabilities to design and architect for observability with a goal to converting system performance to business performance.

Applied Observability – Converting System Performance to Business Performance