Knowledge graphs and GraphQL: What DevOps pros need to know

Enterprise digital transformation has caused an explosion of business data — and with it, a re-examination of data management and analytics techniques that has taken on significant implications for IT ops teams.

Traditional data management and analytics tools were built on relational databases, which use rigid schemas to organize data. Knowledge graphs, often in the form of graph databases, instead make subtler inferences in context about relationships between groups of data sets. Data scientists access such contextual data models through specific forms of compatible data catalogs and federated APIs, the best-known of which is open source GraphQL

Graphs, and the contextual data analysis they make possible, aren't new inventions. But they are a major trend in managing and analyzing data amid the growth of artificial intelligence and data-driven decision-making. In fact, Gartner predicted in April that by 2025, context-driven analytics and AI models will replace 60% of existing models built on traditional data.

While initial enterprise applications for knowledge graphs have primarily emerged in data science and business intelligence, in 2022 they have begun to trickle into products from established IT observability and AIOps vendors as well. AIOps vendor Dynatrace, for example, feeds data from the Grail data lakehouse it rolled out last month into its Smartscape tool, which creates a graph view of relationships among IT resources, applications and observability data sets such as logs, metrics and traces. VMware revamped its vRealize IT monitoring and management tools with a new system accessed through a graph API called Aria, unveiled in August.

“Current observability [data] volume is beyond human comprehension,” said Andy Thurai, vice president and principal analyst at Constellation Research. “Even an army of IT analysts can't correlate siloed data and make meaning out of it, especially logs, [which] are very messy if they [are] siloed and disjointed. This is where technologies such as graph or other search mechanisms can help.”

IT management applications for knowledge graphs aren't limited to observability tools. Other IT vendor products increasingly depend on or interact with knowledge graphs and graph APIs, from Salesforce's Genie to Solo.io's application networking platform, which includes security and traffic management for graph APIs. Graphs are poised to infiltrate data governance as well, through a CNCF Software Bill of Materials project built on the Neo4j graph database.

IT ops implications of knowledge graphs

The good news for IT operations teams as knowledge graphs and graph APIs gain popularity lies in the way they can improve access to data for analysis without requiring that they be moved into a central repository.

“Moving logs to search cannot only be very time consuming — they can take days to weeks to move — but it will be expensive to egress or ingress data [among cloud providers] before the search can even begin,” Thurai said. “Cheap storage solutions such as Amazon S3 can be very appealing, but searching logs there is not an easy task.”

This is where graph APIs, and the knowledge graph models that normalize disparate data sets to present them for API queries, come in.

“Providing contextual data based on the incident and correlating the entire observability data set including incident logs, traces and metrics is key to solving incidents faster,” Thurai said. “All log platforms are moving toward centralized search with a distributed storage.”

Reorganizing data sets into a graph model, or ontology, requires not only data science expertise, but also distributed data storage with carefully managed I/O performance characteristics. Both graph APIs and data repositories that back them require updated approaches to secure access, namely zero-trust security, which relies on fine-grained user and machine identities rather than data center or application perimeters.

Because of the complexities of all these approaches, some data management experts predict that most enterprises will tap knowledge graphs via SaaS offerings, such as Dynatrace's Grail, which does not have a self-managed version.

“GraphQL operates on the running assumption that information wants to be free,” said Tony Baer, principal at independent database and analytics consulting firm DbInsight LLC in New York. “It lacks built-in security, which requires separate tooling. … In practice, I'd expect that most implementation will be SaaS, both for reasons of infrastructure but also for [simplicity]. To make something simple externally requires a lot of complexity under the hood. Call it the Google principle.”

Indeed, Google and other major cloud providers such as Microsoft are building knowledge graphs and graph API access to systems such as Microsoft 365.

Because organizations are moving to cloud, and [within that] multi-cloud, that in itself is made up of distributed data. Knowledge graphs are the one of the only ways to analyze that data at scale.

Shaun RollsGlobal head, Open Knowledge Graph Lab, EDM Council

“Because organizations are moving to cloud, and [within that] multi-cloud, that in itself is made up of distributed data,” said Shaun Rolls, global head of the Open Knowledge Graph Lab at the Enterprise Data Management (EDM) Council, an industry association headquartered in New York that hosts graph testing and training resources and has produced a reference graph ontology for businesses in the financial industry. “Knowledge graphs are the one of the only ways to analyze that data at scale.”

Despite predictions that most enterprises will go the SaaS route with graphs, Rolls said he's seen an “ecosystem explosion” in tools and demand among large organizations that want to deploy their own knowledge graphs.

But as with so many new technologies, finding professionals with the required skill sets to build and manage knowledge graphs within enterprises can be difficult, Rolls said.

“You can buy the tooling and you can buy the graph databases, but these don't do the knowledge graph [architecture] for you,” he said. “You have to create the knowledge graph models within the organization. … You really have to know what catalogs work on the cloud and which ones work with certain knowledge graph patterns, depending on the stack that your company has purchased. … It's been a big issue.”

BT investigates graphs for BI, IT resilience

One of the enterprises working with EDM Council on knowledge graph training is BT Group, which became a member in October 2021 and has created an internal graph guild with about 80 employees to learn about graphs and apply them to business analytics and IT management.

“The terminology in this space is rather confusing because the term graph can mean so many different things to different people,” said Ben Clinch, principal enterprise architect for data governance and information architecture at BT, who leads the telco's graph guild. “The first question I ask when I hear the word graph is, ‘Are you using a [World Wide Web Consortium] W3C [Resource Description Framework] RDF — compliant graph, or not?'”

If the answer is yes, Clinch said that means the system will easily connect with others within BT's graph ontology, which he said is the most crucial component of a knowledge graph.

“An ontology is a model that allows semantic data context to be applied, and AI and machine learning can then infer more meaningful outcomes, because they can understand the data more effectively than if it was in a pure relational model,” Clinch said.

For example, analysis of relational models wouldn't easily be able to discern whether a series of numbers is related to a postal address or a pair of letters is a person's initials, but graph ontologies could. That could lead to new insights about corporate data, as well as IT systems, Clinch said.

BT's IT teams also use Dynatrace AIOps tools, and Grail could potentially feed data into BT's graph, according to Clinch.

“[Dynatrace Smartscape] is an incredibly powerful tool for harvesting technical metadata across the [IT] estate,” he said. “You could combine trace information with the context of the business processes that systems are supporting and the semantic meaning of the data flowing across the connections that Dynatrace is detecting.”

Among the uses for such a data set would be creating a digital twin or virtual copy of the company's IT environment, to identify opportunities to optimize network performance or make disaster recovery plans.

“You can draw network diagrams for specific domains, or the entire enterprise, quite rapidly, and you can then see if anything is missing, or if there are opportunities to simplify your domain,” Clinch said. “You can run [failure] scenarios against a digital twin to say, ‘What would happen if this system went down or all of these nodes in the network went down?'”

Securing and maintaining graph systems isn't fundamentally different for IT ops teams than traditional relational systems, where access controls and data security measures are well understood, Clinch said.

He acknowledged that the patterns for applying them to graphs will have some differences. However, because graphs separate the access layer from back-end data sets, they can make some aspects of IT governance easier work with, according to Clinch.

“The shape of the graph often is non-sensitive, [even if] the underlying data may be sensitive,” he said. “You can sort of explain what's available … and ‘contact so-and-so to be able to access it.' And then [users will] need to probably state the purpose for which they need access and how [they're] going to be storing it and so forth.”

Beth Pariseau, senior news writer at TechTarget Editorial, is an award-winning veteran of IT journalism. She can be reached at [email protected] or on Twitter @PariseauTT.

This diagram by business analytics vendor Sisense shows how a knowledge graph maps relationships between different data sets.