AI succession crisis: Why AI knowledge isn’t easily transferable

IT leaders now see AI knowledge as critical to success. What’s not so widely known is that such knowledge isn’t transferable. This poses a huge problem, one that seems to inevitably deepen over time into a new, complex layer of tech debt.

Traditional IT systems are usually tied to documented logic: code, configurations, workflows, and system architecture, while AI knowledge is more context-dependent, said Jamie Dearnley, CTO at enterprise risk and incident management software company Resolver. “You can look at the code or documentation to understand how they work and why they behave the way they do.”

AI is different, however, since its behavior depends not just on code, but also on trained models, data, prompts, and settings that change over time. “A lot of what makes an AI system work comes from testing, trial and error, and the experience of the people building it,” Dearnley noted.

This means that it’s not enough to document the technology;. Oorganizations also need to document the decisions behind them. “Otherwise, important knowledge can be lost when people leave or move onto new roles,” he said.

With traditional software, the code is the truth, said Guy Maliar, CTO at AI developer Latent AI. “A new engineer, for example, reads the code and can reconstruct why the system does what it does, because the behavior is deterministic and it’s all right there.” AI breaks this type of model. The behavior isn’t really in the code — it emerges from the model, the prompts, the data, and a thousand small judgment calls the team made along the way, he explained.

Almost none of this gets written down, Maliar said. “Since the system is non-deterministic, you can’t just read it and predict what it’ll do, so you’ll have to rebuild a feel for its quirks.” That requires watching the system work and fail, so the organization retains understanding if and when the original team walks out the door.

Next steps

To make AI knowledge more accessible, start by creating an inventory of every AI system currently used across your organization –, including shadow AI tools adopted by individual teams without formal approval, Dearnley said. Many organizations have more AI in use than they realize.

“Once you know what exists, identify who owns each system, which business process it supports, which data it relies on, and what level of risk it presents,” he said. “You can’t govern or effectively transfer knowledge about something you don’t know you have.”

Treat the hard-won empirical knowledge as a real deliverable, not a byproduct, Maliar recommended. “In practice, this means keeping versioned eval sets that capture what ‘correct’ looks like … and keeping decision traces so successors can see the reasoning behind the design.” The evals matter most, he added, since they’ll let a new team confirm they haven’t broken anything, even before they fully understand it. “For AI, knowledge transfer is less about explaining the system and more about preserving the ability to test it.”

Create an AI system inventory, recommended Sanjay Kukreja, CTO with business process management and technology consulting firm eClerx. “Know which models, agents, datasets, prompts, tools, and business processes exist, as well as who owns them and what decisions or actions they support,” he advises.

Seeking solutions

It’s not just about keeping a record but being able to assess the quality of the record. Build the evaluation harness first, Maliar recommended. If you can’t answer “how do we know this thing is still working?” via an automated, repeatable test, no amount of documentation will save you, he said. “The eval set is the foundation everything else sits on, since it captures the team’s judgment in a form that actually survives people leaving.”

The biggest risk isn’t that AI makes mistakes — it’s that organizations don’t know why it made them, observed Prateek Mishra, CTO at Joveo, an AI-focused recruitment firm. “If users can’t understand the decisions, trust breaks down immediately,” he said. “As we move toward more agentic AI, meaning systems that can stitch together point solutions and execute complex high-volume workflows, transparency becomes increasingly critical,” Mishra noted.

Organizations now need visibility into whatever data influenced a decision and where humans need to remain in the loop. “We have to do a lot of testing and research to ensure these systems are explainable,” he said. “It’s a prerequisite for deploying AI responsibly.”

The black box issue

In AI, the “black box” problem refers to the inability to see or explain exactly how an AI system processes inputs to arrive at a specific output. In complex models, such as deep neural networks, billions of mathematical interactions create decisions that are virtually impossible for human developers to manually trace or interpret. This can create a serious and complex problem.

Organizations are piling up a nasty kind of technical debt, Maliar warned. The system runs fine right up until it doesn’t — a model update, some data drift, an input nobody anticipated — and at that point no one has the context to figure out what went wrong. “You end up dependent on something you can neither fix nor safely change, which pushes you toward one of two bad places: too scared to touch it or rip it out and rebuild because understanding it costs more than starting over,” he said. “If you can’t explain why the system made a decision, good luck defending it to a regulator, a customer, or even a court.”

A final thought

Get creative about how knowledge moves between people, advised Jen Clark, managing director at the Eisner Advisory Group. “Use generative tools like Claude, Copilot or ChatGPT to turn what one person knows into something transferable.” That includes guides, documentation, and short, recorded walkthroughs that move from one team member to the next. Shared repositories and shared projects work well too, Clark added, since knowledge gets captured as the work happens rather than reconstructed after the fact.