Is Explainable AI Explainable Enough Yet?
For technology leaders, explainable AI is not a model feature to admire after deployment; it is a governance control to decide before scaling. The practical issue is less whether a system can produce a plausible explanation and more whether the organization can rely on that explanation for audit, escalation, and user trust. That shifts XAI from an experimentation topic into an operating-model question: who approves the level of explainability required for each use case, and who owns the residual risk when the model cannot fully justify itself?
The trade-off is real. More transparency can expose data lineage, business logic, or model limitations that teams would prefer to keep hidden, including intellectual property or workflow fragility. But weak explainability creates a different cost: slower adoption, harder incident review, and greater exposure in regulated or high-stakes decisions. Leaders should therefore separate use cases into tiers, with stricter requirements for customer-facing, financial, safety-related, or employment decisions than for low-risk internal analytics.
The next management question is not โCan we explain the model?โ but โWhat evidence would satisfy the decision?โ That implies asking vendors for lineage, test artefacts, red-team results, versioning discipline, and clear statements of what their explanations do and do not prove. It also means defining whether explanation is needed for end users, auditors, regulators, or internal model risk teams, because each audience needs a different depth and format.
For portfolio and governance leaders, XAI should be funded as part of AI controls, not as optional documentation after go-live. Useful metrics include traceability coverage, percentage of high-risk models with documented human override paths, time to investigate an adverse output, and the share of AI initiatives blocked by insufficient evidence. If those numbers are not tracked, explainability remains an aspiration rather than an operating standard.
As artificial intelligence usage continues to increase, thereโs a problem lurking in the background growing larger by the day: Itโs the ability of AI to explain itself so itโs clear what led to an outcome. Deep learning opacity has driven demands for explainable AI (XAI). It doesnโt help that researchers canโt explain how the models they build work and why. The increasing complexity of deep learning and large language models is making XAI even more difficult to achieve, and a higher priority.
โWe have been able to achieve explainability through techniques like LIME, SHAP, and attention visualization, but that only provides a partial understanding of complex AI systems,โ says Aditi Godbole, senior data scientist at enterprise application software provider SAP, in an email interview. โEven though we have mechanisms to explain certain aspects of many deep learning modelsโ decision-making, providing comprehensive and intuitive explanations is still challenging.โ
Several developments are needed to improve the state of explainable AI, such as the development of more advanced techniques and establishment of industry standards to ensure consistency and comparability, she says. Also necessary is interdisciplinary collaboration among fields like computer science, cognitive psychology, and communication for more effective explanations, as well as well-defined regulatory frameworks with clear guidelines and requirements.
โMore often than not, the higher the accuracy provided by an AI model, the more complex and less explainable it becomes, which makes developing explainable AI models challenging,โ says Godbole. โThe premise of these AI systems is that they can work with high-dimensional data and build non-linear relationships that are beyond human capabilities. This allows them to identify patterns at a large scale and provide higher accuracy. However, it becomes difficult to explain this non-linearity and provide simple, intuitive explanations in understandable terms.โ
Other challenges are providing explanations that are both comprehensive and easily understandable and the fact that businesses hesitate to explain their systems fully for fear of divulging intellectual property (IP) and losing their competitive advantage.
โAs we make progress towards more sophisticated AI systems, we may face greater challenges in explaining their decision-making processes. For autonomous systems, providing real-time explainability for critical decisions could be technically difficult, even though it will be highly necessary,โ says Godbole. โWhen AI is used in sensitive areas, it will become increasingly important to explain decisions that have significant ethical implications, but this will also be challenging. As the complexity of systems using multiple interacting AI agents increases, explaining the resulting behavior could become extremely complex.โ
Explainable AI is rapidly evolving as more businesses express the need for it, according to Maitreya Natu, chief data scientist at SaaS-based autonomous enterprise software provider for IT and business operations Digitate.
โGoing forward, it wonโt just be a wise choice, but a necessity. Industries would have to look for creative ways to find a golden middle between the transparency of the white-box solutions and the accuracy of the black-box solutions,โ says Natu in an email interview. โFormal design methods will evolve to bring explainability by design from the very early stages of developing AI solutions, and elaborate frameworks will get set to evaluate the explainability levels of AI solutions and define its tangible benefits with regard to trust, adoption, regulatory compliance, etc. Explainable AI will set the foundation for new ways of augmented intelligence through human-machine collaboration.โ
XAIโs key drivers include the issues black-box solutions introduce, such as overfitting or spurious correlations, difficulty in maintaining performance, unexpected behavior, potential unfairness and more. Explainability will prevent these issues and allow human experts to more effectively train, provide feedback to, and course-correct an AI model, Natu says. Explainable AI can open-up new metaphors of augmented intelligence where AI models truly benefit from the intuition of an expert and the experience of a seasoned practitioner.
When designing complex analytics pipelines, careful decisions should be made to leave enough hooks for backward traceability and explainability. In other words, consider an application as a complex flow-chart, and if hooks are provided at every decision point of this flow-chart, then trace from the leaf to root can offer a fair degree of explainability behind the generated output.
โBlack-box algorithms are inevitable for many scenarios that are inherently complex such as fraud detection or complex event predictions. But these algorithms can be complemented with post-hoc explainability methods that enable interpretability by analyzing the response function of a machine learning model,โ says Natu. โThere will be cases where comprehensive reasoning cannot be inferred but even in such cases, techniques can be used to infer the contributions of different features in the output of an ML model.โ
Language Challenges
Humans and machines speak different languages, which is why analytics vendors have been transforming human language queries into SQL, but the transformation in reverse can be problematic. For example, every SQL statement has a from clause that states where the data came from, which goes into the query result. According to Mike Flaxman, VP of product and spatial data science practice lead at advanced analytics company HEAVY.AI, itโs easier to identify where the data came from, the harder part is the transformations. โWhen you select sales from a table, itโs self-evident when you see the SQL. The number came from that column, and that is the semantic answer,โ says Flaxman. โThe problem is that corporations donโt have one sales table, they have 400 and three different definitions of sales. Those are real dilemmas in the semantics of how that information was computed, as well as who did it, when and where, which brings up the issue of providence. SQL is explicit, but the answer might be a really complicated formula.โ For example, a heat reading from a thermometer provides a simple answer. However, heat indexes are complicated because they have as many exceptions as rules. Moreover, there are many heat indexes, so if one does not understand which heat index number is used, one canโt really understand what the heat index means. โYou can document the top-level thing and make that explicable, but then one step below that is where did that column even come from and what was the formula or science that went into it?โ says Flaxman. โSo, I think SQL is a partial answer, and then explain the results in [human language].โData Challenges
Data powers AI. If the data quality is bad, the outcome of an analytical result, recommendation or answer will likely be erroneous. Data governance is also important to ensure proper data usage. Vyas Sekar, Tan Familyโฏprofessorโฏof electrical and computer engineering in the ECE department at Carnegie Mellon University and chief scientist at data analytics company Conviva, believes the advent of large foundation and multimodal models has had a negative effect on XAI efforts. He also underscores the importance of data. โWe don’t know what goes in and can’t understand what comes out or why it came out!โ says Sekar in an email interview. โWe need to understand the training data that went into building the model as well as the algorithm/model being used to respond and the kinds of questions we are asking. As we deal with larger datasets and more complex models and multimodal interactions, there are going to be challenges.โ
Vyas Sekar, Carnegie Mellon University, Conviva
The increasing deployment of large language models (LLM) over the past two years has made it increasingly challenging to maintain auditable systems. To address the issue, Panagiotis Angelopoulos, chief technology officer at motivation AI company Persado says itโs crucial to focus on the auditability of the training data itself — ensuring that the training corpus is free from biases to enable fair and reliable AI systems.
โMoreover, rigorous testing, including red teaming, is vital for these complex models. This involves subjecting the AI system to various extreme scenarios post-training to observe how it reacts and what decisions it makes under different conditions,โ says Angelopoulos. โSuch testing helps in evaluating the robustness and decision-making processes of AI systems, ensuring they operate reliably and ethically in real-world situations.”
Visualizations Could Help
HEAVY.AI has been experimenting with visualizations that advance explainability, but itโs a challenging endeavor. โWe can agree to the SQL answer to a question in a lot of cases, but we canโt agree on a good map or visualization. There are no benchmarks to my knowledge, and few standards in terms of explaining data and analytical results,โ says HEAVY.AIโs Flaxman. โThe graphic becomes part of the next question, and I believe the models [can do] that. Itโs just that we have little theory to go on, so weโre trying to piece it together as we go, but I think that conversational ability is quite important.โ Diby Malakar, VP of product management at data intelligence company Alation, believes a critical step forward involves creating a clearer visual map of the data lineage, including tracking the technical, system and business metadata as data flows across systems and into AI models. โMapping is essential for understanding and explaining the AIโs decision-making process, which is especially important in high-stakes environments where accountability and trust are crucial,โ says Malakar in an email interview. โThe future of explainable AI lies in powerful AI governance frameworks that include data lineage as a fundamental component. Ensuring transparency across the entire lifecycle of data — from its origin to its role in AI decision-making — is key to enhancing explainability. This will likely involve the development of new tools and methodologies that provide a visual representation of how data is used across systems, making it easier for data scientists, end users, business leaders, and public stakeholders to understand AI decisions.โBottom Line
XAI is becoming increasingly necessary as organizations rely on AI more than they have in the past. While not all AI techniques are considered โblack box,โ deep learning models and LLMs continue to become more complex and therefore more difficult to explain. Many things need to happen to realize the vision of XAI, and thereโs no simple or immediate fix to all the issues. Advancing it requires organizations to prioritize it, experts who can advance it, sound regulation and unflinching market demand.Enjoyed this article? Sign up for our newsletter to receive regular insights and stay connected.