Finance AI Trust Gap Critical as Explainability Becomes Non-Negotiable

The debate over artificial intelligence in finance has shifted. It is no longer about whether AI can perform accounting tasks. It is about whether finance leaders can trust, explain and defend the outputs their AI systems produce.

That was the central message from Sage’s “From Black Box to Glass Box” panel at Sage Future 2026 in San Francisco, where Sage Chief Brand and Corporate Affairs Officer Amy Lawson, CTO Aaron Harris, tech journalist Kara Swisher and analysts from IDC and GrowCFO examined why AI adoption in finance hinges on transparency rather than raw capability.

“You won’t use AI if you don’t trust it,” Harris says. “You need to believe AI is being built competently and respectfully.”

The Explainability Imperative in Finance

Finance has always demanded precision. Now it demands proof.

According to IDC analyst Kevin Permenter, more than half of finance professionals report an improved view of AI after being shown glass box systems, those that surface reasoning and sources alongside every output.

However, Permenter pushed back against the framing of explainability as a standalone feature. “Explainability is not as important as functionality; it’s part of it,” he says. “If I can’t figure out why it did what it did, then it’s not functional. It’s a paperweight.”

That standard raises the bar for every vendor competing in the finance AI market. According to a 2026 survey cited by Ncontracts, 72% of financial institutions are only partially aware of which vendors are using AI, and not a single organization feels “extremely confident” managing AI-related risks. The implication for ERP buyers: A platform that wraps transparency around an existing model as an afterthought will not pass the audit test. Trust, Permenter says, “Equals revenue.”

Harris reinforced the technical stakes. Finance AI, he argues, must be deterministic and invariant in the same way sound accounting principles are. “Finance and programming are elegant and deterministic,” he says. “They must always hold true to be valid.”

Sage’s response is a system architecture built on three layers: an integrated UI, an agent operating system and what Harris calls an “arbiter,” which is a dedicated layer that sits between the user and the AI to detect hallucinated content, jailbreak attempts, prompt injection and toxic outputs before they surface in financial workflows.

How Agentic AI Changes the Finance Workday

For finance professionals and ERP practitioners, the arrival of agentic AI is not a future scenario. It is a present operational reality reshaping the daily workflow in ways that demand immediate evaluation.

Sage AI Labs processed 40 million predictions in 2025. That figure has climbed to 400 million in 2026, according to Harris. The company’s Sage Intacct platform already deploys AI for overdue invoice identification, variance analysis and real-time reconciliation support.

Harris used his own hands-on experiment to illustrate where the technology stands and where it breaks. After building a personal accounting agent that consumed 220 million tokens in a single week through Anthropic’s API, he found off-the-shelf models are insufficient for enterprise accounting tasks. The challenge, he stressed, is not accuracy alone. It is the model’s capacity for competent, safe judgment within the specific semantics of finance. Harris noted finance has its own linguistic context, and the arbiter layer within Sage’s architecture is designed to translate that context before AI outputs reach decision-makers.

GrowCFO founder Michael O’Reilly framed the human dimension. “Adoption of AI is going to be dictated by people’s willingness,” he says. “Creating an environment where people see it as of service to them” is the foundational work no vendor can shortcut.

What Finance, ERP Leaders Must Evaluate Now

The panel’s message for technology and finance leaders was direct: vendor selection criteria must evolve beyond feature sets.

O’Reilly noted how CFOs must be able to explain AI-driven outputs. That accountability cannot be delegated. “If you put the numbers in the spreadsheet, they’re your numbers. AI is the same thing. It’s not a get-out-of-jail free card.”

ERP buyers should therefore evaluate whether AI trust is architected into the core platform or applied as a surface-layer control.

Permenter identified the reputational dimension as the most underappreciated risk in the conversation. “The output wasn’t right and it damaged their personal brand,” he said, noting that AI exposure goes deeper than inaccurate forecasts. It affects individual credibility. Mature ERP platforms must enforce least-privilege access for agents, separate reasoning from execution and build immutable audit logs of every agentic decision pathway.

Swisher offered the broadest lens. Pointing to the concentrated power dynamics shaping AI development, she warned a small number of companies are positioned to determine how the technology evolves. “It’s not a bottom-up revolution. It’s a top-down one.”

For enterprise buyers, that concentration makes vendor due diligence a governance obligation.

Harris closed with a practical litmus test: Productivity in AI-assisted finance equals confidence. Systems that can show their work, trace every recommendation to source data and hold up under audit scrutiny are the only ones that belong in the financial close.