The weekend before go-live after a major ERP migration is rarely peaceful. For CIOs and IT leaders, it is often a period of nervous anticipation as they wait to see whether months of labor and millions of dollars will yield a seamless Monday morning or a disastrous halt to production.
According to industry statistics, the odds are not favorable. More than 70% of ERP migrations fail to meet their original objectives, and the culprit is rarely the software itself. Instead, failure often stems from a key misunderstanding of the unglamorous but essential discipline of testing. “From my perspective, it’s not so much that testing goes wrong, but that testing is more of an afterthought,” says Carl Andrews, CEO of testing and quality assurance (QA) provider Original Software.
In an interview with ERP Today, Andrews unpacks why modern cloud migrations are exposing the cracks in traditional QA strategies and why the industry’s rush toward AI might be premature if the foundations are not fixed first.
The Frequency Shift
Historically, organizations operating on-premises ERPs, such as legacy SAP ECC environments, lived by a comfortable, sedate cadence. During that time, organizations performed a significant upgrade—and the requisite testing—every few years. These were large and meticulously planned events that were executed then forgotten until the next cycle. However, Cloud ERP disrupted that cadence entirely.
“With today’s cloud ERP, that frequency of change is totally different,” Andrews notes. The shift to SaaS and cloud-native environments means updates happen continuously. If a business retains the old-world mentality of waiting until the last minute to test, they are walking into a trap. “By the time it gets round to testing, it’s too late and the project inevitably slips or, in some cases, worst,” he warns.
Andrews adds that this creates a dangerous friction. “Leaders under pressure to deliver on time and under budget are tempted to take shortcuts on QA. But in a modern ecosystem, a shortcut is often a dead end,” he says.
The Spaghetti Ecosystem
The complexity of the modern IT landscape compounds the risk, as an ERP system no longer sits in isolation. It is connected to upwards of 50 other critical business applications, from CRMs to legacy green-screen apps.
“One small change in one system or application might break a business process, and without that end-to-end test, you might not know until it’s too late,” Andrews explains.
This is where the human element becomes vital. Teams could fail to understand the gap between how the old system worked and how the new one must work. “When organizations cut corners on documenting these processes, they lose the institutional knowledge required to verify that the business can actually function on Day 1,” Andrews adds.
AI: Helper, Not Holy Grail
Inevitably, the conversation turns to AI. The industry hype suggests AI should be able to sweep in, write all the tests, and fix the bugs with a single click.
Andrews is quick to temper those expectations. “There is no ‘magic red button’ you can press to get all your testing done in one go,” he says. “AI doesn’t inherently know a company’s unique business processes or how its myriad systems integrate. That knowledge resides with the Subject Matter Experts (SMEs) who know the workflows inside and out.”
However, once that human foundation of documenting and understanding processes is laid, AI shifts from hype to a powerful accelerator. For example, Original Software uses AI to visually spot the difference in automated regression tests and identify changes between software versions that the human eye might miss.
“It’s a massive time-saver,” Andrews notes. “We’ve seen results with recent businesses where they multiplied the tests they can run by 10 once they move to automation.”
The QA Team’s Future
Does this mean the end of the human tester? Not yet, says Andrews, who views AI’s current role as augmentation rather than replacement.
“By offloading the laborious data crunching and regression checking to algorithms, human SMEs are free to return to their actual day jobs instead of being pulled into endless testing cycles,” Andrews explains. “In one case, this shift allowed one of our customers to return six full-time employees to their primary roles, saving the business in excess of £100,000 per year.” That is the equivalent of over $130,000.
Andrews believes the future of testing lies in speed, and the goal for companies like Original Software is to take organizations from low testing maturity to high maturity in weeks, not years. As ERP vendors continue to push businesses toward the cloud, the volume of change will only increase. “The organizations that survive will be the ones that stop treating testing as a checkbox at the end of a project and start treating it as a continuous, strategic discipline,” he concludes.
What This Means for ERP Insiders
Risk perception within organizations often needs recalibration. Many CIOs aim for zero risk, but project timelines suggest otherwise. Achieving a safe go-live requires treating testing as an essential activity rather than an afterthought or expendable budget line item. The frequency of change in Cloud ERP environments demands a continuous testing strategy. Effective testing should begin during the planning phase of an ERP migration, not just before deployment.
Documentation is the precursor to automation. Automation cannot proceed without a thorough understanding of existing processes. Prior to adopting advanced tools, Andrews recommends organizations having their SMEs manually test and document current business processes. This creates the dual benefit of building a library of training materials for new staff and providing the precise roadmap required to build reliable automated tests later.
Reliance on a “Magic Red Button” mindset can be misleading. AI functions as a force multiplier rather than a complete strategy. It performs well in areas such as parsing release notes, prioritizing changes, and completing visual regression tasks, but it still depends on human oversight to confirm correct system behavior. ERP Insiders must get the basics right, then use AI to scale capacity—not to fix a broken process.
