Dated but still feted: The Big Data universe

Big Data is big. You just won’t believe how vastly, hugely, mind-bogglingly big it is.

Big Data is big. To paraphrase the late, great Douglas Adams on space: you just won’t believe how vastly, hugely, mind-bogglingly big it is. And that’s nothing compared to the hype around it. However, like all major technology shifts, paradigms and platforms, innovation and evolution generally consign most elements of IT to legacy status after a while. So, has something changed?

In the era of web-scale everything and the massively parallel processing backbones that span the cloud database era, has the Big Data furor subsided? As the use of multi-cloud computing fabrics now spans every industry vertical and we look forward to a future of data pipelines fueled by AI-enriched filtering, standardization and integration, can we now say that Big Data is dead and that ERP systems spanning this space now represent a bigger entity in, and of, themselves?

Whatever amplifying term we use to describe really large, weighty, voluminous or big quantities of data, the reality is that information will continue to spiral and grow, so the whole notion of Big Data was probably just endemic of the 1990s and our proclivity for snappy nametags in the wake of the dot com bubble.

Beyond the buzz

Infor-Phil-Lewis_BW_WEB-240x300.png" alt="" width="240" height="300" />The next frontier is all about harnessing and exploiting data  – connecting it, analyzing it, injecting it with intelligence, mining it and squeezing every drop of value  – Phil Lewis, Infor

Senior vice president for solution consulting (EMEA and APJ) at Infor, Phil Lewis broadly agrees and says that of course Big Data isn’t actually dead, it just isn’t a buzzword anymore.

“That reality is largely born out of the fact that working with vast quantities of data has become the norm for most companies of any reasonable size. The next frontier is all about harnessing and exploiting data – connecting it, analyzing it, injecting it with intelligence, mining it and squeezing every drop of value it can offer the business,” says Lewis.

With technology organizations in the ERP space, like Infor, now working to apply huge amounts of process intelligence, new strains of RPA and of course the now-ubiquitous application of generative AI to their platform, we can see why the conversation might have elevated beyond “look at all that data” today. It’s clearly time to accept the size of the known universe and work out how to navigate it.

“Faster, more accurate, intelligence-based and increasingly automated decisions will accelerate business performance at the coalface of software application development and data analytics today,” clarifies Infor’s Lewis.

“We should now expect ambitious organizations to thrive and continuously improve via the development and application of hyper-automation tools, processes and functions; all of which will drive productivity, new data-driven processes, AI-driven insights, revenue streams and services unthought of a few years ago. In modern ERP systems and throughout the entire fabric of enterprise software, data-driven decision-making will become instantaneous and will accelerate the way work gets done. When exactly the right information is injected into both the business process and the decision-making process at exactly the right moment, work happens in a more fluid, composed and efficient manner. We all live in an intelligence-driven world and Big Data is still right at the heart of it. Even if we drop the term itself, there’s still a whole lot of bigness out there.”

Dated, but still feted

Big Data was never full equipped to handle the quality of the data. That’s where the combination of ERP and AI comes in – Stu Johnson, Rootstock

It feels like Big Data might be a slightly muted data label by now, but the rationale behind the term being coined in the first place is still very much in place. As we continue to dive into the morass of new information streams, many of which will end up being channeled towards voluminous data lake repositories, we can reasonably expect Big Data-like technologies to pervade for a long time.

“Big Data has indeed become a dated term; it was first used to describe unstructured data and then eventually evolved to encompass much of the work involved with analytics as organizations grappled with the volume, velocity and variety of data being ingested. Unfortunately, Big Data was never fully equipped to handle the fourth V: veracity, which refers to the quality of the data. That’s where the combination of ERP and AI comes in,” says Stu Johnson, VP of product marketing at cloud ERP company Rootstock.

Because we know that almost all of a reasonably sized organization’s data is stored in its ERP system, Johnson says that businesses are increasingly looking to take advantage of AI. They need cloud-based ERP systems that can collect the various Vs of data while also applying it to inform better business decisions. The confluence point between AI and ERP at the edge of the Big Data chasm is what will make the difference.

“Today we can say that AI tools, including large language models (LLMs), are aided by the contextualization that ERP provides. As a result, Big Data analysis is democratized; any individual is able to interpret data in various contexts and roles,” clarifies Johnson. “The most powerful ERP solutions are capable of ingesting streams of data signals coming from all directions and interpreting those signals against a vast store of historical data. The next frontier will occur when LLMs – fueled by the data these ERP systems provide – are combined with predictive analytics, unleashing a whole new wave of signals that will drive intelligent decisions.”

It feels like Big Data is still on a journey then. One where it will be inevitably swept up into the vortex formed by new AI models, the rise of real-time data analytics, the creation of new smart application services and the trajectory that all these technologies take as they pass over the existing topography of ERP deployments. So, basically, something of an evolution needs to happen.

Bare metal, laid bare

‘Traditional’ approaches to Big Data are becoming an impediment to business success – James Sturrock, Nutanix

“It’s important to realize that what we might consider to be ‘traditional’ approaches to Big Data (use of dedicated data silos, often located on bare metal cloud servers) are becoming an impediment to business success due to long time-to-value, complex lifecycle management and poor performance,” says James Sturrock, director of systems engineering for UK and Ireland at Nutanix.

Clearly keen to tell us that we need to get out of the bare metal era and invest in hyperconverged infrastructure (HCI) solutions for Big Data analytics, Sturrock makes the point that we’re moving rapidly to an always-on continuous computing fabric where many data resources lose value comparatively quickly. As a direct consequence, the way we approach the Input/Output (I/O) capabilities of the systems we use to manage Big Data – especially in the gargantuan realms of an enterprise ERP landscape – really starts to matter now.

“It’s imperative that organizations embrace Big Data analytics solutions capable of handling what we like to call the I/O blender effect,” explains Nutanix’s Sturrock. “Because the firehose stream of I/O requests coming from demanding Big Data jobs results in a random mix of I/O that can defeat storage ‘read-ahead’ algorithms, making it difficult for a central storage system to operate efficiently. We add to the challenge when we realize that there are mixed I/O needs, i.e. some Big Data jobs require high streaming performance while others create random I/O and most storage arrays don’t handle both types equally well.”

By consolidating all data services for an ERP deployment on the same platform alongside the core compute function, Sturrock suggests that an organization can simplify provisioning and management while increasing resource utilization. Combine these techniques with a real understanding of data locality and intelligent data tiering (to rank mission criticality across a whole spectrum of levels) and it is argued that Big Data can be handled in environments that enjoy optimum performance without constant tuning.

Quality data, big or otherwise

There is no way to layer an algorithm over the top of a bunch of ERP data and just expect it to deliver meaningful insights – Claus Jepsen, Unit4

The ERP Big Data story is no open-and-shut case. If anything, both disciplines have come under scrutiny, fire and abuse for being clunky, outdated and anachronistic at times. Chief product and technology officer at Unit4, Claus Jepsen, is very much of the mind that we should exert a little more balance when we consider these two technology streams concurrently.

“I don’t think the use, implementation or management of Big Data is the central issue and, likewise, I don’t think ERP is the sole enterprise software solution that we should consider in the mix,” says Jepsen. “It was only a few years ago that everyone was saying ERP is dead – and I don’t think today we’re in a position to expect an ERP system to manage all an organization’s data management and analysis requirements. The key issue is whether a business has access to the right data to make effective decisions.”

Jepsen’s comment is reflective and illustrative of the wider sentiment here; people don’t want to talk about Big Data anymore unless it’s business-relevant accurate data that can track a clearly defined path to a positive business outcome. He reminds us that the value of a good ERP system is that the data within it is highly structured, sanitized and contained in order to drive the information systems in an organization. When all these factors are in line, he says a business can then deliver deeper insights and apply ML algorithms to its information streams, because it has confidence in the structure and semantics of its dataset, big, medium, small or otherwise.

“However, the reality is that we operate in a world of multiple data sources and types, which has created so-called data lakes that are a more convenient way to store and analyze information. Big Data offers the ability to accumulate data from those types of subsystems across multiple domains, to be analyzed alongside any data in the ERP system to get a complete picture of a business,” explains Jepsen. “But this cannot be done in the ERP system itself, as it does not necessarily have sight of the data in these other subsystems.”

Because of this unavoidable truth, Jepsen advises IT practitioners to realize the critical need for any ERP system to have the requisite level of interoperability to share data with other applications. An IT shop can standardize the Big Data analytics solutions in use, but ultimately, if the business is not analyzing the right data then the whole exercise is pointless. Data quality and integrity are critical foundations to ensuring this whole process works, but there is a realization that accumulating vast amounts of Big Data is useless if you don’t know what to do with it.

“We find companies are investing more time in the data management planning phase. This is because they want to know the questions they need to ask before accumulating data for analysis from across their IT systems,” says Unit4’s Jepsen, referencing the real-world projects his firm interacts with on a daily basis. “An organization must work out what it is trying to understand, what the data represents and what its attributes and semantics are. There is no way to layer an algorithm over the top of a bunch of ERP data (or any other source) and just expect it to deliver meaningful insights. This is why we are seeing more focus on developing specific use cases for data analytics today – and it’s a trend that must surely continue.”

A fuel for ERP systems

Complex blending and deciphering processes, with an ever-increasing amount of data, still mean data is essentially big – Lee An Schommer, insightsoftware

Agreeing that ERP had been erroneously consigned to the “where are they now” file some years back, chief product officer at insightsoftware, Lee An Schommer, suggests that the current ERP renaissance still embodies a degree of disconnect at the data level. Reminding us that ERP systems are at the heart of business data locales, she says that users themselves still struggle with information tools and feeds at certain levels.

“These ERP users continue to struggle with enterprise data streams, needing to decipher what insights are ERP-specific (versus those that come from different third-party solutions) in order to effectively help the C-suite understand the health of the business and where to act,” asserts Schommer.

In fact, many mid-market and enterprise companies have more than one ERP by design, in order to address corporate global needs versus those of certain world regions.

This requires complex blending and deciphering processes with an ever-increasing amount of data, which still means data is essentially big. It begs the question, who will make sense of all this data?

As commentators in this space have underlined, while AI and ML will continue to learn and provide business insights, any given model is only as good as the data it can feed upon. It will be up to the business itself to provide clean, relevant data to produce the best outputs. AI models may then also assess synthesized data, point out anomalies and suggest corrections. For Schommer’s money, the secret sauce is having people trained to run different models based on unique business goals and then convince senior leaders that it’s time to adjust the plan.

“This is often where data exercises fall apart,” she says. “The data is collected and then AI is interjected and applied, but afterwards… we tend to see management go back to business as usual because they are not quite ready to trust the technology and its output. As AI evolves and additional filters create quicker connectivity to better-organized data, team leaders must determine how they can validate and then take advantage of the resulting insights. With this in mind, as AI technology and ERP systems continue to come together, it can only be expected that these systems continue to grow and phase out Big Data entirely.”

The modern data firehose

Cloud computing allows companies today to streamline data processing in a way that was almost unthinkable a decade ago – Phil Le-Brun, AWS

It seems clear that the Big Data evolution issue is perhaps nowhere more clearly illustrated than in the ERP space. After all, ERP systems are a) inherently big, b) full of data, c) characterized by the volume of analytics they are able to process and d) typified by their proximity to systems of record and transaction, both of which continuously feed the modern data firehose. Who better to drive home the question of how ERP should dovetail with a big, medium or small data strategy than AWS?

“Organizations today create and store petabytes – or even exabytes – of data from a vast variety of sources, from online transactions and customer interactions through to connected devices and product inventory,” says Phil Le-Brun, director of enterprise strategy at AWS. “Cloud computing allows companies today to streamline data processing in a way that was almost unthinkable a decade ago – and to easily adapt and evolve as an organization’s needs change. This is important because building a data strategy that evolves as the business expands and changes is key to success.”

Le-Brun says that he understands just how big that Big Data mountain might feel to different firms and that every organization’s data needs are unique, which is why it’s important to have access to a variety of tools so they can leverage the right ones depending on their specific needs. “A modern business will require the data spread across their organization to be integrated and connected, so they can act on their data no matter where it lives. Ultimately, they must eliminate structural and departmental data silos to be sure that all the right people can access data at the right time and with the right controls,” concludes AWS’s Le-Brun.

Whether we drop the Big Data term tomorrow or hang on to it by some romantic attachment, its time on Earth has been worthwhile because it has taught us that we can quickly find ourselves with too much to eat at the information buffet or smörgåsbord. A progressive business today needs an IT strategy, a data strategy (which encompasses analytics) and of course now an AI strategy, all of which must exist in symbiotic unity. So long and thanks for all the bits.