The IoT sweet spot – how much data is too much data?

The Internet of Things (IoT) is no longer news. The Gartners of the world have made their predictions about the billions of connected devices and sensors that are on their way to run our world – and most of those predictions have come true. 

However, it is not always clear whether those IoT buildouts are delivering value now – or if they ever will. It’s a problem of too much of a good thing. No one questions that data is good and more data is generally even better. But with the cost and complexity of creating huge IoT networks, there has to be a substantial payback to make it worthwhile. Furthermore, there is the question of whether every last bit of data is really going to be worth collecting and analysing. Maybe it’s a case of more garbage in, produces even more garbage out.

To be sure, IoT is producing data like nothing else in history. A recent article, sourcing the information to the International Data Corporation, says total data generated by 2025 is expected to be around 175 zettabytes (ZB) with about 80 ZB attributable just to IoT devices. 

And there are certainly many examples of IoT success – preventative maintenance for capital equipment being one of the most frequently cited. But not every IoT project produces gold.

In some cases, IoT is introduced as a solution in search of a problem. “By focussing on collecting as much data as possible and then analysing it, many companies end up drowning in their ‘data lake’ as it is difficult to extract value if you do not know what you are looking for,” says Mats Samuelsson, the chief technology officer for Triotos, a provider of IoT reference solutions that help companies build IoT capabilities. 

“A better way is to start by focussing on events of interest and then collect data associated with them. This focusses the ‘data analytics’ task on meaningful results instead of wild goose chases,” Samuelsson adds.

“Start by focussing on events of interest and then collect data associated with them”, Mats Samuelsson, Triotos

Mats Samuelsson


Evolving IoT

So, where to start? Consider how we got here, says Greg Schulz. He is a senior advisory analyst at StorageIO and he suggests looking at what IoT has evolved from and where it is going. He says some IoT is built on the classic IoT-type activities that have been around for decades, such as SCADA systems. These systems integrate control activities and information, and were traditionally comparatively limited in scope and scale. Then there is the more modern IoT which focusses on collecting and controlling, as well as gaining insights. Modern IoT also offers tremendous scalability. “The problem is that the modern technology has become so much more affordable and ubiquitous that there is a tendency to go looking for things to apply it to,” says Schulz.

“In other words, we have gone from unlocking the potential of processes, many of which were being done manually for years, to now looking for totally new ways to apply the technology,” he says.

And, notes Schulz, this begs the question: what are you going to do with all that data? 

It’s not always the first question asked when companies are rushing headlong towards something that is sometimes pitched as a cure-all. “Are you processing data and leveraging it successfully?” Schulz continues.

“Are you doing it for the sake of doing it or are you leveraging it to actually unlock some better potential,” he asks. Too often, in a quest to apply IoT to achieve cost savings and betterment, companies may actually be increasing backend costs, he warns. “Is that really saving money or is it just a wash?” These are the questions that Schulz says must be asked. 

“Are you doing it for the sake of doing it or are you leveraging it to actually unlock some better potential”, Greg Schulz, Storageio

Greg Schulz


Of course, no strategic initiative or investment should be judged on its ability to provide immediate return on investment. “If it is just a ‘wash’ financially, it may still be worth doing if it enables something that couldn’t be done before,” he says. For example, using IoT may get you to results and insights faster. Therefore, from an opportunity cost standpoint, it could be worthwhile even if there is no direct or obvious savings. It needs to be looked at from a broader perspective or longer timescale.

But another aspect of IoT that can go wrong has nothing to do with the soundness of the concept and everything to do with the real world. Rich Karpinski, senior research analyst, 451 Research, part of S&P Global Market Intelligence, says while the desire to leverage IoT data to help transform an organisation is valuable, it must be accompanied by the ability to deliver: “There is often a tipping point where projects move to production scale and things get real and then get challenging.” He says that’s because the number of endpoints and the amount of data generated by IoT deployments can be massive, “so you need a plan”.  

As projects grow, that is where the associated costs appear, he says. Those are the costs related to moving, storing, and analysing data. “You really want to make sure you are analysing the right data for the right reasons and in the right location within the right cost structure,” he explains.

Specifically, he says, organisations need a strategy for determining which data is most important and an IT approach to help manage it all. “That includes sensors and machine endpoints, both for how much data they send as well as being able to keep them secure,” he says.

The edge is typically the key. In other words, addressing all the issues around data generation typically needs to be engineered into the edge, close to the sources of data.

“We often hear about edge computing in the context of IoT because it can support characteristics such as high performance and low latency, but its ability to help to manage the amount of data that needs to be digested and the cost of that data is just as important,” says Karpinski. The key job of edge computing, he says, is aggregating and filtering data and only keeping what you really need. “That is probably the biggest thing you need to get right,” he adds.


Plan for success

If you want to avoid the pain, you need to have a plan in place from the start. “You need the right people internally and at your vendors, and you need to start thinking about those scaling issues from the start,” says Karpinski.

Fortunately, you don’t have to start entirely tabula rasa. According to Karpinski, there are definitely several broad categories of deployment types, typically related to how they handle edge computing issues. “You can also characterise applications by the amount of data they handle and the type of analysis that’s needed,” he said.

The scale of the company and its technical readiness also need to weigh into decisions, because these will determine much about how successfully ambitious plans can be pursued.

In some instances, cloud can provide a short cut. “You can do some IoT with SaaS and by depending on vendors and systems integrators, even if your organisation is not tech savvy, you can still get things done,” he said. However, he notes, large organisations that want to make a big difference in their operations have found “you have to take the architecture challenges seriously from the start because it will be the biggest challenge you face,” says Karpinski.

“You have to take the architecture challenges seriously from the start because it will be the biggest challenge you face”, Rich Karpinski, 451 Research

Rich Karpinski


And, at any size firm, there will be orchestration issues, especially when cloud is involved, and that is where IT should focus. And the business or operations side of the organisation should be recognising and describing what they need so that IT will know what it should be building,” he says.


Keeping the right perspective on IoT

So, the takeaway is the need to get past the dazzle and buzzwords and make sure you are clearly understanding how IoT can benefit you – and then build accordingly. Schulz says companies naturally want to be ‘first and best’ and can be tempted to see IoT investments as a way to fulfill that goal. But, even if they can justify the investment, they also need to be aware of the risks, particularly with cyber security. “Can you actually accomplish your goals in a safe and secure manner?” he asks. 

You need to keep that perspective as you scale up from small to large and increase the complexity of what you are doing. How are your tools helping with that and with security? In your quest to save money, are you opening yourself to new costs through cyberthreats and ransomware?

Ultimately, you need to make sure your IoT initiative is either solving a business problem or creating a genuine business opportunity, says Schulz.