What Is The Value Of A Big Data Project?

According to software vendors executing the big data projects, the answer is clear: More data means more options. Then add a bit of machine learning (ML) for good measure to get told what to do, and the revenue will thrive.

This is not really feasible. Therefore, before starting a big data project, a checklist might come in handy.

Gain actionable insights

Make sure that the insights gained through machine learning are actionable. Gaining insights is always good, but it is even better if you can act on this new knowledge.

Example: Analyzing the data of a local ice cream parlor might show that sunny weather doubles the revenue. The ML algorithm would then suggest the sun should shine all year. True, that would help, but it is hard to achieve.

Take actions that make sense

A shopping basket analysis shows which products are sold together. What to do with that information?

Companies could place the two products in opposite corners of the shop, so customers walk through all areas and will find other products to buy in addition. Or they could place both products next to each other so each boosts the sales of the other. Or how about discounting one product to gain more customers?

As all actions have unknown side effects, companies have to decide for themselves which action makes sense to take in their case.

Clean big data of needless noise

Some conclusions are obvious. If the competition offers the same product at half the price and while enjoying higher brand awareness, you will have a harder time selling this product.

In other words, there are three variables in this example – price, brand, revenue – with a high degree of dependency. No fancy IT system is needed for that kind of analysis. However, it gets more complicated the more variables are involved.

The main advantage of machine learning is that it can deal with more variables and still find meaningful interdependencies and relationships, e.g. between product quality, marketing budget, availability, sales channels and many more – but only if there is a pattern.

Way too often, the noise in the data superimposes any potential pattern. As a result, the ML logic does not find any pattern or worse, believes to have found one.

Make sure the data in your big data project is cleaned of needless noise as much as possible to ensure the best possible results.

Make sure you get the full picture

An interesting fact was uncovered: With a high probability, the motor of a conveyor belt will break at 8 AM.

A detailed analysis would show that in the morning, starting for the first time that day, the motor is still cold, and the torque is the highest. The result is a failure in the motor’s main bearing. The solution would be to start the belt more slowly.

But what if the motor neither has a temperature sensor nor a torque sensor? How would any ML program find the root cause if it is not included in the data?

Make sure you get the full picture on all relevant machine parts, equipment and so on to enable machine learning programs to identify the root cause of problems – otherwise, why even employ ML, right?

Do not expect machine learning to replace human experts

A common situation is to spend a lot of money on an IT solution to find weak spots – which is fine, if it actually uncovers new insights.

For example, let’s imagine a capacitor on a PCB that is frequently broken. The ML analysis shows it clearly, hence its suggestion would be to replace it with another capacitor of higher quality from now on. Well done, ML – only problem: the repair teams might be upset because they suggested the same thing years ago and nobody was listening.

The best ML algorithm in the world are skilled people. They have the data, the understanding, intuition and are available already. ML cannot completely replace the know-how of human experts, so don’t expect it to.

​Use ML for optimization, but don’t expect it to think outside the box

ML algorithms recognize patterns well, which can be useful for optimization. But thinking outside the box is beyond their capabilities as they would need an understanding of the problem for that.

Example: A hotel group asks their customers what their biggest pain point was during the stay. The answer is the check-in process. How would one solve that problem? ML would probably recommend more people, better training or better software. These are all valid approaches.

But what the hotel decides to do instead is to get rid of the check-in counter altogether. Each guest is greeted at the door, hands over the credit card and while walking them to their room, hotel staff take care of the check-in.

Machine learning would have optimized the existing process, human decision makers got rid of it and established a new, better suited one. If you want outside-of-the-box thinking, ML is not the way to go.

Conclusion

If there are so many potential problems, the best approach to any big data project is to keep the costs down. With more practical experience, the value will be seen quickly and quite often in areas not anticipated initially. A project with little costs is also a quicker project with an higher return on investment.