Big Data is giving organizations the ability to acquire a lot of actionable insights. But with so much data in hand, some companies are overwhelmed. Experts share six tips to stop the Big Data overload.
It is no secret that the amount of data is growing exponentially. According to EMC, the digital universe is going to grow to more than 40,000 exabytes by 2020. The number alone is intimidating. Making the most of data can be even more daunting.
Click here to read the full article.
Throughout Monday’s plenary sessions, a key message from panelists was the need for directors to blend quantitative—harder—data with qualitative—softer—complements. For example, a focus on shareholder return but with a stakeholder view, the intersection of situational awareness and the ability to use intuition, or the need to harness qualitative data with application of context. In an interview with Jeffrey M. Cunningham, managing director and senior advisor of the National Association of Corporate Directors, RichRelevance Co-founder and CEO David Selinger shared how directors can bring big data into the boardroom.
Click here to read the full article.
In this volume, we conclude with Privacy and Security.
Privacy and Security
For our final examples I want to dig into the notions of privacy and security in big data settings. These are and always will be critical concerns.
We begin with financial information and e-commerce. In the early days of Amazon, there were a significant number of customers who were very concerned about the security implications of entering their credit card numbers online. The specific concerns varied, but they almost always involved the possibility of a gang of nefarious hackers gaining access to credit card numbers and using them to make fraudulent purchases.
By Doug Bryan, Data Scientist, RichRelevance.
The oldest and largest marketing data aggregator, Acxiom, recently opened up its database to let individuals see and edit the data that the company has collected about them. Data scientists and marketers have been viewing the data and, often, chuckling at its inaccuracy. Acxiom admits that up to 30% of its data is wrong, and yet it’s the industry leader. How can Acxiom make so much money with data that’s only 70% accuracy? Simple: Most marketers’ data is worse, and in the land of the blind the one-eyed man is king.
In this volume, we discuss Data Mining and The Birthday Paradox.
Data Mining and The Birthday Paradox
We’ve all heard of the Birthday Paradox. Put 23 randomly chosen people in a room and there is a 50% chance that two or more have the same birthday. Put 57 people in the room, and the chance is 99%. Do those people have anything more in common because they were born on the same day of the year? Astrologers will say yes, but most scientists would say there is no evidence to support that claim.
What does this have to do with big data? The answer is that generalizations of the math behind the birthday paradox tell us that we will—not just that we can but that with near 100% certainty we will—draw meaningless conclusions if we just look at enough variables. In fact, we can show that if we generate a large number of streams of completely random data, some of them will look like others.
The problem is, we can easily forget this when we look at big data sets with lots and lots of variables. These are the kind of things we see in what is called data exhaust. Data exhaust is the vast stream of data gathered and logged by digital devices ranging from mobile phones to engine sensors in cars to video cameras in public spaces to instruments on particle accelerators.
Look at a lot of this data, and you will find spurious correlations. This is what Principle 3 is all about. Statisticians have known about Principle 3 for decades, and have techniques for trying to deal with it. The best technique, however, is and always has been a controlled scientific experiment, as Principle 2 advocates.
In this volume, we discuss Product Recommendations.
Product Recommendations
Product recommendations span far more than just films. We have all seen these recommendations hundreds of times while shopping online. You are looking at one product, and five or ten others are recommended, either as alternatives to consider, or perhaps accessories, based on what shoppers who are in some way like you have done in the past.
Many frame the product recommendation problem as one of prediction. How can I predict which product you are likely to buy next, and recommend it to you? That’s actually a lot better than what contestants were asked to predict for the Netflix Prize. Unfortunately, prediction is not really the problem at all. If I am 100% accurate in my prediction, and you buy exactly what I predicted you would, I haven’t changed anything. I haven’t generated any incremental value for the retailer, or created much immediate value for the shopper, other than perhaps saving them a bit of time.