The Big Data revolution at Google, Facebook, Amazon and Apple has transformed the consumer journey across channels, and even helped turn a presidential election. This transition simultaneously poses tremendous challenge and opportunity to retailers.
In our session “How Relevance Can Get Your Brand Elected” at last week’s NRF Big Show, Rayid Ghani (Obama for America’s Chief Scientist) and I shared how data scientists have utilized key methodologies for capturing value and turning Big Data to smart data—using it to win votes and create profit. The analogies between data usage in the Obama campaign and my experience at Amazon and RichRelevance are deep: there are three ways to use data in operations, and it is necessary to take small steps toward a very specific goal using an agile, fail-fast methodology.
The methodologies for data usage are analytics; prediction and interruption; and optimization. They transcend the emergence of “Big Data” and extend in their impact from the last century, as do the learnings from their failures. The “big transformative project” is doomed to fail; only iterative implementations see the huge ROI of data-oriented technologies.
As I traveled the exhibit hall, I reached my personal predictions for 2014:
- 2014 will test us: We must know our identity—who are we to our customers and how we maximize that. This challenge exists in a highly competitive and unfriendly ecosystem: Amazon is on a tear and will not stop; will Google, eBay, Apple and Facebook turn out to be friend or foe?
- Omni-channel ROI at scale: 2014 will be the first year with enough historical case studies of successful omni-channel strategies to support systemic investment. Retailers can invest in 5-10 major initiatives and each can build on the successes/failures of past years.
- Emergence of Big Data, Year 2 of 5: Big Data is a long-term trend. We are still in the early adopter phase, where practitioners familiar with the technology and business can be successful, but not enough business people fully understand the technology’s capabilities, ROI or risk in order for 2014 to be the year of mass scale. That said, Big Data is complex and will require numerous years of investment and learning to reach maturity. Executives know that, so this year will see significant investment from both capital and time budgets.
I invite you to learn more about how you can turn your Big Data into smart data for a relevant brand experience by visiting https://richrelevance.com/nrf/.
In this volume, we conclude with Privacy and Security.
Privacy and Security
For our final examples I want to dig into the notions of privacy and security in big data settings. These are and always will be critical concerns.
We begin with financial information and e-commerce. In the early days of Amazon, there were a significant number of customers who were very concerned about the security implications of entering their credit card numbers online. The specific concerns varied, but they almost always involved the possibility of a gang of nefarious hackers gaining access to credit card numbers and using them to make fraudulent purchases.
In this volume, we discuss Data Mining and The Birthday Paradox.
Data Mining and The Birthday Paradox
We’ve all heard of the Birthday Paradox. Put 23 randomly chosen people in a room and there is a 50% chance that two or more have the same birthday. Put 57 people in the room, and the chance is 99%. Do those people have anything more in common because they were born on the same day of the year? Astrologers will say yes, but most scientists would say there is no evidence to support that claim.
What does this have to do with big data? The answer is that generalizations of the math behind the birthday paradox tell us that we will—not just that we can but that with near 100% certainty we will—draw meaningless conclusions if we just look at enough variables. In fact, we can show that if we generate a large number of streams of completely random data, some of them will look like others.
The problem is, we can easily forget this when we look at big data sets with lots and lots of variables. These are the kind of things we see in what is called data exhaust. Data exhaust is the vast stream of data gathered and logged by digital devices ranging from mobile phones to engine sensors in cars to video cameras in public spaces to instruments on particle accelerators.
Look at a lot of this data, and you will find spurious correlations. This is what Principle 3 is all about. Statisticians have known about Principle 3 for decades, and have techniques for trying to deal with it. The best technique, however, is and always has been a controlled scientific experiment, as Principle 2 advocates.
In this volume, we discuss Product Recommendations.
Product Recommendations
Product recommendations span far more than just films. We have all seen these recommendations hundreds of times while shopping online. You are looking at one product, and five or ten others are recommended, either as alternatives to consider, or perhaps accessories, based on what shoppers who are in some way like you have done in the past.
Many frame the product recommendation problem as one of prediction. How can I predict which product you are likely to buy next, and recommend it to you? That’s actually a lot better than what contestants were asked to predict for the Netflix Prize. Unfortunately, prediction is not really the problem at all. If I am 100% accurate in my prediction, and you buy exactly what I predicted you would, I haven’t changed anything. I haven’t generated any incremental value for the retailer, or created much immediate value for the shopper, other than perhaps saving them a bit of time.
Big Data has become the subject of Big Hype, much as Social Media and Mobile were recently. Our goal today is to peel back the hype and discover some of the key principles behind Big Data so we can make the best possible decisions about when, where, and how to apply it.
My background with Big Data has predominantly been in retail, as Principal Engineer in Personalization at Amazon, and now Chief Scientist at RichRelevance, so I will use several retail examples. However, the principles behind these examples are without question more broadly applicable. These principles are:
- Before we look at any data, we have to have a clear and well-defined goal. Otherwise we are likely to find very clever solutions to the wrong problems.
- Smart data science requires the same fundamental scientific method—hypothesis, experimentation, and analysis—as every other science.
- Correlation is not causation. We all know this, but in a big data world it is much easier to confuse the two.
- Data are economic assets. Understanding them as such helps us understand how to motivate all participants in the data economy, from individuals to corporations to governments and non-profits.
The Netflix Prize
The Netflix Prize has done more to bring Big Data and data science in general to the public mind than any other event. This has been great for increasing the visibility of the field, but I’m sad to say, miserable for actual practice. The saddest part is that the winning algorithms are not in use at Netflix today, and are unlikely ever to be.
This week, RichRelevance Shopping Insights™ released an infographic with some keen insights into which social networks drive retail traffic.
Perhaps least surprising was the finding that Facebook dominates as a source of traffic. Shoppers who click through from Facebook account for the overwhelming majority of shopping sessions and also browse more and buy more often. The transactional, sticky nature of Facebook means that shoppers may already be in the mindset of browsing and buying when they click on to a retail site. But while shoppers who come to retail sites from Facebook and Twitter purchase more often, Pinterest users spend dramatically more than either ($168.83 average order value vs. $94.70 for Facebook and $70.84 for Twitter)—nearly double that of other social channels.