Big Data has become the subject of Big Hype, much as Social Media and Mobile were recently. Our goal today is to peel back the hype and discover some of the key principles behind Big Data so we can make the best possible decisions about when, where, and how to apply it.
My background with Big Data has predominantly been in retail, as Principal Engineer in Personalization at Amazon, and now Chief Scientist at RichRelevance, so I will use several retail examples. However, the principles behind these examples are without question more broadly applicable. These principles are:
- Before we look at any data, we have to have a clear and well-defined goal. Otherwise we are likely to find very clever solutions to the wrong problems.
- Smart data science requires the same fundamental scientific method—hypothesis, experimentation, and analysis—as every other science.
- Correlation is not causation. We all know this, but in a big data world it is much easier to confuse the two.
- Data are economic assets. Understanding them as such helps us understand how to motivate all participants in the data economy, from individuals to corporations to governments and non-profits.
The Netflix Prize
The Netflix Prize has done more to bring Big Data and data science in general to the public mind than any other event. This has been great for increasing the visibility of the field, but I’m sad to say, miserable for actual practice. The saddest part is that the winning algorithms are not in use at Netflix today, and are unlikely ever to be.
RichRelevance hosted its first official Hack Week last week, and I challenged our engineers with a few simple rules before they embarked on unleashing their inner creativity to dig into their personal projects.
- You do NOT need to create a new product. Hack Week is for research, ideas, or improvements!
- Use Hack Week strictly for your projects.
- You are encouraged to collaborate with other teams within the company (Client Services has some great ideas and skills!)
- No matter what Pink or AMT (our product management directors) tell you, don’t work on “work.” Ignore them : )
- Most importantly, HAVE FUN!
In my last blog, I demonstrated how the noise in online sales data could be correlated (to some extent) to the weather. What I didn’t tell you was that I was saving my more interesting findings for a separate post. It’s the nature of science that many discoveries are accidental, and so it was for me when I discovered (by checking my y-axes of all things) that the effect of weather pales in comparison to the effect that day of week has on our shopping.
This is a bit obvious in retrospect; I for one frequently buy electronics early in the week in the hopes that I’ll have my cables or components by the weekend, but I know quite a few weekend electronics shoppers. So given an entire population’s varying predilections, how does the timing of our purchases pan out on the national scale?
Ecommerce on an Olympic level: Darren Hitchcock, VP of EU/UK at RichRelevance, shares three important tips.
There has been a lot of discussion—since 2008, in fact—about the consequences of intensified demand on London’s data centres during the Olympics. As a retailer, you may have taken note of the warnings and put a plan in place long ago to ensure your infrastructure’s uptime and reliability. You’re sorted. Right?
Maybe not. If you haven’t considered how your third-party application providers could impact your website during the Olympics you could be putting your business at risk. Recommendations, reviews, video, gamefication—any time you’re adding technology to your site through a partner, you need to consider what precautions they’re taking to ensure their own data centre solutions are reliable. If one of your features goes down it could disappear from your site or, worse, cause your site to slow down significantly.
It’s too late to make serious changes before the Games, but the Olympic challenge retailers are about to face does shine a much needed spotlight on this issue. So what can to ensure your website is as good as gold?
With the Olympics just around the corner, it’s not just athletes that are undergoing intense preparations in anticipation of record-breaking performance, but also many technology and infrastructure providers. Just in the past two months, outages from Amazon Web Services prompted outcries from many startups (including Netflix, Pinterest, Instagram, Quora) and their customers. It’s no wonder that infrastructure uptime and reliability are top of mind these days.
The intensified demand and consequent risk for London’s data centres during the Olympics was identified as early on as 2008, with some companies even suggesting that the official energy supplier to the 2012 Games would have to limit the energy it supplied to other organisations in central London in order to guarantee supplies to the Olympic Park and other venue. One such company, Interxion, is actually going so far as to install sleeping pods at its London data centre campus, so that on-call engineers can be on site 24-7 to ensure uptime for its customers.
Tech Crunch talks to RichRelevance CEO David Selinger about the rise of personalization in online retail.
“Personalization was really important in enabling Amazon to differentiate itself and grow in past ten years,” David Selinger, CEO and co-founder of RichRelevance. Selinger also was Amazon’s Manager, Consumer Behavior Research and helped build some of the site’s personalization features a number of years ago. “Personalization will be the differentiating factor in e-commerce and digital commerce going forward, especially for multichannel retailers and new entrants online.”