Product recommendations, also known as “recs,” are a cornerstone to an effective ecommerce merchandising strategy. When fully optimized, recs typically increase retailer revenues by up to 5%. That’s a substantial contribution when extrapolated across annual sales in the hundreds of millions or even billions of dollars. However, one thing I’ve noticed while navigating the world of ecommerce personalization is that the word “performance” is often misused in the context of recs. The reality is that there are multiple meanings to this word, and for each meaning, there is a discrete method for assessing its relationship with your shoppers and your revenue. So, let’s decode your recs performance.
At RichRelevance, we group performance into three analytical categories: value, engagement, and cohort analyses. While each type is useful, each tells us vastly different things. Misappropriating the type, like confusing an engagement analysis for a value analysis, can lead to disastrous business decisions. I’m here to protect you from making such mistakes.
First, let me summarize the three types of analysis:
- Value Analysis: this indicates the incremental revenue impact of recs, and is usually the report of most interest to retailers. The key metrics are typically revenue per session (RPS), conversion, average order value (AOV), and the lift contribution is determined by running an A/B or multivariate test that compares key performance indicators (KPIs) of populations based on their exposure to recs.
- Engagement Analysis: this quantifies shopper utilization of recs, and can be a signal for relevance. The key metrics are recs sales and clickthrough rate (CTR). Recs sales represents the sale of items clicked on in recommendations.
- Cohort Analysis: this profiles the spend patterns of shoppers that choose to use recommendations vs. those that do not—comparing the RPS, conversion, and AOV of shoppers based on their engagement with recs.
Most retailers stumble with using engagement or cohort analyses interchangeably as indicators of value. I get it; these reports are readily available, whereas conducting a true value analysis requires an A/B test which can take weeks and expose the retailer to undue opportunity costs. However, the reality is that these reports don’t convey the incremental benefit of recommendations, and here’s why:
Disassociating Engagement and Value Analyses
An engagement analysis tells us how much shoppers use recs and, to some extent, indicates relevance. If the recommendations are random, shoppers won’t click or buy from the retailer. That said, striving to maximize recs sales or CTR is equal to saying that recommendations are the most important piece of content on your site—and we all know that they’re not. What’s most important is getting shoppers to convert with the highest level of spend. Recs merely support that objective.
In fact, certain kinds of recs engagement can have a neutral or negative impact on your business. It’s the responsibility of the recommendation technology and how it’s configured to mitigate these instances. To shed more light on the matter, here are three events you should be aware of that explain why value is not a function of recs engagement or sales:
- Overlap: Sale of items that would have happened anyway, even in the absence of recommendations. Frankly, the vast majority of sales that happen through recommendations aren’t incremental, so indiscriminately increasing recs sales does not guarantee more cash in your coffers.
- Cannibalization: Reduction in AOV due to the presence of recs. If recommendations cause a shopper to buy a cheaper SKU than they otherwise would have purchased, that takes money out your pocket as a retailer.
- Shopper Distraction: Reduction in conversion due to recs. If recommendations are optimized to attract as many clicks as possible, they can bait shoppers to click in perpetuity until they become fatigued and leave your site without converting. No one wants this.
And of course, we have copious amounts of data to demonstrate that maximizing recs engagement has a detrimental impact on revenue. We’ve plotted recs CTR and recs sales against RPS lift for a multitude of RichRelevance A/B tests, and the resulting scatterplot shows no positive correlation between engagement and incremental revenue. In fact, the data suggests that at extreme levels of recs engagement, RPS lift can be severely compromised. For more details, view this interesting TED Talk on the “paradox of choice”, which implies that oversaturating a shopping experience with product options and baiting shoppers into excessive exploration can result in non-conversion.
Quite simply, it’s not about maximizing recs engagement; it’s about driving the right level of engagement that minimizes the aforementioned events and maximizes your per session revenue.
Disassociating Cohort and Value Analyses
A cohort analysis tells us the RPS, conversion and AOV of shoppers that choose to use recs versus those that do not. Most often the shoppers that choose to use recommendations have substantially higher key performance indicators (KPIs), which can lead one to presume an extreme level of benefit from recs. That said, please consider that although recommendations do generate substantial revenue lift when properly optimized, a cohort analysis does very little to quantify that impact.
In the business-to-consumer world, shoppers that utilize recommendations tend to have higher spend propensities, so it’s not uncommon for recs users to have 2x the RPS of non-recs users. Intuitively, this makes sense. When dealing with household consumers, willingness to engage with your catalog is a primer for purchase intent. In the business-to-business (B2B) world, the inverse is true. B2B buyers have predictable requisition lists that they rarely deviate from. Consider an office manager that’s restocking his company’s office supplies or an IT manager that’s purchasing specific laptop models for her business. These professionals know exactly what they want, order in bulk, and do not use recommendations.
So, these non-recs users have high RPS, conversion, and AOV. Conversely, the recs users represent those shoppers that are less committed—those that may or may not know what they want, and if they do convert, it’s with much less spend. For sites like Staples, Office Depot, or Dell, these are most usually your household consumers, and a cohort analysis would report much lower KPIs all around. Therefore, due to a self-selection bias, whether a cohort analysis shows higher or lower KPIs for recs users is independent of the value the recs technology actually delivers. Rather, it’s indicative of the types of users that decide to use recommendations.
Are recs impacting your best shoppers? Do recs help monetize individuals least likely to convert, or do they have broad resonance across all levels of spend in your shopper base?
So, there you have it—your performance analyses decoded to help you learn a bunch more about your recs and answer these questions: Who is using recs (Cohort Analysis)? How much are they using it (Engagement Analysis)? What incremental value does this utilization provide (Value Analysis)?
Today, we’re happy to announce that the Relevance Cloud is out of beta and available to all with the 15.02 release. This release introduces Build (API-based personalization building blocks) and delivers enhancements to our Recommend and Discover products, helping you to personalize every step of your customers’ purchase journey and setting you up for an exciting 2015!
Let’s start with Build. Build is an entirely new approach to personalization that uses the power of cloud APIs to give you real-time access to the building blocks of personalization: customer data, product data and contextual events. Not only are these APIs the foundation on which we have built our own applications, they also power some of the most innovative customer experiences for top retailers such as Marks and Spencers. Watch the webinar on how to reshape your CX using Build or download this white paper to learn more.
We have launched a terrific set of new features with this release:
Gain full view of the customer in real time through Profiles API.
Track every click, view, search and purchase as they happen through the Event Stream API.
Record your consumers’ likes and dislikes explicitly with the preference API.
Turn on product or search and browse personalization on any channel: web, mobile or store.
Preview “complete the look” or “compatible products” for Advanced Merchandising rules before deploying them.
- Improve testing for coat-tailing rules with Discover.
Log in to your RichRelevance dashboard to read more details in the release notes.
Our improved DX for your enhanced CX
Developers know that it can be extremely frustrating to access complete and up-to-date documentation. It slows down the process of trying to figure out how to work on all the new, exciting features. However Jen Kollmer, our product documentation champ, has got you covered. Jen has single-handedly created a site dedicated to enhancing our developer experience, so that you can focus on innovating your customer experience.
Check it out at: http://developer.richrelevance.com. Comment here on the blog if you would like to thank Jen or give her any ideas for improvement.
Constantly fine-tuning your personalization command center
Along with the Relevance Cloud release, we’ve made some critical changes to our dashboard so that features and functions are better organized around tasks, and you have a more streamlined and productive experience.
Finally, if you find it to be more productive, you can even customize our dashboard in Japanese!
Personalization is what empowers retailers to create a 1-1 relationship with customers online. By tracking engagement and other KPIs, you can quickly take stock of how are you doing with those relationships.
The new RichRelevance dashboard is the perfect tool for putting richer reports with more data at your fingertips, so that you’re always in tune with your site personalization. The new dashboard pulls back the curtain to showcase your data in a more digestible, visually appealing way. When your reports are eye-friendly, organized and automated, you can assess the status and impact of your site personalization within minutes.
Here are three quick ways to dive into the dashboard.
1. Check and share the vitals
Our brand new Lookback Summary is a detailed tile band that displays an at-a-glance summary of your sales and recommendation metrics. You can configure the lookback period to be a day, week, month or any number of days up to 90. You can also click any tile to access an expanded view of the graph. Take a screenshot and share metrics with key stakeholders in just a few clicks!
2. Track the value add
The Attributables Report tile is another new addition to help you track the impact of RichRelevance recommendations on your site–specifically through change in average order value (AOV) and items per order (IPO).
3. Access more detailed data and then some
Other significant enhancements include the Page Type Performance tile, where you can view the performance of different metrics for various pages on your site, and the Viral Products tile, which displays the most viewed products and takes you to detailed Merchandising Reporting. Best of all, feature documentation can be accessed within the feature page itself.
The new RichRelevance dashboard is designed to upgrade your experience and provide site health visualization, easy controls and usability improvements throughout. If you haven’t done so already, we encourage you to log in to your dashboard and play around. We believe that in order to provide you with more of the tools that you love, we need to work together. So, tell us what you’d like to see in the next set of improvements using the comments section below.