While most people were getting ready to indulge for the Thanksgiving Holiday here in the U.S., the {rr} Ops Team was stocking up on their Mountain Dew supply and compiling lengthy playlists to weather the busiest shopping period of the year. As we geared up for these events, I wondered what we would see over the upcoming days given the heavy focus on ‘Cyber Week’ in lieu of in-store Thanksgiving shopping, how much would traffic and sales spike across our 240+ retailers come Black Friday and Cyber Monday? The anticipation was exhausting. We added 20+% in capacity, a new datacenter in London, England as well as Sydney, Australia. We doubled our network capacity, and so much more to prepare for these 6 days.
The {rr} Ops team tracks a number of metrics like requests per second, response times, model builds, transfer rates and system loads during the shopping rush to ensure we are on par with how retailers expect performance. Records were quickly broken and as in years past, the numbers spoke loud and clear. In the past six days, we’ve delivered more than 38.9 billion recommendations (up from our previous record of 19.5 billion), over 630 million page views on Black Friday (up from 532M last year) and managed more than 18,330 peak requests per second on our servers. In general, we saw much higher overnight (USA) troughs in our charts, showing a larger engagement overseas during Thanksgiving night and early Black Friday. In fact we saw our EU datacenters enjoying high numbers until 1130p to midnight GMT and USA shopping up to 2am PT. Through all the holiday extended weekend we enjoyed 100% uptime as well.
Here’s a quick recap of our ride as we scaled the peaks of the Devil’s Horns in 2016. My seventh at RichRelevance – climbing these peaks never become old.
November 23: T minus 1 to Thanksgiving
- In the two-day ramp up to Black Friday, we saw a 20+% increase in requests per second (from an average of 10,000 RPS last year to over 11,975 RPS this year).
- On Monday of Thanksgiving week, we saw over 343M page views, compared to 300M page views last year
- Traffic doubled and ran higher and longer globally starting in early November. A normal day from September, as shown on the below left, would show a defined growth to a peak, short idle, then a drop —think of a hill or small mountain’s shape. Whereas our troughs (nights) now rested around a sustained 6000 requests globally per second, a month ago they were below 3000. Seems like those earlier, longer holiday shopping periods sure are evident online.
November 24: Turkey with a side of shopping?
- In all the years my team has been tracking this, we typically we see higher page views on Thanksgiving compared with the rest of the weekend, but more purchases on Black Friday and Cyber Monday. Until last year. And now…
- Today, we hovered around 12,500 to 13,800 RPS all day. We peaked at 13,950K RSP which means, Thanksgiving 2016 had a lot more Pageviews and clicks than last year!
November 25: Black Friday is Upon Us
- At 7:30am PT on Black Friday, we hit over 18,3000 requests per second, breaking the record of 17,240 set in 2014. The long high flat plateaus of Thanksgiving, Black Friday and Cyber Monday signal lots of traffic. 300RPS brings more than 1M more requests in an hour – all those clicks and views add up.
- Page views for Black Friday were 16% higher than last year at >630M. Our former high was previous black Friday with about 532M page views. After two years of this pattern, we are converting our thoughts to Black Friday being the king of Page Views, usurping Thanksgiving from its former glory.
In the graph below, the purple shows current page views in millions up through the end of day black Friday. Orange is the projections through the end of the year. Each color below represents the previous 8 years (2008 to current).
The Weekend
Time for resting, restocking our Dew, and refreshing our playlists. The typical “eye of the storm” Cyber weekend occured. Saturday’s and Sunday’s high is down well below the trough of Thursday night. We just floated around 10-11K on the Load Balancers(edge) RPS all day.
November 30: Cyber Monday
- Our high for Cyber Monday Page Views were decent at 503M (503,359,235 to be exact).
- Monday beat Black Friday on Units (total aggregated items placed in purchased carts over the day) inferring possibly more value/’Dollars’ on the day, but lower on clicks, pageview and the rest of the stats.
- We didn’t break our 18,300 RPS peak from Black Friday as I had thought we would, hitting just 17,140 RPS, but still up from 2015 Cyber Monday RPS of 16,530.
- Below is the Devil’s Horns for Cyber Monday. As I mentioned above, we hit 503M page views as we continue our upward trend year over year (499M on Cyber Monday 2015 and 215M on Cyber Monday 2014)
The shopping season is in full force, and almost a week longer this year with the early Thanksgiving, but to summarize a great holiday shopping season opening weekend, in 6 days we had:
- 2.7B page views,
- 4.4B placement views,
- 38.9B recommendations served,
- 100% uptime on all 14 of our front-end data centers,
And a RPS peak of 18,330; that’s 65M requests per hour, more than 1M a minute.
Some things just don’t change. Black Friday 2016 was a great shopping day as expected, but brought whole new twists to existing patterns. Traffic climbed earlier in the day, plateauing high across the day and ran high for longer periods; the troughs were high, higher than any past year. We saw more traffic from overseas, and not just from growth in our client base it was also much longer and much higher in EU than any year past, even APAC shopped tons. Interestingly, mobile made up a much larger percentage of sales, the National Retail Federation said more than half of all smartphone and tablet owners used those devices to shop over the holiday weekend. With Black Friday 2016 over, we prepare to wash, rinse, repeat. 2017, bring it.
I do get a bit reflective this time of year. I’m thankful to be blessed with one of the best Operations Team in my career who are like family to me. My team has four members with more than 7+ years with RichRelevance, and we have an average of four or more years with the Company. You’d think this would just be second hand for them. But these highly skilled people worked tirelessly over each of the Thanksgiving/Black Friday/Cyber Monday weekends together and remained as focused as our first time around. Just the right amount of caution, risk, nervousness and confidence ensured that we were up 100% again, for seven years and counting.
For a few years now Europe has adopted the Black Friday shopping tradition, with many retailers launching massive sales on what is now the official start to Christmas shopping across the world.
Given the chaos in the shops experienced on Black Friday in 2014, it’ll come as no surprise that many European shoppers chose to do their buying from the comfort of their armchair. What’s more surprising is they increasingly chose to do it via their phone rather than larger devices.
Crunching the numbers across RichRelevance’s global network of retailers, we saw virtually double the number of purchases online over the Black Friday to Cyber Monday weekend in 2015, compared to 2014. View all our facts and figures for this holiday weekend in this infographic.
Black Friday still proved the most popular day of the weekend with four times as many sales on Black Friday compared to Cyber Monday. Across Europe, the peak shopping hour proved to be 10-11am on Black Friday, with €300 million in sales taken during these 60 minutes on the RichRelevance platform.
In stark contrast to 2014 when the majority of research was done on a desktop, 2015 saw nearly half of all research done via mobile and tablet devices. And out of the two it was the smaller device, which saw the most research compared to its larger tablet cousin.
While the majority of purchases were made via desktops, again it was the phone rather than the tablet, which generated the most sales volume. In fact sales volume on tablets halved year on year. Similar insights were highlighted by IBM in ‘Black Friday goes mobile,’ a recent article in Marketing Week.
The jump in popularity of online research and purchases caught some retailers off guard as they struggled to cope with the volume of traffic and transactions made online this year. However, those retailers who go their online offers (and infrastructure) right this year were the big winners. For example Curry’s PC World recorded 5 online orders per second, up 56% on last year and John Lewis recorded its biggest ever single day’s trade which they noted was mainly driven by johnlewis.com*.
A key challenge for retailers with such high demand online is the cost of fulfilment, and potential volume of returns. At least some retailers have learnt the lessons of 2014 when problems arose with missed deliveries due to inability to cope with demand. This year, couriers have been much more organised, working closely with retailers on the volume they are able to take, for example limiting the number of ‘next day delivery’ capacity to avoid disappointing consumers**.
Reviewing EMEA’s results for this Black Friday suggest it is a tradition that is here to stay and one that is certainly tipped to replace the previous Boxing Day sales of old. After the chaos of 2014, retailers this year coped with the increased demand admirably. The clear winner is the consumer who benefits from great deals on Christmas purchases wherever and whenever they choose to shop.
*Source Marketing Week
**Source Retail Week
There was a lot of talk this year about Black Friday being “less black” and less of a bellwether for holiday shopping. In addition to REI’s #OptOut campaign, an increasing number of retailers held sales earlier, and have turned Cyber Monday into “Cyber Week” by stretching their sales out longer. So how did it all pan out behind the scenes?
In search of what this meant for our 230+ global customers (and their 550+ sites), my team loaded up on Mountain Dew and music and buckled down for the wild weekend ride—from Thanksgiving on through Cyber Monday. As mentioned in my last blog, my IT Operations team tracks a number of metrics like requests per second, response times, model builds, transfer rates and system loads during this time.
This year saw the implementation of product features and innovations that quickly multiplied our monitoring and measurement needs. To support this, we added in ThousandEyes, which provided deep visibility into all the global networks we touch and into our application delivery. We also built deeper measurements for things like cross-data center latency, global carrier monitoring, API connections, mean response times, records processed, SEO and…oh yeah, we still measure page views. For page views, we generate the now famous RichRelevance Devil’s Horns graph—where the left “horn” in the graph shows Thanksgiving/Black Friday views and the right one shows Cyber Monday.
As in years past, the numbers spoke loud and clear. In the past 6 days, we’ve delivered more than 19 BILLION recommendations (up from our previous record of 17 BILLION), over 533 MILLION page views on Black Friday (up from 400M last year) and managed more than 16,560 peak requests per SECOND on our servers.
Here’s a quick recap of our ride as we scaled the peaks of the Devil’s Horns in 2015.
November 25: T minus 1 to Thanksgiving
- In the two-day ramp up to Black Friday, we saw a 20+% increase in requests per second (from 8200 RPS last year to over 10,000 RPS this week).
- On Monday of Thanksgiving week, we saw over 300M page views, compared to 215M page views last year.
- Traffic doubled and ran higher and longer globally starting in early November. Typically, a normal day would show a defined growth to a peak, short idle, then a drop—think of a hill or small mountain’s shape. Whereas our troughs (nights) now rested around a sustained 4000 requests globally per second, a month ago they were below 2000. Seems like those earlier, longer holiday shopping periods sure are evident online.
2015 looks like this (high at 9K)
Instead of this (high at 4K in 2014)
November 26: Too stuffed with turkey to hit the “buy” button?
- In all the years my team has been tracking this, we typically we see higher page views on Thanksgiving compared with the rest of the weekend, but more purchases on Black Friday and Cyber Monday.
- Today, we hovered around 10,700 to 11,500 RPS all day and ran about 90,000 concurrent sessions worldwide. That RPS is much higher than last year’s plateau and represents a lot of shopping and browsing (which translates into a lot of page views).
- In fact, we beat our previous Thanksgiving page view record by nearly 26% (> 376M page views vs 300M in 2014).
- In spite of a major carrier issue with an Internet backbone provider in Europe (detailed below), I’m proud to say our centers were 100% up during their outage
November 27: Who says EU doesn’t have a Black Friday?
- At 6:44 AM on Black Friday, we hit over 15K RPS, growing to almost 16K RPS an hour later. This was led by our European data centers in Amsterdam, Frankfurt and Stockholm, with New York, Chicago and Virginia on the US east coast just below them. No question that it was a big online shopping day in Europe. Just look at the trough of the US night covering Thursday to Friday, it rests around 9000 requests per second all night, with Europe leading with about 3500 of that right into the middle of the US day. EU shoppers were banging their keys until after 11:00 PM GMT. Welcome to Black Friday, Europe!
- Page views for Black Friday were 33% higher than last year at 532M. Our former high was the previous Black Friday, with about 400M page views.
In the graph below, the purple shows current page views in millions up through Thanksgiving. Orange is the projections through the end of the year. Each color below represents the previous 7 years (2008 to current).
November 30: Cyber Monday
- The whole weekend was solid with midday plateaus at a healthy 10-12K RPS. No issues, a Steady Eddie kind of weekend.
- Page views peaked on Friday at 532M. Our high for Cyber Monday was 499M (498,573,302 to be exact).
- We didn’t break our 17,120 RPS peak from last year, hitting just 16,530 RPS, but as noted above, we saw wider higher plateaus without the pointy peaks. This results in more views, more clicks, more recommendations and more value to our merchants. Note that many mobile apps don’t capture a “Page View,” so the large increase in mobile use would imply that the actual page view number is higher. Below is the Devil’s Horn’s for Cyber Monday. Look at how much higher the purple (this year) is to the navy (2014). We hit 499M page views vs 379M for Cyber Monday 2014. That’s more than double 2013’s 215M! It appears that the dynamic has flipped. For 5 years, Cyber Monday had higher page views. The last two years it has been Black Friday. That pretty much tells you that Black Friday is trending out of the malls and into your home and not by small numbers.
- Why was Cyber Monday lower? It all in the overnight numbers. That high Thursday night trough carried the numbers for the day, even with the higher plateau yesterday. The 4500 RPS difference overnight for 4-5 hours makes up the difference easily.
I mentioned above that we greatly expanded our monitoring and ability to see further into our solutions than ever before. Our newest toy, ThousandEyes, captured a regional anomaly on the European public Internet backbone before we even saw it affect our merchant’s traffic. The simple chart below is one of their traffic path visualizations. To the left is some of our global datacenters and the far right is our Frankfurt data center. The central red circle is the public Internet European hub that degraded for about 20 minutes due to high congestion and packet loss. In between, we can investigate and test every hop our traffic takes – sharing it in real time with our customers. This very cool, highly interactive chart provides a whole new layer of transparency to our service.
This new tool is just one example of how we constantly invest in new technology. As we move deeper into SOA this year, we will dive deeper and deeper into our services. Along with DataDog, this tool will be central to measuring our varied metrics.
So to summarize a great holiday shopping season opening weekend, in 6 days we had:
- 2.4B page views
- 3.5B recommendation placement views
- 19.5B recommendations served
At 100% uptime on all 11 of our front-end data centers.
As I mentioned last year I am blessed with a team that is like family to me. These highly skilled people worked tirelessly over our 6th Black Friday weekend together and remained as focused as our first time around. Just the right amount of caution, risk, nervousness and confidence ensured that we were up 100% again, for six years and counting. I am also excited for fantastic new product leadership at RichRelevance, bringing experience and even more thought leadership.
Some things just don’t change. Black Friday 2015 was a great shopping day as expected, but brought whole new twists to existing patterns. Longer periods of browsing, no longer sharp peaks, but long high flat periods with more interaction from EMEA. And now here we are at Cyber Monday +1 and we are already discussing our new scaling, expansion of Service Oriented Architecture, deep BI capabilities and of course more power <manly grunts>! Wash, rinse, repeat… here we go, 2016. We’re ready. Bring it.
We’ve finally slowed down enough to reflect on how we did in 2014 and the final holiday push: pretty darn great if I may say so myself. We ended another year at 100% uptime all year with record loads and HUGE growth.
The graph below shows 2014 traffic, as measured in requests per second (RPS), highlighting the final Devil’s Horns through Jan 5. You can see Thanksgiving through Cyber Week and the dips in shopping on Christmas and New Years pretty clearly. Historically, the traffic we settle at by mid-January becomes the baseline for the next year. We typically gain online traffic each year from new customers but also see organic growth in the number of shoppers.
Each color below represents a different data center location.
Below is a close up of the Devil’s Horns with page views on the left, and days across the bottom. The colors represent the years 2008 to present. We peaked a bit over 400M RPS (graphing smooths the numbers by averaging across time, making them look lower).
Finally, here are some fun stats for December 2014:
We served 10,297,583,013 impressions, converting over 51B recs at 100% uptime!
The month’s top search terms were:
- Laptops
- Sale
- Tablets
- Mens
- Clearance
- Mens clearance
- iPad
- Boots
- Frozen
- Lego
We kicked off 2015 with the launch of the Relevance Cloud and look forward to another year of continued innovation in personalized customer experiences with our customers! Here’s to great shopping and good health to us all.
________________________________________________________________________________________________
Original Blog Post from 12/03/2014
It’s been a mere 36 months since I wrote my last blog entry on Black Friday/Cyber Monday traffic. So what has changed? Nothing, and A LOT! We are still up 100% on our front-end services (52 months and counting). We still have the industry leading response rates and infrastructure platform.
But in the past 5 days, we served:
• More than 17 billion recommendations (up from our previous record of 8B)
• Over 400 million page views on Black Friday (up from 215M last year)
• More than 17,000 peak requests per second on our servers (vs 7,000 in 2011)
And oh yeah, our speeds never wavered from our blazing-fast, industry-busting, top-of-the-line service. We’ve also graduated from Mountain Dew to espresso, from Rush tapes to downloaded Urban tunes. We’ve gone from five data centers in the USA and two data centers in Europe to 12 data centers, directly supporting South America, Eastern Europe and Asia.
This year, we also invested in a great monitoring product, DataDog, as well as a fantastic scalable infrastructure. With DataDog we have the ability to track granular and custom metrics for that new infrastructure in a single place. Every year, I track the metrics of a number of IT operations that include requests per second, response times, model builds, transfer rates, load and page views (what I call the Devil’s Horns), for our 220+ global customers on over 500 sites for the most pivotal shopping weekend of the year. On my “Devil’s Horns” graph, the left “horn” shows Black Friday views and the right one shows Cyber Monday, with the dip further to the right being Christmas and Boxing Day spike following. In the graphs below, the gray shows current page views in millions, with each color below representing the previous 6 years (2008 to current).
Here are some highlights from our journey tracking the Devil’s Horns of 2014.
November 25: T-minus 2 days — Countdown to Thanksgiving.
Here’s how we looked as we started to map the Devil’s Horns graph last week.
- We surpassed 215M page views on Monday of Thanksgiving week; by this time in 2012 we were only in the108M range.
Gray is 2014 recorded traffic for the days leading up to Thanksgiving. Light blue is projected traffic estimated from past patterns and current sizes. Scale = Millions of Page Views.
- We hit nearly 10K RPS (requests per second—over two times what a typical shopping day sees), two days before Thanksgiving evening. By comparison, we peaked at 13K on Black Friday morning last year. So barely below with 48hrs of growth to come.
November 27: Is Christmas Creep eating into Thanksgiving?
- We beat our previous Thanksgiving record by nearly 40% (> 300M page views vs 255M in 2012).
- As of 7:40 AM PDT Thanksgiving morning, we were at 9.4K Requests per Second (RPS), which was already higher than 2013’s peak of 9K RPS.
- We ran the whole day at a plateau of around 10K RPS, with our peak being 11,800 at 4pm.
Gray is 2014 recorded traffic for the days leading up to and including Black Friday. Light blue is projected traffic estimated from past patterns and current sizes. Scale = Millions of Page Views.
It seems Thanksgiving has creeped its way to establish a solid Thanksgiving/Black Friday/Cyber Monday retail shopping trifecta! #ChristmasCreepIsReal. But we had another reason to give thanks today: no calls or merchant problems raised. That’s great news.
November 28. Black Friday continues to set records.
- Page views for Black Friday were over 400M. That’s not a typo, that’s Four. Hundred. Million. Our former high was the previous Cyber Monday with about 255M page views.
- Our steady states were really high at 7 – 9K per second with a plateau of 12K RPS, and we set a new record of 15,580 RPS at 8:20 AM PDT. The 7+K steady state overnight is double the HIGH mark of a regular shopping day; most normal overnight rates are about 1.5 – 2K and daily highs are 3.5 – 4.5K.
- We had a weird spike at 1 AM PDT on Black Friday: 7,500 RPS mostly coming out of EMEA (The dark blue and purple shaded area represent Amsterdam and Frankfurt data center traffic in the below charts). This corresponds with morning/midday hours in that region. This year the EU entered the USA Black Friday craze setting records both in store and especially online. In the UK, where Black Friday was fully adopted by an ever-increasing number of UK retailers, several websites crashed on Black Friday amidst shopping frenzy and near riots. Not us! Welcome to Black Friday, UK.
A mystery midnight mini peak.
Above is the overnight RPS graph zoomed in to show that weird sudden peak. I’m hoping the {rr} team in the EU was buying my team some treats.
Let’s also take a quick look below at the same RPS Graph worldwide for that 48 hours, from Thursday through Friday, to see that overnight traffic. That’s a lot of sleepy shoppers.
48-hour view of Thanksgiving and Black Friday.
December 1: Cyber Monday
- The whole weekend was solid with midday plateaus at a healthy 10K RPS. No issues, Happy merchants and shoppers.
- We saw page views peaked Friday at 400M. Our high for Cyber Monday was about 374M. A bit lower.
- We hit our all-time high today of 17,120 RPS at 12:00 Pacific time, compared to the previous record 13,300 RPS on Cyber Monday 2012 — about a 22% growth, 2012 vs 2014. This is up 15% from the 15,580 RPS this last Friday—So while more RPS at peak, the higher “low water” mark of Black Friday morning and night allowed it to surpass Cyber Monday for total page views. Note that many mobile apps don’t capture as a “Page View,” so the large increase in mobile use implies that the true page view number is higher for both days.
Gray is 2014 recorded traffic for the days leading up to and including Cyber Monday. Light blue is projected traffic estimated from past patterns and current sizes. Scale = Millions of Page Views.
I mentioned above that we greatly expanded our monitoring and our ability to see further into our solutions than ever before. DataDog, our new monitoring tool is very granular—down to the millisecond when we need it to be—and it provides our developers the ability to build custom monitoring into their code. In fact, in 2015, you’ll want to keep a lookout for a public-facing status page on our metrics.
I do get a bit reflective this time of year. I’m thankful to be blessed with one of the best Operations team in my career; the core group has been with me through five of these past holidays (I did say 52 months and running at 100% uptime, right?). I’m thankful to have the best engineers, architecture and infrastructure in the industry and the phenomenal vision of our Product Team and CEO; and of course, thankful to have the best retail merchants as my customers
As I said in the beginning, some things just don’t change. As we wind down the trifecta of online shopping events, I’m already excited for 2015 — build, scale-up, new pricing models, merchandise, engage, more customers and of course more power <manly grunts>! Wash, rinse, repeat… here we go, 2015. We’re ready. Bring it.
Jan 2011 after the 2010 holiday season….
Still in the glow of 100% uptime, response rates around 80ms, 5.5K requests per second served at peak.
We’re all sitting around in IT Ops excited, pumped and ecstatic for our performance. We’re beating our chests and basking in the glow of success. 5500 Requests per Second (RPS) peak—our measurement of capacity reached and slayed. Hey guys, I have the year-over-year page-view graphs. Look at that, over 147M page views in one day—wow.
Then someone chimes in, “but, guys, we are bringing on more than five of the top 20 retailers this year, and we’ll double that traffic. In Europe we will easily double what we did in 2010. What if online shopping takes off even more, and we TRIPLE that?”