Category Archives: Uncategorized

The Chief Architect’s Purpose

A few years ago I was attending the O’Reilly Software Architecture conference in London where I had prepared a talk on Target’s new vision for the future, a Target Retail Platform. After the conference I had a couple days of vacation and I stopped by an art exhibit at the Saatchi Gallery on Alexander Calder. It was there that I learned that Calder defined the Chief Architect’s purpose better than anyone.

The Calder quote at the top says: “I think I am a realist, because I make what I see. It’s only the problem of seeing it. If you can imagine a thing, then you can make it and ‘tout de suite you’re a realist. The universe is real, but you can’t see it. You have to imagine it. Once you imagine it, you can be realistic about producing it.

This quote struck me as I had spent the last few days thinking about architecture, and how the architecture conference I’d been attending rarely talked about architecture. I had attempted to create a presentation on Target’s architecture vision, nothing about how we would build it or the technologies used, but simply the vision, principles and structure that would guide a 3000 engineer strong team to leave its current reality and build a new one, a retail platform. I would guess that 80% of the presentations at the conference were on microservices. It was 2017 and all I could think was architects are behind the curve if this is their first introductions to learning about microservices. However, the microservices presentations were far better attended than my own, so, joke was on me.

I loved this quote because I often used the term Architecture Realist when people would ask me about my architecture style. I was only interested in creating architectures that would and could be implemented. I had found that most Chief Architects rarely present anything that engineering teams can actually use to guide their development, and instead fall back to edicts on technologies.

But what really struck me was the simple definition of Calder’s artistic approach, which coalesced for me into the Chief Architect’s purpose.

“The universe is real, but you can’t see it. You have to imagine it.”

This is it, this is the purpose of being a Chief Architect. It’s your role to take yourself out of your current reality and imagine a new one.

“Once you imagine it, you can be realistic about producing it.”

Unlike Calder who could then go about executing his vision via a painting or mobile, producing what you imagine within a large enterprise is not a straightforward task.

The presentation I had created, “Platform Architecture for Omnichannel Retail” was my attempt to convey what I had imagined for Target. Communicating to 3000 individuals to follow a vision is the Chief Architect’s job, not its purpose. I prepared the presentation for O’Reilly to force myself to document the vision in a way that could be consumed by a technical workforce.

Over the ensuing three years I’ve given some version of this presentation hundreds of times to small and large groups within Target. In the real world, producing an architecture means understanding and engagement from teams, which is best done in small groups where people feel comfortable asking hard questions.

What you’ll find is you don’t always have the right answers, your reality, which you thought was perfect, was half baked. But if your principles were sound, working through the hard questions to answers that follow the principles of the architecture builds the patterns and practices necessary to produce the new reality.

For Target in 2020, the new reality is here, we have a Retail Platform.

The Simple Formula to IT Modernization

Having instigated and led through the digital transformation of BestBuy.com, and then moving to Target where we have been completely modernizing IT across the entire company, I’ve learned the simple formula to IT success.  While the formula is simple, the execution is what is inherently difficult.

In my first year at Best Buy, I spent months pitching how we would rebuild BestBuy.com and move it to the cloud back in 2010 and 2011.  At the time, this was a radical idea as clouds were new and unstable, and were considered highly insecure by corporate IT leaders.  The pitch was centered on the standard IT formula of People, Process and Technology.  This formula is good, but it also has a big miss, there is no emphasis on outcomes.  The pitch worked and I was granted $13M to start the rebuild of BestBuy.com, a project that took four years and well over $100M to complete.

While we continued to use the People, Process and Technology formula in our decks and communication to leadership, I managed my team of 200+ engineers to outcomes.  We had to deliver a flexible, maintainable, layered cloud ecommerce platform that scaled infinitely or we were failures.  We implemented Product Management and Agile and morphed into a Product Engineering team that brought BestBuy.com into the modern world and did our part in the overall turnaround of the company.

For more on that see my 22 part series on the Digital Transformation of BestBuy.com.

The People, Process, Technology formula was great for selling to VPs and EVPs, but the dissonance between the sales pitch and the implementation kept me wondering about a better way.  Then I moved to Target which helped me understand that I had found a better way, I just didn’t have the name for it yet.

At Target, our CIO is the most architecture centric CIO that’s ever existed.  Most CIOs pay lip service to architecture, but then hand it off to an Enterprise Architecture team and say “go implement architecture.”  But when the architects try, they are constantly overruled or ignored because all good architecture decisions require tradeoffs in feature delivery in the short term.  Without an overarching vision, no business or IT leaders will make feature tradeoffs.

With little support or understanding from the CIO, the various IT VPs are free to flout architecture rules or governance, and therefore go off and implement their locally optimized solutions.  This is the core cause of IT inconsistency and sprawl, and why every SOA ever designed failed at enterprise scale.

At Target, we’ve used a different formula consistently for the last four years:

  • Architecture First
  • Team Second
  • Value Third

How does this compare to People, Process and Technology, let’s analyze it.  

People and Team sound the same but they have different inferences.  People is generic and generally boils down to something about only hiring A players, and B players hire C players because they are insecure.  This assumes you already somehow have a bunch of A players, and that you are an A player too.  This is obviously ridiculous.

Team, however, is about getting people to work together of all types and capabilities.  It’s about maximizing talent by the group coming together and creating something more than its parts.  At Target we’ve strived to create a learning culture where the most granular breakdown of the organization is the team rather than the individual.  Inside the team, there are ranges of capabilities, often based on experience, the environment encourages helping each other through pairing or mobbing, increasing everyone’s capabilities.

Process isn’t even part of the new formula though it is important.  Process actually gets rolled up into Value.  Value is what you are striving for, it’s the outcome of working on features and technology.  But if a feature doesn’t resonate with the customer, no actual value is delivered, although we have learned something that didn’t work.  Value, in the end, is how the customer perceives it, and how you measure it.  Delivery of value uses a process, in Target’s case Product and Agile.  Making Value measurable is the hard part.  Saying you delivered value by adding a new payment type is great, but measuring the impact in incremental sales through an experiment which tests whether a new payment type actually increases sales is better.

Technology and Architecture are often tied together, but the reality is Architecture is technology independent.  Technologies are tools, or, as architects like to say, implementation details.  Architecture is the vision, strategy and principles underlying and overlaying how every system is built and how it fits into the larger picture.  Getting the architecture right gives every engineering team a place to fit their work into how Target’s guests benefit.  Too often, engineering teams have no idea how they fit into the enterprise, so they make choices and build solely to please themselves and their sponsors.  But if the team understands how they benefit the company, they have a higher calling and are willing to make architecture tradeoffs.  

Getting the architecture right allows the company to achieve both known and unknown outcomes.  If we learned anything from the last two months of COVID lockdowns, a good architecture allows you to flex, scale and build new capabilities overnight.  It allows you to withstand an instant 30% channel shift from store customer to online customer.  

Architecture, Team, Value is the simple formula to IT modernization.  Just look at the recent outcomes.  

WFH Chronicles – 2020 May – agile manifesto revisIted

A number of things have reminded me of Agile lately.  The new HBR article on The Agile C-Suite only emphasizes that the little agile cults we formed in the early 2000s to build software was the right direction.  As one of the executives interviewed for the study, it’s great to see agile evolving to managing entire companies.

I can remember the exact day I started my Agile journey, it was January 2, 2003.  I was long enough into my occupation as a software engineer to know that the dominant Waterfall methodology was painful, slow and often resulted in poor outcomes.  I had been working as a consultant for about two years at this point, and on the first workday of the new year, we were starting a project to rewrite a 25 year old mainframe application for workforce management.  West Telemarketing in Omaha, Nebraska was the largest outsourced call center manager in the world with over 25,000 agents across the globe.  It was a big project for everyone and we had just hired two new consultants with Agile backgrounds to help us on the journey.

I was certainly skeptical up front, but as we got into the daily processes and mindset it felt like someone had finally designed a process that fit the real world.  It didn’t take long until I was a convert and devoured Alistair Cockburn’s Agile Software Development to get a better understanding of the process.  

For the last 17 years I’ve only worked in Agile shops.  Sometimes I had to create them, sometimes I joined existing agile teams.  But through it all, I think I’ve sent the URL to the Agile Manifesto 1000 times to engineers and managers to get them introduced to the concepts.  

But I hadn’t really read it for many years.  When I was first introduced to the manifesto, I was a consultant and the “this over thats” made sense.  We recently went through product training that brought the manifesto back in front of me, and was drawn into a discussion about it.

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

Reading and discussing them again as Chief Architect for Target, I find that they don’t resonate as well.  Target has converted to largely in-house product and engineering, so no more contracts.  When working within an enterprise where hundreds of product teams rely on mutual timeframes to deliver large scale initiatives, sorry, we need some type of plan.  We also need documentation for long term ownership of the products.  However, many people and teams read the manifesto and said “no more plans, no more docs!”  The manifesto struck me as very oriented towards outsourced engineering practices.

Here’s a modern take on the manifesto learned while building high scale distributed platforms through the digital transformations of Best Buy and Target:

People matter most, processes and tools are enablers

Working software, maintainable over time

Continuous collaboration across all teams, always

Plan, build, learn, change, plan

WFH Chronicles – 2020 – April

Taking an intern out to lunch in 2020 suddenly became much harder to do, luckily, I already got it done prior to the WFH shift.

After a month now of WFH, it’s official, I can’t love it.  I don’t think anyone would classify me as outgoing and gregarious, but I do like the separation of work and home, and find in-person whiteboarding far more collaborative and energizing than trying to work out architecture with Confluence diagrams and PowerPoint. 

For the first two weeks of WFH, I had no place to work in my house.  I’ve never liked working from home and thus have no office at home.  That left me, like many others, trying to find a quiet space in the house to have meetings for eight+ hours each day on Zoom.  I also have a college freshman trying to do online school from home, and our house doesn’t have many isolated places.  As a music percussion major, we now have a marimba in our living room which takes out 2/3 of the house when he’s playing it.  My Zoom meeting participants have gotten used to light marimba in the background of many of my meetings in the afternoons.

Without a great setup, I was stuck at the dining room table, or my wife’s studio in the mornings until she wanted it back.  I finally created a standing desk out of an old table (but very nice teak table from the 70s), with a footstool one of the kids built in shop class in high school, and two large books.  The books are Volume 2 of a Shorter Oxford English Dictionary (who knows where volume 1 went) and the 98th edition of the Handbook for Chemistry and Physics.  I’m assuming it’s better than the 97th edition but how would anyone tell?  These two books add about seven inches of height for me.

The best part about my new setup is it’s in a room we call the playroom, a four-season porch added to the back of the house circa 1954.  It’s the playroom because it started out has a place where toddlers could play and be well contained without any noise filtering to the rest of the house.  It continued that way through high school housing the many gaming consoles and late night tv sessions.  But the room looks out on the backyard which is just starting to revive from winter, and has many birds that have appeared with the start of spring. 

I now can observe our cherry tree, and see a pair of purple finches, who have created a nest in the neighbor’s arbor vitae.  They seem to like to sit in the cherry tree and take in their domain.  There’s also some of the many neighborhood rabbits and squirrels that traverse the yard throughout the day.  I’m getting to know their habits.

But the bird watching has now become a hobby/obsession.  As you can see from my setup, I have a bird watching guide and a pair of binoculars, not for creeping on neighbors, but for checking out the birds that come through.  Besides the finches, so far there’s plenty of cardinals and robins, but also a yellow-bellied flycatcher (at least I think it was).  With the sparrows and black capped chickadees, you might find me stepping away from my Zoom meeting to grab the binoculars to get a better look.  I also have a digital camera ready but have found the telephoto isn’t strong enough and taking pictures of birds is tough.  An iPhone just doesn’t cut it either.

A sad DIY standing desk

You’ll also see a 2.5kg soft weight, I’ve found that I get fidgety standing for most meetings all day long, and a light weight to toss around keeps me occupied and better able to focus, while possibly providing some health benefits.

Final item in the picture is a serious digital metronome, I’m sharing the space with the snare drum as it’s still the most sonically isolated area of the house, and if you’ve ever heard a snare drum in practice, they are loud!

It seems we’ll be at this for some time so my unexpected outcome is intimate knowledge of my backyard and a minor ornithological obsession.

Take an Intern out to Lunch in 2020

Twenty odd years ago as I worked at my first couple  jobs, people with much more experience in the workplace spent a bit of time and money and took me out to lunch.  I was reminded of this when I took out a Target intern for lunch and it brought back these memories. Here are those stories.

My first internship was at Wisconsin Power & Light (WP&L) in Madison, WI, a now defunct power utility.  It was a program for graduates interested in careers in power and a friend of the family who worked for WP&L had let me know about it.  It was the summer of 1992 and I found myself in a 10 story office tower in downtown Madison. There were about 8 interns who were all working in different parts of the company, I was in a group that attempted to calculate the best mix of power production to meet the needs of the utility’s customers.  The VP of the group, Don, had a nice office with a big wooden desk and a posh leather chair. We met every few weeks to talk about what I was working on.  

I spent my time trying to create the first spreadsheet that could calculate the cost of electricity from a cogeneration power plant.  Cogeneration is when you use the waste heat from generating power to produce heat for heating or cooling buildings. It uses more of the heat energy that would otherwise be expelled into the environment, and lowers the cost of electricity to the utility.  Cogeneration was just starting to catch on in the early 90s.

Rather than write a simulator in either of the two languages I was learning, Pascal and C, the chosen technology was macros in Lotus 123.  Lotus 123 was basically the Excel of the time. It turned out to be quite fun to build up a spreadsheet of inputs such as fuel costs, efficiency, etc and create a bunch of calculations that would output the cost of electricity from a potential plant.

At the end of the summer, Don scheduled a lunch to review how my summer went, he told me to pick the restaurant.  I chose Wah Kee Noodle, probably the best Chinese noodle shop in Madison which was in business for over 30 years before the owners retired in the summer of 2019.  Don had rarely been to Chinese restaurants so it was a new experience for him. We had a nice lunch, slurping a big bowl of noodles, where he asked a bunch of questions about my work, what I had learned that summer, and whether I would come back in the future.  I was starting my PhD program in the fall so I didn’t think I would return, but thanked him for the experience and for lunch. I still remember the genuine interest in my future Don showed even though I was just an intern passing through his office. Being an executive now, I realize what a great leader Don was in showing his team that everyone in his office mattered.

A few years later as I decided to exit my PhD program in Nuclear Fusion, I needed some work to fill in the gap between January and May of 1995.  Being part of the University of Wisconsin, my mom knew someone who was looking for a programmer to create the software to run a PLC network at the UW Biotron.  The Biotron is a building that houses climate controlled experiments usually on plants. The project to install the PLCs was going to automate the collection of environmental data that was being done by lab techs.  When it was finished, they would be letting go of two of the four lab techs. It was my first introduction to automation replacing humans in jobs. I heard that even 20 years later, the software and PLCs were still doing their jobs.

Even though the lab techs knew I was working on the system to replace them, the lead tech, Matt, made it clear to his team that they were to treat me like one of the team.  They were unionized and they would come find me at break time, and make sure I took my break. We would hang out in the break room and chit chat about the weather, which was generally below 0F, and how I managed to bike there every day.  

As the winter progressed and I began building my own protocol for the C based PLCs to talk to the master controller and relay their data, as well as build the control systems that would adjust the environment of each room, everyone stayed friendly.  Even when it became clear I would get the project done, no one showed me any animosity.  

At the end of the effort, Matt took me and the team out to lunch, I think it was at a Burger King, and given that I was building software to put a couple people on his team out of work, we had a surprisingly good time.  I really appreciated Matt’s kindness and leadership, it could have been a much more hostile environment under a different leader.  

These lunches were over 20 years ago, but I remember them well, much better than any lunches I’ve had in the past few years.  If you are a supervisor and you have an intern in your group, take them out to lunch, it will make a big impression on them that may last for their entire careers.

R.I.P. Kenilworth Trail

This may be slight deviation from normal, but the end of the Kenilworth Trail is coming today (May 13, 2019), which is a bike path that has featured highly in my life. Whatever you think of the light rail that will be built in the corridor for the next three years, it will destroy one of the most highly used bike paths in the US so that suburban commuters can get downtown faster and bypass Minneapolis. That’s what the light rail is doing, it’s routed through the far west side of Minneapolis through one of the least populated portions because it’s faster than taking it through Uptown and down Nicollet, which might actually serve some Minneapolitans.

I estimated today that I’ve ridden portions of the Kenilworth trail over 10,000 times in the last 20 years. How did I get to that number:

  • Bike commuted downtown for 13 years, 2x per day, ~200 times (yes I biked all winter for many of these years. ~5200 rides.
  • For 20 years, from April to November, I generally ride to do daily errands: trips to Vertical Endeavors, Target, Walgreens, Punch Pizza, Barnes and Noble, Whole Foods, Lunds, Byerly’s, etc. On weekends it’s probably 4 trips per day, weekdays and extra 0.5 trips per day. ~7290
  • For 10 years estimate 30 trips downtown to Twins games, fireworks, restaurants and brew pubs in the evenings per summer: ~300
  • Probably another 1000 rides for fun and exercise with the kids, dog and friends

So in the past 20 years, I’ve ridden portions of the Kenilworth trail approximately 13,790 times. That’s likely an underestimate if anything.

There’s a lot of memories on that trail. I used to rollerblade with a Burley with two kids (in the 2-6 range) in it around Cedar lake. One time I’m not sure what I was thinking but I was swinging the kids side to side for fun. I got a bit to aggressive and flipped the Burley on it’s side and proceed to crash myself. Luckily, the kids were fine, and I escaped with minor roadrash.

I used to run our dog, Callahan, down the path to get him some quick exercise. I would ride my bike, he would run alongside, he was very well behaved. One day though, as I was riding back down the path, I slipped the leash over my handlebar by accident. Callahan abruptly stopped to do some business, and in these cases I usually just dropped the leash. This time though when I dropped it, it caught the handlebar and pulled the front wheel sideways. I took a header over the bars and somehow landed it on my feet, jamming my hip and causing future arthritis. These were my only two crashes on the Kenilworth.

A number of friends and I would go brew pub hopping in the evening for many years. We’d ride to Northeast and hit a few pubs and ride back, often without lights. There’s nothing like riding the Kenilworth at night with no lights, it’s pitch black and you can barely see the path. One night, and I swear this is true, after turning off the path and passing the Cedar Lake South Beach around 1AM, there were about 30 naked teenage girls getting ready to go skinny dipping. It was like riding past Ulysses’s Sirens, except that there was one older lady standing under the streetlight with a clipboard, likely ensuring the Sirens earned their skinny dipping merit badges.

It’s funny but there are a lot more stories simply about riding and walking down an amazing urban bike path. These don’t exist in other cities, and now it will no longer exist in ours. I filmed my last ride down the Kenilworth, it’s not the greatest, but this is what is now gone:

On Learning New Things

Holiday is mostly past us going into the new year of 2019 which means we retailers are starting to have some time again.  The start of the year is when people think about making resolutions with the intent of improving themselves.  For the past ten years at least, my resolution has always been to “eat more pie.”   It’s an easy resolution to keep and it makes me happy to accomplish it every year.

It’s better to be more intentional about what you are learning, so skip the resolution and take some action.  In the last year I took a deep dive into Distributed Ledgers (DL, better known as Blockchains) to determine how we might use them at Target.  This turned out to be a much harder problem that it sounds, use you can use a DL for anything really, but it only makes sense in specific contexts where you are actually sharing data with external partners.

Second, I gave a presentation on technical culture rather than technology.  It was a first to talk about building engineering culture, even though it’s what I’ve done for most of my career.  But it was important to put Target on the map as a leading technical retailer.  Five minutes from the keynote is below, but there’s also a 45 minute talk if you have access to Pluralsight.

Outside work, I learned silver smithing and attained the rank of amateur.  I generally need something inside and outside work to keep me busy.  Plus, you can give the finished products away and it makes people happy.  An almost finished ring below:

Commerce Tomorrow Podcast

I recently had the opportunity to sit down with Kelly Goetsch and Dirk Hoerig to record an interview on Target’s engineering culture as part of their Commerce Tomorrow podcast.  Kelly was in town for the Open Source North conference which we were both speaking at.  We sat down in the Target recording studio to tape the show.  I’ve been using the studio to create an internal podcast for Target Engineering so we had an audio engineer and figured out how to include Dirk from Germany.

Click here to hear the podcast.

A Digital Ecommerce Transformation – Holiday – This Year is Always the Most Important One Ever – Part XII

Part XII of a multipart story, to start at the beginning goto Part 1.

Since we’re in the middle of Holiday 2017, I thought a digression on Holiday was in order.

If you have never worked in retail, than you’ve have missed out on the grand experience we call “Holiday”. On the other hand, you’ve probably actually enjoyed the time of year from mid-November to Christmas while you celebrate with your friends and family, and take advantage of thousands of days of deals from the many retailers trying to get their share of wallet from you.

Holiday, with a capital H, is something that has to be experienced to be believed. In my first Holiday at TWLER in 2010, I was on a team that had just started writing code and had very little in production leading into Thanksgiving. The only offering we supported was the failure site, if the main TWLER.com went down, we would quickly spin up the browse only site so consumers would be able to at least see what products we sold, and where our stores were located. In 2010, this was actually a pretty good thing since the ecommerce site was still less than 5% of revenue.

When you work in IT in a retailer, your entire year is judged on whether or not the systems you support survive the shopping onslaught of Holiday. In the online space, an ecommerce site might make 30% of its revenue in the five days from Thanksgiving to Cyber Monday. TWLER.com also experienced the third highest traffic of North American retailers during that time. This massive scale up to 20X normal daily traffic was largely accomplished without clouds in the 2000s. You had to take a really good guess as to how much infrastructure was needed, build it all out over the course of the year, and hope you weren’t overwhelmed by consumer behavior. You could easily receive 1M requests per second at the edge, and 100,000+ requests per second to your actual systems. If those requests were concentrated on the wrong systems, you could easily take down your site.

TWLER counts how long you’ve been at a company by the number of Holidays you’ve experienced. If someone asks how long you’ve worked there, you might say “four Holidays.” And every Holiday is the most important one yet, because those six weeks account for 50% or more of yearly revenue.

After a few Holidays, you realize the second the current year’s Holiday is over, you are immediately planning for the next one. There is no break. It’s like a giant tsunami that is slowly approaching, day by day. You can look over your shoulder and it’s always there, waiting to crash down on you and ruin your day. Once this year’s tsunami passes, you turn around and can see next year’s on the horizon.

In my six Holidays at TWLER, we experienced numerous outages, usually caused by either internal stupidity, or unexpected consumer behavior. In our first few years, we would purposely force our ecommerce site to use “enterprise” services because they were the “single source” for things like taxes, or inventory. This is a great notion, but only if the “enterprise” services were actually built to support the entire Enterprise. Since TWLER was store focused, this meant the “enterprise” services were often down at night for maintenance, or were not built to withstand massive surges in traffic. One million people refreshing a PDP to check for inventory on a big sale every few seconds quickly overwhelmed these services. So we often turned these services off and flew semi-blind, rather than have the site completely fail.

In other instances we tried to use various promotion functions embedded in our ATG commerce server. These seemed like useful things to easily setup a promotion like buy one get one. But when millions of people come looking for the sale, the vendor built commerce engines go down quickly by destroying their own database with the same exact calls, over and over again.  They hadn’t heard of caching yet, I guess.

We would sometimes publish our starting times for various sales, saying a big sale is starting at 11AM and send out millions of customer emails. The marketing teams loved the starting times and the technology teams hated them. We warned that setting a hard start time is a sure route to failure. Yet we did it multiple times and incurred multiple failures as the traffic surge brought down the site. There are physical limits even in clouds, you can only spin things up so fast and 10M rqs will bring down most sites. After a few of these episodes, we did convince the marketing teams that it wasn’t the way to go and learned how to have sales with gradual ramp-ups in requests rather than massive surges.

Around 2013, the Black Friday shopping was so intense in the evening across the nation that the credit card networks themselves slowed down. Instead of taking a few seconds to auth a credit card, it started taking one or two minutes. This was across all retailers. However, the change in time caused threads to hang up inside our ecommerce systems and all of a sudden we ran out of threads as they were all tied up waiting for payments to happen. For the next year, we changed our payment process to go asynchronous so that would never happen again.

There are many more stories of failure, but from every failure we learned something and implemented fixes for the next year’s wave. This is why Holiday in retail is such fun, every year you get to test your mettle against the highest traffic the world can generate. You planned all year, you implemented new technologies and new solutions, but sometimes the consumer confounds you and does something totally unexpected.

The last story is one where the consumer behavior combined with new features took us down unexpectedly. In 2014 we implemented “Save for Later” lists where you could put your items on a list that you could access later and add them to your cart. As Thanksgiving rolled around and the Black Friday sale went out at around 2AM, our Add to Cart function started getting pounded at a rate far higher than we had tested it for. We were seeing 100K rqs in the first few minutes the sale was happening, it rapidly brought the Add to Cart function to its knees and we had to take a outage immediately to get systems back together and increase capacity.

This was completely unexpected consumer behavior so what happened? It turned out that customers used the Save for Later lists to pre-shop the Black Friday sale and add all the things they wanted to buy into the lists. Then when 2AM rolled around, they opened their Save for Later lists and started clicking the Add to Cart buttons one after the other. A single customer might click 5-10 Add to Cart buttons in a few seconds. With hundreds of thousands of customers figuring out the same method independently, it led to a massive spike in Add to Cart requests, we effectively DDOSed our Add to Cart function with simultaneous collective human behavior.

I feel like I could keep going on Holiday for another two pages, but that’s enough for this year, maybe we’ll do it again in the all important next year.

Goto Part XIII