Category Archives: Uncategorized

Take an Intern out to Lunch in 2020

Twenty odd years ago as I worked at my first couple  jobs, people with much more experience in the workplace spent a bit of time and money and took me out to lunch.  I was reminded of this when I took out a Target intern for lunch and it brought back these memories. Here are those stories.

My first internship was at Wisconsin Power & Light (WP&L) in Madison, WI, a now defunct power utility.  It was a program for graduates interested in careers in power and a friend of the family who worked for WP&L had let me know about it.  It was the summer of 1992 and I found myself in a 10 story office tower in downtown Madison. There were about 8 interns who were all working in different parts of the company, I was in a group that attempted to calculate the best mix of power production to meet the needs of the utility’s customers.  The VP of the group, Don, had a nice office with a big wooden desk and a posh leather chair. We met every few weeks to talk about what I was working on.  

I spent my time trying to create the first spreadsheet that could calculate the cost of electricity from a cogeneration power plant.  Cogeneration is when you use the waste heat from generating power to produce heat for heating or cooling buildings. It uses more of the heat energy that would otherwise be expelled into the environment, and lowers the cost of electricity to the utility.  Cogeneration was just starting to catch on in the early 90s.

Rather than write a simulator in either of the two languages I was learning, Pascal and C, the chosen technology was macros in Lotus 123.  Lotus 123 was basically the Excel of the time. It turned out to be quite fun to build up a spreadsheet of inputs such as fuel costs, efficiency, etc and create a bunch of calculations that would output the cost of electricity from a potential plant.

At the end of the summer, Don scheduled a lunch to review how my summer went, he told me to pick the restaurant.  I chose Wah Kee Noodle, probably the best Chinese noodle shop in Madison which was in business for over 30 years before the owners retired in the summer of 2019.  Don had rarely been to Chinese restaurants so it was a new experience for him. We had a nice lunch, slurping a big bowl of noodles, where he asked a bunch of questions about my work, what I had learned that summer, and whether I would come back in the future.  I was starting my PhD program in the fall so I didn’t think I would return, but thanked him for the experience and for lunch. I still remember the genuine interest in my future Don showed even though I was just an intern passing through his office. Being an executive now, I realize what a great leader Don was in showing his team that everyone in his office mattered.

A few years later as I decided to exit my PhD program in Nuclear Fusion, I needed some work to fill in the gap between January and May of 1995.  Being part of the University of Wisconsin, my mom knew someone who was looking for a programmer to create the software to run a PLC network at the UW Biotron.  The Biotron is a building that houses climate controlled experiments usually on plants. The project to install the PLCs was going to automate the collection of environmental data that was being done by lab techs.  When it was finished, they would be letting go of two of the four lab techs. It was my first introduction to automation replacing humans in jobs. I heard that even 20 years later, the software and PLCs were still doing their jobs.

Even though the lab techs knew I was working on the system to replace them, the lead tech, Matt, made it clear to his team that they were to treat me like one of the team.  They were unionized and they would come find me at break time, and make sure I took my break. We would hang out in the break room and chit chat about the weather, which was generally below 0F, and how I managed to bike there every day.  

As the winter progressed and I began building my own protocol for the C based PLCs to talk to the master controller and relay their data, as well as build the control systems that would adjust the environment of each room, everyone stayed friendly.  Even when it became clear I would get the project done, no one showed me any animosity.  

At the end of the effort, Matt took me and the team out to lunch, I think it was at a Burger King, and given that I was building software to put a couple people on his team out of work, we had a surprisingly good time.  I really appreciated Matt’s kindness and leadership, it could have been a much more hostile environment under a different leader.  

These lunches were over 20 years ago, but I remember them well, much better than any lunches I’ve had in the past few years.  If you are a supervisor and you have an intern in your group, take them out to lunch, it will make a big impression on them that may last for their entire careers.

R.I.P. Kenilworth Trail

This may be slight deviation from normal, but the end of the Kenilworth Trail is coming today (May 13, 2019), which is a bike path that has featured highly in my life. Whatever you think of the light rail that will be built in the corridor for the next three years, it will destroy one of the most highly used bike paths in the US so that suburban commuters can get downtown faster and bypass Minneapolis. That’s what the light rail is doing, it’s routed through the far west side of Minneapolis through one of the least populated portions because it’s faster than taking it through Uptown and down Nicollet, which might actually serve some Minneapolitans.

I estimated today that I’ve ridden portions of the Kenilworth trail over 10,000 times in the last 20 years. How did I get to that number:

  • Bike commuted downtown for 13 years, 2x per day, ~200 times (yes I biked all winter for many of these years. ~5200 rides.
  • For 20 years, from April to November, I generally ride to do daily errands: trips to Vertical Endeavors, Target, Walgreens, Punch Pizza, Barnes and Noble, Whole Foods, Lunds, Byerly’s, etc. On weekends it’s probably 4 trips per day, weekdays and extra 0.5 trips per day. ~7290
  • For 10 years estimate 30 trips downtown to Twins games, fireworks, restaurants and brew pubs in the evenings per summer: ~300
  • Probably another 1000 rides for fun and exercise with the kids, dog and friends

So in the past 20 years, I’ve ridden portions of the Kenilworth trail approximately 13,790 times. That’s likely an underestimate if anything.

There’s a lot of memories on that trail. I used to rollerblade with a Burley with two kids (in the 2-6 range) in it around Cedar lake. One time I’m not sure what I was thinking but I was swinging the kids side to side for fun. I got a bit to aggressive and flipped the Burley on it’s side and proceed to crash myself. Luckily, the kids were fine, and I escaped with minor roadrash.

I used to run our dog, Callahan, down the path to get him some quick exercise. I would ride my bike, he would run alongside, he was very well behaved. One day though, as I was riding back down the path, I slipped the leash over my handlebar by accident. Callahan abruptly stopped to do some business, and in these cases I usually just dropped the leash. This time though when I dropped it, it caught the handlebar and pulled the front wheel sideways. I took a header over the bars and somehow landed it on my feet, jamming my hip and causing future arthritis. These were my only two crashes on the Kenilworth.

A number of friends and I would go brew pub hopping in the evening for many years. We’d ride to Northeast and hit a few pubs and ride back, often without lights. There’s nothing like riding the Kenilworth at night with no lights, it’s pitch black and you can barely see the path. One night, and I swear this is true, after turning off the path and passing the Cedar Lake South Beach around 1AM, there were about 30 naked teenage girls getting ready to go skinny dipping. It was like riding past Ulysses’s Sirens, except that there was one older lady standing under the streetlight with a clipboard, likely ensuring the Sirens earned their skinny dipping merit badges.

It’s funny but there are a lot more stories simply about riding and walking down an amazing urban bike path. These don’t exist in other cities, and now it will no longer exist in ours. I filmed my last ride down the Kenilworth, it’s not the greatest, but this is what is now gone:

On Learning New Things

Holiday is mostly past us going into the new year of 2019 which means we retailers are starting to have some time again.  The start of the year is when people think about making resolutions with the intent of improving themselves.  For the past ten years at least, my resolution has always been to “eat more pie.”   It’s an easy resolution to keep and it makes me happy to accomplish it every year.

It’s better to be more intentional about what you are learning, so skip the resolution and take some action.  In the last year I took a deep dive into Distributed Ledgers (DL, better known as Blockchains) to determine how we might use them at Target.  This turned out to be a much harder problem that it sounds, use you can use a DL for anything really, but it only makes sense in specific contexts where you are actually sharing data with external partners.

Second, I gave a presentation on technical culture rather than technology.  It was a first to talk about building engineering culture, even though it’s what I’ve done for most of my career.  But it was important to put Target on the map as a leading technical retailer.  Five minutes from the keynote is below, but there’s also a 45 minute talk if you have access to Pluralsight.

Outside work, I learned silver smithing and attained the rank of amateur.  I generally need something inside and outside work to keep me busy.  Plus, you can give the finished products away and it makes people happy.  An almost finished ring below:

Commerce Tomorrow Podcast

I recently had the opportunity to sit down with Kelly Goetsch and Dirk Hoerig to record an interview on Target’s engineering culture as part of their Commerce Tomorrow podcast.  Kelly was in town for the Open Source North conference which we were both speaking at.  We sat down in the Target recording studio to tape the show.  I’ve been using the studio to create an internal podcast for Target Engineering so we had an audio engineer and figured out how to include Dirk from Germany.

Click here to hear the podcast.

A Digital Ecommerce Transformation – Holiday – This Year is Always the Most Important One Ever – Part XII

Part XII of a multipart story, to start at the beginning goto Part 1.

Since we’re in the middle of Holiday 2017, I thought a digression on Holiday was in order.

If you have never worked in retail, than you’ve have missed out on the grand experience we call “Holiday”. On the other hand, you’ve probably actually enjoyed the time of year from mid-November to Christmas while you celebrate with your friends and family, and take advantage of thousands of days of deals from the many retailers trying to get their share of wallet from you.

Holiday, with a capital H, is something that has to be experienced to be believed. In my first Holiday at TWLER in 2010, I was on a team that had just started writing code and had very little in production leading into Thanksgiving. The only offering we supported was the failure site, if the main went down, we would quickly spin up the browse only site so consumers would be able to at least see what products we sold, and where our stores were located. In 2010, this was actually a pretty good thing since the ecommerce site was still less than 5% of revenue.

When you work in IT in a retailer, your entire year is judged on whether or not the systems you support survive the shopping onslaught of Holiday. In the online space, an ecommerce site might make 30% of its revenue in the five days from Thanksgiving to Cyber Monday. also experienced the third highest traffic of North American retailers during that time. This massive scale up to 20X normal daily traffic was largely accomplished without clouds in the 2000s. You had to take a really good guess as to how much infrastructure was needed, build it all out over the course of the year, and hope you weren’t overwhelmed by consumer behavior. You could easily receive 1M requests per second at the edge, and 100,000+ requests per second to your actual systems. If those requests were concentrated on the wrong systems, you could easily take down your site.

TWLER counts how long you’ve been at a company by the number of Holidays you’ve experienced. If someone asks how long you’ve worked there, you might say “four Holidays.” And every Holiday is the most important one yet, because those six weeks account for 50% or more of yearly revenue.

After a few Holidays, you realize the second the current year’s Holiday is over, you are immediately planning for the next one. There is no break. It’s like a giant tsunami that is slowly approaching, day by day. You can look over your shoulder and it’s always there, waiting to crash down on you and ruin your day. Once this year’s tsunami passes, you turn around and can see next year’s on the horizon.

In my six Holidays at TWLER, we experienced numerous outages, usually caused by either internal stupidity, or unexpected consumer behavior. In our first few years, we would purposely force our ecommerce site to use “enterprise” services because they were the “single source” for things like taxes, or inventory. This is a great notion, but only if the “enterprise” services were actually built to support the entire Enterprise. Since TWLER was store focused, this meant the “enterprise” services were often down at night for maintenance, or were not built to withstand massive surges in traffic. One million people refreshing a PDP to check for inventory on a big sale every few seconds quickly overwhelmed these services. So we often turned these services off and flew semi-blind, rather than have the site completely fail.

In other instances we tried to use various promotion functions embedded in our ATG commerce server. These seemed like useful things to easily setup a promotion like buy one get one. But when millions of people come looking for the sale, the vendor built commerce engines go down quickly by destroying their own database with the same exact calls, over and over again.  They hadn’t heard of caching yet, I guess.

We would sometimes publish our starting times for various sales, saying a big sale is starting at 11AM and send out millions of customer emails. The marketing teams loved the starting times and the technology teams hated them. We warned that setting a hard start time is a sure route to failure. Yet we did it multiple times and incurred multiple failures as the traffic surge brought down the site. There are physical limits even in clouds, you can only spin things up so fast and 10M rqs will bring down most sites. After a few of these episodes, we did convince the marketing teams that it wasn’t the way to go and learned how to have sales with gradual ramp-ups in requests rather than massive surges.

Around 2013, the Black Friday shopping was so intense in the evening across the nation that the credit card networks themselves slowed down. Instead of taking a few seconds to auth a credit card, it started taking one or two minutes. This was across all retailers. However, the change in time caused threads to hang up inside our ecommerce systems and all of a sudden we ran out of threads as they were all tied up waiting for payments to happen. For the next year, we changed our payment process to go asynchronous so that would never happen again.

There are many more stories of failure, but from every failure we learned something and implemented fixes for the next year’s wave. This is why Holiday in retail is such fun, every year you get to test your mettle against the highest traffic the world can generate. You planned all year, you implemented new technologies and new solutions, but sometimes the consumer confounds you and does something totally unexpected.

The last story is one where the consumer behavior combined with new features took us down unexpectedly. In 2014 we implemented “Save for Later” lists where you could put your items on a list that you could access later and add them to your cart. As Thanksgiving rolled around and the Black Friday sale went out at around 2AM, our Add to Cart function started getting pounded at a rate far higher than we had tested it for. We were seeing 100K rqs in the first few minutes the sale was happening, it rapidly brought the Add to Cart function to its knees and we had to take a outage immediately to get systems back together and increase capacity.

This was completely unexpected consumer behavior so what happened? It turned out that customers used the Save for Later lists to pre-shop the Black Friday sale and add all the things they wanted to buy into the lists. Then when 2AM rolled around, they opened their Save for Later lists and started clicking the Add to Cart buttons one after the other. A single customer might click 5-10 Add to Cart buttons in a few seconds. With hundreds of thousands of customers figuring out the same method independently, it led to a massive spike in Add to Cart requests, we effectively DDOSed our Add to Cart function with simultaneous collective human behavior.

I feel like I could keep going on Holiday for another two pages, but that’s enough for this year, maybe we’ll do it again in the all important next year.

Goto Part XIII

Internal Open Source Projects

What does it mean to start an open source project internal to an organization?  Does that make any sense?

Many large organizations have very large systems within them, systems which are mission critical to the delivery of their business model.  These systems are often bottlenecks for development as, in some fashion, they cannot be avoided and some development work is needed to complete any enterprise scale capability.  They limit the change available to an organization.

What if there were a way to unlock the capacity limit on these systems?  There is, open source the project.

If you open source a project internal to a company you are opening up the codebase for anyone in the company to work on.  Instead of supplying a dedicated development team, you now need a dedicated team of system stewards, people that ensure the stability of the system and that the code being added meets the criteria of the project’s sponsors.

You can now do this fairly easily with Git based source control, where anyone in the company could write a module or patch and submit a pull request.  The stewards review the pull request and whether the code takes them in the direction of their roadmap for the project and potentially accept the request into the main repo.

If done correctly you’ve opened up the system to the teams with the greatest need, while still maintaining control over the system and its direction.  If done incorrectly you’ll probably have the biggest mess of your life.  To push an entire enterprise forward at higher velocity the risk may be worth it.