I will be presenting an overview of BestBuy.com’s cloud architecture at the SATURN conference at the beginning of May. It is generally an academic oriented conference so I’m glad they decided to hear about a real-life case study. Attend if you can!
We will be part of the OpenStack Summit keynote in mid-April. We’ll be presenting about our internal cloud we’ve developed over the past year.
One interesting phenomena we’ve noticed on long running projects and teams (> 1 year), is that when we take the same people and make new mini teams to attack certain problems the results are suddenly better. The results seem to last as the team is disbanded and the individuals return to their previous roles, they suddenly have different ideas and a better overall view of the problems faced by their peers.
This has obviously been done for ages with “tiger” teams and secret task forces. However, it got me thinking about a different way to manage agile software development at large scale. Instead of creating 5 or 10 small teams with 6 or 8 people and managing a backlog over the course of years, you could take the same set of people and continually reconfigure teams to address a set of stories or defined epics. Thus, the teams would always be changing and no one could fall back into a comfort zone of a known set of people and interactions. I envision it as an extension to the idea of constantly switching pairs, only this time constantly switching up entire teams.
This likely sounds like hell to a certain set of people that value stability and the pride of working for long periods of time and knowing a certain area of software extremely well. There is also another set of engineers that would jump at the chance to continually be building new things with new people. Trying to find the healthy balance is the key.
Some new potential benefits would be that engineers are forced to build as simply as possible, knowing that new people would be picking up their code within weeks rather than years. It would also force more communication and a better understanding of the entire system on all participants.
I’m afraid to try this as I can also see it devolving into a giant mess where everyone is unhappy. It is also difficult to change up teams that have been together for a year and are hitting a serious stride in productivity. Studies on teams show that those teams that are together longer get more productive so this may be a terrible idea.
Still, the focus and discipline that comes from a small team knowing they only have two or three weeks/months to get a fixed set of work done has tremendous value. We’ll have to experiment with this a little more before taking it further.
As we remake BestBuy.com into a new platform, we are building a culture of architecture at the same time. Previous to 2010, BestBuy.com had no holistic architecture team guiding its development. Instead, a long series of projects simply bolted on more and more functionality until the resulting system was impossible to deterministically change. With little test and low regression coverage, any change in the system often resulted in unintended consequences.
In 2010 an architecture team was built and claimed ownership over BestBuy.com. We began to involve ourselves in projects that affected the site. We built a path and vision to remake the site into a next generation eCommerce platform. But over all, we established that architecture mattered, and agile architecture would be our culture. Our group of architects share similar architecture values, high involvement in development, decoupled flexible systems, TDD, small focused teams, high quality engineers, and letting architects lead projects rather than delivery managers.
The path of architecture has worked, teams with projects come find us now and we are involved with all aspects of the site. We are slowly working our way towards an infinitely scaling cloud/datacenter SOA. It is the architects who intervene when necessary, set engineering direction and mediate between all parties. To make it work, the culture of architecture must be in place first.
Innovation follows a pattern. If you know the pattern it can help you identify innovation occurring in your own company, it can also help you manage an innovate project. First of all, you identify the different phases of innovation and have knowledge on how to respond to difficult situations. It will also set your expectations correctly in regard to starting and running an innovative product development program.
The ideas behind the pattern of innovation are found in academic research published by Andrew Van de Ven in The Innovation Journey. Through long running study of numerous innovative product development cycles, Van de Ven and team have amassed a great deal of knowledge around how innovation works and how to manage it.
I’m reading Harvard Business Review again after taking a few years hiatus after finishing an MBA. There’s an excellent article in the Nov 2012 issue about innovation and work and having an additional operating system to manage innovation (ACCELERATE!, by John Kotter). This got me thinking about how a company should act more like a Cloud.
The notion in Accelerate! is that to innovate a company needs a second operating system running along side the normal hierarchical structure. This second operating system is a network of company appointed change agents along with volunteers that work for the cause. Essentially it allows workers trapped in their box of a hierarchy to work on innovative projects in their “spare” time. The networks spin up as need and as quickly dissipate back into the organization.
To me this sounds essentially like the same reasons Cloud computing has become so popular. Instead of having compute units (workers) with additional cycles available languishing within a hierarchy (datacenter), we build a network that can tap into those units. When we need to use them we throw a topic onto to the queue and see which compute unit picks up the work (master/worker).
This also greatly mirrors the Agile concept of self-organizing teams. The network teams have no hierarchical leader and therefore must determine how to do the work amongst themselves. This gives them both the power to decide and the mandate to implement their decisions at the same time.
If a company acted more like a Cloud than a hierarchy, who knows what it could do.
Purely based on circumstantial experiential data, Agile always dies in the Twin Cities. That being said, the same people pack their bags and move on together, or to the other current Agile refuges, to continue working the way they want to work. I’m terming this the pedal-pub Agile development model of the Twin Cities, the party is always moving on.
When I say Agile I’m not talking about the latest IT initiative to make all their development projects Agile. But the true grass roots, visionary leader who managed to convince their boss that Agile is the way to go. Agile meaning developer empowered self-managing teams where no project manger or VP is asking for a five page status report each week. Real Agile where everyone knows to just go look in the story tracking tool and see the burndown chart and what people worked on recently. Than that same VP/PM goes and asks why the velocity is down this iteration and how they’d rather have the changes to the REST commerce services than the text updates in the UI.
Why Agile Fails as a Process
The biggest problem with Agile over the last five years is that everyone is adopting it as the new great process to deliver software. Alistair Cockburn gave a whole talk on how Agile is dead a few years ago. The problem is when IT professionals get ahold of Agile they only take the process aspects and try to apply them. Since to me, Agile is a state of mind which happens to have some well known process elements that re-enforce this mindset, Agile solely as a development process always fails. You just have a bunch of developers that don’t care about processes or work environments enough to vote with their feet and move on to a truly Agile project.
Even when grass roots Agile exists, the existence is always ephemeral. Why is this Twin City developers? Because Agile processes are at their best with new development with lots of unknowns and new technologies being tried out. For your cookie cutter website builds agile works well, but as the project shifts into production the Agile team needs to shift into both a development and production support model. DevOps has appeared to help solve this problem but I don’t know of anyone in the Twin Cities doing DevOps yet.
Back to the failure pattern, whenever there is the need for a large program that involves complicated web development and lots of new technologies, business teams are willing to take a flyer on some dude spouting Agile development as the way to do it. Why are they willing? Because anyone who has tried to implement a huge program before knows that they are certainly going to fail if they try to do it in whatever fashion the organization currently uses, be it waterfall/agile in-house or handing off to some “world class” service provider. So having heard that Agile has saved someone else’s job, the business sponsors are willing to try anything to save their lives.
The Agile teams show up and assuming that they are well run deliver something of value. But then reality sets in and new bleeding edge technologies are now old and gray, the money has stopped flowing because everything is complete and works, and the organization spotlight moves off somewhere else. This is when Agile dies.
Waiting in the wings are all those managers and financial folks who swoop in and say “this project is out of compliance with corporate standards.” Since you’ve lost your high level sponsors, you have no recourse to change anyone’s minds and the Universal Development Process flavor-of-the-day must now be applied and the project also needs to meet the corporate mandate for 60% off-shore and blended rate of $64/hour. At this point you’ve had all your going away parties for the prima-donna (but really good) developers and are starting to wonder where to go next.
Best Buy is the Place
This is truly shameless touting of my current workplace but someone has to do it. Someday, and that someday is right now, we’re establishing an Twin Cities Agile culture that will last for years and potentially forever. Once Agile is established at one entire company the size of Best Buy, there will be no going back. I’m saying it, I’m doing it, BestBuy.com is going to be the first true Agile Engineering culture in the Twin Cities.
And if it’s not, text me the location of the pedal pub.
I’m not afraid to admit it, I have weird kids. I was a weird kid too so it is not surprising that my own are considered weird as well. How do I know they are weird? Lots of meetings with various school specialists in three different districts. My kids have been mentally poked and prodded for years trying to figure out why kids that tested out as very intelligent, were having such a hard time completing their school work. One of the psychological assessors tried hard to find words to describe their test results once and ended up with “unique.” Basically, the message was they don’t fall into any of the known psychological categories so their was nothing they could actually do to help them.
Anyway, now that we’ve established they are weird, let’s get on to KhanAcademy ruining bedtime. In 4th grade, my son Alistair was finally challenged in math class (mentally challenged that is, not just challenged to get his work done). We had moved him to his third school in 5 years as each of the previous schools were unable to get through to him. (He’s in Minnetoka Minnesota school district now which is the most phenomenal school district ever in addressing individual kid’s needs).
But he was finally challenged, and because of that he started asking me questions about math at bedtime. At first it was a couple simple questions and then we moved on to bedtime stories. After a few weeks he said “Let’s just talk about math instead of stories tonight.” Wow! As a engineer it was like music to my ears. So we talked about math most nights for a number of months. I would always ask him whether he wanted to talk about something new or something he was working on. New stuff was introducing summations, probability functions, imaginary numbers, and other fun topics for 4th graders.
After a few months of this, we were introduced to KhanAcademy. I thought this was one of the coolest sites every and quickly got addicted to the math section myself. But to get my son’s interest I had to pay him to work on KhanAcademy for a few weeks until he became addicted himself. That’s when the problems started.
The math questions stopped coming at bedtime anymore. After a few days I asked why we weren’t talking about math anymore? He said he didn’t need to ask anymore because he could just ask KhanAcademy. KhanAcademy is apparently better at teaching math concepts than I am.
Bedtime had become less fun.
Fast forward one year later. I found a piece of paper on his desk with chemistry equations written on them. I love chemistry as well and asked him if he had any questions about stoichiometry? He said no, he was learning it from KhanAcademy.
Damn you KhanAcademy, you’ve ruined bedtime!
Learning to Ride a Fixie
One aspect of riding a fixie that was unexpected was that I discovered the little switch in my head. While first riding a fixie, a little signal went off every time I thought I should shift gears. On my geared bikes I simply shifted gears when that happened and never noticed the signal. However, without any gears to shift, I noticed the signal because I couldn’t complete the action required. My built in instincts for bicycling were no longer relevant.
Coasting can be Painful
Even more exciting was finding out how often I coasted when riding a freewheeled cycle. When first learning to ride fixie, one of the hazards to one’s knees is that every time you try to coast, the pedals attempt to separate your knee joint. Since the pedals don’t stop, the coasting switch can be quite painful. After a few knee jolts, that signal is overridden quite effectively.
Braking no longer Necessary
There’s one more switch in my head for braking, however, with a fixie you can brake just by pedaling slower. Since the pedals are directly attached to the rear wheel, your pedal cadence is directly related to your velocity. A slower cadence means you go slower. To slow down on a fixie you simply slow down your pedal cadence. But since the braking switch is wired to your hands you have to overcome the urge to hand brake and instead slow your legs down. It is fascinating how you can rewire your own brain.
Results are Better
The overall result once you’ve managed to turn off the switches in your head are a much more intimate relationship with the bicycle. You no longer have to do anything but pedal, to go fast pedal fast, to go slow pedal slow. The connection is absolute and immediate. Now, you just think fast or slow and your legs do all the work.
Connection to Software
The process of learning to ride a fixie is the same as learning a new technology. Switching from procedural to OO coding styles requires you to rewire your instincts. There is a large change in thinking when going from RDBMS land to NoSQL stores. When learning about serious scaling, such as supporting the traffic at Best Buy, some of those things you thought you knew no longer apply. But you often don’t notice the rewiring process because there is no immediate painful feedback when you do it wrong.
Learning to ride a fixie helps you recognize the little switch in your head that tells you to do something that’s become hard wired. When it is no longer the correct action, fixie riding gives you immediate negative feedback. The cognizance of the little switch in my head has helped me extensively when creating the new architecture of BestBuy.com. When the switches go off around architectural notions, I can feel them happening and can take a mental step back and evaluate whether it is actually still the right switch to flip. If it is, than no action is needed, if not then the process of rewiring starts again.
So buy a fixie and start riding, it will help your software architecture. However, since it is actually a dangerous thing to do without a little background, review those dangers first here.