If you’re a Riak fan or interested in innovation at large companies I’ll be speaking at Ricon 2012 in October. Riak is a great tool and is one of the new breed of NoSQL systems currently in use at Best Buy. We’ve done a lot of great things on Riak so come to my talk at the conference to learn more.
I’m not afraid to admit it, I have weird kids. I was a weird kid too so it is not surprising that my own are considered weird as well. How do I know they are weird? Lots of meetings with various school specialists in three different districts. My kids have been mentally poked and prodded for years trying to figure out why kids that tested out as very intelligent, were having such a hard time completing their school work. One of the psychological assessors tried hard to find words to describe their test results once and ended up with “unique.” Basically, the message was they don’t fall into any of the known psychological categories so their was nothing they could actually do to help them.
Anyway, now that we’ve established they are weird, let’s get on to KhanAcademy ruining bedtime. In 4th grade, my son Alistair was finally challenged in math class (mentally challenged that is, not just challenged to get his work done). We had moved him to his third school in 5 years as each of the previous schools were unable to get through to him. (He’s in Minnetoka Minnesota school district now which is the most phenomenal school district ever in addressing individual kid’s needs).
But he was finally challenged, and because of that he started asking me questions about math at bedtime. At first it was a couple simple questions and then we moved on to bedtime stories. After a few weeks he said “Let’s just talk about math instead of stories tonight.” Wow! As a engineer it was like music to my ears. So we talked about math most nights for a number of months. I would always ask him whether he wanted to talk about something new or something he was working on. New stuff was introducing summations, probability functions, imaginary numbers, and other fun topics for 4th graders.
After a few months of this, we were introduced to KhanAcademy. I thought this was one of the coolest sites every and quickly got addicted to the math section myself. But to get my son’s interest I had to pay him to work on KhanAcademy for a few weeks until he became addicted himself. That’s when the problems started.
The math questions stopped coming at bedtime anymore. After a few days I asked why we weren’t talking about math anymore? He said he didn’t need to ask anymore because he could just ask KhanAcademy. KhanAcademy is apparently better at teaching math concepts than I am.
Bedtime had become less fun.
Fast forward one year later. I found a piece of paper on his desk with chemistry equations written on them. I love chemistry as well and asked him if he had any questions about stoichiometry? He said no, he was learning it from KhanAcademy.
Damn you KhanAcademy, you’ve ruined bedtime!
Learning to Ride a Fixie
One aspect of riding a fixie that was unexpected was that I discovered the little switch in my head. While first riding a fixie, a little signal went off every time I thought I should shift gears. On my geared bikes I simply shifted gears when that happened and never noticed the signal. However, without any gears to shift, I noticed the signal because I couldn’t complete the action required. My built in instincts for bicycling were no longer relevant.
Coasting can be Painful
Even more exciting was finding out how often I coasted when riding a freewheeled cycle. When first learning to ride fixie, one of the hazards to one’s knees is that every time you try to coast, the pedals attempt to separate your knee joint. Since the pedals don’t stop, the coasting switch can be quite painful. After a few knee jolts, that signal is overridden quite effectively.
Braking no longer Necessary
There’s one more switch in my head for braking, however, with a fixie you can brake just by pedaling slower. Since the pedals are directly attached to the rear wheel, your pedal cadence is directly related to your velocity. A slower cadence means you go slower. To slow down on a fixie you simply slow down your pedal cadence. But since the braking switch is wired to your hands you have to overcome the urge to hand brake and instead slow your legs down. It is fascinating how you can rewire your own brain.
Results are Better
The overall result once you’ve managed to turn off the switches in your head are a much more intimate relationship with the bicycle. You no longer have to do anything but pedal, to go fast pedal fast, to go slow pedal slow. The connection is absolute and immediate. Now, you just think fast or slow and your legs do all the work.
Connection to Software
The process of learning to ride a fixie is the same as learning a new technology. Switching from procedural to OO coding styles requires you to rewire your instincts. There is a large change in thinking when going from RDBMS land to NoSQL stores. When learning about serious scaling, such as supporting the traffic at Best Buy, some of those things you thought you knew no longer apply. But you often don’t notice the rewiring process because there is no immediate painful feedback when you do it wrong.
Learning to ride a fixie helps you recognize the little switch in your head that tells you to do something that’s become hard wired. When it is no longer the correct action, fixie riding gives you immediate negative feedback. The cognizance of the little switch in my head has helped me extensively when creating the new architecture of BestBuy.com. When the switches go off around architectural notions, I can feel them happening and can take a mental step back and evaluate whether it is actually still the right switch to flip. If it is, than no action is needed, if not then the process of rewiring starts again.
So buy a fixie and start riding, it will help your software architecture. However, since it is actually a dangerous thing to do without a little background, review those dangers first here.
Architect Driven Development, or ADD (just add the H in the appropriate place for most of us), is my new methodology of choice. In the large Enterprise context, where Agile is difficult to implement across many teams, ADD hits the gap by positioning the Architect to enforce the development methodology as well as the software architecture. Let’s face it, as Architects we end up being the people that, at least at first, are ensuring the development methodology is followed.
In the Enterprise context where there are numerous teams spread across many mini-empires, one of the few roles that can be consistent is the Architect. In my role at Best Buy, I run a 70 person development team. That team is broken into eight different workstreams all lead by top tier architects. As long as the architect team is aligned on both a methodology and high level architecture goals, we get consistent results from our teams. It helps a lot that we all believe in Agile self-managed teams as it gives the developers the freedom to do the work how the team sees fit. But the architects carry the consistency through the development process from start to finish.
There are also a couple hundred developers working on various parts of BestBuy.com that are not under architect control yet. These teams are largely driven by outsourcers who have a vested interest in keeping Best Buy architects away from their work. If we were there, they wouldn’t get to staff their $200/hr architect on the team or their 10 offshore counterparts. The problem here is the outsourced architect is not on board with our development methodology or high level goals, and given our structure there’s little incentive for them to even care.
So as we strategized how to get better control over the dotcom platform, I put forth that as long as we controlled the architects and the process, nothing else mattered. Inherent in that control is control of resources as well. Since our hiring standards are tough, we only get top notch developers which makes everything else easier. But without a strong architect voice to push a methodology, pick resources and keep the architecture vision, the process would fail.
Next time you are thinking about restructuring your development environment, stop thinking about what methodology you’re PMO is going to use and start thinking about getting strong technical architects to carry the vision and methodology. You’ll find that the rest of the process falls into place afterwards.
I just did my first video interview about what BestBuy.com is trying to build and how we are trying to build it. It was fun but I came out of it thinking about the overall strategy Best Buy has applied toward technology for the last 10 years. In a nutshell, and this is certainly not new information, Best Buy has decided that thinking is the most important part of technology strategy. Therefore, we only staff managers who think about business problems, and farm out the doing to the cheapest vendor we can find. If you ever try to do anything technology related at Best Buy, you have to have a big IT vendor do it for you.
The problem with this attitude is that no value is placed on the technical solution. When technology is driving the base strategies of so many eCommerce companies, it is hard to understand why we don’t place value on technology. Last year alone we taught a number of our vendors how to build elastic systems in the Cloud. When our projects were over, all those people we taught went off and built elastic systems in Clouds for other companies, likely our competitors as well.
Not only Think. Not only Do. Think and Do.
If you only staff Think you are reliant on someone else to Do. You are giving away you’re strategies and technologies. Most importantly, you’re giving away your minds that have learned new technologies and new strategies.
If you only staff Do you may pursue interesting technological pursuits, but their business value may be suspect. From a tech person standpoint, just staffing Do is very seductive. I’ve been on plenty of teams where the implementing team knew the business requirements better than the business.
Think and Do. Tech and business teams highly coupled and dependent on each other can produce amazing results. One of my favorite projects was with a highly engaged business customer who took a high functioning Agile team in directions we never would have expected. The results were superior to anything the tech team could imagine.
But the best of the best comes from Think, Do, and let the Doers think too. This is the core of the Agile process where decisions are distributed to the people that know the system best.
Think, Do, and let the Doers think too.
I’ve hired over 100 developers and architects throughout the years. I’ve worked at around 20 companies, some very briefly, as both an employee and consultant. I like fast development teams that ride the fine line between excellence and chaos. But I’ve recently come to understand that a lot of my attitudes around being a developer in the Twin Cities are vastly different than those of our West Coast brethren. This post is full of vast generalizations so your particular situation may not apply, bear with.
Twin Cities Career Path
As a software developer in the Twin Cities, the general path is to get your foot in the door anywhere that will hire you. Small companies, large companies, whoever will pay you to learn enterprise scale, team oriented development. Honestly, your first 5 years as a developer probably cost the company you work for more money to have you there than your worth in business value. Even so, you’ll add the most value at a small Agile shop where you get to do everything from development to deployment to maintaining the Cloud infrastructure. Your learning experience at these places will set you up for your next 10 years of work.
Anywhere after your 5 year mark you get the choice to become a career developer. Around here is where all the people that can’t really cut it become PMs or BAs or some other manager. This is a fine choice as the life of development can be tough. But the choice for developers is to continue to work as a cog in the corporate machine or to become an independent or sponsored (employee at a body shop) consultant.
When you go the employee route you get perceived stability in exchange for little control of your own destiny and work. Due to the nature of financial funding of development projects, employees are often relegated to maintaining existing systems or as the token employee or two on the new development projects staffed by contractors. You’re only on these projects because they want someone to maintain the system after all the consultants leave.
Why is this the case?
Budgets at corporations split development into two buckets, capital and expense. Projects that can be capitalized allow the company to depreciate the cost of the development which saves them money on taxes. Projects that are expense go right to the bottom line of the budget and are a drain on that division’s profitability. From a bean counter’s perspective, adding flexible staff to write new development projects makes the most sense as the cost of consultants is higher than the cost of full time staff. That higher cost then gets depreciated and the ongoing maintenance expense goes to cheaper full time or offshore resources.
Why does this limits the career path in Minneapolis?
What this means to you as a developer is that, since your value is lower to the company as a pure expense, the amount company’s are willing to pay you is limited. Thus the salary of a full time developer in the Twin Cities is capped not far north of $100,000.
As a consultant, you can make significantly more with a similar amount of experience. The rates vary greatly depending on your skill set and experience level but rates to the consultant between $80 and $120 per hour are fairly common. Also, when a company pays this much for development resources, you are more likely to land a new green field project as the project is capitalized (and can be depreciated).
Given the general climate of Twin Cities development, the more aggressive and generally higher quality developers go into consulting once they realize the limitations of working in the Twin Cities. But, while the pay is better the career path is over at this point and one risks being a developer hopping from contract to contract for one’s whole career or until your skillset is no longer valued.
West Coast Difference
Having not lived there, I’m now going on conversations with the west coast engineers that I know. In general, they’ve described the opposite situation as exists in the Twin Cities, where the vast majority of developers stay full time employees and the number of contractors is limited. Also, that the best development talent is always scooped up by the numerous tech companies in the area at high salary plus stock options. Plus, the culture of the startup breaking out is always there and quite tangible. The latest story being Instagram. The Instagram writeup in the New York Times drives home the point of what is wrong with the Twin Cities. This excerpt sums it up:
The extraordinary success of Instagram is a tale about the culture of the Bay Area tech scene, driven by a tightly woven web of entrepreneurs and investors who nurture one another’s projects with money, advice and introductions to the right people. By and large, it is a network of young men, many who attended Stanford and had the attention of the world’s biggest venture capitalists before they even left campus.
What is wrong with the culture of the Twin Cities tech scene? These structures just plain do not exist here and that is the main culprit behind the exodus of top talent in the Twin Cities to mercenarial endeavors. With all those smart minds going to short term financial reward, there will be no culture of risk taking, the financial availability won’t appear and the notion that ideas for the next billion dollar startup are ubiquitous will never materialize.
How we can start this in the Twin Cities will be difficult. It involves getting the venture capitalist and angel’s around here to start taking chances on smart college kids and catching them before they hit the Twin Cities career path. Once they are in, our culture will take over and they’ll be consultants before you know it. And the next generation of risk takers will be lost.
Yes, I work at Best Buy and despite our recent stumbles (CEO resigns amid personal conduct probe, big miss of earnings in 2011, forgetting to deliver products by Christmas, everyone saying we’ll go out of business slowly, than quickly) I’m quite proud of the company. As the Chief Architect of the .com website, we definitely have our work cut out for us over the next few years.
However, what bothers me is the feeling that many people seem ecstatic over the potential demise of the last big box electronics retailer. While you may think our customer service is bad and that driving to a store and finding the item you wanted is out-of-stock (even though the website said the store had it in stock) is extraordinarily annoying, if Best Buy were gone it would put 180,000 or so people out of work and another big hole in Minnesota’s economy.
Even if you don’t care about that, hopefully you care about social engagement and corporate social responsibility. One thing I can say after two years at Best Buy is that, as a whole, Best Buy has its heart in the right place. We may fail but we try hard to put our customers first, we care about the employee environment, and we care about the communities in which we do business.
Just to list a few CSR items that I’m aware of:
- We recently completed a complete audit of our supply chain for labor abuses.
- We collect used electronic equipment for free at all stores.
- We donated $5 million to Fukushima relief along with millions more donated to other causes.
- We sent hundreds of care packages to our U.S. military last December.
And you don’t have to believe me, Forbes was impressed with our efforts if not all the results. Or take a look at our Annual Sustainability Reports. While we’re not to the point of triple bottom line reporting yet, we’re doing a lot more than paying lip service to sustainability.
Finally, it’s just a cool place to work; for the first time ever, my kids think I have a cool job. Having worked at over 20 different organizations, I’ve also experienced many substandard environments. Best Buy is highly employee centric from ROWE to the general environment. The campus was created to promote networking and the general buzz around campus is energetic. Despite our woes people are engaged and aggressively trying to work our way out of this current trough. You can argue whether the strategies we’re embarking on are valid but you can’t argue with the will to survive and thrive.
So while there may be more efficient ways to deliver electronics to the market and many find the push to upsell everyone to protection plans and buy back reprehensible, if Best Buy disappeared it would be one more data point that treating your employees like adults is a failing strategy. Just one less reasonable workplace in the world hopefully makes you at least a little sad.
Keeping Up with the Speed of the Outside World
In a world where anyone or any business unit within a company can build an application on Google App Engine, Heroku or AWS, offering only heavy weight web development as the answer to the business needs is clearly inadequate. Just as individuals forced the switch from Blackberry to iPhone in the enterprise, a similar shift is coming in web application development.
Groups have always bypassed IT based development as too slow and too expensive. Every Fortune 100 company I’ve worked in has hired development teams outside IT to get crucial work done in a timely manner. The better companies acknowledge this and try to at least train the outside developers in secure coding and process for their enterprise. These outside groups were often stymied because they eventually had to deploy in the company datacenter for data access turning that quick 2 month project into a bitter 10 month project. But the real problem is the mismatch between expectations driven by innovation outside the business, and the ability to execute within the business.
Bridging the Gap
The key for development service providers within the organization is to offer capabilities similar to what’s available externally yet customized to the unique information and systems available to the organization. Basically, we need to offer Heroku to our internal business units to build applications, while also giving them APIs into all the enterprise data, content and transactional systems. Thus the need for web development Platforms within companies themselves.
The right internal platform offers several key capabilities:
- Transparent operations and elastic scalability
- SDKs and SLAs for UI and Service components
- Built-in monitoring and reporting dashboards
- Smart data caching
- Simple UI driven interfaces
- As far as possible, technology agnosticism (e.g. Java, Ruby, .Net should all be viable on the platform)
- Security from common attack patterns (XSS,SQL Injection, etc.)
- Metering and throttling
- Priority for run the business applications
- Automated failover
- Disaster Recovery
What this is Not
We are not talking about an internal cloud, ESB or web portal. We are talking about an internal development platform that likely sits on an internal (or external) (or both) cloud. By being on the platform and developing as a registered user, you gain instant access to all internal services and data sources. Where these are and who runs them hardly matters if they all have defined SLAs that are met. Maybe there is an ESB back there somewhere but essentially no one cares if their service calls are managed correctly.
Unshackling the Business
The whole point of a flexible platform that barely dictates and mostly enables is to allow the business folks to get a little crazy. In the past development was often the voice of reason guiding crazy ideas towards implementable ideas given time and money constraints. Those days are over, it is time to let the business go unfettered by reason. At today’s pace of change, the crazy ideas are the ones that may catch on and the platform of the future will allow those innovations to occur. As technologist supporting business in this new reality, building internal platform capabilities is best way to meet those needs.
Platform Architectures are the next change coming to Web applications, though plenty have existed up to now. Salesforce.com is a prime example of a pure platform architecture but Facebook, Apple, Amazon and Google all are Platforms as well (as described in Phil Simon’s book The Age of the Platform).
However, for the first post let’s talk about my favorite platform, and to me that’s the bicycle. What makes the bicycle a platform? You can change it to fit your current needs and desires, all its component parts are individually upgradeable, it uses a common set of standards for all these parts, and it is up to your will or imagination to make the most of it.
A bicycle allows you to do more than you could before you had the bicycle. You can now get from here to there faster. You can meet your friends and go for a ride. You can work on it in your spare time. But ultimately, you decide what works best for you, and when to use (or not use) the platform (bicycle). That’s the essence of the new breed of platform, the customer will decide whether or not it is useful to them.