Category Archives: Software Development

Open Source North, A Great Start for a Tech Conference in MN

The OpenSourceNorth conference put on this weekend by Solution Design Group was a rocking success.  Local design, local speakers and just a little local beer made for a good time.

IMG_2174Best Buy had a recruiting booth at the conference and hopefully you had a chance to stop by and chat either with a recruiter or an engineer.  If not check out the jobs located here.

osn_pic2I had a chance to give a new presentation on how and why engineers should be engineering managers, at least a few of them.  If we ever want more great places for engineers to live and work, someone needs to create them.  Only individuals that understand how software engineering really works can create those places.  Here’s a picture from the conference.

If you couldn’t make it this year, put it on your calendar for next year.

Revolutionary’s Revenge

Sometimes I think I studied the wrong field in college, I should have been a political scientist or a historian. Having studied in those fields it may have helped me understand that the worst thing that can happen to a revolutionary is to win the war and end up governing.

A true revolutionary is not doing what they do because they are seeking power, they are championing an idea, a culture or a different way of life. They believe it; to the point where they’ll risk their lives in a real war, or their professional career if they work in a corporation. We’ll stick to corporate revolutionaries for now, it’s a little bit safer.

The thing to remember is that most revolutions fail, which is great because the corporate revolutionary can move on to the next place with their ideals intact and their pride unblemished. They haven’t had to face real decisions that would affect how they view themselves and their core beliefs.

But what if the unthinkable happens and the revolution succeeds? Now you’ve got a bunch of idealistic, stubborn, socially awkward and politically unsavvy people running your organization. No one wants that, you’ll be replaced by “real managers” the second you make a big mistake.

First, believe that a powerful force is waiting on the sidelines to determine whether the result of the revolution is a new paradigm that improves the old, or a purveyor of ideals that must be snuffed out quickly lest they infect the entire organization. As the new governing body, you need to exude ineptitude while clandestinely shoring up political position and making some new friends of old enemies. Once you’ve established a culture and moved systems beyond the point of no return, you can show your true hand, your competency, and institute the revolutionary ideals that were the driver of all the change.

This is where reality sets in. Running a revolution and governing a semi-stable organization are two distinct beasts. In revolutionary times, it’s us against them, there’s always a big enemy out there to drive camaraderie and take the brunt of negative emotions as they crop up. However, once the enemy is gone, the emotions turn inwards and now you are left arbitrating disagreements between former allies and making decision you know are bad, but are necessary to prevent chaos.

The skills of the revolution are not the skills of maintaining and building incrementally on top of established practices. If you cannot make the shift to sustaining governance, everything you fought for will eventually collapse and regress to the mean. Temperance, consensus, collaboration, compromise are all words that would cause the revolution to fail, but are necessary to sustain the new system over time. Operating in the same environment for multiple years is massively more difficult than moving to new gig every year. It is also massively more rewarding to see a system build, grow and become the new normal. After five years it is firmly in place and no longer can be replaced, that is, until the next revolutionary comes along…

Internal Open Source Projects

What does it mean to start an open source project internal to an organization?  Does that make any sense?

Many large organizations have very large systems within them, systems which are mission critical to the delivery of their business model.  These systems are often bottlenecks for development as, in some fashion, they cannot be avoided and some development work is needed to complete any enterprise scale capability.  They limit the change available to an organization.

What if there were a way to unlock the capacity limit on these systems?  There is, open source the project.

If you open source a project internal to a company you are opening up the codebase for anyone in the company to work on.  Instead of supplying a dedicated development team, you now need a dedicated team of system stewards, people that ensure the stability of the system and that the code being added meets the criteria of the project’s sponsors.

You can now do this fairly easily with Git based source control, where anyone in the company could write a module or patch and submit a pull request.  The stewards review the pull request and whether the code takes them in the direction of their roadmap for the project and potentially accept the request into the main repo.

If done correctly you’ve opened up the system to the teams with the greatest need, while still maintaining control over the system and its direction.  If done incorrectly you’ll probably have the biggest mess of your life.  To push an entire enterprise forward at higher velocity the risk may be worth it.


I’m In the Library!

This may be a bit self-promotional (but then it is a blog) but I made it into the library!  Wow, you say.  Well let’s step back to childhood and growing up in an academic family where the measure of worth was not just how many Bachelors degrees a person had, that was small potatoes, but how many PhDs, law or medicine degrees a person had.  In my extended family, two or more was the norm.

So imagine the horror when I dropped out of the PhD program for Nuclear Fusion at the University of Wisconsin.  Yes, I am a failure at life.

Fast forward 20 years or so and I just wrote an article for IEEE Software magazine.  When I was writing it I didn’t realize that IEEE Software is one of those journals that appears in libraries in Universities around the world.  However, I discovered that today!  I have an actual citation!  I’ve finally done something the family can be proud of!

Screen Shot 2014-06-07 at 12.22.29 PM

The Art of Large Systems

I was discussing art with my daughter, someone who is extremely talented at what most people consider art.  That is, being able to draw and paint things that look amazing and you know you could never do yourself, not in a million years.

In any case, she made the comment that all artists hate their work.  This is an intriguing statement because if it were true than there is no incentive to actually make art.  If you know you’ll hate it what’s the point.  With further questioning we clarified that the statement really means an artist is never happy with the outcome.  This is far more reasonable, nothing ever turns out like the perfect image you have in your head for how something should be.  Try as you might, you know if you could just figure out how to get there, the piece would be infinitely better.

Now this makes sense to those of us who work on large dynamic constantly changing systems.  Systems that are constantly under varying stressors such as high volumes of traffic appearing in ways that were not anticipated, frequent code releases that can cause unintended consequences, and the connection to numerous other systems that often go awry.

All of these things are done in code, and we all know that if we could just puzzle it out, there’s a better way to construct this system; something which is eminently simple.  Maybe we’re using Tinker Toys when we should be using Lincoln Logs.  There’s a seismic shift that can happen if we could just force our brains to make a jump, we can often feel it out there waiting to be discovered.   But the reality of having a job and a deadline kick in and we have to deliver something that works.  Perfection not achieved, again.

When writing code you get used to this feeling because you are delivering every day, often on a long timeline and things just have to get done, sometimes badly.  We’re not happy about it but the world moves on.

Looking back at the large systems you’ve built you can always point out the things you wish you could change.  Sometimes you get the opportunity to refactor them, possibly finding out the idealized new architecture actually was worse than the original system.  Sometimes your amazing ideas fall down when confronted with the complexity inherent in large systems.  However, sometimes the new system is fantastic, there’s just some things that could still be better…

It is hard to love the outcome, it’s the child that rebelled and ran away from home after stealing all your money and taking your car.

Creating large systems is essentially a form of art.  There’s no defined methods to ensure a positive outcome.  For works like, the work is in the public domain and constantly being judged by individuals and the media.  Some people love it, some hate it, but everyone has an opinion.  And, finally, you can sell it (well technically the company could sell it).



The Never Permanent Team

One interesting phenomena we’ve noticed on long running projects and teams (> 1 year), is that when we take the same people and make new mini teams to attack certain problems the results are suddenly better.  The results seem to last as the team is disbanded and the individuals return to their previous roles, they suddenly have different ideas and a better overall view of the problems faced by their peers.

This has obviously been done for ages with “tiger” teams and secret task forces.  However, it got me thinking about a different way to manage agile software development at large scale.  Instead of creating 5 or 10 small teams with 6 or 8 people and managing a backlog over the course of years, you could take the same set of people and continually reconfigure teams to address a set of stories or defined epics.  Thus, the teams would always be changing and no one could fall back into a comfort zone of a known set of people and interactions.  I envision it as an extension to the idea of constantly switching pairs, only this time constantly switching up entire teams.

This likely sounds like hell to a certain set of people that value stability and the pride of working for long periods of time and knowing a certain area of software extremely well.  There is also another set of engineers that would jump at the chance to continually be building new things with new people.  Trying to find the healthy balance is the key.

Some new potential benefits would be that engineers are forced to build as simply as possible, knowing that new people would be picking up their code within weeks rather than years.  It would also force more communication and a better understanding of the entire system on all participants.

I’m afraid to try this as I can also see it devolving into a giant mess where everyone is unhappy.  It is also difficult to change up teams that have been together for a year and are hitting a serious stride in productivity.  Studies on teams show that those teams that are together longer get more productive so this may be a terrible idea.

Still, the focus and discipline that comes from a small team knowing they only have two or three weeks/months to get a fixed set of work done has tremendous value.  We’ll have to experiment with this a little more before taking it further.

A Company Should Act Like a Cloud

I’m reading Harvard Business Review again after taking a few years hiatus after finishing an MBA.  There’s an excellent article in the Nov 2012 issue about innovation and work and having an additional operating system to manage innovation (ACCELERATE!, by John Kotter).  This got me thinking about how a company should act more like a Cloud.

The notion in Accelerate! is that to innovate a company needs a second operating system running along side the normal hierarchical structure.  This second operating system is a network of company appointed change agents along with volunteers that work for the cause.  Essentially it allows workers trapped in their box of a hierarchy to work on innovative projects in their “spare” time.  The networks spin up as need and as quickly dissipate back into the organization.

To me this sounds essentially like the same reasons Cloud computing has become so popular.  Instead of having compute units (workers) with additional cycles available languishing within a hierarchy (datacenter), we build a network that can tap into those units.  When we need to use them we throw a topic onto to the queue and see which compute unit picks up the work (master/worker).

This also greatly mirrors the Agile concept of self-organizing teams.  The network teams have no hierarchical leader and therefore must determine how to do the work amongst themselves.  This gives them both the power to decide and the mandate to implement their decisions at the same time.

If a company acted more like a Cloud than a hierarchy, who knows what it could do.



The Little Switch in my Head

Learning to Ride a Fixie

One aspect of riding a fixie that was unexpected was that I discovered the little switch in my head.  While first riding a fixie, a little signal went off every time I thought I should shift gears. On my geared bikes I simply shifted gears when that happened and never noticed the signal.  However, without any gears to shift, I noticed the signal because I couldn’t complete the action required.  My built in instincts for bicycling were no longer relevant.

Coasting can be Painful

Even more exciting was finding out how often I coasted when riding a freewheeled cycle.  When first learning to ride fixie, one of the hazards to one’s knees is that every time you try to coast, the pedals attempt to separate your knee joint.  Since the pedals don’t stop, the coasting switch can be quite painful.  After a few knee jolts, that signal is overridden quite effectively.

Braking no longer Necessary

There’s one more switch in my head for braking, however, with a fixie you can brake just by pedaling slower.  Since the pedals are directly attached to the rear wheel, your pedal cadence is directly related to your velocity.  A slower cadence means you go slower.  To slow down on a fixie you simply slow down your pedal cadence.  But since the braking switch is wired to your hands you have to overcome the urge to hand brake and instead slow your legs down.  It is fascinating how you can rewire your own brain.

Results are Better

The overall result once you’ve managed to turn off the switches in your head are a much more intimate relationship with the bicycle.  You no longer have to do anything but pedal, to go fast pedal fast, to go slow pedal slow.  The connection is absolute and immediate.  Now, you just think fast or slow and your legs do all the work.

Connection to Software

The process of learning to ride a fixie is the same as learning a new technology.  Switching from procedural to OO coding styles requires you to rewire your instincts.  There is a large change in thinking when going from RDBMS land to NoSQL stores.  When learning about serious scaling, such as supporting the traffic at Best Buy, some of those things you thought you knew no longer apply.  But you often don’t notice the rewiring process because there is no immediate painful feedback when you do it wrong.

Learning to ride a fixie helps you recognize the little switch in your head that tells you to do something that’s become hard wired.  When it is no longer the correct action, fixie riding gives you immediate negative feedback.  The cognizance of the little switch in my head has helped me extensively when creating the new architecture of  When the switches go off around architectural notions, I can feel them happening and can take a mental step back and evaluate whether it is actually still the right switch to flip.  If it is, than no action is needed, if not then the process of rewiring starts again.

So buy a fixie and start riding, it will help your software architecture.  However, since it is actually a dangerous thing to do without a little background, review those dangers first here.

Architect Driven Development

Architect Driven Development, or ADD (just add the H in the appropriate place for most of us), is my new methodology of choice.  In the large Enterprise context, where Agile is difficult to implement across many teams, ADD hits the gap by positioning the Architect to enforce the development methodology as well as the software architecture.  Let’s face it, as Architects we end up being the people that, at least at first, are ensuring the development methodology is followed.

Why ADD?

In the Enterprise context where there are numerous teams spread across many mini-empires, one of the few roles that can be consistent is the Architect.  In my role at Best Buy, I run a 70 person development team.  That team is broken into eight different workstreams all lead by top tier architects.  As long as the architect team is aligned on both a methodology and high level architecture goals, we get consistent results from our teams.  It helps a lot that we all believe in Agile self-managed teams as it gives the developers the freedom to do the work how the team sees fit.  But the architects carry the consistency through the development process from start to finish.

There are also a couple hundred developers working on various parts of that are not under architect control yet.  These teams are largely driven by outsourcers who have a vested interest in keeping Best Buy architects away from their work.  If we were there, they wouldn’t get to staff their $200/hr architect on the team or their 10 offshore counterparts.  The problem here is the outsourced architect is not on board with our development methodology or high level goals, and given our structure there’s little incentive for them to even care.

So as we strategized how to get better control over the dotcom platform, I put forth that as long as we controlled the architects and the process, nothing else mattered.  Inherent in that control is control of resources as well.  Since our hiring standards are tough, we only get top notch developers which makes everything else easier.  But without a strong architect voice to push a methodology, pick resources and keep the architecture vision, the process would fail.

Next time you are thinking about restructuring your development environment, stop thinking about what methodology you’re PMO is going to use and start thinking about getting strong technical architects to carry the vision and methodology.  You’ll find that the rest of the process falls into place afterwards.

Think, Do, and let the Doers Think Too

I just did my first video interview about what is trying to build and how we are trying to build it.  It was fun but I came out of it thinking about the overall strategy Best Buy has applied toward technology for the last 10 years.  In a nutshell, and this is certainly not new information, Best Buy has decided that thinking is the most important part of technology strategy.  Therefore, we only staff managers who think about business problems, and farm out the doing to the cheapest vendor we can find.  If you ever try to do anything technology related at Best Buy, you have to have a big IT vendor do it for you.

The problem with this attitude is that no value is placed on the technical solution. When technology is driving the base strategies of so many eCommerce companies, it is hard to understand why we don’t place value on technology.  Last year alone we taught a number of our vendors how to build elastic systems in the Cloud.  When our projects were over, all those people we taught went off and built elastic systems in Clouds for other companies, likely our competitors as well.

Not only Think.  Not only Do.  Think and Do.

If you only staff Think you are reliant on someone else to Do.  You are giving away you’re strategies and technologies.  Most importantly, you’re giving away your minds that have learned new technologies and new strategies.

If you only staff Do you may pursue interesting technological pursuits, but their business value may be suspect.  From a tech person standpoint, just staffing Do is very seductive.  I’ve been on plenty of teams where the implementing team knew the business requirements better than the business.

Think and Do.  Tech and business teams highly coupled and dependent on each other can produce amazing results.  One of my favorite projects was with a highly engaged business customer who took a high functioning Agile team in directions we never would have expected.  The results were superior to anything the tech team could imagine.

But the best of the best comes from Think, Do, and let the Doers think too.  This is the core of the Agile process where decisions are distributed to the people that know the system best.

Think, Do, and let the Doers think too.