One thing I learned from three days at the Openstack Summit is that I have always misconstrued the definition of Hybrid cloud architecture. When we started making plans for our cloud architecture, I always thought of it as a Hybrid cloud. At Openstack, there were numerous presentation on Hybrid cloud and all of them revolved around using the cloud to provide additional scaling for an application that runs in the datacenter. In all cases, the datacenter architecture stack was simply recreated in the cloud and used for peak load. The database master is in the datacenter and a slave exists in the cloud. The Hybrid cloud architecture simply means using a cloud to elastically horizontally scale an existing application.
When I originally thought about Hybrid cloud I thought of an application that has one or more layers in the cloud, and the remaining layers in the datacenter. I now call this a Layered Cloud architecture. In our case we built our new product browse capability in the cloud and kept the remaining application in the datacenter. All the data in the cloud was non-secure, basically public data so there was little to no security issues. We are keeping the commerce pipeline in the datacenter simply because it is easier to keep the commerce data and transactions in our secure datacenter.
This is a good example of assumptions clouding my view of reality. I’ve read plenty of articles and information about Hybrid cloud, but until I was sitting in a presentation having someone tell me about Hybrid cloud, I never noticed my definition was incorrect. Than after recognizing this, I watched every presentation to determine which definition was used more frequently. Unfortunately for me, all the definitions were the same and they did not support my original view.
I will be presenting an overview of BestBuy.com’s cloud architecture at the SATURN conference at the beginning of May. It is generally an academic oriented conference so I’m glad they decided to hear about a real-life case study. Attend if you can!
I’m not afraid to admit it, I have weird kids. I was a weird kid too so it is not surprising that my own are considered weird as well. How do I know they are weird? Lots of meetings with various school specialists in three different districts. My kids have been mentally poked and prodded for years trying to figure out why kids that tested out as very intelligent, were having such a hard time completing their school work. One of the psychological assessors tried hard to find words to describe their test results once and ended up with “unique.” Basically, the message was they don’t fall into any of the known psychological categories so their was nothing they could actually do to help them.
Anyway, now that we’ve established they are weird, let’s get on to KhanAcademy ruining bedtime. In 4th grade, my son Alistair was finally challenged in math class (mentally challenged that is, not just challenged to get his work done). We had moved him to his third school in 5 years as each of the previous schools were unable to get through to him. (He’s in Minnetoka Minnesota school district now which is the most phenomenal school district ever in addressing individual kid’s needs).
But he was finally challenged, and because of that he started asking me questions about math at bedtime. At first it was a couple simple questions and then we moved on to bedtime stories. After a few weeks he said “Let’s just talk about math instead of stories tonight.” Wow! As a engineer it was like music to my ears. So we talked about math most nights for a number of months. I would always ask him whether he wanted to talk about something new or something he was working on. New stuff was introducing summations, probability functions, imaginary numbers, and other fun topics for 4th graders.
After a few months of this, we were introduced to KhanAcademy. I thought this was one of the coolest sites every and quickly got addicted to the math section myself. But to get my son’s interest I had to pay him to work on KhanAcademy for a few weeks until he became addicted himself. That’s when the problems started.
The math questions stopped coming at bedtime anymore. After a few days I asked why we weren’t talking about math anymore? He said he didn’t need to ask anymore because he could just ask KhanAcademy. KhanAcademy is apparently better at teaching math concepts than I am.
Bedtime had become less fun.
Fast forward one year later. I found a piece of paper on his desk with chemistry equations written on them. I love chemistry as well and asked him if he had any questions about stoichiometry? He said no, he was learning it from KhanAcademy.
Damn you KhanAcademy, you’ve ruined bedtime!
Architect Driven Development, or ADD (just add the H in the appropriate place for most of us), is my new methodology of choice. In the large Enterprise context, where Agile is difficult to implement across many teams, ADD hits the gap by positioning the Architect to enforce the development methodology as well as the software architecture. Let’s face it, as Architects we end up being the people that, at least at first, are ensuring the development methodology is followed.
In the Enterprise context where there are numerous teams spread across many mini-empires, one of the few roles that can be consistent is the Architect. In my role at Best Buy, I run a 70 person development team. That team is broken into eight different workstreams all lead by top tier architects. As long as the architect team is aligned on both a methodology and high level architecture goals, we get consistent results from our teams. It helps a lot that we all believe in Agile self-managed teams as it gives the developers the freedom to do the work how the team sees fit. But the architects carry the consistency through the development process from start to finish.
There are also a couple hundred developers working on various parts of BestBuy.com that are not under architect control yet. These teams are largely driven by outsourcers who have a vested interest in keeping Best Buy architects away from their work. If we were there, they wouldn’t get to staff their $200/hr architect on the team or their 10 offshore counterparts. The problem here is the outsourced architect is not on board with our development methodology or high level goals, and given our structure there’s little incentive for them to even care.
So as we strategized how to get better control over the dotcom platform, I put forth that as long as we controlled the architects and the process, nothing else mattered. Inherent in that control is control of resources as well. Since our hiring standards are tough, we only get top notch developers which makes everything else easier. But without a strong architect voice to push a methodology, pick resources and keep the architecture vision, the process would fail.
Next time you are thinking about restructuring your development environment, stop thinking about what methodology you’re PMO is going to use and start thinking about getting strong technical architects to carry the vision and methodology. You’ll find that the rest of the process falls into place afterwards.
Keeping Up with the Speed of the Outside World
In a world where anyone or any business unit within a company can build an application on Google App Engine, Heroku or AWS, offering only heavy weight web development as the answer to the business needs is clearly inadequate. Just as individuals forced the switch from Blackberry to iPhone in the enterprise, a similar shift is coming in web application development.
Groups have always bypassed IT based development as too slow and too expensive. Every Fortune 100 company I’ve worked in has hired development teams outside IT to get crucial work done in a timely manner. The better companies acknowledge this and try to at least train the outside developers in secure coding and process for their enterprise. These outside groups were often stymied because they eventually had to deploy in the company datacenter for data access turning that quick 2 month project into a bitter 10 month project. But the real problem is the mismatch between expectations driven by innovation outside the business, and the ability to execute within the business.
Bridging the Gap
The key for development service providers within the organization is to offer capabilities similar to what’s available externally yet customized to the unique information and systems available to the organization. Basically, we need to offer Heroku to our internal business units to build applications, while also giving them APIs into all the enterprise data, content and transactional systems. Thus the need for web development Platforms within companies themselves.
The right internal platform offers several key capabilities:
- Transparent operations and elastic scalability
- SDKs and SLAs for UI and Service components
- Built-in monitoring and reporting dashboards
- Smart data caching
- Simple UI driven interfaces
- As far as possible, technology agnosticism (e.g. Java, Ruby, .Net should all be viable on the platform)
- Security from common attack patterns (XSS,SQL Injection, etc.)
- Metering and throttling
- Priority for run the business applications
- Automated failover
- Disaster Recovery
What this is Not
We are not talking about an internal cloud, ESB or web portal. We are talking about an internal development platform that likely sits on an internal (or external) (or both) cloud. By being on the platform and developing as a registered user, you gain instant access to all internal services and data sources. Where these are and who runs them hardly matters if they all have defined SLAs that are met. Maybe there is an ESB back there somewhere but essentially no one cares if their service calls are managed correctly.
Unshackling the Business
The whole point of a flexible platform that barely dictates and mostly enables is to allow the business folks to get a little crazy. In the past development was often the voice of reason guiding crazy ideas towards implementable ideas given time and money constraints. Those days are over, it is time to let the business go unfettered by reason. At today’s pace of change, the crazy ideas are the ones that may catch on and the platform of the future will allow those innovations to occur. As technologist supporting business in this new reality, building internal platform capabilities is best way to meet those needs.
Platform Architectures are the next change coming to Web applications, though plenty have existed up to now. Salesforce.com is a prime example of a pure platform architecture but Facebook, Apple, Amazon and Google all are Platforms as well (as described in Phil Simon’s book The Age of the Platform).
However, for the first post let’s talk about my favorite platform, and to me that’s the bicycle. What makes the bicycle a platform? You can change it to fit your current needs and desires, all its component parts are individually upgradeable, it uses a common set of standards for all these parts, and it is up to your will or imagination to make the most of it.
A bicycle allows you to do more than you could before you had the bicycle. You can now get from here to there faster. You can meet your friends and go for a ride. You can work on it in your spare time. But ultimately, you decide what works best for you, and when to use (or not use) the platform (bicycle). That’s the essence of the new breed of platform, the customer will decide whether or not it is useful to them.