Monthly Archives: March 2017

Architecture, Digital Ecommerce Transformation, Minneapolis, Platform

A Digital eCommerce Transformation – A Multipart Series

10 Comments

It took six long years but we successfully transformed a monolithic 10 year old ATG based commerce system into a completely distributed, infinite scaling eCommerce platform.  I’ll call the place TWLER, or the world’s largest electronics retailer.

I joined TWLER in 2010 as a Hadoop Architect on the small team that was tasked with rewriting TWLER.com.   I interviewed in May and was offered the role, but didn’t start until July. About two weeks before I started the VP called me and let me know that the Hadoop project was cancelled and if I declined the offer they would understand. I had spent the last two years learning about Hadoop and installing one of the first Hadoop cluster in the Twin Cities on a bunch of old Dell towers for a company called Peoplenet. The VP there had the idea that they would build a remote data collection system that could take in one million messages per second back in 2008. I tried to startup a Hadoop consulting arm for the consulting company I was working for at the time, Object Partners. However, it was a bit too early in the Twin Cities for companies to be interested in BigData and NoSQL. We spent a lot of time talking to companies about the technology, but no contracts were forthcoming. I put together an Introduction to Hadoop presentation and gave that numerous times, all to no avail. So when TWLER came calling with a Hadoop Architect position, I jumped at the chance to actually use Hadoop at a large company. All that to say I was quite disappointed when the role fell through and seriously considered declining the position as I had not yet resigned, and after two years of pushing a technology I did want to see Hadoop in action. But I was quite tired of my second stint in consulting and ready to move on.

So I arrived at TWLER with no assigned role. At the time we were five architects under an Operations Vice President, and I had no idea of the organizational politics that were hindering a major system rewrite. The cancelling of one project should have been a clue to the future as already, funds were being removed from this team.

We started out under the tutelage of Michael N., a renowned architect that had worked on the initial version of TWLER.com. Our first task was to setup a modern development pipeline using Chef in AWS. A reminder that this was in 2010 when a large retailer doing anything in AWS was highly uncommon, and Chef was practically brand new.

The pipeline we stood up at the time was fairly standard, Atlassian Stash for Git, Crowd for user management, Confluence for knowledge management, Jira for issue tracking, Bamboo for continuous integration, and Artifactory for artifact storage and a Maven repo. We used Chef to completely automate the deployment of these tools into AWS. While none of these was new to TWLER, they had not been combined together, made externally accessible or offered to anyone who wanted to use them throughout the company.

We used this infrastructure to start a small AWS based project to put together a small site that could be used during outages. The site would allow customers to lookup locations, products and prices, but full commerce capabilities would not be available. It would reside in the cloud, ready to be deployed within minutes. It says something that the first thing we built was an outage site, TWLER.com was not a stable platform at the time.

GOTO Part II

Architecture, Software Development

What Architects Do? – Part 2

Governance or Strategy

I’ve fielded many questions lately on how I am governing and reviewing the architectures of Target to make sure they conform to enterprise standards. This is a common question asked of Architecture teams. After all, many people believe the main responsibility of Enterprise Architecture is governance.

But governance is the last thing I like to discuss about what Architects actually do. Governance or discussions about being governed means that I’ve actually failed to deliver a simple, clear and implementable technology strategy. If I’ve clearly communicated a technology strategy with desired, but not necessarily mandated, system structure, implementation stacks and supporting platforms, than engineers will happily fit into the architecture to deliver their systems.

Let’s unpack that last statement and define point #2 of what architects do:

Answer #2: Architects define the types of systems and their boundaries that are possible within an organization.

When we discuss technology strategy my goals are to deliver composeable systems which meet current and future business demands. The key to this statement is that we will meet unknown future demands without rebuilding our current systems, or having to build large new systems. There’s an assumption here that your future unknown demands are extensions of your current capabilities. If you’re headed into new business models, you’ll likely need a few new systems.

In today’s world, unknown future demands are presented every day, and the expectation is that they can be delivered in days, weeks or months, not years. For an organization to stay relevant with their consumers, a technology strategy must meet this demand for speed above all else. To achieve speed at scale the systems are constrained to deliver one thing only. A system is generally composed of services (or microservices if you like) that deliver either data or process through an API. The only governance that architects do is to ensure that data or process is comprehensive and unique. Data systems are comprehensive when they are complete for an organization, and take all changes generated throughout the organization. Process systems are comprehensive when they contain the basic services necessary to complete a process, and can be flexibly orchestrated by any entity within the organization.

Done correctly, an organization is composed of hundreds of APIs that each implement a narrow set of functionality. Systems don’t continue to expand to take on more business demands and processes. Actually, systems never want to do this, people expand systems to become more important in their organizations. Governance falls out of strategy, the only governance necessary is ensuring teams don’t construct competing data systems or alternative processes. If the current systems don’t meet their needs, fix them by doing the work and submitting a pull request. That’s also much easier than standing up a new system and engineers generally like this solution.