A Digital Ecommerce Transformation – First Architecture Forays – Part IV

Part IV – Start at the beginning with Part I

As the first team of architects to work for TWLER.com, we started mapping out the current situation and planning for the future. ATG was the base eCommerce engine and Oracle the RDBMS. Some attempt to provide enterprise services had been undertaken in the past and tax, inventory and payment had all been removed from the ATG codebase and were now called as enterprise services. There were a number of other integrations but these were the main services that caused issues in the digital world due to the disconnect in service levels that were present in the TWLER (The Worlds Largest Electronics Retailer) environment.

The basic architecture of where we started is below, I take no responsibility for the IT side of the architecture:

 

It’s worth mapping out the organizational structure to start to understand the additional frictions present in TWLER’s attempts to run a digital eCommerce system.

The Digital teams were separated out from the IT teams many years ago and had remained in that state.  Digital was viewed as unimportant since stores drove the revenue, Dotcom was a sideline play, even in 2010.  Digital was run like a business and software development was matrixed out to the IT team. The Digital team had taken over its own operations at some point because the IT teams were unable to support 24×7 operations in their model. There were dotted lines between the Digital VP Operations and the IT VP Digital Portfolio, as well as between the Digital Chief Architect and IT Senior Director Digital.

The Business VPs in the Digital team drove the business projects, with input from the wider enterprise business teams in marketing and merchandising. Between the Digital Business and IT was a Business Relationship Manager, who was supposed to translate the business asks into IT requirements. The IT requirements were then shipped off to an enterprise integrator to spin up a new team and deliver the projects. Your success varies.

In addition, internally to the Digital team was an innovation team that had spun up a completely separate Mobile site on ATG, and managed the Mobile Apps. This team was quite clear in their goals to replace all of Dotcom with their mobile site at some point in the future. So now I knew that there were two teams with the goal to replace the main www.TWLER.com eCommerce site.

Finally, there was the Senior Director of Digital in the IT team who was mainly responsible for hiring Accenture to manage projects, and WiPro/TCS to execute projects. This was a very important role as someone had to manage our vendor partners and ensure quality delivery (since I can’t get sarcasm across in writing, read this last line with as much sarcasm as you can imagine). This Senior Director also had plans to rewrite the Dotcom site, he had not gained support from the Digital VPs because they were extraordinarily unhappy with the quantity and quality delivered thus far.

To recap, in the first six months at TWLER I learned that there were at least three teams who felt they were responsible for rewriting TWLER.com. Additionally, the software development on the ATG platform was spread across multiple divisions, multiple vendors and multiple countries. All features were added by project teams who appeared when the money started, and vanished the second the money stopped, all support fell to the Digital VP of Operations. This VP was forced to contend with the mess of code and integrations that 1000s of developers were contributing to every day. I finally understood why she had moved forward with hiring her own team to rewrite TWLER.com. We owned the codebase but lacked capital, in year one we had $7M to get the team started. By the time 2011 rolled around, that amount was cut back to $3.5M.

GOTO Part V

A Digital Ecommerce Transformation – A Little More Background Before We Get Started – Part III

Part III – Start at the beginning with Part I

The TWLER.com (The Worlds Largest Electronics Retailer) architecture team started with the chief architect and five additional people, myself for high scale Java applications and NoSQL, a TWLER consultant/architect that had lived through the entire life of TWLER.com and recently converted to an employee, a second TWLER employee architect specializing in APIs and Product catalogs, a cloud infrastructure architect, and another systems thinking high scale Java architect. With a six person team, we tasked ourselves with converting TWLER.com from its massive monolithic state to a new not yet known state. What we decided early, or was decided for us, was that we were going to evolve out of the current state to a future state. It was decided for us by two constraints, not enough capital to build a new system separately and run the current system, and the stipulation that business must continue unaffected by our efforts. At this time revenue was around $1.5 billion.

There were multiple projects that we started that first year: infrastructure automation of a modern build infrastructure deployed in a cloud, cloud based outage site, distributed product catalog and QA automation. There was one more task assigned to me as exploratory which was to look into the ATG Ant build and see if there was any way of automating it with a dependency management framework.

The selection of these projects was built around the strategy to establish a modern engineering infrastructure that we could then leverage with all our future work. Having a robust Continuous Integration environment, universally available Git and SVN version control systems, artifact repository, wiki, task management and user authentication system would allow us to move quickly in the future. The distributed product catalog was built off the theory that to exit ATG, the basis of that plan required a new distributed product catalog outside of the ATG system. QA automation was the final card needed to move past six week manual release testing windows so that we could run thousands of regression tests in hours and speed up the release process.

My main project was QA automation, which was a successful disaster. Successful in that we were able to automate many of the manual test cases using Selenium and JBehave, but that the UI was so inconsistently rendered that all fields had to be accessed directly with XPath, and even then the pages sometimes rendered differently causing the tests to fail. Additionally, we were unable to setup test data in a consistent manner causing tests to fail randomly when underlying data was changed or deleted. In reality it was still better than the manual tests, but we struggled to maintain even 80% test coverage for more than a few days.

The most interesting work for me was delving into the ANT build file for ATG. I had plenty of experience with ANT in its heyday in the late 90s and early 2000s. I thought I knew ANT pretty well, but when faced with the 20+ ANT files that made up the build of 14 separate ATG applications, I had met my match. In my spare time I started tracing through the ANT files, determining how variables were setup up and finding the main path through the files. I had to diagram out the actual workings of the files as there was multiple instances of recursion occurring within the build process. My goal was to figure out how to convert the build over to Maven, which I’m sure many of you hate, but bringing in dependency management and forcing a standard file system layout we felt were the most important considerations.

After a few weeks, it was my estimate that it would probably take a couple people 2-3 months to convert the ATG build from ANT to Maven, and that we could tackle it in smaller pieces by starting with the 13 builds that were not the main TWLER.com site. We shelved this idea for the time being, but when we did get back to it a year later, my estimate proved exceedingly optimistic.

GOTO Part IV

A Digital Ecommerce Transformation – Front End Madness – Part II

Part II – Start at the beginning with Part I

In 2010, the cloud was not new but it was ignored by large companies, particularly in non-technology focused segments such as retail. While it is clear now, at the time large retailers had not yet awakened to the new reality that a company’s prowess in software might decide its future outcomes.

As an example, in my first year at TWLER (The Worlds Largest Electronics Retailer), store sales during December were not going well. Since many retailers make 50% or more of their annual sales in November and December, this spelled impending disaster. Traffic was down in stores and the company’s reaction was to propose that the digital channel stop its free shipping offer. This clearly showed the company leadership’s inability to fathom online shopping. The logic would be that if a customer could buy it online and have it shipped for free, they would not go to a TWLER store. Seems reasonable, but if the customer is shopping online and we did not offer free shipping, they would simply click to the next store, probably Amazon, and buy their electronics from them with free shipping. This customer was not going to a TWLER store that Holiday, ever. We were actually saving sales for the company, it just wasn’t understood.

A minor digression on the state of the ATG system is necessary to understand what we were dealing with. The original ATG system was built in 2003, at the time it was an excellent decision for a mid-sized retailer to build its first ecommerce engine. But over time, and numerous one-off projects, the codebase had morphed, intertwined and been generally neglected and abused. As an example, one ongoing project when I arrived was to widen the product detail page (PDP) and move the Add To Cart button from the left side of the page to the right side. This seemed fairly innocuous, but it took six months and well over one million dollars to accomplish this task. This seems ridiculous to me so as an architect, I dug into why this was happening.

It turns out there were multiple reasons why this project was practically impossible to complete. To start with, there were nine separate versions of the PDP, each made for a different category such as TVs, Music, Computers, etc. The nine separate PDPs all had common origins in some ancestral PDP, but after years of projects aimed at the individual categories, they had all strayed in different ways, including using different Javascript frameworks and versions to accomplish dynamic page elements. These PDPs were written in JSP/Javascript and were each well over 10,000 lines long intermixing actual Java into the JSPs themselves. Imagine trying to figure out how to change nine different pages all implemented slightly differently in monstrous JSP files, with no test automation to determine if you broke anything in the process.

This sounds bad enough, but the executable for ATG was in the GB range, built as an ear, with a special Ant file which only one or two people understood (more on this later). It was necessary to build and run the entire ear to determine if the page changes worked since the JSP code was so intertwined with the server side code. However, it was impossible to actually run this ear on a developer’s machine because it also required a full working copy of the Oracle database. No one had actually figured out how to make all these things work yet on a single desktop or laptop machine.

Instead, there was one shared development server for the entire Dotcom division. This server was a large Unix box but still too small to serve thousands of developers trying to build and run the ATG codebase. This server alone routinely failed due to lack of disk space and not enough CPU. But, it was the only place to build and run code you had worked on in your IDE, so everyone had to deal with it.

If you weren’t crying yet, the next step was to actually deploy the code to the staging environment (skipping the integration environment altogether) because the reality was the front end code only worked if it could access the Internet as there were so many externally downloaded components to the page. Even though you deployed the code to the shared developer environment, you couldn’t actually run it there. The staging build happened once every night.

To sum this up, the normal front end development cycle is change some code, save it locally, have it automatically picked up by your running app server and test it. This cycle time should be in the seconds range so you can work quickly and efficiently through all the little tweaks necessary to make a UI page look good and work as expected. The cycle at TWLER was change some code, save it locally, do your best to make sure the page compiled, check in the code, push to your developer environment, do your best to check it compiled, wait for the overnight stage push (go home), come back the next morning and see if the change worked in stage (assuming the push didn’t fail, which it often did). Instead of a cycle time of seconds for each code change, the cycle time was one day. One entire 24 hour day!

Did I mention zero automated regression tests?

Now I bet you think that $1M was cheap. In fact, I still don’t know how anyone actually got any work done in these conditions, but I do know that the churn in the front end development team was enormous.

GOTO Part III

A Digital eCommerce Transformation – A Multipart Series

It took six long years but we successfully transformed a monolithic 10 year old ATG based commerce system into a completely distributed, infinite scaling eCommerce platform.  I’ll call the place TWLER, or the world’s largest electronics retailer.

I joined TWLER in 2010 as a Hadoop Architect on the small team that was tasked with rewriting TWLER.com.   I interviewed in May and was offered the role, but didn’t start until July. About two weeks before I started the VP called me and let me know that the Hadoop project was cancelled and if I declined the offer they would understand. I had spent the last two years learning about Hadoop and installing one of the first Hadoop cluster in the Twin Cities on a bunch of old Dell towers for a company called Peoplenet. The VP there had the idea that they would build a remote data collection system that could take in one million messages per second back in 2008. I tried to startup a Hadoop consulting arm for the consulting company I was working for at the time, Object Partners. However, it was a bit too early in the Twin Cities for companies to be interested in BigData and NoSQL. We spent a lot of time talking to companies about the technology, but no contracts were forthcoming. I put together an Introduction to Hadoop presentation and gave that numerous times, all to no avail. So when TWLER came calling with a Hadoop Architect position, I jumped at the chance to actually use Hadoop at a large company. All that to say I was quite disappointed when the role fell through and seriously considered declining the position as I had not yet resigned, and after two years of pushing a technology I did want to see Hadoop in action. But I was quite tired of my second stint in consulting and ready to move on.

So I arrived at TWLER with no assigned role. At the time we were five architects under an Operations Vice President, and I had no idea of the organizational politics that were hindering a major system rewrite. The cancelling of one project should have been a clue to the future as already, funds were being removed from this team.

We started out under the tutelage of Michael N., a renowned architect that had worked on the initial version of TWLER.com. Our first task was to setup a modern development pipeline using Chef in AWS. A reminder that this was in 2010 when a large retailer doing anything in AWS was highly uncommon, and Chef was practically brand new.

The pipeline we stood up at the time was fairly standard, Atlassian Stash for Git, Crowd for user management, Confluence for knowledge management, Jira for issue tracking, Bamboo for continuous integration, and Artifactory for artifact storage and a Maven repo. We used Chef to completely automate the deployment of these tools into AWS. While none of these was new to TWLER, they had not been combined together, made externally accessible or offered to anyone who wanted to use them throughout the company.

We used this infrastructure to start a small AWS based project to put together a small site that could be used during outages. The site would allow customers to lookup locations, products and prices, but full commerce capabilities would not be available. It would reside in the cloud, ready to be deployed within minutes. It says something that the first thing we built was an outage site, TWLER.com was not a stable platform at the time.

GOTO Part II

What Architects Do? – Part 2

Governance or Strategy

I’ve fielded many questions lately on how I am governing and reviewing the architectures of Target to make sure they conform to enterprise standards. This is a common question asked of Architecture teams. After all, many people believe the main responsibility of Enterprise Architecture is governance.

But governance is the last thing I like to discuss about what Architects actually do. Governance or discussions about being governed means that I’ve actually failed to deliver a simple, clear and implementable technology strategy. If I’ve clearly communicated a technology strategy with desired, but not necessarily mandated, system structure, implementation stacks and supporting platforms, than engineers will happily fit into the architecture to deliver their systems.

Let’s unpack that last statement and define point #2 of what architects do:

Answer #2: Architects define the types of systems and their boundaries that are possible within an organization.

When we discuss technology strategy my goals are to deliver composeable systems which meet current and future business demands. The key to this statement is that we will meet unknown future demands without rebuilding our current systems, or having to build large new systems. There’s an assumption here that your future unknown demands are extensions of your current capabilities. If you’re headed into new business models, you’ll likely need a few new systems.

In today’s world, unknown future demands are presented every day, and the expectation is that they can be delivered in days, weeks or months, not years. For an organization to stay relevant with their consumers, a technology strategy must meet this demand for speed above all else. To achieve speed at scale the systems are constrained to deliver one thing only. A system is generally composed of services (or microservices if you like) that deliver either data or process through an API. The only governance that architects do is to ensure that data or process is comprehensive and unique. Data systems are comprehensive when they are complete for an organization, and take all changes generated throughout the organization. Process systems are comprehensive when they contain the basic services necessary to complete a process, and can be flexibly orchestrated by any entity within the organization.

Done correctly, an organization is composed of hundreds of APIs that each implement a narrow set of functionality. Systems don’t continue to expand to take on more business demands and processes. Actually, systems never want to do this, people expand systems to become more important in their organizations. Governance falls out of strategy, the only governance necessary is ensuring teams don’t construct competing data systems or alternative processes. If the current systems don’t meet their needs, fix them by doing the work and submitting a pull request. That’s also much easier than standing up a new system and engineers generally like this solution.

What Do Architects Do? – Part 1

Having been Chief Architect for Best Buy and currently Chief Architect for Target, I get this question frequently.  The question occasionally comes at your neighborhood party but generally it’s from people in Engineering or Marketing.  At the neighborhood party you attempt to answer this question at risk of becoming known as the “boring tech guy.”

What do architects do?

People ask this question because they truly don’t understand.  They are really asking “Are architects necessary?”  They ask this question because they have rarely seen value from the architects they’ve seen in the past.  This is the sad reality of much of the architecture world, enterprise architecture in particular.  My opinion on why architecture has devolved to a place where many companies are eliminating the practice altogether is simple.  Most architects in the upper echelons of companies were never software engineers.

Why this is important and is a point I harp on continuously, is that if you haven’t spent your 10 years writing code and building and running systems with higher and higher business complexity, you cannot do the first thing architects should do.

Answer #1:  Architects create the environment for engineering culture to thrive.

To create an engineering-centric culture, you have to have been an engineer.  You have to have a few large scale Agile/DevOps systems under your belt.  You have to understand what drives and motivates engineers that want to work on six person teams tackling the toughest problems facing enterprises.  You need to feel it in your gut when one person on the team is a hack and can’t pull their weight and your management won’t address the problem.  You must have found the rock and through sheer force of will, pushed it up the hill, leading in such a way that the rest of the team helps push it along with you.   If you haven’t done these things over the course of years, you haven’t been a software engineer.

Architects that understand how engineers want to work, spend all of their time and energy creating an environment where hard problems are solved, new solutions are found, and everyone sleeps well at night.  Architects make decisions based on whether engineer’s will understand them and choose them regardless of what else is available.

Architects also create the constraints that allow engineers to solve their problems quickly.  With the near infinite array of tools, frameworks and packages available, to remain economically competitive, enterprises need to limit the scope of technology in some way.  Make no mistake, architects select and limit technologies, but only those that involve large expenditures.  If a technology selection will cost a company in excess of $1M for licenses, subscriptions or maintenance over a five year lifespan, architects should be involved.

Architects essentially act as the aggregated will of the engineers.  Architects are there to make engineer’s lives easier.  Architects in effect are servants of both engineers, and the enterprise and walk the fine line that brings the maximum value to both.

 

Peak Architecture

I believe peak architect occurred somewhere around 2013, that was the year with the most people employed with architect titles ever. Since 2013, we are now entering the post-big-architect era, where architect roles are quickly shrinking in the tech world. This is a good thing.

The number of roles with architect titles in industry was getting ridiculous. I can riff 20 titles in less than a minute if I try: Data, Technical, Web, Digital, Application, Cloud, Security, Infrastructure, Network, Business, User Experience, Information, Middleware, Enterprise, Solution, Sales, Product, Project, Principal and my personal favorite, Sharepoint. The question is, what did all those architects do?

My contention is that they completely messed up most companies by proclaiming they knew how to select and manage technology at large companies, and forced vast numbers of engineers to conform to a highly limited set of technologies in the name of fiscal conservatism. Then retreated to their own worlds to rest and consider the next great set of technologies they would enshrine five years from now.

What is the optimum number of architects for a company?

The answer to this, like all math problems, is 0, 1 or some hard to derive number. Zero and one are actually quite good answers. Zero is complete engineering anarchy, you’ve decided to let the engineers do whatever they want. If they’re good engineers, they might actually make an amazing system by developing external APIs which hide all the technology choices from the outside world.

One is also a great answer, you’re basically saying someone needs to think about the company as a whole, and one insanely smart person can get a whole company pointed in the right direction, or they can spend a lot of time thinking and watching engineers do whatever they want. One architect can spawn ideas, if they’re in touch with the engineers, that just may be enough to get people pointed in the right direction.

A better number is some ratio to the number of engineers. But what ratio to pick? Too low and you have architects running over each other to tell engineering teams what to do. Something like the commissioned sales model, “that team was mine, I talked to them first.” Too few and you’re not much better than 0 or 1. My personal favorite ratio right now is 1:100, one architect for every 100 engineers. That seems to be enough to get some rational standardization (if you don’t like standardization, deal with it, every company beyond startup mode has some set of standards), while allowing engineers to fight the power and pick new technologies when necessary. Since technology is progressing faster and faster, it’s essential to keep new technologies flowing into an organization, and old ones leaving (sometimes 6 months is old!). Relying on a few individuals to follow these trends and pick the right tools for everyone is impossible. But sourcing newness from those engineers willing to push a new idea over a few intentional roadblocks will keep your organization current enough to remain relevant.

Peak architecture has passed, time to jump on the bandwagon and become engineers again!

Open Source North Rides Again

We had another fun and exciting rendition of Open Source North in June earlier this year (2016). It’s a conference that eventually will be recognized as the premier technology event of the year in the Twin Cities. Well, that’s my prediction at least.

I had a great time and got to give my presentation/group discussion on the state of Architecture in the Midwest and how the practice needs to be redefined. Architecture means nothing these days, Enterprise Architect in it’s traditional form is dying, and Engineers are taking over the space once occupied by Architects. Why have an architect if you can have an engineer do the same job and deliver the working software as well?

Hope to see you again next year!

Tech Cities 2016 – Quick Fun at the Carlson School

Tech Cities 2016 turned out to be a quick half-day conference, high on fun and low on pretense. I mistakenly thought we had an hour to play the Agile Architecture game, which was already a short time for explaining the rules and playing once through, but it turned out we only had 45 minutes.
techcities-1 Generally we like to have 90 minutes and 2 hours works best. Kevin Matheny and I cranked through our short presentation on Agile Architecture and the rules of the game and jumped right into playing it with ~40 of our new friends.
techcities-3

We always played the game with software people in the past, so there were a few more questions about how to play than we’ve seen in previous games. But with a lot of individualized attention from ourselves and our three additional proctors, everyone was able to get through the game without being fired! Some even earned some gold pirate coins for completing their objectives!

If you have an interest in learning what life is like for an Agile Architect, let me know. Kevin is putting together these workshops going forward and I’ll be helping him out when I have the time.

Using the Altimeter on iPhone 6S with Swift

I was playing around with the new xcode 7.2 which allows you to beta test applications on your own phone without an Apple developer account.  Add to that the barometer available on the iPhone 6s and I just wanted to see how hard it was to write an app that can show you your altitude.

This seems like it would be pretty straightforward but it turns out there’s practically zero documentation on how to use the altimeter, and for a Swift newbie, this turned out to be quite a problem.

All I wanted to do was create an app with one button to start the altimeter, and a label which constantly updated with the relative altitude.  It seemed like something that might take an hour or two max.  However, with the incomplete solutions available and cryptic problems with refreshing the display while in a closure, it became much harder than I ever anticipated.

Which is why I’m publishing the code, however amateur, because it does actually work.

@IBOutlet weak var altitudeLabel: UILabel!

lazy var altimeter = CMAltimeter()
lazy var queue = NSOperationQueue()

@IBAction func trackButton(sender: UIButton) {

 if CMAltimeter.isRelativeAltitudeAvailable() {

    altimeter.startRelativeAltitudeUpdatesToQueue(
        queue, 
        withHandler: {( data: CMAltitudeData?, 
                       error: NSError?) in

    // Needed to refresh the screen 
    // from inside the closure
    dispatch_async(dispatch_get_main_queue(), {
        self.altitudeLabel.text = 
           String("%.2f feet",
           ((3.28 * 
           ((data?.
             relativeAltitude.doubleValue))!)))
        })
    })
  } else {
    self.altitudeLabel.text = 
       "No barometer available"
  }
}

It did finally work and looks like this:

AltimeterScreen