Reliable Systems

We spend most of our time deeply engaged in our client’s projects.  We work hard to assimilate quickly in to our client’s culture and challenges to make sure we can deliver the most value we can.  This doesn’t leave us with a lot of opportunities to do team building within our own company.  We’re committed that employees of eSymmetrix feel a part of something a lot more than just our clients - they are part of the eSymmetrix way of getting things done, which they can be proud of.

In the past several months the three partners of our firm have been scattered to handle all of the challenges of our growing business, but with the completion of a major engagement with a client we had the opportunity to spend some time back together to focus on more long range concerns.  We had intended to spend the time working on product and marketing strategy for our upcoming commercial release of Gibraltar, but instead on a whim decided to do something we called the Code Monkey Challenge.

The goal was to make something complete and useful within a very short period of time.  Complete for us meant we were going to create a software component, test it, package it, and publish it to a new web site with descriptive content all in 48 hours.  We didn’t even have a hosting company set up beforehand.  To be useful it had to be usable by people outside of our company as delivered, and fulfill a real need, at least for a set of users.

We started at 9:00 AM with the goal of publishing the final version to a new web site by 6:00 PM the following day.  Our initial approach was to have one hour sprints of work divided between each team member, then meet and review and set up the next sprint.  In the end we used three hour sprints, delivering to each other through our source code system at the end of each sprint.  We made our goal - you can see the results at www.gibraltarsoftware.com.  Most importantly, we achieved our result of bringing together our team members.

We eliminated any distraction we could so that we could apply as much of the 48 hours as possible to the project.  We worked well into the evenings and had others provide some support to make sure we had food, coffee, and no outside distractions.

The intense time pressure gave us a good tool to eliminate most conflict: there simply wasn’t time for much philosophy on how to approach the different technical aspects.  The lack of time removed ambiguity that otherwise would be the source of conflict.  There wasn’t much time for yak shaving either - we had to produce results at the end of each sprint.  Still the first few sprints were spent mostly in exploration and validation of the approaches with our first rough cut of each element done by the end of the first day.  The second day was then about refining and documenting.

While we did skip over some aspects that we would normally do on a larger scale project, we did include most elements of our preferred development process.  We did coordinated sprints of the team separated by an after action review to adjust our plan.  We distributed code and results between the team through our source code control system, we wrote automated unit tests to verify key aspects of the system, and did peer design reviews.  If anything, the lack of time made us less likely to bypass most parts of the process because we knew we wouldn’t have time to dig our way out of any problems we created.

The real goal of the exercise was to bring us back together as a leadership team and establish more of the shared experiences that define interpersonal relationships.  It’s helpful to bridge the gaps that easily develop between architects and developers, developers and designers, engineering and marketing.  We all had to come together to achieve our result because it was clear that we’d all fail if we couldn’t.  Most business projects are sufficiently fuzzy that it isn’t nearly as clear what the cost of not working together is, and politics can overwhelm cooperation.  It’s an experience I’d recommend in any software team, particularly if you’re in a company that can mix skills and responsibilities.

The next time your shop is done with a project, or when you need a break consider doing your own Code Monkey Challenge.  It only takes two days.  Here’s the rules:

• Produce a Real Product: Scope out a product that is really useful to a specific audience.  Sure, you’re giving it away for free, but it still needs to stand up to scrutiny.
• Marketing Material Too: What good is a product if no one knows about it or understand how or why to use it?  Even though you’re giving it away you want people to be able to understand and take advantage of your effort.
• Help and Usage Information: Like a real product, it has to be usable without you sitting over the customer’s shoulders.  Depending on what it is, you need to tell them how to install it, program with it, and provide guidance on recommended usage scenarios.
• In Little Time: Extra time is the enemy - it will reduce the pressure to work together and encourage unnecessary bells and whistles while removing the catalyst that gets everyone to work together.
• Publish to the World: If you’re  a large company, perhaps it’s just the company.  Wherever it is, it has to feel like real stakes to the team:  Your results will be on display.

To set it up, be sure that you can eliminate anything that would distract the team - they can work into the evenings, have food brought in, whatever.  The closer it is to a total immersion experience the better.  It improves the sense of camerarderie developed within the team and ensures the most creative energy is directed to the project. 

If you try it out, or have another software team building story, please drop me a line and tell me how it went, or leave a comment below.

In the past two years I’ve heard the term Technology Debt thrown around to justify a number of technology decisions.  In an effort to come up with a term that would bridge the business-technology gap, someone came up with Technology Debt to indicate that you were basically creating a future liability that would have to be paid back - rewriting a section of code or switching out a module, whatever.  Since business folks deal with assets and liabilities routinely, expressing subtle technology problems in financial terms has a lot of appeal.  The basic concept is most frequently used in relationship to software: Let’s say you defer something to a future release so you can get the release out the door.  Perhaps the feature will only work for very few customers but you don’t have time to generalize it, or the solution won’t scale as new users are added.

This is fine as far as it goes, but like any metaphor while it may be a way of explaining some aspects where two things are in common it’s very easy to overextend because in fact it isn’t a true financial liability.

Take a hypothetical example: You are supposed to add federated identity to your web application. What you want to do is create an identity broker that will allow your SaaS application to connect with Microsoft’s ADFS, OpenID, and a few variations of SAML. You believe this will get you the market reach you need, and by creating your own identity broker you can decouple your application from changes in this still evolving space, as well as support your own native security technique.

As you dig into it, you realize that this just isn’t feasible in the time you have: You need to get a solution done in two months to meet a commitment to a customer, and it turns out this customer just uses ADFS. It just so happens that your web framework can easily work with ADFS directly, so to save the schedule you drop back and just do ADFS. From a development standpoint this is a hack - you are doing something quickly you can’t extend to meet the original requirement and you’re pretty sure that you’ll need to undo this later and do it right, which will cost more than just doing it right the first time. This is Technology Debt.

Metaphors have Limited Application

The metaphor doesn’t hold for long.  First, unlike real debt there is no external requirement to pay this back.  Perhaps you never will need to support more than ADFS, or that after all of the talk customers just won’t adopt federated identity.  At the start of each release cycle, you can look at the competitive market and see what is the correct, most important work to be done.  It might be that you’ll have to make good on something you deferred, but you might not.  If you didn’t, it never was debt.  How confident are you that you can tell the difference right now?

Second, all technology has future liability. The more code you write, the more you have to maintain and support. That has a cost as well for the future. Depending on the nature of your product it could mean that you have to support questionable past API decisions or obscure and intricate features of your product. Every feature has a cost to maintain, every line of code you use or reference in a third party library has weight. Only talking about deferred development as technology debt implies that what you have right now is all asset, but it isn’t.

Rampant Misuse

The biggest challenge I’ve seen is that the metaphor is used to push development team goals on the business without having to adequately justify them.  By handing business folks something they can easily relate to, it’s easy to gloss over the underlying technical implications.

For example, say you have an application that’s currently written in Visual Basic 6.  You can justly claim that Microsoft has dropped support for it and that the day is coming when it will no longer run on the latest operating system (Microsoft has committed that it will run on Vista and Windows Server 2008, but that will likely be the end). This sounds very alarming indeed!  Naturally, the answer is to rewrite the entire application in .NET 3.5. Yes, this will take longer and cost a lot more than just adding the features you need to the existing code, but it eliminates all of the technology debt represented by that old nasty VB6 code.

Now look at it a little more objectively. Yes, you will need to eliminate that VB6 code at some point - notably when you are upgrading from Vista to whatever’s next.  When will that be?  Well, if you believe the hype that people love Windows XP and may just consider Vista in the future, you have some calendar time. As long as it’s done by then, Microsoft dropping support isn’t a real issue.  We’ve yet to work with a client who’s VB6 application didn’t work just fine on Vista, thank you very much.

Second, with rare exception every modern technology has a half-life. The notable exceptions are COBOL, C, and C++.  You can with confidence know that these languages are still going to be in active use in 15 years and that code you create now will work with minor to modest adjustment on current platforms at that time.  I would dispute if a similar claim can be made for the latest crop of languages.  A major reason for this is that modern languages are really whole environments - a programming language and an API/runtime.  While it may be theoretically possible to write a C# compiler without toting along the .NET runtime, it isn’t going to happen.  Likewise for PHP, RoR, and even Java.  Java’s framework is much more shallow which ironically will likely give it more life because it can adapt to radical changes in computing environments and there is already an intent to decouple most of the code from specific frameworks.

This means that today’s latest and greatest environment will be tomorrows VB6.  Just listen to people that jumped on .NET 1.1 discuss how they’re going to upgrade to .NET 2.0 or later, and that’s really not a particularly dramatic move.

Fundamentally, when you purchase software you’re making a bet in the vendor and community behind it. This is one place where the LAMP stack has a number of advantages because of the openness of the environment and the institutionalized tolerance of actively supporting releases for a long time. This is fundamentally enabled by their open source nature, but even if you had the source code for a commercial product it’s not likely you could do much with it; unless it’s fairly simple the burden of maintenance is likely higher than the cost of replacing it. Counteracting this are a lot of hidden costs to open source that may be difficult for a small team to absorb.

Easy Metaphors Seldom Produce Great Outcomes

Next time you’re trying to bridge the gap from technology to business, try to stay away from simple metaphors like Technology Debt. Instead, have real conversations to articulate the potential business impacts of the technical decisions and then hear from the business what they are concerned about. With a real dialog, everyone is in on why you deferred work to later and what bet is being made, and no one will be surprised when you do show up in a year to start porting that VB6 app to .NET.

In software development, you’re always being asked to estimate things:  How long will the whole project take?  Just this feature?  What if we changed this feature to remove this aspect?   This is all part of the feedback cycle that is fundamental to product creation:  We have a certain amount of time and money for a given set of functionality, but if there’s something really juicy and it just takes a little more time, then maybe we adjust, or if we can get a lot of value for a little effort, perhaps we do a little mini release first.  The business decisions feed the development estimates which in turn inform new business opportunities.

Getting the feedback wrong can be disastrous: The right functionality late can kill your business; perhaps half a loaf earlier is better.  On the other hand, some things the market won’t accept half way so a full loaf it is.  The business needs to trust the information it’s getting isn’t random or capricious to make good decisions, and the development team needs to be able to provide a best guess without fear of misunderstanding.

It’s natural to pad estimates with the idea that it’s better to under promise and over deliver - so that means to guess long and come in well early.  But in the end, is that really any better?

You’re Guessing, and Possibly Lucky

We’ve been experimenting with the new Evidence Based Scheduling features of FogBugz (which we use internally for managing our software development) and one thing that it highlights quickly is that estimation isn’t good if you’re early, bad if you’re late - it’s about getting your average as close to the mark as you can.  Take a look at a graph of most of my estimates:

Ideally, the graph line would have a 1:1 slope, indicating that on average you are accurate.  Further, you want your estimates clustered pretty near to the line itself.  What you can see from my estimate curve is that I’m uneven - I tend to underestimate shorter tasks (under 1.5 hours) and overestimate longer tasks (and that’s after removing some really bad outliers…).  But notably if you draw a ruler on the 1:1 line you’ll see that I’m not even close. Don’t let the hash lines fool you - look at the numbers to see.  The thing is, the other developers in our company that are all regarded as skilled, senior developers aren’t particularly more accurate on any one estimate, and the averages work out similarly.

So what?

Why isn’t it a great thing when we beat our estimates?  There are several potential pitfalls:

Features based on Effort

We’d all like to believe in the rosy model that our customers ask for features and then we build them into the software, so if we’re done early then it’s a pure win.  It’s my experience that it’s much more like a game of Tetris:  What features we take on is dependent on how much effort we think they’ll take.  Every feature has an amount of effort above which it isn’t worth it any more.  When hashing out what makes it and what doesn’t, the effort estimate is a big factor.

If we overestimate the effort of features, then we are slanting the project management decisions away from customer-selected features in favor of the developer’s whims.  This is because some features will be estimated to take more effort than they’re worth, and a more invisible internal team dynamic.  If a project is doing well on schedule, it’s very human to take advantage of this to try out newer, riskier things, over-engineer a feature, or do other things within the team that would otherwise be successfully argued against because of their effort.  In general, the more time available, the more yak shaving the team will do.

Once a schedule is accepted, the business will tend to act on it as fact:  Customers that can’t wait for it will end up being turned away and others will be promised a schedule.  This is a necessary but painful aspect:  Developers are generally optimists and will not want to say no to a customer even though it’s generally not in the best interest of either the company or customer to rush a feature. You want the business to stick with your decisions and not pass the buck on saying no to the customer, but you also need them to trust that it’s a fair trade.

Approach based on Effort

Within the development team, decisions are made at every level on how to implement a feature with an eye towards both the feature’s estimate and the overall project’s status.  Even when not explicitly laid out, a team that believes the overall schedule is tight will feel pressured to find ways to reduce the effort on anything they do.  This means when deciding between a careful implementation that may take longer but be more scalable or easier to support the team will often opt for a more direct path to completion even if the estimate was based on the more careful approach.

If done as a conscious decision in consultation with the project’s sponsors, this can be a way of bringing the project back on track but really it’s just another way of cutting functionality to make schedule:  You’re going to cut out something you intended to deliver (say a more generalized, upgradeable framework for reporting) even though you meet the letter of the requirement in front of you (delivering a few reports).  This can lead to nasty surprises for the team down the road when your sponsor’s find out that they didn’t get what they thought they would.

Alternately, if you overestimate one feature it may have put another feature under pressure so a more expedient and risky approach was adopted for it to fit it into the schedule.  If the true effort had been known, a different decision could have been made.

Make Your Guesses A Coin Toss

In the end, being early and being late just have different ways they create problems for your development project.  Your goal when estimating is to not try to find the estimate that has the highest probability of being sufficient to get the job done but instead the estimate that is equally likely to be high as low.  In aggregate if you have enough of these items on your project (say more than 25) then you’re entire project’s estimate should also be just as likely high as low.

There is still a place for the traditional high estimate: When you move outside of the project sponsor and the development team to users that need a guarantee.  There the downside of missing a date is much worse than the impact of being early.

On your next project, try out the 50/50 approach and make it clear to both the development team and the business.  You’ll probably notice that people develop a more subtle appreciation for the fact that estimates are based on probabilities.  This can help you skip over the discussions that aren’t useful about why you are where you are and instead keep focusing on the business goals for your current situation.

Have you had the experience of looking at a situation and knowing - just knowing, that something was wrong? Perhaps it was a user interface design or a software diagram or project schedule. I’m not talking about a dispassionate concern but an emotional response - you recoiled inside and just knew. Then you had to explain to the person or team that presented the situation why this was the case. Most likely you can draw on your experience and come up with a few very convincing explanations for your gut reaction, but usually you’ll walk away unsettled. It still isn’t right, but you just couldn’t put your finger on it.

Most professionals develop an instinctive ability to size up situations within their core expertise. For example, a seasoned product manager that has worked up through the ranks of developers can often look at a schedule and get a quick feel that it can or can’t be met. Most of the time this intuition expresses itself as a strong gut reaction that you can’t explain. Malcolm Gladwell wrote the excellent book Blink talking about this phenomenon, well worth reading so that you know what to do when you have this reaction the next time.

The basic challenge is this: Even if your instinctive reaction is correct, it isn’t particularly useful until you can explain why. It’s very tempting to try to verbalize the rationalization for your reaction, but don’t. The problem is that your rational mind has no idea why you had an emotional reaction. The emotional reaction is your clue - your intellect and emotions aren’t really connected that well. They need to arrive at their conclusions independently: For you to intellectually justify your reaction you need time to perform a dispassionate intellectual process. Your instinctive emotional reaction didn’t need that time, but it can’t explain itself.

The Path Through The Woods

So if your emotional reaction is equally as reliable as your intellectual evaluation, but you can’t articulate it right away what should you do? The first thing is to develop a code phase for your team that indicates that this is your blink reaction, and not something else. This lets everyone weigh it correctly: It isn’t because you just don’t want to do it or you like another option better, it’s that there’s something instinctively wrong with it. Your team should give it equal weight to you having just expressed a cogent, real argument for there being a problem. Second, the reaction can’t be challenged - at least not directly. It’s an emotional response, so any challenge will drive the participants strait into conflict from collaboration. In our shop, we just say “I have a blink about this…” That cues everyone in.

At the same time, it’s important to recognize that your blink can and will be wrong, sometimes a great deal. It’s no more accurate than a rational analysis of the same circumstance, and that means if the circumstance is something notoriously hard to predict like an election, a project schedule, or a roll of dice then it’s not going to do better than spending some quality time in analysis. This means that when you have a blink reaction then your team should continue with the assumption that the reaction is dead on, but casually seek out the proof.

Casually Find The Evidence

Once you’ve articulated your instinctive blink reaction, have the team take the case that it’s true and then as discussions continue identify facts and data that support the reaction. Eventually, one of a few things will happen:

1. Killer supporting evidence will emerge: In the process of going through subsequent analysis, you’ll find the evidence that suddenly has the reason behind your reaction clear to the team. It’s now an intellectually reasoned response; you just got there faster instinctively.
2. You’ll realize your reaction was wrong: As time passes, your mind will keep attempting to align facts with your reaction seeking to prove or disprove it. Eventually you, or your team, will see what it was that your mind originally caught on and understand that it doesn’t apply in this case: The reaction was wrong, and now you can proceed. This was still an important exercise because you now have validation on your course of action (”we can ignore this previous best practice because it no longer applies because….”)
3. You’ll look at the problem tomorrow and get over it: Perhaps what you felt was just the normal human fear of change. So be it- now that you aren’t feeling threatened any more, your rational mind can reassert itself and look at the item more objectively and be open to new possibilities.

Each of these is a powerful collaboration result because it lets the team and the individual practice and demonstrate the characteristics of a trusting, supporting environment. The team showed the respect for the individual participant and leveraged the experience of everyone, not just those with great oratorical skills. The individual gets to articulate something they feel very deeply without being embarrassed or having to justify and then later defend something that they just can’t. Everyone is in on it - and when the instinct resolves into intellect later everyone will have better insight into the experience and thinking of the person that voiced it. This insight develops trust between the individuals that will live beyond the team.

Finally, by the team not pushing for an immediate defense and then challenging it demonstrates and reinforces that it’s a safe environment to voice ideas and opinions, and builds credibility for when a miss-step happens.

Gaining Speed

Over time as your team works together people may be able to help each other articulate their blink reactions into concrete issues to be resolved based on understanding the key emotional drivers each participant brings to the table. In our company, we have some folks that tend to have reactions over usability, others over ultimate performance (we’ve dubbed one “the keeper of the nanosecond”), and others over code simplicity. Knowing these points helps us not miss-read emotional reactions to ideas and to help each other understand what we really need to do to create the best outcome.

With practice, you’ll find your team can get to great, collaborative conclusions faster and generate the buy-in from the participants with little or no effort.

What Gets You to Blink?

Looking back, when did you have a blink reaction to something? What did you do with it? Share your story by posting a comment or dropping me a line.

Editor’s Note: This is the final article in a three article series. For a list of the entire series, see the Article Series page.

In the final article of our series we’ll look at how to create an automated build incrementally and make it a natural evolutionary process of your team, providing both immediate and ongoing value. Really, it’s no joke and it’s not going to require upending your technology or team.

Components of Build Automation

When introducing an automated build, we recommend pursing components in the following order:

1. Compile files from Source: Retrieve all of the input data from the configuration management system, and compile everything that needs to be compiled. If you’re working in a language or technology that doesn’t require compilation, just retrieve and label all of the input files. Compiling may include activities such as automatically assembling release notes from the defect tracking system or any other action necessary to create a file that is distributed. Check compiled results back into the configuration management system.
2. Assemble Build from all Files: Take all of the files needed to create your distribution including every dependency, release notes, etc. and create your distributable build. The distribution (a.k.a “build”) ultimately should be exactly one file per target platform. Copy to a central networked location either named with the unique build number or otherwise clearly identified.
3. Automated Unit Test: Using a third party framework (recommended) or your own custom framework perform automated tests designed to exercise the individual components of your system in a very detailed way. This is often easier than a real system test because the tests will tend to be more stable over time and more compartmentalized.
4. Install and Smoke Test: On a green system, perform a fresh installation and basic smoke test of the system.
5. Automated System Test: Perform a full automated test of the public surface area (that reachable by users) of the system.

Compiling Files from Source Control

The first task to take on is to automate compiling all of the files from source. This is required before the build can be centralized, and really is the cornerstone of the process. This step will require the most investment before you realize any return. We recommend taking a developer with IT administration experience and dedicate them to the task. Many IT administrators are used to automating tasks and working with installing and cleaning up software, and a solid understanding of administration can be very handy for this step. In our experience it can take as little as a few days of time to as much as a few weeks depending on the experience of the developer and the complexity of the product.

It’s important that the build process be idempotent to be valid: Regardless of where the build gets started, you should be able to restart it and have it recover, cleanup, and then work. Generally it’s best to clean up the failed build then proceed with a good build instead of trying to pick up where the previous build left off (it’s more deterministic). This approach also lets you develop the build iteratively with a failed build recovery stage added to the start of a normal build process.

The build process must ensure that it generates the right labels to meet the traceability goals of a build. The best way to do this is to generate a unique build number, label the source code, then pull the source code based on the label. Even in source code management systems that aren’t completely transactional, retrieving source code by label is going to be consistent which is the key goal. This allows the build to run at any time without requiring developers withhold from checking in or out source code due to fears of interfering with the build. For best consistency, the build process should label all of the necessary source code in as short an interval as feasible (to guard against drift between projects) and then it can pull the source code as needed during the build process.

The build should be easy to extend with new projects. This requires spending a little time considering how to externalize what projects to compile, where to get them, and where to put the output from the raw build process itself. We spent some time writing a standard build script that integrates with our product of choice which uses a single data file to tell it all of the information it needs for any one product. This lets us set up new products very quickly and amortizes the development effort of making the build process over multiple projects.

Some example products that can help you fully automate your build:

1. ANT: Free and capable. Probably the best freely available build system. Available in many flavors to match your technology (like NANT).
2. Visual Build Pro: We’ve used this on several projects and it’s what we use internally. Sports a great GUI for developing and debugging the build process and comes with build-in interfaces for pretty much anything you want to talk to on Windows, and can be easily extended. Great for folks that prefer an IDE. Very cost effective.
3. MSBuild: Visual Studio ships with an internal build environment that can be extended to handle a number of tasks and, with enough force of will can be used to do most anything you want. Debugging and extending it to perform tasks beyond basic compilation and file copying can be a challenge, and your time is probably better served using a dedicated build framework.

Centralizing the Build

Once you’ve automated your build, you can move it to a central server. To centralize the build on a common server, you need to have a mechanism that satisfies at least the following:

1. Anyone can trigger a build remotely: Anyone (subject to some basic security authorization) can trigger a build, and do so without any particularly specialized knowledge. The build system will automatically know if it’s safe to build (such as ensuring conflicting builds aren’t run at the same time). This has to be available from anywhere developers are.
2. Easy access to the status of builds remotely: It should be trivial to know if a build is underway and to know the historical success of the build process. This has to be available from anywhere developers are.
3. Works logged off and through restarts: The build process should not require the system be left logged in or require manual steps to bring online after a computer restart.
4. Runs as a unique user: The build process shouldn’t use anyone else’s identity to log into the source code management system or access other resources so it’s very clear from an audit perspective when the build did something.

Some example products that can centralize your build:

1. Cruise Control: Pretty much the standard. Free, capable, and satisfies the requirements. We use the .NET oriented version, Cruise Control .NET for our in-house build system. The user interface is fairly primitive, and the non-web client is unduly cranky, but it is a good remoting system. It isn’t a particularly good build system - you’ll want to use one of the products listed below.
2. Automated Build Studio: This commercial product for the windows platform is reasonably capable and is both a build product and a build centralization system. While it is capable, the pricing is fairly high when you compare it to Cruise Control + Visual Build Pro (see below) which is substantially as capable. This is because you really don’t need may Visual Build Pro license(s) on a typical team, but you will need a number of Automated Build Studio licenses to allow anyone to invoke and monitor a build. The centralization capability is relatively new in the product. The main reason to go this route is that one configuration IDE can give you centralization and automation, so you are trading money for time.

Challenge Your Team To Fill It Up, Then Buy A Bigger One

While it is very tempting to recycle some old developer system or server as the build server because it doesn’t feel like performance should matter, investing in a high speed build server can pay back quickly by allowing the build process to be optimized for strictness and reliability instead of performance while keeping it fast enough to preserve the development team’s attitude that builds are free. When your build process gets long (up to 1 hour) your team gets a great boost in productivity by purchasing just one system. Consider that a great new build server should top out at $5,000. If you replaced it every 18 months it’d double in performance at the same price.

When specifying a build server, you want to emphasize single processor performance and disk performance. Memory is generally not a particular issue, so invest in the fastest pair of disks you can find with a good hardware RAID controller (for the very best disk throughput) and the highest gigahertz single socket processor you can get. Build processes can very rarely take advantage of multiple cores, so anything beyond two cores isn’t going to speed up a single build, but gigahertz will. Finally, make sure it has a big network pipe to the configuration management system because it will spend a fair amount of its time pushing things in and out of it.

Automated Unit Tests

Connect your existing unit test system to the build system to automatically perform the unit tests on each build and post the results. This can generate some useful metrics that you can use to understand the quality and progress of your development:

1. Increase in Proportion to Features: As you are claiming victory adding features to your system, you should see a linear increase in unit tests. For example, if you have 100 unit tests and the current system has say 200 design features then in broad terms for every 20 design features you should see 10 unit tests. It isn’t completely accurate- it’s a guide line. However, if you see only 5 unit tests and 25 features, you know something is up
2. Indication of Design Complexity Problems: If you are seeing unit tests routinely break that previously worked, or a single unit test repeatedly break, this will tend to indicate that the design of the system (the architecture or software patterns or implementation) is unreasonably complicated for its feature goals. In the abstract it’s often hard to have team discussions in these points because it’s all a matter of tradeoffs, experience, and opinions. This will give you empirical evidence that the software is overly complicated to keep functional.
3. Indication of Performance Issues: Every unit test, when tracked over time, gives you a clear trend to understand the performance of your system. It provides a highly standardized test case where the exact same routine was run on the same hardware in the same way, and timed. If you see the duration of your unit tests change (up OR down) it’s worth investigating - it may be failing quickly or slowly internally, or you may have counter-optimized the code.

Most Automated build systems can directly run one or more unit test frameworks. The build products discussed above each can do this. If your build system can’t, you might look at one of these. You should definitely check out:

1. NUnit: Unit testing for .NET. And yes, we get the irony that the site is written in PHP. This is our unit test framework of choice, however it’s worth noting that we’d be a lot less enamored with it if it wasn’t for ReSharper’s ability to act as a dramatically better test runner.
2. JUnit: Unit testing for Java. Don’t let the nearly comical web site fool you, this test system is all meat.
3. TestComplete: A commercial product from the same folks that made Automated Build Studio. It goes way beyond unit testing (as do most commercial test products) and integrates with both Visual Studio and Automated Build Studio, so if you go with ABS you might look to use these together. It earns our honorable mention because it’s reasonably affordable, easy to use, and very approachable.
4. Hundreds of others: There are many automated testing products out there. We recommend you start with something very simple and straightforward and stay away from the large enterprise testing systems. These are really not meant for the needs of small and mid-sized development teams, particularly where your QA staff are largely development trained.

Run It Every Day

Every development project should be built every day if there’s any change made to the source it comes from. The build centralization systems mentioned above can detect if there is any change to the affected projects checked in and automatically queue a build for a fixed time of day (for a scheduled, automatic build) or a few minutes after the change is checked in. The latter is a great approach for a team that is fully adopting agile development practices: Configure it to automatically start a build after any checkiin after there haven’t been any checkiins for a few minutes. This gives rapid feedback to the team and encourages good configuration management discipline: What you check in better build and pass tests, so don’t check it in until you’re ready.

This frequent execution will ensure that build changes need to be coordinated with code changes because they’ll fail the build immediately otherwise. This quickly will instill the discipline within the development team to keep the build clean, which in the end takes up the least time. Just like it’s easier to maintain something than fix it, it will take the least time across the team to keep the build live and accurate than deal with the downstream consequences of not having your house in order.

Get There In Stages

The great thing about centralizing and automating your build process is that you can get there incrementally. You can make minor investments in time from various team members and each investment will add a little value to your automated build process which in turn will generate a return from there forward. There aren’t a lot of development practices that can be incrementally adopted in such small chunks yet produce steady returns.

The incremental approach is highly recommended because it helps conquer some objections by not requiring a major change in developer habits or a major investment in developer time up front. You can even usually get it done with such small investments in time that it doesn’t need to be a formal project or even a formal part of your project plan, if you are concerned about internal resistance to spending work on non-development activities.

Wait, What about Visual Studio Team System?

Microsoft’s Visual Studio Team System (VSTS) does provide all of the infrastructure you need to automate your build process - centralization, configuration management, testing, reporting, the whole package. If you can afford its licensing fees and the investment in time and resources it takes to set it up, it will fit the bill. VSTS is aimed at larger development teams - on the small side 20+ developers (people actually writing code). The tools and techniques discussed in this article are intended to provide value on teams down as small as two developers. Unless you have large development teams or free VSTS licenses, it’s probably not your best bet.

About Product Recommendations

This article features specific product references and recommendations. Neither the authors of this article nor eSymmetrix are affiliated with any company mentioned, nor have they received any consideration at any time from a party with an interest in these products.