Reliable Systems

In the past two years I’ve heard the term Technology Debt thrown around to justify a number of technology decisions.  In an effort to come up with a term that would bridge the business-technology gap, someone came up with Technology Debt to indicate that you were basically creating a future liability that would have to be paid back - rewriting a section of code or switching out a module, whatever.  Since business folks deal with assets and liabilities routinely, expressing subtle technology problems in financial terms has a lot of appeal.  The basic concept is most frequently used in relationship to software: Let’s say you defer something to a future release so you can get the release out the door.  Perhaps the feature will only work for very few customers but you don’t have time to generalize it, or the solution won’t scale as new users are added.

This is fine as far as it goes, but like any metaphor while it may be a way of explaining some aspects where two things are in common it’s very easy to overextend because in fact it isn’t a true financial liability.

Take a hypothetical example: You are supposed to add federated identity to your web application. What you want to do is create an identity broker that will allow your SaaS application to connect with Microsoft’s ADFS, OpenID, and a few variations of SAML. You believe this will get you the market reach you need, and by creating your own identity broker you can decouple your application from changes in this still evolving space, as well as support your own native security technique.

As you dig into it, you realize that this just isn’t feasible in the time you have: You need to get a solution done in two months to meet a commitment to a customer, and it turns out this customer just uses ADFS. It just so happens that your web framework can easily work with ADFS directly, so to save the schedule you drop back and just do ADFS. From a development standpoint this is a hack - you are doing something quickly you can’t extend to meet the original requirement and you’re pretty sure that you’ll need to undo this later and do it right, which will cost more than just doing it right the first time. This is Technology Debt.

Metaphors have Limited Application

The metaphor doesn’t hold for long.  First, unlike real debt there is no external requirement to pay this back.  Perhaps you never will need to support more than ADFS, or that after all of the talk customers just won’t adopt federated identity.  At the start of each release cycle, you can look at the competitive market and see what is the correct, most important work to be done.  It might be that you’ll have to make good on something you deferred, but you might not.  If you didn’t, it never was debt.  How confident are you that you can tell the difference right now?

Second, all technology has future liability. The more code you write, the more you have to maintain and support. That has a cost as well for the future. Depending on the nature of your product it could mean that you have to support questionable past API decisions or obscure and intricate features of your product. Every feature has a cost to maintain, every line of code you use or reference in a third party library has weight. Only talking about deferred development as technology debt implies that what you have right now is all asset, but it isn’t.

Rampant Misuse

The biggest challenge I’ve seen is that the metaphor is used to push development team goals on the business without having to adequately justify them.  By handing business folks something they can easily relate to, it’s easy to gloss over the underlying technical implications.

For example, say you have an application that’s currently written in Visual Basic 6.  You can justly claim that Microsoft has dropped support for it and that the day is coming when it will no longer run on the latest operating system (Microsoft has committed that it will run on Vista and Windows Server 2008, but that will likely be the end). This sounds very alarming indeed!  Naturally, the answer is to rewrite the entire application in .NET 3.5. Yes, this will take longer and cost a lot more than just adding the features you need to the existing code, but it eliminates all of the technology debt represented by that old nasty VB6 code.

Now look at it a little more objectively. Yes, you will need to eliminate that VB6 code at some point - notably when you are upgrading from Vista to whatever’s next.  When will that be?  Well, if you believe the hype that people love Windows XP and may just consider Vista in the future, you have some calendar time. As long as it’s done by then, Microsoft dropping support isn’t a real issue.  We’ve yet to work with a client who’s VB6 application didn’t work just fine on Vista, thank you very much.

Second, with rare exception every modern technology has a half-life. The notable exceptions are COBOL, C, and C++.  You can with confidence know that these languages are still going to be in active use in 15 years and that code you create now will work with minor to modest adjustment on current platforms at that time.  I would dispute if a similar claim can be made for the latest crop of languages.  A major reason for this is that modern languages are really whole environments - a programming language and an API/runtime.  While it may be theoretically possible to write a C# compiler without toting along the .NET runtime, it isn’t going to happen.  Likewise for PHP, RoR, and even Java.  Java’s framework is much more shallow which ironically will likely give it more life because it can adapt to radical changes in computing environments and there is already an intent to decouple most of the code from specific frameworks.

This means that today’s latest and greatest environment will be tomorrows VB6.  Just listen to people that jumped on .NET 1.1 discuss how they’re going to upgrade to .NET 2.0 or later, and that’s really not a particularly dramatic move.

Fundamentally, when you purchase software you’re making a bet in the vendor and community behind it. This is one place where the LAMP stack has a number of advantages because of the openness of the environment and the institutionalized tolerance of actively supporting releases for a long time. This is fundamentally enabled by their open source nature, but even if you had the source code for a commercial product it’s not likely you could do much with it; unless it’s fairly simple the burden of maintenance is likely higher than the cost of replacing it. Counteracting this are a lot of hidden costs to open source that may be difficult for a small team to absorb.

Easy Metaphors Seldom Produce Great Outcomes

Next time you’re trying to bridge the gap from technology to business, try to stay away from simple metaphors like Technology Debt. Instead, have real conversations to articulate the potential business impacts of the technical decisions and then hear from the business what they are concerned about. With a real dialog, everyone is in on why you deferred work to later and what bet is being made, and no one will be surprised when you do show up in a year to start porting that VB6 app to .NET.

If you spent several hours some afternoon researching on the web what technology is the best for your next project, you’d probably come to the following conclusions:  Linux is the best, or perhaps the Macintosh…  Of course everything can be written in PHP or Ruby on Rails.  If you’re feeling very stuffy, you might be old fashioned and use Java or .NET, Windows or any flavor of Unix that isn’t Linux.  For your database you should just store everything as XML files, but if you feel compelled to have a database use MySQL.  But if you’re still a slave to the 1990’s then you might decide to keep using the corporate dinosaur- Oracle or Microsoft SQL Server.  In the end, the only constant is that whatever you’ve used in the past is certainly out of fashion and certainly a slow, archaic approach to solving problems.

Nearly all teams work within a relatively closed ecosystem - the technologies and people represent only a minor subset of all technologies available today.  Even within these small groups the number of choices at every level are daunting.  Even if you’ve selected your OS, language, framework, and database - what architecture model are you going to use?  What is your data access and caching strategy?  At each turn you’ll want to pick the best option but have a flood of choices to select from.  Many people can tell you about how they led a project that used any particular technology and it worked like a champ with no drawbacks.  On the surface it makes it possible to defend just about any new technology as the way to get your next project done.

What is very hard to find is a real comparative analysis that highlights in comparable situations the results with different technologies.  It’s not a surprise this is so - such analysis is very expensive and time consuming, and few companies would try to solve the same exact problem using competing technologies because it’s not the business they’re in.

Where does this leave you?  What technology should you use on the next project?  Most likely, you’ll have the best success with an incremental improvement on the technologies you already have in your toolkit.  Why?

Infinite Solutions in Infinite Diversity

Most conversations in technology on the web are exclusive - they advocate X over Y or Y over X.  The truth is much more that X or Y can both solve the problem but do it in different ways and with different levels of effort for a team starting from scratch.  What’s much more useful is to ask why you can’t solve the problem effectively with the tools and technologies that are a natural fit for your environment.  Even if a new technology may be the easiest way to solve the specific problem you’re looking at it may not be the best choice when you consider everything that goes into creating the entire software system and maintaining it over time.

When confronted with a strong advocate for a technology shift, keep the conversation focused on the benefits of shifting away from the natural or familiar selections for your organization.

New Methods are Expensive

When you introduce new technologies or tools there is always a short term hit.  Most respected research indicates that even technologies and tools that have a substantial improvement in effectiveness are at best neutral on the first project that uses them.  The most successful technologies and tools are ones that are evolutions of things your team already knows.  The more divergent it is from that, the more time it will take to get over the learning curve, establish best practices, and generally become effective.

Known Problems vs. Unknown Problems

A common challenge when comparing a new technology against existing methods is not recognizing that while you know of all of the problems with your existing technologies, you don’t know of the problems with the new ones.  This can lead to a comparison that shows a number of critical problems with the current technology, and none on the new technology.  It isn’t that there aren’t problems with the new technology, it’s that you don’t know what they are yet. Whenever you put in a new technology, no matter how promising it is, it is going to have new, unexpected problems.

What’s worse is that your organization most likely has workarounds for every problem you’ve encountered, so they don’t really have the impact of a new problem.  Your development team may not know about them, but talk to the operations staff, support staff, and your users before you assume that a technology problem that worries you really is at the top of their list.  It may be that the big memory leak in a third-party library that has you wanting to rewrite a subsystem is conquered in production by having a script reset the service every night.

Existing Code and Libraries

It’s very easy to underestimate the value of existing libraries and practices in effectively solving problems.  When faced with a new assignment, your developers can draw from a large pool of existing, tested solutions to a range of the more mundane, plumbing aspects of the solution.  This includes storing user information, reliably working with data storage, security systems, and other functional requirements that aren’t unique to the problem at hand.  These software libraries accrue over time as developers face similar problems even in development shops that don’t place a high emphasis on modularity and reusability - as long as you have source code control and developers that aren’t paid by the line of code, they’ll naturally find ways to adapt and remold things they’ve already done to fit new needs.

When you have a major technology shift, losing the use of this common body of code will require the first project to reinvent it.  On the surface this may seem straightforward but it’s usually held up by a desire to understand exactly how to best accomplish the same common tasks in the new environment.  For example, you might have written your own security system for your previous environment which you’ll then need to either re-implement or drop in favor of a built-in capability of the new environment you’re targeting.  What’s worse is you need to make these critical decisions at the time when you have the least experience with the new environment:  Is its built in security system really sufficient for your needs?  What about logging?

Your Customers Don’t Care

With the exception of a narrow range of situations (such as developer tools), your customers really don’t care what technology you use to implement your solution.  After all, they’re buying your solution not the technology you wrote it in.  Even if the IT representative of the evaluation team in a potential customer objects because your entire solution is written in a technology they don’t like, in the end they are often overruled unless they can point to a practical implication that you can’t mitigate.  For example, you may get overruled because it’s a Unix shop and they won’t accept a solution that only runs on Windows.  Even in the most extreme cases, if you provide enough customer value it will conquer any customer technology objection.  If the prospect has no Windows servers, that translates into a finite cost for them to support a unique system in their environment.  If your value well exceeds that, then it isn’t the key challenge to crossing the chasm to that prospect.

We often hear developers discussing internals of software development and giving them the weight of user requirements.  If it isn’t visible to the customer, it isn’t a requirement. In the end, your customers don’t pay you to have a beautiful object model.  They don’t care how hard it is for you to create your product or what hoops you have to run through.  For them, it’s a cash for capabilities decision.  It may be true that doggedly sticking with an old technology will mean you can deliver fewer features with each release, or you won’t be able to run on the latest operating system but in the eyes of your customers the question is still how compelling the functionality is and whether you can run on the operating systems they use.

Ignore the Pundits

If you’re part of a shop that has a track record of producing results, be proud.  Don’t worry about what is all the rage at producing the next social networking site, focus on what is effective for you.  For projects that can afford the risk, take the opportunity to incrementally improve your technologies and methods:  Try out a new version of the development framework or new capabilities of the latest database version.  Just remember, you can always tell the pioneers:  They’re the ones lying on the ground with the arrows sticking out of their backs.  Unless you’re part of a dedicated research team, most often you’ll get the best results by waiting for the first round of adopters to figure out what did and didn’t pan out with the newest release and then benefit from that experience.  There’s no satisfaction in burning six months working out the kinks of version 1.0 just to have everything addressed in version 1.1 published a month later.

What’s Your Experience?

Have a great story about being the pioneer, working a project that was packed to the gills with the latest and greatest, only to fall on its face?  Or perhaps you found raging success completly severing your ties with the past?  Drop me a line or leave a comment about it.

Consider this simple point:

Effort ≠ Value

Think about it for a few minutes and it seems patently obvious: Just because something’s difficult doesn’t mean it has great value. For example, if I want to mail 50 letters to clients and I put an individual stamp on each one instead of using an automatic postage machine I’ve achieved the same value: I can now send these letters to each of my customers. They’ll get there just as fast, the postage is just as valid. Therefore, if it takes me 15 minutes to put the stamps on one by one vs. about 1 minute to run it through a machine I’ve spent 15 times as much effort to achieve the same value.

It works in reverse as well: Just because something has great value doesn’t mean it’s intrinsically difficult. It may be exceedingly valuable to me to get a message to a client that lives on the other side of the country, and yet it’s really easy to do: In just a few seconds with my cell phone I can reach out any time of the day, from virtually anywhere. Low effort, high value.

Obvious, and yet we ignore the implications of this every day. We naturally assume that anything worthwhile takes effort, and that anything that takes a lot of effort was worthwhile.

Good examples of low effort, high value

Cosmetic defects are a classic example of this. It isn’t unusual at all to go through a new software application and find a substantial number of cosmetic defects: Alignment issues, inconsistencies in language (is it login, logon, or user id? Do you click or press that button?), spelling or language errors and a range of items that aren’t application behavioral issues (like tab order). Developers tend to instinctively minimize these issues: They’re trivial to resolve and they don’t prevent the application from working. They aren’t anywhere near whatever hideously complicated part of the system the developer is really worried about, and they’ll take no time to get right later. They can’t be that important, so development teams tend to not talk about them or work them. Even the term “cosmetic defect” is often used as a label for trivial or low value: “that’s no big deal, it’s just a cosmetic issue. Now let’s talk about that rare crash on every other leap year if you attempt to delete a customer with no records!”. This perspective isn’t even particularly unreasonable if you’re looking at the development process from a risk management perspective: You know the issues can be cleaned up quickly and without a lot of technical risk, and if you clean them all up now you’ll still have to do a recheck of the system before release because new ones will show up.

Now look at it from the standpoint of an end user of the system. The system is a black box: They don’t see the really artful code that figures out automatically when they enter a name as Last, First or First Last or how you managed to make a really fast look-ahead search system despite the large number of records you have to work with. Instead, they’ll see what’s right in front of them: The user experience of the application itself. If they start it up and notice immediately that things aren’t lined up vertically & horizontally or there are spelling errors it will bring rise to the classic line of reasoning that if you didn’t get this right, what hope is there that the black box is right? The more you protest that this is easy to fix the worse it gets: If you couldn’t get the easy to fix simple stuff right then there’s now no way the detailed 12 step process for determining how much to bill a client is going to be right, and the user is going to have to check it all before you regain their trust.

The good news is that this direction is the easiest to avoid as a manager or team member of a software development project. Once you’ve had the above experience once or twice, you will start to get wise and do a cosmetic issue pass at strategic points in the time line - usually just a few builds before it’s going to be seen by people outside of the team. You’ll be surprised at both how many items show up each time, and how easily they clean up. Then, while you’re sweating during the big demo about whether you’re going to get a runtime error you’ll at least have the comfort that what they are seeing while they’re waiting for the next page represents the good work your team did in a way that communicates to the average user that can’t see behind the curtain.

Good examples of high effort, low value

This trap is more dangerous and harder to avoid. At many points through the development process you’ll have opportunities to chose architectures, designs, algorithms and other items that will either increase or decrease the effort it’ll take to complete the project. You might chose to not use that built-in dialog to open files and instead make your own dialog because of one annoying behavior you really want to avoid. Or decide that you want to make a better column sizing routine for the grids you display so that you can avoid either trying to cram too much on a small screen or having acres of empty space on a large one. None of these are on their own bad ideas necessarily, and that’s part of the trap: Most development processes by design tend to focus team attention on the things that are hard, high risk, or just time consuming because these have the biggest ROI for project management activities. This reinforces our built in instincts to presume that the harder the work, the greater the value.

What this ignores is that the value is essentially constant regardless of effort: Any particular feature or capability has a set value in the eyes of the user. Our goal is to realize that value with if not the minimum effort then something that appears (prior to construction) to be the minimum effort that has an acceptable risk. In its most direct form, this means that the user places the same value on a five thousand line algorithm to determine optimal column width and using a method built into a control to get it right, as long as the outcome achieves their expectation.

<tip>Corollary: Be sure what’s important to you is also important to your users before investing a lot of time. Perhaps they don’t care if there’s a bunch of empty space on their 24″ widescreen monitor as much as you do. Get evidence commensurate to the effort you think it’ll take to resolve the issue.</tip>

Understand the trap

This issue tends to manifest itself in some classic ways. One is when a developer argues passionately in favor of a complicated algorithm even in the face of peer review that casts substantial doubt on its necessity. Typically the developer caught one small aspect of the problem and has ruthlessly optimized for it, and uses that one point as the proof of why simpler approaches don’t work (”If users are constantly switching back and forth between these two displays it’s 30% faster to do it this way than what’s built in”). These items also tend to be defect prone and difficult to explain to others.

Complicating this trap are a few factors:

• There are hard problems to solve: You can’t assume every hard problem is really an overcomplicated solution. Most applications will have at least two places where there is some real trickery and engineering to get the right result each and every time. If there weren’t, your users probably wouldn’t want the application in the first place.
• There are low value problems to solve: There are hard problems that have to be solved, and some of these are even relatively low value to the customer but are still a requirement. Consider this example: The customer places relatively low value on your application not crashing when they run it. Don’t get me wrong - if it starts crashing they will be very upset, but they simply assume that it won’t crash. All joking about Microsoft aside, any application you write is virtually guaranteed to be more crash happy than Microsoft Office is. So you’re going to end up investing a lot in something users don’t really place a lot of incremental value on.
• Developers are Optimists: Developers like hard problems (after all - hard problems are valuable problems according to our instincts) and want to solve them. They will underestimate the effort going in and overstate the value of the journey. If it’s a new problem, it’s unlikely that their estimate is particularly great even if they aren’t focused on why a particular complicated approach is necessary.

Striking the Balance

How to avoid this trap? First, For these problems to get out of hand it usually requires the ability for one or two developers to go off away from the herd for long enough to cook up a complicated idea, justify the effort to themselves, and then get far enough into the swamp to be in real trouble. Depending on your specific software development approach, find ways to catch the telltale signs before developers sink enough effort into the solution to get permanently attached to it.

Second, be fully prepared to throw out an already developed solution regardless of how much code or effort it is. In other words, the decision on whether or not to back up and take another approach should be largely blind to how many lines are being thrown out as long as they are all part of the same solution. Even at this point the instinctive desire to equate effort and value will creep into the entire team’s thinking: people will look at the large block of code and assume that it must be necessary, we’re just missing the subtlety of why it is the light & the way. This is what source code control is for (you do use source code control, don’t you?). It enables you to with no fear reject a bird in hand for a simpler bird that may converge faster, have fewer issues, and ultimately be a more cost effective way of providing the value your customers are expecting. Remember that even if you’ve taken a complicated implementation through initial unit testing, there is still a substantial investment that will be made in that code over time.

A production implementation is worth many theories

This article has been talking about high effort ways of achieving value in software, and the approach shouldn’t be generalized into applying to applications simply because they are large or complicated, or even any particular solution that is large and complicated - provided it got there incrementally over time. While it’s often tempting to look at a few hundred line block of code that does just one thing and think that in this age of objects, partial template classes, interfaces, and reflection there just has to be a cute, simple implementation that’s less than half the size and complexity of the current solution there are two key issues with this thinking:

1. Large, stable code already achieved value: If the block of code is substantially stable and accepted by the customers then it has achieved its value and any effort spent on refactoring it that doesn’t also deliver more value to customers isn’t improving the value of the application
2. Refactoring introduces defects: It’s virtually guaranteed that in the process of refactoring the existing routine sufficiently to make a good dent in its complexity you’re going to introduce some new defects just due to conceptual or implementation oversight during the process. It’s generally not considered a great justification to management that you introduced defects that then require expense to clean up in an effort to avoid possible future expense maintaining code.
3. Code gets larger because it handles very subtle points: If the code organically grew over time to be a complicated routine, it probably did so because it was progressively asked to handle a number of interesting boundary cases that experience with the application proved necessary. In the minds of the users, these subtle behaviors can be some of the greatest value they place on your application - and yet they’ll never mention the feature when talking with you about it. Therefore, when you refactor it you have to preserve every little behavior, and that is often infeasible (with the exception of true defects in design of the original code - well isolated routines that can be replaced with provably equivalent code)

If it’s already in production and doesn’t have a critical flaw that the business needs addressed, leave it alone.

Clearly communicate the end-user value within your team

The best technique to avoid this problem is to make sure your development team has a tradition of discussing the end-user value of the work being done. This tradition would mean that anyone gets to clarify what the end user value of any work that’s being done is - that’s a free question never met with ridicule. It may take some practice to make sure the question is asked correctly, e.g. it’s asked in a way that gets the entire team to back up and be clear about why the work is important. Within that questioning, its important to make sure the discussion is based on what’s important to the users of the system, not the developers or other non-users. There are times to focus on maintenance or other non-user issues but with rare exception it isn’t the reason you’re writing the system.

With some practice, this will become a strong self-regulation mechanism for the team, ensuring that your discussions about design and approach are grounded in the needs of your customer. It creates a good mental yardstick for how much time to invest in a solution before going back to the customer to re-verify the requirement.

What’s your experience?

Have a good story to share? Have a critique? Post your comments or drop me a line to continue the conversation.