Now step back for a minute and look at it from the employee’s perspective.  When someone needs access to IT resources within your company, there are really three interests you have:

1. Access: People need equipment, software, accounts, and access to IT support.  This is the basic block & tackling that you are handling now.
2. Effective: The tools you provide need to deliver on the business needs the user has.  Whatever’s in the way of that - defective equipment, user training, or suitability to task needs to be addressed.  The best computer with the right software in the hands of a user that doesn’t know how to use them is worthless.
3. Security: Access to your network means access to all of the data and work products within it.  You need people to understand how you approach security, what they are and aren’t allowed to do, and how you’re going to work with them to maintain security.

Setting the Stage

Employees often develop a personal and possessive feeling about the equipment provided to them by a company.  They think of it has their computer, just like they have a computer at home.  This creates a range of problems for your organization by extension:  If it’s their computer then when there’s a problem they’ll want their computer fixed, not a different computer that’s suitable.   They’ll come to regard problems personally, not objectively.

Instead, you want users to look at the equipment they’re provided to do their jobs as just that - tools that enable them to be more effective.  Stepping back into the big picture, a computer isn’t any different than a wrench or a filing cabinet.  It isn’t their computer or phone, it’s the company’s - designed to make them effective at producing whatever the company needs.   When your user community gets this, they’ll self censor their support requirements:  Watching a DVD movie on the company laptop won’t feel support-worthy.

The best time to establish this is to set the right expectations up front:  Have this conversation before the user gets their network account.  The goal is to make sure they understand that:

• You’re committed to their success.  You’re passionate about making sure they’re effective.  You have a support system designed to make sure that their issues get resolved quickly, and you have provisions for support off hours and when they’re on the road.  If they aren’t sure if you can or should help with an item, you want them to engage you anyway - You’ll let them know.
• The technology is there to make them effective at their job. Your job (IT) is to make sure that they are as effective as possible at that.
• They are responsible for their effectiveness. If they need something - training, repair, whatever - it’s their responsibility to get it, and they can get it.
• They are responsible for their user account. Anything anyone does with that account is their responsibility.   That means if someone figures out their password, or they leave their computer unlocked, or otherwise treat their user account with less than the respect it deserves then they are going to be held responsible by the company for that.

Support Your Local Sheriff!

It’s painful to hear on Monday that a user was trying to get something important done and couldn’t due to a simple issue you could have resolved.  Perhaps they knew they could have contacted you for support - but didn’t for whatever reason.  What users will remember is that they had a problem, and it kept them from getting things done.  All of the work you do to support users - special on call staff, phone numbers, email contact, whatever - didn’t work because they never got called upon.

To address this, you want to address as many of the human factors that keep people from calling on support as you can.

1. Make sure you’re always available: The cost of setting up a toll-free number for users to contact support is trivial.  If you don’t already have an on-call rotation, set one up and make sure there’s someone to answer that toll-free number at all times.  The same person can answer an email address designed for support.
2. Make sure they know all the ways: In the past, we’ve published business cards with the 800# for support and email address, and we put these cards everywhere:  In laptop bags, in a card holder at the front desk, anywhere that we could think of so that there’d be one around when a user needed to know how to contact support.
3. Talk to users about it: Be cheery.  Make sure they know that you personally are driven to make sure they’re successful, and you look at it as an honor to help them out after hours.  They need to really get that you want that phone call, because you need to conquer the very human desire not to bother or inconvenience other people.

We really recommend making up a business card that has all of the key information a user needs - the contact information for support, company fax number and main phone number, remote dial in for voice mail, common URLs for external access to email and other services, pretty much anything they need to know on the road.  I’m sure you have it all committed to memory, but if you’re an employee that doesn’t travel every day you probably don’t.  Little steps like this can dramatically affect the general user population’s opinion about IT.

Security Begins at Home

You want to make sure that each user gets how seriously your organization takes security.   People often don’t treat their user account with the same respect they’d treat a physical key or card.  Most users wouldn’t give a stranger the keys to their office or building but would give their password out over the phone to someone who claimed they need it.

People worry a lot about security threats from the Internet, but most break-ins - overwhelmingly - happen from inside.  Most of these are done either through social engineering (where the intruder convinces someone to give them access) or by a disgruntled employee.

To address these common threats, you need to address the key social aspects of security.  In addition to normal sensible security practices,  we recommend establishing a few policies:

1. IT Personnel NEVER ask for passwords: Make it clear to your IT Support organization and every user that no one in IT will ever ask them for their user ID or password.  Therefore, if anyone calls you asking for that information you know one thing - they aren’t authorized to it.  If they give their password to IT, or IT hears that they gave it to someone else, their password will be reset.
2. No one will use their account but them: If IT needs to do something logged in as you, they’ll do it in your presence - after all, you are still accountable for what happens with your account.

The second one may cause some heartburn with your desktop support staff- they’re probably used to solving a range of user problems by accessing the computer as the user, and anything that’ll get in the way of that is a problem.  While it may cause some inconvenience - you aren’t going to be able to do work that requires logging in as the user if they aren’t around - the message this sends to your users about how serious you are about security is essential.  You need to be cleaner about the rules than they are.

What about Non-Employees?

What should you do with contractors or others that need access to your network, even temporarily?  If they are getting a user account, they should go through the same procedure.  You have the same goals:  You want them to be effective and not compromise your environment.

Finally, Ditch the Input Devices

Most computers come with mice and keyboards that are dirt cheap.  If this is what you’re using and you’re recycling a computer, please - get a new mouse and keyboard.  Most computer companies do the same thing when they process returns.  The fact is that keyboards get filthy quickly, and while I may not mind the crumbs from my pop tarts, it certainly isn’t going to create the right impression if I get one that’s full of someone else’s.  You should be able to score new ones for your HP, Dell, or whatever for not more than $40 and really - with what employees cost in salary and other expenses, don’t you want them to know you care?

Have a story about how you support your new users?  Share it in the comments below or drop us a line to tell us about it.

In the past several months the three partners of our firm have been scattered to handle all of the challenges of our growing business, but with the completion of a major engagement with a client we had the opportunity to spend some time back together to focus on more long range concerns.  We had intended to spend the time working on product and marketing strategy for our upcoming commercial release of Gibraltar, but instead on a whim decided to do something we called the Code Monkey Challenge.

The goal was to make something complete and useful within a very short period of time.  Complete for us meant we were going to create a software component, test it, package it, and publish it to a new web site with descriptive content all in 48 hours.  We didn’t even have a hosting company set up beforehand.  To be useful it had to be usable by people outside of our company as delivered, and fulfill a real need, at least for a set of users.

We started at 9:00 AM with the goal of publishing the final version to a new web site by 6:00 PM the following day.  Our initial approach was to have one hour sprints of work divided between each team member, then meet and review and set up the next sprint.  In the end we used three hour sprints, delivering to each other through our source code system at the end of each sprint.  We made our goal - you can see the results at www.gibraltarsoftware.com.  Most importantly, we achieved our result of bringing together our team members.

We eliminated any distraction we could so that we could apply as much of the 48 hours as possible to the project.  We worked well into the evenings and had others provide some support to make sure we had food, coffee, and no outside distractions.

The intense time pressure gave us a good tool to eliminate most conflict: there simply wasn’t time for much philosophy on how to approach the different technical aspects.  The lack of time removed ambiguity that otherwise would be the source of conflict.  There wasn’t much time for yak shaving either - we had to produce results at the end of each sprint.  Still the first few sprints were spent mostly in exploration and validation of the approaches with our first rough cut of each element done by the end of the first day.  The second day was then about refining and documenting.

While we did skip over some aspects that we would normally do on a larger scale project, we did include most elements of our preferred development process.  We did coordinated sprints of the team separated by an after action review to adjust our plan.  We distributed code and results between the team through our source code control system, we wrote automated unit tests to verify key aspects of the system, and did peer design reviews.  If anything, the lack of time made us less likely to bypass most parts of the process because we knew we wouldn’t have time to dig our way out of any problems we created.

The real goal of the exercise was to bring us back together as a leadership team and establish more of the shared experiences that define interpersonal relationships.  It’s helpful to bridge the gaps that easily develop between architects and developers, developers and designers, engineering and marketing.  We all had to come together to achieve our result because it was clear that we’d all fail if we couldn’t.  Most business projects are sufficiently fuzzy that it isn’t nearly as clear what the cost of not working together is, and politics can overwhelm cooperation.  It’s an experience I’d recommend in any software team, particularly if you’re in a company that can mix skills and responsibilities.

The next time your shop is done with a project, or when you need a break consider doing your own Code Monkey Challenge.  It only takes two days.  Here’s the rules:

• Produce a Real Product: Scope out a product that is really useful to a specific audience.  Sure, you’re giving it away for free, but it still needs to stand up to scrutiny.
• Marketing Material Too: What good is a product if no one knows about it or understand how or why to use it?  Even though you’re giving it away you want people to be able to understand and take advantage of your effort.
• Help and Usage Information: Like a real product, it has to be usable without you sitting over the customer’s shoulders.  Depending on what it is, you need to tell them how to install it, program with it, and provide guidance on recommended usage scenarios.
• In Little Time: Extra time is the enemy - it will reduce the pressure to work together and encourage unnecessary bells and whistles while removing the catalyst that gets everyone to work together.
• Publish to the World: If you’re  a large company, perhaps it’s just the company.  Wherever it is, it has to feel like real stakes to the team:  Your results will be on display.

To set it up, be sure that you can eliminate anything that would distract the team - they can work into the evenings, have food brought in, whatever.  The closer it is to a total immersion experience the better.  It improves the sense of camerarderie developed within the team and ensures the most creative energy is directed to the project. 

If you try it out, or have another software team building story, please drop me a line and tell me how it went, or leave a comment below.

This is fine as far as it goes, but like any metaphor while it may be a way of explaining some aspects where two things are in common it’s very easy to overextend because in fact it isn’t a true financial liability.

Take a hypothetical example: You are supposed to add federated identity to your web application. What you want to do is create an identity broker that will allow your SaaS application to connect with Microsoft’s ADFS, OpenID, and a few variations of SAML. You believe this will get you the market reach you need, and by creating your own identity broker you can decouple your application from changes in this still evolving space, as well as support your own native security technique.

As you dig into it, you realize that this just isn’t feasible in the time you have: You need to get a solution done in two months to meet a commitment to a customer, and it turns out this customer just uses ADFS. It just so happens that your web framework can easily work with ADFS directly, so to save the schedule you drop back and just do ADFS. From a development standpoint this is a hack - you are doing something quickly you can’t extend to meet the original requirement and you’re pretty sure that you’ll need to undo this later and do it right, which will cost more than just doing it right the first time. This is Technology Debt.

Metaphors have Limited Application

The metaphor doesn’t hold for long.  First, unlike real debt there is no external requirement to pay this back.  Perhaps you never will need to support more than ADFS, or that after all of the talk customers just won’t adopt federated identity.  At the start of each release cycle, you can look at the competitive market and see what is the correct, most important work to be done.  It might be that you’ll have to make good on something you deferred, but you might not.  If you didn’t, it never was debt.  How confident are you that you can tell the difference right now?

Second, all technology has future liability. The more code you write, the more you have to maintain and support. That has a cost as well for the future. Depending on the nature of your product it could mean that you have to support questionable past API decisions or obscure and intricate features of your product. Every feature has a cost to maintain, every line of code you use or reference in a third party library has weight. Only talking about deferred development as technology debt implies that what you have right now is all asset, but it isn’t.

Rampant Misuse

The biggest challenge I’ve seen is that the metaphor is used to push development team goals on the business without having to adequately justify them.  By handing business folks something they can easily relate to, it’s easy to gloss over the underlying technical implications.

For example, say you have an application that’s currently written in Visual Basic 6.  You can justly claim that Microsoft has dropped support for it and that the day is coming when it will no longer run on the latest operating system (Microsoft has committed that it will run on Vista and Windows Server 2008, but that will likely be the end). This sounds very alarming indeed!  Naturally, the answer is to rewrite the entire application in .NET 3.5. Yes, this will take longer and cost a lot more than just adding the features you need to the existing code, but it eliminates all of the technology debt represented by that old nasty VB6 code.

Now look at it a little more objectively. Yes, you will need to eliminate that VB6 code at some point - notably when you are upgrading from Vista to whatever’s next.  When will that be?  Well, if you believe the hype that people love Windows XP and may just consider Vista in the future, you have some calendar time. As long as it’s done by then, Microsoft dropping support isn’t a real issue.  We’ve yet to work with a client who’s VB6 application didn’t work just fine on Vista, thank you very much.

Second, with rare exception every modern technology has a half-life. The notable exceptions are COBOL, C, and C++.  You can with confidence know that these languages are still going to be in active use in 15 years and that code you create now will work with minor to modest adjustment on current platforms at that time.  I would dispute if a similar claim can be made for the latest crop of languages.  A major reason for this is that modern languages are really whole environments - a programming language and an API/runtime.  While it may be theoretically possible to write a C# compiler without toting along the .NET runtime, it isn’t going to happen.  Likewise for PHP, RoR, and even Java.  Java’s framework is much more shallow which ironically will likely give it more life because it can adapt to radical changes in computing environments and there is already an intent to decouple most of the code from specific frameworks.

This means that today’s latest and greatest environment will be tomorrows VB6.  Just listen to people that jumped on .NET 1.1 discuss how they’re going to upgrade to .NET 2.0 or later, and that’s really not a particularly dramatic move.

Fundamentally, when you purchase software you’re making a bet in the vendor and community behind it. This is one place where the LAMP stack has a number of advantages because of the openness of the environment and the institutionalized tolerance of actively supporting releases for a long time. This is fundamentally enabled by their open source nature, but even if you had the source code for a commercial product it’s not likely you could do much with it; unless it’s fairly simple the burden of maintenance is likely higher than the cost of replacing it. Counteracting this are a lot of hidden costs to open source that may be difficult for a small team to absorb.

Easy Metaphors Seldom Produce Great Outcomes

Next time you’re trying to bridge the gap from technology to business, try to stay away from simple metaphors like Technology Debt. Instead, have real conversations to articulate the potential business impacts of the technical decisions and then hear from the business what they are concerned about. With a real dialog, everyone is in on why you deferred work to later and what bet is being made, and no one will be surprised when you do show up in a year to start porting that VB6 app to .NET.

Nearly all teams work within a relatively closed ecosystem - the technologies and people represent only a minor subset of all technologies available today.  Even within these small groups the number of choices at every level are daunting.  Even if you’ve selected your OS, language, framework, and database - what architecture model are you going to use?  What is your data access and caching strategy?  At each turn you’ll want to pick the best option but have a flood of choices to select from.  Many people can tell you about how they led a project that used any particular technology and it worked like a champ with no drawbacks.  On the surface it makes it possible to defend just about any new technology as the way to get your next project done.

What is very hard to find is a real comparative analysis that highlights in comparable situations the results with different technologies.  It’s not a surprise this is so - such analysis is very expensive and time consuming, and few companies would try to solve the same exact problem using competing technologies because it’s not the business they’re in.

Where does this leave you?  What technology should you use on the next project?  Most likely, you’ll have the best success with an incremental improvement on the technologies you already have in your toolkit.  Why?

Infinite Solutions in Infinite Diversity

Most conversations in technology on the web are exclusive - they advocate X over Y or Y over X.  The truth is much more that X or Y can both solve the problem but do it in different ways and with different levels of effort for a team starting from scratch.  What’s much more useful is to ask why you can’t solve the problem effectively with the tools and technologies that are a natural fit for your environment.  Even if a new technology may be the easiest way to solve the specific problem you’re looking at it may not be the best choice when you consider everything that goes into creating the entire software system and maintaining it over time.

When confronted with a strong advocate for a technology shift, keep the conversation focused on the benefits of shifting away from the natural or familiar selections for your organization.

New Methods are Expensive

When you introduce new technologies or tools there is always a short term hit.  Most respected research indicates that even technologies and tools that have a substantial improvement in effectiveness are at best neutral on the first project that uses them.  The most successful technologies and tools are ones that are evolutions of things your team already knows.  The more divergent it is from that, the more time it will take to get over the learning curve, establish best practices, and generally become effective.

Known Problems vs. Unknown Problems

A common challenge when comparing a new technology against existing methods is not recognizing that while you know of all of the problems with your existing technologies, you don’t know of the problems with the new ones.  This can lead to a comparison that shows a number of critical problems with the current technology, and none on the new technology.  It isn’t that there aren’t problems with the new technology, it’s that you don’t know what they are yet. Whenever you put in a new technology, no matter how promising it is, it is going to have new, unexpected problems.

What’s worse is that your organization most likely has workarounds for every problem you’ve encountered, so they don’t really have the impact of a new problem.  Your development team may not know about them, but talk to the operations staff, support staff, and your users before you assume that a technology problem that worries you really is at the top of their list.  It may be that the big memory leak in a third-party library that has you wanting to rewrite a subsystem is conquered in production by having a script reset the service every night.

Existing Code and Libraries

It’s very easy to underestimate the value of existing libraries and practices in effectively solving problems.  When faced with a new assignment, your developers can draw from a large pool of existing, tested solutions to a range of the more mundane, plumbing aspects of the solution.  This includes storing user information, reliably working with data storage, security systems, and other functional requirements that aren’t unique to the problem at hand.  These software libraries accrue over time as developers face similar problems even in development shops that don’t place a high emphasis on modularity and reusability - as long as you have source code control and developers that aren’t paid by the line of code, they’ll naturally find ways to adapt and remold things they’ve already done to fit new needs.

When you have a major technology shift, losing the use of this common body of code will require the first project to reinvent it.  On the surface this may seem straightforward but it’s usually held up by a desire to understand exactly how to best accomplish the same common tasks in the new environment.  For example, you might have written your own security system for your previous environment which you’ll then need to either re-implement or drop in favor of a built-in capability of the new environment you’re targeting.  What’s worse is you need to make these critical decisions at the time when you have the least experience with the new environment:  Is its built in security system really sufficient for your needs?  What about logging?

Your Customers Don’t Care

With the exception of a narrow range of situations (such as developer tools), your customers really don’t care what technology you use to implement your solution.  After all, they’re buying your solution not the technology you wrote it in.  Even if the IT representative of the evaluation team in a potential customer objects because your entire solution is written in a technology they don’t like, in the end they are often overruled unless they can point to a practical implication that you can’t mitigate.  For example, you may get overruled because it’s a Unix shop and they won’t accept a solution that only runs on Windows.  Even in the most extreme cases, if you provide enough customer value it will conquer any customer technology objection.  If the prospect has no Windows servers, that translates into a finite cost for them to support a unique system in their environment.  If your value well exceeds that, then it isn’t the key challenge to crossing the chasm to that prospect.

We often hear developers discussing internals of software development and giving them the weight of user requirements.  If it isn’t visible to the customer, it isn’t a requirement. In the end, your customers don’t pay you to have a beautiful object model.  They don’t care how hard it is for you to create your product or what hoops you have to run through.  For them, it’s a cash for capabilities decision.  It may be true that doggedly sticking with an old technology will mean you can deliver fewer features with each release, or you won’t be able to run on the latest operating system but in the eyes of your customers the question is still how compelling the functionality is and whether you can run on the operating systems they use.

Ignore the Pundits

If you’re part of a shop that has a track record of producing results, be proud.  Don’t worry about what is all the rage at producing the next social networking site, focus on what is effective for you.  For projects that can afford the risk, take the opportunity to incrementally improve your technologies and methods:  Try out a new version of the development framework or new capabilities of the latest database version.  Just remember, you can always tell the pioneers:  They’re the ones lying on the ground with the arrows sticking out of their backs.  Unless you’re part of a dedicated research team, most often you’ll get the best results by waiting for the first round of adopters to figure out what did and didn’t pan out with the newest release and then benefit from that experience.  There’s no satisfaction in burning six months working out the kinks of version 1.0 just to have everything addressed in version 1.1 published a month later.

What’s Your Experience?

Have a great story about being the pioneer, working a project that was packed to the gills with the latest and greatest, only to fall on its face?  Or perhaps you found raging success completly severing your ties with the past?  Drop me a line or leave a comment about it.

1. This new system fits into an existing business, possibly replacing a prior application, so you can predict with some accuracy the different aspects of scalability that apply to it.
2. It doesn’t, and you can’t.

The second scenario is the most interesting one. First off, let’s face it - your new system isn’t going to be the next Facebook, MySpace, or eBay. In short, you don’t need to worry about having your system needing to be designed front to back as a super-scalable system. This is good because the options at that level are time consuming and resource intensive.

The key question you need to understand when laying out a new software system is to what degree it needs to scale without being re-written? This scale is unlikely to be your “best case” business size, because scalability has opportunity cost. This scale should be defined as specifically as reasonable, and clearly understood and validated by both business and technical staff. This ensures that if your business grows beyond expectations that it won’t come as a surprise if you need to make even major changes to your system.

Creating facts from Air

Let’s say you’re starting to develop an application that fits into the second category above. You still need to work out what your scalability target is.

To make any decision that is better than random, you have to work out some aspects of the expected scaling of the application. In the absence of real facts to extrapolate scalability from, you need to cooperate with the business side to established presumed facts of the scalability requirements. This may sound a lot like assumptions, but they really go beyond that because these will become facts as you develop the system. As a starting point, make it clear to all involved that:

1. If the targets are low, it should be assumed you’ll have to turn away business because the system can’t scale above them.
2. If the targets are high, the system will cost more and take longer to create.

In most businesses, the second outcome is worse than the first. Why? Because the second is a price you pay up front, before the system goes into service. The first is based on an assumption: you might have to turn away business. You also might be able to realize it in time and address the issue. From a business standpoint, this is a better trade off. Finally, there’s the non-technical aspects:

1. The sooner you have a working system, the sooner the business can validate the market and start getting real data on uptake to adjust your scalability goals
2. Unless the product is a failure, you expect demand to eventually exceed the capacity of the system, it’s just a matter of when. If it does, then you should be able to afford rewriting all or part of the system. In other words, the funds to solve the problem should be available if you have the problem.

From this comes an axiom of scalability:

The system needs to be based on the lowest scale that will provide enough time and money to replace it with a new system.

Put another way, a system that is faster or more scalable than it needs to be for the business was more expensive and took longer to develop than necessary. Think of it like a race car: The ideal Indy Car would fall apart just after the judges validated it won without breaking the rules. Any stronger and that strength could have been put into something else. The time you spent making it more scalable than necessary could have added more features, fixed more defects, or gotten it out the door sooner.

Establish a Growth Curve

The growth curve needs to be sufficient to inform the developers of what decisions to make at each point. To get there, start with describing the scale from the business stand point. During design of the actual system you can keep translating this into the specific requirements for speed, storage, and capacity based on the behavior of the actual system. This will prevent you from achieving technical goals that don’t satisfy the business goals.

For most systems, you want to establish the business goals for:

1. Number of Possible Users: How many accounts will there be on the system? This is an upper bound of the number of people that could access the system if they wanted to.
2. Number of Simultaneous Users: Number of accounts that will be accessing the system at the same time. For most applications, at the same time is likely best thought of as in the same 15-30 minutes.
3. Number of Customers: For most applications delivered to businesses the number of customers (e.g. businesses) drives the scalability of some parts of the system (such as configuration and data storage) will scale based on the number of customers, not the number of accounts those customers have.
4. Data In and Out: If the system is going to have any imports and exports that aren’t user-driven (such as EDI feeds or a public API) then the number of partners (other entities that will exchange information with you) and the frequency of exchange need to be determined.

Things to not bother with:

1. Response Time: For customer interactive products, response time is dictated by what end users will tolerate and is not really going to be a business decision (aside from deciding if you’re going to produce something your customers are willing to use). For non-interactive products or back-end this may need more discussion with the business, but again - the business is going to expect you to be able to figure out what will make it a success.
2. Data Retention: Assume it all has to be kept and more indefinitely. In the end, storage is cheap and this design decision rarely costs a lot of made up front but is expensive to reverse. Data also has the amazing power to make heroes out of IT when the business starts posing questions later and you can answer them. Generate as many facts as you can now to help you out later.

These items are past the point of diminishing returns with the business. You should work them out within the development team and document them, but you shouldn’t believe that any business sign off you might get is binding or useful.

Build to the Scale

Once you’ve established your growth curves, pick your candidate architecture and translate the growth curves into system performance requirements.

Hypothetical Example: If you need to support 1000 simultaneous users for a web application, determine the dynamic web hits per second by determining how often an average user will request a dynamic page (say ever 5 seconds, which is very fast for most dynamic applications) These two numbers would give you a dynamic hits per second of (1000/5) = 200. Then add how long each page will take to calculate (make a goal of say 250ms) to get how many requests you need to be able to process at the same time: (200 * 0.250) = 50. This is the key scale point for your web application: When deployed, it must support 50 requests being processed in parallel. You’ll need to get to this point by either making it really scalable on a single server, or splitting the load over multiple servers.

One thing that should jump out of the math behind this is that anything you can do to make the calculation time of a single page drop pays big dividends: If you drop the average calculation time by half (125ms) then the number of requests in parallel drops by half (200*0.125) = 25. This in turn may well cut the number of servers you need in half, easing your maintenance and deployment cost. If you can’t do this, reduce the number of dynamic pages requested per second by either making more static pages (such as pre-rendering pages that change but don’t change frequently) or caching dynamic pages that have some predictable consistency (which really makes them static pages). This is often much trickier to do and test, so your best first option is to reduce the time for each page.

Side Point: This also highlights an easy way to accommodate guessing low on a system that’s been in service for a year or more: If you’re processor bound you can replace that hardware with current units and often pick up 30% per year it’s been since you purchased the original hardware. This won’t save you from network problems, disk storage problems, or some memory problems, but it is surprisingly handy.

As you look at each candidate architecture, look at each component and determine the critical “how much, how fast, how often” factors based on the business inputs. If you change your architecture or external interface design (the user interface or import/export capabilities) you need to re-evaluate if you’ve moved the targets as well because your design goals no longer reflect the business growth curves.

Really, to the Scale

Within your development team you will typically have two types of developers you need to watch: Those that never consider scale and those that obsessively consider scale. The former will build it however and then wait to see if there is a performance problem. The latter will try to make every system the next Amazon. Neither situation is good. Identify early people’s tendencies and work to manage them to the center. Remember that the system is only as scalable as its slowest part, and there is always a slowest part.

You can get good results by having the people that are most concerned about scalability move around on the project to different subsystems. This will tend to keep them too busy to earn the keeper of the nanosecond award on any one system (which they will do if you let them stay put and just work on one system) and will make it unlikely that more cavalier developers can hide a problem. It will also help the team learn from each other: It often isn’t worth making a specific feature as fast as possible, and it is always worth thinking about what will make a feature fast before coding it.

Finally, budget time in the development team to fix scalability issues. Regardless of how much work you put into it, once the real system is build and tested you’ll find places that are slower and less scalable than you expected. If nothing else, you need to develop an accurate model of how the system should perform in production so you can check the real world against it later. As your business grows, you need to be able to get ahead of it and understand when it is time to make the code faster, add hardware, or do something else to stay one step ahead.

Disk is Your Friend, but Beware the Network

If you’ve gone over the system from nose to tail and you’re disk bound, you’ve probably optimized that design as well as you can. Disk has gotten faster at a much slower pace than memory or processor, and being disk bound means you’re getting all the requests where they need to go in a timely manner and are able to process the inputs and outputs, so now it’s in the hands of the hardware. Unfortunately at that point there generally isn’t much more you can do: The difference in performance between server drives and the fastest drives money can buy isn’t very much.

If you’re finding that you aren’t disk bound and you aren’t processor bound then be worried. You’re either network throughput bound or you’re network latency bound. If you’re network throughput bound, you can probably fix it cost effectively with some basic engineering either in how you select what to send across the network or what you cache so you don’t need to send it across. You should try to give yourself some headroom here for growth, but faster networks can be purchased and you can generally tweak the software to mitigate this in minor updates.

Being network latency bound is a more serious issue because it often means that you are at the practical scalability limit of your application. The difference in network latency between relatively cheap hardware and the best hardware isn’t very much, and has been essentially constant for the last 10 years. You can’t buy your way out of this problem. It also is typically caused by a badly designed interface between components of the system which will need to be substantially or entirely rethought and rebuilt to address, which isn’t easy to do with a running system. If you find yourself in this situation and you aren’t sure you have met your business goals you should rethink your approach immediately. Because no amount of money on hardware can get you out of this problem, caution is the word of the day.

Getting the feedback wrong can be disastrous: The right functionality late can kill your business; perhaps half a loaf earlier is better.  On the other hand, some things the market won’t accept half way so a full loaf it is.  The business needs to trust the information it’s getting isn’t random or capricious to make good decisions, and the development team needs to be able to provide a best guess without fear of misunderstanding.

It’s natural to pad estimates with the idea that it’s better to under promise and over deliver - so that means to guess long and come in well early.  But in the end, is that really any better?

You’re Guessing, and Possibly Lucky

We’ve been experimenting with the new Evidence Based Scheduling features of FogBugz (which we use internally for managing our software development) and one thing that it highlights quickly is that estimation isn’t good if you’re early, bad if you’re late - it’s about getting your average as close to the mark as you can.  Take a look at a graph of most of my estimates:

Ideally, the graph line would have a 1:1 slope, indicating that on average you are accurate.  Further, you want your estimates clustered pretty near to the line itself.  What you can see from my estimate curve is that I’m uneven - I tend to underestimate shorter tasks (under 1.5 hours) and overestimate longer tasks (and that’s after removing some really bad outliers…).  But notably if you draw a ruler on the 1:1 line you’ll see that I’m not even close. Don’t let the hash lines fool you - look at the numbers to see.  The thing is, the other developers in our company that are all regarded as skilled, senior developers aren’t particularly more accurate on any one estimate, and the averages work out similarly.

So what?

Why isn’t it a great thing when we beat our estimates?  There are several potential pitfalls:

Features based on Effort

We’d all like to believe in the rosy model that our customers ask for features and then we build them into the software, so if we’re done early then it’s a pure win.  It’s my experience that it’s much more like a game of Tetris:  What features we take on is dependent on how much effort we think they’ll take.  Every feature has an amount of effort above which it isn’t worth it any more.  When hashing out what makes it and what doesn’t, the effort estimate is a big factor.

If we overestimate the effort of features, then we are slanting the project management decisions away from customer-selected features in favor of the developer’s whims.  This is because some features will be estimated to take more effort than they’re worth, and a more invisible internal team dynamic.  If a project is doing well on schedule, it’s very human to take advantage of this to try out newer, riskier things, over-engineer a feature, or do other things within the team that would otherwise be successfully argued against because of their effort.  In general, the more time available, the more yak shaving the team will do.

Once a schedule is accepted, the business will tend to act on it as fact:  Customers that can’t wait for it will end up being turned away and others will be promised a schedule.  This is a necessary but painful aspect:  Developers are generally optimists and will not want to say no to a customer even though it’s generally not in the best interest of either the company or customer to rush a feature. You want the business to stick with your decisions and not pass the buck on saying no to the customer, but you also need them to trust that it’s a fair trade.

Approach based on Effort

Within the development team, decisions are made at every level on how to implement a feature with an eye towards both the feature’s estimate and the overall project’s status.  Even when not explicitly laid out, a team that believes the overall schedule is tight will feel pressured to find ways to reduce the effort on anything they do.  This means when deciding between a careful implementation that may take longer but be more scalable or easier to support the team will often opt for a more direct path to completion even if the estimate was based on the more careful approach.

If done as a conscious decision in consultation with the project’s sponsors, this can be a way of bringing the project back on track but really it’s just another way of cutting functionality to make schedule:  You’re going to cut out something you intended to deliver (say a more generalized, upgradeable framework for reporting) even though you meet the letter of the requirement in front of you (delivering a few reports).  This can lead to nasty surprises for the team down the road when your sponsor’s find out that they didn’t get what they thought they would.

Alternately, if you overestimate one feature it may have put another feature under pressure so a more expedient and risky approach was adopted for it to fit it into the schedule.  If the true effort had been known, a different decision could have been made.

Make Your Guesses A Coin Toss

In the end, being early and being late just have different ways they create problems for your development project.  Your goal when estimating is to not try to find the estimate that has the highest probability of being sufficient to get the job done but instead the estimate that is equally likely to be high as low.  In aggregate if you have enough of these items on your project (say more than 25) then you’re entire project’s estimate should also be just as likely high as low.

There is still a place for the traditional high estimate: When you move outside of the project sponsor and the development team to users that need a guarantee.  There the downside of missing a date is much worse than the impact of being early.

On your next project, try out the 50/50 approach and make it clear to both the development team and the business.  You’ll probably notice that people develop a more subtle appreciation for the fact that estimates are based on probabilities.  This can help you skip over the discussions that aren’t useful about why you are where you are and instead keep focusing on the business goals for your current situation.

Things change. Over the past eight years broadband in some form has become available in most cities across the nation. This bandwidth has made dedicated remote access a thing of the past. Now you can provide remote access to your employees over your Internet connection. Traditionally, IPSec has been the technology of choice to provide a virtual private networking solution for your employees but over the past two years there’s been a new game in town - SSL VPNs.

if you are using IPSec for your mobile users, you owe it to them and you to check out one of the SSL VPN options at your disposal. We’ve used IPSec VPNs for network to network access reliably, but they’ve always been tough to support for mobile users. Offhand, there isn’t any specific reason this should be true, but it is. For mobile users, we seem to consistently run into a few problems:

• Installation: The success rate for an average user being able to install an IPSec client and get the VPN tunnels to work, even with phone support, was around 15%. Most of the time the user had to bring in the computer or we had to send a tech on site.
• Compatibility: Different physical network technologies - notably DSL - run into performance problems with IPSec in many configurations, requiring adjustments on the client, routers, or other things that you just can’t expect end users to understand.
• Portability: IPSec is very easy to block on a network. In fact, it took some time for most network routers to be compatible with IPSec. Now try to get it to work at 8 PM over a wireless network in a hotel in Buffalo.

In contrast, a few years ago at the urging of Watchguard (our resident firewall vendor) we tried out their SSL VPN product, which was basically a version of the Citrix Access Gateway SSL VPN solution running on a Watchguard hardware appliance. Out of the box it worked - every time, and even faster than IPSec. We had resisted the option because we preferred standards-based solutions, and this sounded like yet another proprietary security technology. We used a demonstration appliance for a month but the feedback from our users was so strong we purchased a unit after a few weeks. Upon reflection, there really is a good bit of sense to why it works so well:

• SSL is Simple, IPSec is complicated: SSL is a single TCP/IP socket with a relatively straightforward, self-configuring, and invisible to intervening appliances.
• SSL is essential, IPSec is a threat: No one can afford to block SSL on their network without basically admitting to not having a network at all. It’s very expensive to proxy by decrypting and re-encrypting, so few companies do it. On the other hand, many networks view with suspicion the goal of establishing an encrypted connection out of their network, so blocking IPSec may sound like a good idea.

With the SSL VPN solution we had about an 85% end-user self install rate without support, and a 100% rate of not requiring a tech to go on site. Even better, the reviews from end users was that it was fast to connect, easy to use, and performance was good. Because it was so easy to get set up, many more users started connecting from home in the evenings or in bad weather to get work done. The net cost? While your firewall probably offers an IPSec client for free, you can expect to pay a few thousand for a dedicated SSL VPN appliance and depending on licensing $50-$200 per concurrent additional user after the first five or so. For a company with say 100 users that might have at most 20 concurrent users the cost is in the order of $4,000 to $6,000.

Making the Business Case

Jumping from “free” to $6,000 may seem questionable until you look at it from the value standpoint: A service that was expensive to setup and of questionable reliability became cheap to set up and rock solid. In other words, this is the real cost to provide this service. An unreliable solution isn’t a business solution. If it’s more than your business is willing to pay, wait a little while - the cost has come down by half in the last two years, and some vendors (like Watchguard in their Fireware Pro product) are offering it alongside their free IPSec VPN option.

Why am I still hosting these at home? Really there’s no reason - I’ve shifted hosting for my personal services out to other providers, and our company services are also hosted by hosting companies. I just haven’t moved her stuff out of my house.

We talk with a lot of small and medium sized businesses that are still hosting all of their own services internally for pretty much the same reasons - they originally had them in house when they were much smaller and the market was different, and haven’t considered what it would mean to have those computers live somewhere else. It’s time for a change.

Why It’s time to Use the Cloud

You should look at all of your important business services - things that your business can’t operate without - and work out a plan to no longer host those items in your facility. As a first step, just consider what it means to provide the same applications and services, but have the computers not live within your company. The main goals for moving these services out are:

1. Business Agility: When you use a hosting company it’s easier to change capacity as your needs change, even to bring services up temporarily as a trial run and then shut them down if they don’t pan out. This makes it easy to experiment with new software technology without the traditional problems of hosting getting in the way.
2. Low Cost Reliability: If you want those services available, the cost to outfit a room to provide redundant cooling and power for a single rack of equipment is easily $50,000. To host one rack of equipment in a basic Tier-2 data center can cost around $1,500 to $3000 a month, which includes power and Internet. At that rate, how quickly will you get an ROI on your facility investment?
3. Improved Focus: Getting this equipment out of your shop improves your focus on the things you really need to be spending time on: Projects for the business and end-user support. The rest of it is overhead.
4. Access from Anywhere: When you set up your services so they can live in the cloud and be used from your office, it’s easy to make those same services available to employees from home and from laptops. Not as second class citizens but with all of the ranks and privileges of being in the office. This helps you leverage employee talent wherever it is. It’s also easier to set up rock-solid extranet access for customers and suppliers.

When you start looking at each thing you provide as a service, you might also find that some of them - like Microsoft Exchange - really aren’t worth hosting yourself at all even in a data center, and it’d be ultimately in your best interest to outsource it entirely to a hosted Exchange provider. There are number that can do this very effectively. While the cost may seem high based on what it cost you to purchase your initial Exchange licenses, when you look at the real cash costs for Exchange over two to three years they are very cost effective.

Once you’ve taken the step of taking an existing service and outsourced it entirely, you might even consider a Software as a Service offering for some of your core services (such as a hosted CRM). This is the most aggressive mode of outsourcing and does create a set of unique risks and opportunities.

But I can’t See It

Two common objections we hear from IT administrators about moving services out of their shop, even if it’s just relocating servers into a data center. is that it will make it hard for them to get upgrades when necessary because the business won’t be able to see & feel the new equipment. Out of sight, out of mind as the saying goes. The second main objection is that the IT administrators want to be able to do a laying of hands on the equipment to maintain it. There’s a comfort factor in knowing you can walk into a room and flip the power switch or move a drive or just bask in the warm glow of blinking lights.

Here’s the good news: Both of these reasons are not only suspect in their own right, but are preventing your shop from getting to the next level in IT’s relationship with the business.

First, even though vendors do a good job of making server hardware look serious and fun, in the end it’s just a business appliance: It either is good enough to deliver for the business or it isn’t. With rare exception, there is no extra business value for it to look good, new, or cool. If you find that you need to show the business physical servers to explain your costs, you’re missing out on the critical opportunity to establish a real partnership between business and IT. You need to be sure you’re spending when it’s time to spend and saving when it’s time to save, and have discussions in the language the business would use for any other service it would acquire.

Second, If your IT administration patterns and practices require routinely touching your physical infrastructure then you need to re-examine them. It generally means you either have equipment that is no longer up to the task or that you’re not doing enough automation of IT tasks. If you have trouble-prone hardware, it’s time to either fix the fundamental issue or ditch the hardware. Ironically, this type of problem is often easier in a hosted environment because it generally isn’t your problem: it’s the hosting company’s.

Automation is essential because humans are the most error-prone part of any standard process. Your routine IT administration time shouldn’t be going to consistent tasks - they should be automated, leaving your time for user support and other business value-add services. That’s right - even in your shop with your existing staff you can find more time to spend on projects instead of support events by automating recurring tasks.

Some Things Still Stay

There are some things that should be on site for performance reasons. Regardless of how big your Internet connection is, you’re going to want basic file and printer sharing services to be local. Depending on the size of your site, you’ll probably also want a directory server for whatever your directory system is (e.g. Microsoft Active Directory). Even here the central services help: If you have a reasonable Internet connection, you can have your local file server back itself up to the data center by using one of a few distributed backup systems (such as Microsoft’s Data Protection Manager or a third-party option like NSI Software’s Double-Take). This eliminates the time and attention that local disk backups require.

Perhaps not Now, but Soon - and For the Rest of Your Life

It may not be appropriate to move a number of your services outside yet; If you have only one business site, light access by employees externally, and aren’t expecting that to change then you can host most things yourself. A number of the considerations still apply - but you might just use an external facility for your public web presence and for backing up your essential data for business continuity.

Even if you don’t do much now, you should find some opportunity to put a service outside so you and your company can gain experience at working with external hosting providers and you’ll stay current on the capabilities and costs so that as new business requirements evolve you’re ready to take care of them. You’ll be in a better position to advise your company on when to move things out of the shop, and as you do you’ll discover that instead of focusing your time and talent inward at the routine operations of infrastructure you’ll have time for those projects that really make a difference to your business.

How Has the Cloud Delivered For You?

Have a story about what has and hasn’t worked with hosting? Drop me a line or post a comment to share it.

Ever since in every shop I visit I push for two screens to be the default configuration of any computer. Large screens are great, but there are real practical limits where a larger screen doesn’t do much for productivity, but a second screen will. The best part is that two modest sized screens are invariably cheaper than the same number of pixels in a large screen due to manufacturing cost.

Bring On the Workspace

The primary advantage of two or more screens is they are a very cost effective way of increasing the amount of information your users can see at one time. For between $200 and $400, you can add a 19″ LCD flat panel to a system. With that, users can do several tasks much easier:

1. Comparisons: When you want to compare two things, say two Word documents or spreadsheets, it’s generally ideal to have twice as much desktop area as you normally use for just one. This allows you to interact with each the way you’re used to without having to adjust to navigation and toolbars being in the wrong place because you’re cramming things into a size you don’t normally use (and the document may not be oriented for.
2. Active and Background tasks: A great use of multiple screens is to have one for your active working task and another for background tasks you are monitoring. For example, put Outlook on a secondary screen and your main task (say development, writing an article, or analyzing a spreadsheet) on your main screen. In this configuration, it’s important that your main screen is directly in front of you and your secondary screen is off to the side where it won’t be distracting.

Why Not Just Bigger?

Why not just get one larger monitor? There are a few advantages to multiple displays.

1. Maximize Behavior: When you maximize a window, it goes to the dimensions of the monitor it’s on. This makes it really easy to move windows from screen to screen quickly because you just need to maximize it then drag it, it will size to the full extent of the target window. Get to love maximize - if you can see the background, you’re not displaying as much information as you could.
2. Too big is too big: You should be able to see the entire extent of your screen without moving your head and really without moving your eyes a lot. The entire screen should be in your peripheral vision. With very large screens, you’ll tend to find that you don’t use all the screen because some is too far to the left or right for comfort. This is particularly an issue with the latest very large widescreen monitors (over 24 inches diagonally). If you find that you just can’t use the entire screen, then that monitor is just too big, you really should have another.
3. Optimize for Purpose: Instead of accepting just one generally good shape and set of capabilities you can optimize different screens for different uses. When using two screens, I like to have my primary be a 24″ widescreen and my secondary be a 20″ normal aspect ratio. This fits well with my normal set of activities where I want some extra width for my primary activities but I find that I like the normal ratio for my background task screen where I leave Outlook and put things like online help windows, web sites I’m monitoring, etc. I worked with a quality assurance manager who had three screens with the center one in portrait orientation which fit her work very well - she would have the application being tested on one screen, the QA tracking application on another, and her email or other background window on a third.

Common Objections

The most common objection, particularly from people that aren’t used to working with multiple screens, is that it’s too expensive. This is usually because the cost is viewed as a percentage of increase to new computer cost. From that standpoint it looks significant - if your shop tries to keep individual computers in the $1500 range (which is very easy in the current market for a typical desktop) this will increase the cost by around $250, closing in on 20%! Why, that would require that you increase your budget for desktop PC purchases by 20%, who’s going to pay for that?

This objection completely ignores both the total cost of ownership of a computer and, more importantly, the value of the computer as a tool to the user. A second screen adds very little to the TCO over one screen (LCD monitors are low support cost items) so if you looked at the entire cost of the PC over its life - say $5000 - this cost is much more reasonable.

A second objection is that users will not use it for productivity gains but to instead couple non-work activities with work time or in general just not be able to make effective use of the space.  While users may become very successful on their own with multiple screens, some quick orientation on how to make the best use multiple screens will pay off with users changing how they work with applications to take advantage of the new capability.  If you don’t do this, many will pick it up on their own as they watch their peers.

At the last company I worked at before eSymmetrix, I finally achieved the goal of everyone - even the receptionist - having two screens.  No more was it a privilege for the elite or the engineers; the benefits apply to any information worker so it was done everywhere.

Most professionals develop an instinctive ability to size up situations within their core expertise. For example, a seasoned product manager that has worked up through the ranks of developers can often look at a schedule and get a quick feel that it can or can’t be met. Most of the time this intuition expresses itself as a strong gut reaction that you can’t explain. Malcolm Gladwell wrote the excellent book Blink talking about this phenomenon, well worth reading so that you know what to do when you have this reaction the next time.

The basic challenge is this: Even if your instinctive reaction is correct, it isn’t particularly useful until you can explain why. It’s very tempting to try to verbalize the rationalization for your reaction, but don’t. The problem is that your rational mind has no idea why you had an emotional reaction. The emotional reaction is your clue - your intellect and emotions aren’t really connected that well. They need to arrive at their conclusions independently: For you to intellectually justify your reaction you need time to perform a dispassionate intellectual process. Your instinctive emotional reaction didn’t need that time, but it can’t explain itself.

The Path Through The Woods

So if your emotional reaction is equally as reliable as your intellectual evaluation, but you can’t articulate it right away what should you do? The first thing is to develop a code phase for your team that indicates that this is your blink reaction, and not something else. This lets everyone weigh it correctly: It isn’t because you just don’t want to do it or you like another option better, it’s that there’s something instinctively wrong with it. Your team should give it equal weight to you having just expressed a cogent, real argument for there being a problem. Second, the reaction can’t be challenged - at least not directly. It’s an emotional response, so any challenge will drive the participants strait into conflict from collaboration. In our shop, we just say “I have a blink about this…” That cues everyone in.

At the same time, it’s important to recognize that your blink can and will be wrong, sometimes a great deal. It’s no more accurate than a rational analysis of the same circumstance, and that means if the circumstance is something notoriously hard to predict like an election, a project schedule, or a roll of dice then it’s not going to do better than spending some quality time in analysis. This means that when you have a blink reaction then your team should continue with the assumption that the reaction is dead on, but casually seek out the proof.

Casually Find The Evidence

Once you’ve articulated your instinctive blink reaction, have the team take the case that it’s true and then as discussions continue identify facts and data that support the reaction. Eventually, one of a few things will happen:

1. Killer supporting evidence will emerge: In the process of going through subsequent analysis, you’ll find the evidence that suddenly has the reason behind your reaction clear to the team. It’s now an intellectually reasoned response; you just got there faster instinctively.
2. You’ll realize your reaction was wrong: As time passes, your mind will keep attempting to align facts with your reaction seeking to prove or disprove it. Eventually you, or your team, will see what it was that your mind originally caught on and understand that it doesn’t apply in this case: The reaction was wrong, and now you can proceed. This was still an important exercise because you now have validation on your course of action (”we can ignore this previous best practice because it no longer applies because….”)
3. You’ll look at the problem tomorrow and get over it: Perhaps what you felt was just the normal human fear of change. So be it- now that you aren’t feeling threatened any more, your rational mind can reassert itself and look at the item more objectively and be open to new possibilities.

Each of these is a powerful collaboration result because it lets the team and the individual practice and demonstrate the characteristics of a trusting, supporting environment. The team showed the respect for the individual participant and leveraged the experience of everyone, not just those with great oratorical skills. The individual gets to articulate something they feel very deeply without being embarrassed or having to justify and then later defend something that they just can’t. Everyone is in on it - and when the instinct resolves into intellect later everyone will have better insight into the experience and thinking of the person that voiced it. This insight develops trust between the individuals that will live beyond the team.

Finally, by the team not pushing for an immediate defense and then challenging it demonstrates and reinforces that it’s a safe environment to voice ideas and opinions, and builds credibility for when a miss-step happens.

Gaining Speed

Over time as your team works together people may be able to help each other articulate their blink reactions into concrete issues to be resolved based on understanding the key emotional drivers each participant brings to the table. In our company, we have some folks that tend to have reactions over usability, others over ultimate performance (we’ve dubbed one “the keeper of the nanosecond”), and others over code simplicity. Knowing these points helps us not miss-read emotional reactions to ideas and to help each other understand what we really need to do to create the best outcome.

With practice, you’ll find your team can get to great, collaborative conclusions faster and generate the buy-in from the participants with little or no effort.

What Gets You to Blink?

Looking back, when did you have a blink reaction to something? What did you do with it? Share your story by posting a comment or dropping me a line.