Reliable Systems

Back in the day, corporate remote access meant modem pools that people dialed into from wherever they were. Even then it was like watching a feature film on your IPod; you got a sense of the action but it was ultimately as much frustrating as useful.

Things change. Over the past eight years broadband in some form has become available in most cities across the nation. This bandwidth has made dedicated remote access a thing of the past. Now you can provide remote access to your employees over your Internet connection. Traditionally, IPSec has been the technology of choice to provide a virtual private networking solution for your employees but over the past two years there’s been a new game in town - SSL VPNs.

if you are using IPSec for your mobile users, you owe it to them and you to check out one of the SSL VPN options at your disposal. We’ve used IPSec VPNs for network to network access reliably, but they’ve always been tough to support for mobile users. Offhand, there isn’t any specific reason this should be true, but it is. For mobile users, we seem to consistently run into a few problems:

• Installation: The success rate for an average user being able to install an IPSec client and get the VPN tunnels to work, even with phone support, was around 15%. Most of the time the user had to bring in the computer or we had to send a tech on site.
• Compatibility: Different physical network technologies - notably DSL - run into performance problems with IPSec in many configurations, requiring adjustments on the client, routers, or other things that you just can’t expect end users to understand.
• Portability: IPSec is very easy to block on a network. In fact, it took some time for most network routers to be compatible with IPSec. Now try to get it to work at 8 PM over a wireless network in a hotel in Buffalo.

In contrast, a few years ago at the urging of Watchguard (our resident firewall vendor) we tried out their SSL VPN product, which was basically a version of the Citrix Access Gateway SSL VPN solution running on a Watchguard hardware appliance. Out of the box it worked - every time, and even faster than IPSec. We had resisted the option because we preferred standards-based solutions, and this sounded like yet another proprietary security technology. We used a demonstration appliance for a month but the feedback from our users was so strong we purchased a unit after a few weeks. Upon reflection, there really is a good bit of sense to why it works so well:

• SSL is Simple, IPSec is complicated: SSL is a single TCP/IP socket with a relatively straightforward, self-configuring, and invisible to intervening appliances.
• SSL is essential, IPSec is a threat: No one can afford to block SSL on their network without basically admitting to not having a network at all. It’s very expensive to proxy by decrypting and re-encrypting, so few companies do it. On the other hand, many networks view with suspicion the goal of establishing an encrypted connection out of their network, so blocking IPSec may sound like a good idea.

With the SSL VPN solution we had about an 85% end-user self install rate without support, and a 100% rate of not requiring a tech to go on site. Even better, the reviews from end users was that it was fast to connect, easy to use, and performance was good. Because it was so easy to get set up, many more users started connecting from home in the evenings or in bad weather to get work done. The net cost? While your firewall probably offers an IPSec client for free, you can expect to pay a few thousand for a dedicated SSL VPN appliance and depending on licensing $50-$200 per concurrent additional user after the first five or so. For a company with say 100 users that might have at most 20 concurrent users the cost is in the order of $4,000 to $6,000.

Making the Business Case

Jumping from “free” to $6,000 may seem questionable until you look at it from the value standpoint: A service that was expensive to setup and of questionable reliability became cheap to set up and rock solid. In other words, this is the real cost to provide this service. An unreliable solution isn’t a business solution. If it’s more than your business is willing to pay, wait a little while - the cost has come down by half in the last two years, and some vendors (like Watchguard in their Fireware Pro product) are offering it alongside their free IPSec VPN option.

Right now, the power is out at my home. That doesn’t happen often - in fact, it’s been almost two years since we lost power long enough for my UPS to shut down my home network. Normally this would be a small inconvenience, but I still host a few things for my wife out of my house which are now down. The largest of these is a fairly popular forum for an author she likes, but there are other sites as well.

Why am I still hosting these at home? Really there’s no reason - I’ve shifted hosting for my personal services out to other providers, and our company services are also hosted by hosting companies. I just haven’t moved her stuff out of my house.

We talk with a lot of small and medium sized businesses that are still hosting all of their own services internally for pretty much the same reasons - they originally had them in house when they were much smaller and the market was different, and haven’t considered what it would mean to have those computers live somewhere else. It’s time for a change.

Why It’s time to Use the Cloud

You should look at all of your important business services - things that your business can’t operate without - and work out a plan to no longer host those items in your facility. As a first step, just consider what it means to provide the same applications and services, but have the computers not live within your company. The main goals for moving these services out are:

1. Business Agility: When you use a hosting company it’s easier to change capacity as your needs change, even to bring services up temporarily as a trial run and then shut them down if they don’t pan out. This makes it easy to experiment with new software technology without the traditional problems of hosting getting in the way.
2. Low Cost Reliability: If you want those services available, the cost to outfit a room to provide redundant cooling and power for a single rack of equipment is easily $50,000. To host one rack of equipment in a basic Tier-2 data center can cost around $1,500 to $3000 a month, which includes power and Internet. At that rate, how quickly will you get an ROI on your facility investment?
3. Improved Focus: Getting this equipment out of your shop improves your focus on the things you really need to be spending time on: Projects for the business and end-user support. The rest of it is overhead.
4. Access from Anywhere: When you set up your services so they can live in the cloud and be used from your office, it’s easy to make those same services available to employees from home and from laptops. Not as second class citizens but with all of the ranks and privileges of being in the office. This helps you leverage employee talent wherever it is. It’s also easier to set up rock-solid extranet access for customers and suppliers.

When you start looking at each thing you provide as a service, you might also find that some of them - like Microsoft Exchange - really aren’t worth hosting yourself at all even in a data center, and it’d be ultimately in your best interest to outsource it entirely to a hosted Exchange provider. There are number that can do this very effectively. While the cost may seem high based on what it cost you to purchase your initial Exchange licenses, when you look at the real cash costs for Exchange over two to three years they are very cost effective.

Once you’ve taken the step of taking an existing service and outsourced it entirely, you might even consider a Software as a Service offering for some of your core services (such as a hosted CRM). This is the most aggressive mode of outsourcing and does create a set of unique risks and opportunities.

But I can’t See It

Two common objections we hear from IT administrators about moving services out of their shop, even if it’s just relocating servers into a data center. is that it will make it hard for them to get upgrades when necessary because the business won’t be able to see & feel the new equipment. Out of sight, out of mind as the saying goes. The second main objection is that the IT administrators want to be able to do a laying of hands on the equipment to maintain it. There’s a comfort factor in knowing you can walk into a room and flip the power switch or move a drive or just bask in the warm glow of blinking lights.

Here’s the good news: Both of these reasons are not only suspect in their own right, but are preventing your shop from getting to the next level in IT’s relationship with the business.

First, even though vendors do a good job of making server hardware look serious and fun, in the end it’s just a business appliance: It either is good enough to deliver for the business or it isn’t. With rare exception, there is no extra business value for it to look good, new, or cool. If you find that you need to show the business physical servers to explain your costs, you’re missing out on the critical opportunity to establish a real partnership between business and IT. You need to be sure you’re spending when it’s time to spend and saving when it’s time to save, and have discussions in the language the business would use for any other service it would acquire.

Second, If your IT administration patterns and practices require routinely touching your physical infrastructure then you need to re-examine them. It generally means you either have equipment that is no longer up to the task or that you’re not doing enough automation of IT tasks. If you have trouble-prone hardware, it’s time to either fix the fundamental issue or ditch the hardware. Ironically, this type of problem is often easier in a hosted environment because it generally isn’t your problem: it’s the hosting company’s.

Automation is essential because humans are the most error-prone part of any standard process. Your routine IT administration time shouldn’t be going to consistent tasks - they should be automated, leaving your time for user support and other business value-add services. That’s right - even in your shop with your existing staff you can find more time to spend on projects instead of support events by automating recurring tasks.

Some Things Still Stay

There are some things that should be on site for performance reasons. Regardless of how big your Internet connection is, you’re going to want basic file and printer sharing services to be local. Depending on the size of your site, you’ll probably also want a directory server for whatever your directory system is (e.g. Microsoft Active Directory). Even here the central services help: If you have a reasonable Internet connection, you can have your local file server back itself up to the data center by using one of a few distributed backup systems (such as Microsoft’s Data Protection Manager or a third-party option like NSI Software’s Double-Take). This eliminates the time and attention that local disk backups require.

Perhaps not Now, but Soon - and For the Rest of Your Life

It may not be appropriate to move a number of your services outside yet; If you have only one business site, light access by employees externally, and aren’t expecting that to change then you can host most things yourself. A number of the considerations still apply - but you might just use an external facility for your public web presence and for backing up your essential data for business continuity.

Even if you don’t do much now, you should find some opportunity to put a service outside so you and your company can gain experience at working with external hosting providers and you’ll stay current on the capabilities and costs so that as new business requirements evolve you’re ready to take care of them. You’ll be in a better position to advise your company on when to move things out of the shop, and as you do you’ll discover that instead of focusing your time and talent inward at the routine operations of infrastructure you’ll have time for those projects that really make a difference to your business.

How Has the Cloud Delivered For You?

Have a story about what has and hasn’t worked with hosting? Drop me a line or post a comment to share it.

If you’re involved in IT operations or even in business long enough, you’re going to experience some emergencies. During these emergencies, you’re going to have to balance several conflicting things that will demand your attention simultaneously:

1. Cause of the problem: What is really happening? What device is at the root of the problem (network switch died because an admin configured a loop in the fabric and miss-configured the port)
2. Scope of the problem: Just how bad is it? Problems usually show up in one place (users can’t access Exchange) but those symptoms often represent a larger problem (network switch died)
3. Communicate with users: First, people will be coming in the door to report the problem (do you know that Exchange is down?) and will be expecting updates on what’s going on and when it’ll be resolved (I really need to tell my friend about a party tonight, when will email be back up?)

Even in a shop with healthy staffing, this can be a lot to handle at once particularly because your impulse is going to be to move between the root cause and communication. The first because it’s the real high value item -fix the problem. The last because whenever someone walks in, you’ll want to tell them what’s going on. The higher up the chain of command, the better you’ll want it to sound.

Whenever I’m wondering how to look at an IT Operations problem from a different perspective to gain insight, aviation is the first place I go. Think about the modern air transport system in the United States not from your usual perspective (a passenger on a plane) but from the standpoint of the people that live within it and operate it. For example, the life of a flight deck crew isn’t that different than system support in the sense that you have long periods of routine punctuated by periods of high stress activity. A classic rule taught to pilots when they’re first being trained is Aviate, Navigate, and Communicate - in that order.

1. First, fly the plane. (Be in the middle of the air, not the bottom)
2. Figure out where you are. (Over the White House)
3. Then communicate. (Sorry Tower, would you like us to land?)

To make things easier on commercial planes, you have a pilot and co-pilot that divide these responsibilities by having clear designation of one being the Pilot Flying and the other (called the Pilot Not Flying or Pilot Monitoring) responsible for navigation and communication. This is practiced carefully during training with different parts of each emergency checklist assigned to either the Pilot Flying or Pilot Monitoring.

Now apply this back to a system problem:

1. Create Clear Roles: Have your team know who is going to take on the role of Admin Flying and Admin Monitoring. This shouldn’t always be the same - it may be based simply on rotation (who is “up”) or who gets the trouble ticket or whatever within your shop. The team should declare their role in a situation so everyone knows their role.
2. Perform in Order: If you have an Admin monitoring, it’s their role to intercept external communication while the Admin Flying is working on the problem.
3. Make a Checklist: When there is an emergency isn’t the time to be winging it. During quiet moments, talk as a team about what you would do in a hypothetical situation and work to distill out a basic checklist of things you’re going to run through. Focus on having it be the shortest list that verifies the largest set of items. When a problem shows up, use the checklist.

Problem Checklists

There are a few great advantages to using a checklist for problems:

• Reduce Solution Focus: When diagnosing problem, the general process is to propose a theory then test it to either prove or disprove it. This create cycles where you create theories you have to believe in then your job is to prove yourself wrong. It turns out that people tend to naturally bias towards information that proves themselves right and away from information that’s inconsistent with that diagnosis. Checklists for diagnostics can ensure that a significant breadth of information is available at the start of this process to enable the best theories to be created quickly.
• Creates a Pace: It’s easy to get caught up in an emergency and start working at a pace that really isn’t necessary, but degrades your accuracy and effectiveness. Checklists stop the emotional cycle that reinforces the early stages of emergencies and instead create a steadily paced environment of gathering and verifying facts.
• Establish a Baseline for Improvement: One of the most important parts of any emergency, and the least frequently used effectively, is an after action review. After you’re back up and everyone has calmed down, you want to learn as much as you can from what happened. The existence of a checklist creates a baseline for systematic (As opposed to random or by chance) improvement to your team’s ability to handle future problems. This is true even if the checklist wasn’t used; the fact it wasn’t used is itself an indictment of either the checklist itself or the team’s training.

While initially it may feel corny or even overly dramatic or bureaucratic to create checklists, there is real evidence to back up using them in environments where the downside cost (crash and death) is very steep, and if pressed to admit it most engineer will confess they have a mental checklist they use for standard problems.

Plans are Useless, Planning is Priceless.

Just by creating the checklists (even if they were never used) your team can get a lot of value:

• Cooperative learning: This is a great tool for the team to learn from each other. Each admin will share their best tips and tricks from their mental checklist and be surprised that they don’t line up. Where they don’t, the discussion on which approach is better and why is gold. It’s hard to get the same result with a contrived exercise, so use this opportunity to build the checklist and maintain it as a team.
• Clarifies Automation: While creating the checklist, it will naturally precipitate ideas for how to automatically identify and possibly solve steps in the checklist itself. For example, if a step in the checklist is to verify Internet connectivity, how are you going to accomplish that? Instead of having an ad-hoc mechanism, can an automated mechanism be put in place so that you now can quickly check that data point without variation?
• Encourages Collaboration: If the team collaborates to create the checklist, when a problem occurs they will be more likely to collaborate on resolving the problem because they already have had the experience of working together as a team. This will tend to replace individual ego with group esprit de corps.

An Exercise Left to the Interested Student

A friend of mine also pointed out the principle that if you have a checklist that always ends in the same action, why not automate the action in response to the checklist? In other words, if you can automate the detection steps that lead up to the action, then find a way to automate the resolution. You will often find you get here in inches: You progressively improve your monitoring so that you can find problems faster. Once this is reliable, you start just hooking up alarms to the monitoring so you don’t wait for a call from a real user or a higher level system. Once that’s working well enough, you get tired of performing the resolution manually so you write a script that takes a few arguments to perform the resolution. Now, just connect them together.

Move Forward One Step Today

The best part about this is that you can get there in small steps that even the busiest team can fit into their schedule with a confidence that they will pay back in time saved in the future. With practice, it will become second nature and make it easier for your team to accommodate new processes and service requirements with ease. In the end, isn’t that what you need to ensure your team is viewed as a vital part of your organization?

I was watching Start-Up Junkies the other day which is following a group of people attempting to get Earth Class Mail off the ground. On one recent episode, the main focus was converting their system from being PHP-based to Microsoft .NET.

As you watch the episode, you hear two different reasons given for this transition. The CEO advances the issue that they started with PHP because they needed to get something done quick, but now they need to switch to ASP.NET to make it scale. They always knew they’d have to do it, but now they’re against the wall because they have been invited to demo at a Microsoft conference. The lead engineering staff and operations manager advance a different point: We are about to be demoing in Europe in Microsoft’s booth at a major convention that’s key to our growth, we need to be using Microsoft’s technology for this to happen.

Of these two reasons, which do you think is the better reason for their technology choice:

1. Convert to ASP.NET to scale up because PHP can’t scale.
2. Convert to ASP.NET because we’re getting marketing and sales assistance from Microsoft, which we’ll only get on their platform.

If you picked #1, two things are likely true: First, you haven’t worked with enough technology to know that the choice between PHP and ASP.NET for scalability is pretty far down on the list of “things that control how much we can scale”. Second, you’re probably not balancing business and technology interests effectively.

Go and check out the technology portfolio used at the most scalable web sites in the world - say the top ten super scalable systems, systems that are going to be at least two orders of magnitude greater than anything you’re likely to create (and this isn’t a negative thing; it’s a liberating thing). Notice anything in particular? You don’t tend to see either Microsoft ASP.NET or J2EE in the web infrastructure. In fact, you tend to see a lot of… PHP.

There are a few key reasons that the super scalable sites like these solutions:

1. Open source means they have the source: You can bet that these sites aren’t able to use anything off the shelf. Their needs so outstrip the normal system that it isn’t reasonable that any off-the-shelf framework is going to fit their needs entirely. In fact, if it did that framework is likely seriously over-engineered for the majority of its user base.
2. Licensing costs add up: If you’re a small shop, licensing costs are highly unlikely to be a significant percentage of your total cost of goods for a technology product; bandwidth, hosting, and above all people are the big numbers. If you’re Google, you don’t want to pay even $10 in licensing per server. This is similar to a large manufacturer worry about saving $.05 on a bolt; small incremental costs still add up.
3. Scalability is their first concern: More important than ease of development, cool debuggers, third party component libraries, or anything else. If it can’t meet their scale, it isn’t even a potential solution. Perhaps more importantly, they have to have both the experience and human belief that it will scale. If you’re one of these sites, there is no way a vendor has tested their solution at your scale - you’ll be the first. If you’re going to be the first, you want to have a simple solution that you can adjust and correct.

If you’re honest, your decision matrix isn’t the same as this. It’s highly unlikely you’ll create the next MySpace, even if you are successful. While the principles of scalability are constant, the importance of scalability vs. other constraints changes. More likely, you need to base your technology choices on a mix of:

1. What resources do you have? If you already have a staff of people experienced at technology X, they are likely to produce more results in any moderate interval of time (say one to three months) with this technology than any new one. If you have a large body of existing code in technology X, this is a big accelerator to your project.
2. What resources can you get? When picking a technology, buy the community, not the product. If you can take a number of pieces off the shelf, particularly for things you aren’t attempting to innovate (such as security, content management, grid controls, reporting..) it will accelerate your product curve. Conversely, if you can’t get great people that want to work with a technology, it really doesn’t matter how great the technology is.
3. What religion is your market? Many markets have a non-rational product selection bias. For example, if you want to sell your product primarily to Macintosh users, you probably shouldn’t use ASP.NET. It isn’t that it should make a difference to how the product works for them, but as a group Macintosh users tend to put “Not Microsoft” on their evaluation lists. Similarly Linux users. Conversely, there are several products that are defined as “just like X, but in ASP.NET!” If your market typically has a technology selection criteria that isn’t based on business or practical fundamentals, it’s best to respect it, otherwise you’ll have to focus additional energy during your sales and marketing efforts to overcome what your market will perceive as a natural disadvantage. The coolest technology, developed quickly and cheaply, is no good if your target customer won’t even invite you to the dance.

Back to Earth Class Mail. Could they scale using PHP? Absolutely, others have. Should they switch to ASP.NET? Probably - they wanted to leverage the marketing advantage of Microsoft. I suspect if IBM was the big animal in the space they wanted and a deal could have been made it would have been WebSphere instead of ASP.NET. Each of these technologies can scale, or not scale, depending on how they are used.

I grew up as the third son in our family. When my oldest brother was a newly minted driver, like every new driver he was a little rough. And like any younger siblings, my other brother and I were kind and gentle in our commentary about it. This led him to declaring his first driving rule: No comments on his driving when he was driving. He was in command, and that was it.

One day soon thereafter, he was backing the car out of the garage with my other brother and I in it. Now, he didn’t normally park on the right side of the garage - that was where my dad’s car went. But by whatever fluke, there the big Ford station wagon was - on the right side of the garage. When backing out, you have to start turning right away because the driveway isn’t straight. In fact, you have to start turning left, meaning the front of the car goes to the right. Normally this was not a problem since there was plenty of clearance. But when starting on the right side of the garage, on the right is… the garage. As he started backing out, my brother and I quietly sat there, watched the side of the garage come right up and *bang* *crunch* we hit it. In the “after action review” that followed, my brother exclaimed “Why didn’t you tell me I was going to hit the garage?” You can imagine our response – “Because you had said never comment on your driving.”

At the time, I was smug in my righteousness. We had done exactly what he’d asked, we weren’t the driver and the driver is responsible for the car, so it was a big ‘not-my-problem’.

We were dead wrong.We were in a position to have prevented the problem, and we should have spoken up. Ever since, we’ve had the rule in our family that when riding in a car. The rule is to speak up if you see a problem without fear that the driver will be upset. The potential consequences of not calling a problem to the driver’s attention are too great.

How Do You Play the Blame Game?

The same story often plays out in the aftermath of a technology problem. Hang around a software development team long enough and you’re bound to hear a developer complain “Why didn’t QA find that defect? They should have found it before it shipped.” The difference between an experienced, healthy team and an amateur team is whether the developer is just venting or actually believes they are justified.

We often have a strong desire to try to reduce accountability for avoiding issues to a single party:

• QA is responsible for finding all defects in the software before it is released.
• IT Operations is responsible for keeping all of the servers running.
• The receptionist is responsible for ensuring we don’t run out of coffee.

Before looking at the contentious examples, look at just the last one. Say that you noticed that you pulled the next-to-last box of K-cups out of the supply cabinet. You’re not out of coffee yet - there are 24 individual servings in the box you pulled, one more box on the shelf. In most small companies, that’s at least a day’s worth of coffee. Would you tell the receptionist that you need more coffee? Or just assume that it will be taken care of? Say you then run out of coffee two days later, and everyone has to run out to Starbucks to feed their habit. Would you feel at all responsible for not speaking up when the problem was still avoidable?

You probably would have spoken up - the receptionist is a nice person, it’s an easy enough thing to do and you like your coffee.

Now look at the other two scenarios. The only real difference between them and running out of coffee is that these two will tend to be political and possibly even contractual. While you’d likely also speak up if you saw a defect in your company’s product before it was released or if you saw that a server was just about out of disk space, you wouldn’t want to accept any accountability after the fact if things went bad.

Here’s the elephant in the room: Your customers don’t care who was accountable for avoiding a problem. They care that the problem happened. They pay you for something that works (and has to work according to their definition of what works means). Anything else is just internal noise. If you want to drive your business forward - and really, if you don’t, you need to look to work somewhere else - this needs to be your motivator.

Formal vs. Practical Accountability

What if, instead of looking at issues as someone else’s problem, you followed these two principles?

• If you are in a position to prevent a problem, you are accountable for preventing it.
• If you are responsible for ensuring a problem doesn’t happen, you need to stay in a position to prevent it.

This means that many different people and groups may each be 100% accountable for a problem, because the most useful way to look at accountability is based on the ability and responsibility for preventing the problem. Why the most useful? Because the problem happened, that’s a matter of fact. While recriminations, blame, and shame may be cathartic or fun, they aren’t useful because they don’t further the goals of the team or the company. Put simply, your customers don’t care who’s at fault within your organization, just that you get the seriousness of the problem and you’re making it right. When debriefing your team, the ideal outcome is that everyone in the room sees how they could have prevented the problem, and takes on that they should have prevented the problem. From that, you then work into who was in the best place to prevent it - who could have seen it first, and addressed it while it was cheapest to address. You want to have everyone walk out with a balanced perspective of how they could have prevented it and how to identify when you’re in the best spot to prevent it.

A natural concern with this approach, particularly if it’s new to your organization, is that after action reviews are often a game of musical chairs - while there’s a superficial impression of honesty and openness, the true goal is to not be left without a chair when the music stops. Far from a well-calculated political move, this is really an emotional and ego driven outcome. No one likes admitting they are wrong, and with practice people get very skilled at justifying their emotional responses with pseudo-intellectual reasoning – it is called rationalization.

The next time you’re in this situation, try being the first party to speak up about what you could have done to avoid the problem, and make sure you communicate sincere regret you didn’t catch it. If you are completely open in this - sticking just to what you could have done without any back handedness (that’s right - you can’t say “I couldn’t cover up his incompetence.” That doesn’t count.) you’ll be amazed at how quickly the mood in the room changes. Very quickly others will jump in with what they could have done. You’ve created an environment where people can speak the real fears that are on their mind without posturing.

Once you’ve established this environment, you need to be active in maintaining it. If someone jumps into the attack, speak up and redirect the conversation. This is true particularly if the attack isn’t directed at you. Keep listening to have the conversation stay in even tones and that each party is either talking about what their area could have done or is constructively helping the overall conversation.

Eventually, there will be a fundamentally sticky conversation about which party was in the best position to avoid the problem. At this point it’s going to come down to culture - if your culture is one that learns from mistakes, it will be a clear and short conversation. Depending on how strong the duck-and-cover instinct is in your shop, it can be very painful. In the end - speak up if your team is the one that should have the spotlight. Fear of accountability is often overstated. In practice, managers know that in the end they need people that will be accountable for what happened, and the experience can still be positive in the long term. Great managers actively hunt out people that are quick to learn from their mistakes and own them.