Home » Articles » The Magic in Troubleshooting the Black Box

The Magic in Troubleshooting the Black Box

Sometimes at work I feel like a magician.
This could be you!

This could be you, the magician at work! Now if only Exchange 2003 looked so good …

For instance some Microsoft Exchange or Hyper-V clustering issue bubbles up through the tiers of help desk, engineers and senior engineers that have had hours of troubleshooting and myriad eyes thrown at it without making a lick of progress. Then the issue gets slid onto my plate and 15-30 minutes later I have everything fixed up and a client looking much happier as he is able to get back to work. Sometimes they’ll ask why was I able to fix the issue so swiftly when all these other people spent hours at work with no resolution. That cues what I fondly call the magician moment.

I always wanted to be a magician when I was a kid. This is one way I get a small taste of it. The magician moment is when the audience is wondering how the rabbit was pulled out of the hat, or perhaps how the levitating woman disappeared. Now you can explain to the client how all of the magic happened and sometimes if they’re a truly technical client they will be very interested in the explanation. But the majority of the time I find that they prefer to think that you just have some special magic that the rest of the world does not have access to. It is a pretty good feeling most of the time.

Now there is a secret to being able to pull this off

time and time again even when the odds are stacked against you. It is all about the trail of logic and being able to follow it through the (logical, not physical) closed off black boxes along that trail. In computers as in science every action produces a reaction. So the first step to setting yourself up for success is to make sure that you know everything you can about how your system, and connected systems, at hand operate. You can’t just hide behind your known system be it Exchange or Hyper-V or Active Directory and declare all other territories unknown and not your problem. There has been many a time I have found the solution to a problem with a timely packet capture with WireShark or checking the routing topology on the local router. That falls in the area of the networking team but with applying my own knowledge of networking I was able to take a huge shortcut and point the problem either directly to the network (with solid evidence) or directly to the server and then fix it.

Love the black box

The point is that magic can happen when you break down the black boxes around you. You can’t completely eliminate the black boxes, but if you can reduce them then you can get an idea of how things are functioning in the neighboring black box. Let’s throw in another example. Recently a coworker of mine came to me with a problem he was stuck on. He’s great with server problems but he makes it very clear that networking is not his cup of tea. A server had successfully gone through a P2V and was up and running in the cluster, but a certain service was inaccessible remotely no matter how he looked at things. He made the assumption that things were incorrectly configured on the networking side of things beyond the server. Now there were several ways he could have tested that theory all of which would not involve looking into the black box of the networking equipment, but it does involve knowing what goes into it and what is expected to come out. A ping test would have verified routing and a probe of the port internally would have told him that it was not open on the server. Quickly checking on the server showed that the port was not open on the server’s firewall. An easy check, but since he considered the black box not his problem he was not able to easily and swiftly reach that conclusion.

So in summary make sure you’re always following logic in your troubleshooting and that you are always learning within and without of your realm of expertise. That will set you on the track of the magician as well. Perhaps you’ll be the next one to be asked how that rabbit came out of the hat.

Do you have any magician moment stories as well? Please share them in the comments as I would love to hear them.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

wordpress visitor counter

RSS Subscriptions

Contact Me

%d bloggers like this: