You Probably Should Test That Stuff First

“The more complicated something is to test, the less likely it is to be tested properly or regularly”

If this is not someone’s rule or “law” - I am happy to claim it. Because it is oh-so-true in most IT environments that I see.

So I was watching a Road Runner cartoon the other day with my 8-year old daughter (one of the rewards of parenthood!) and Wile E. Coyote underwent his usual misfortune as some complicated Acme-sourced plan went awry.

As Wile E. got squashed by a plank and spring that should have launched him high into the air, my daughter wryly commented “You probably should test that stuff first…”

As they say “From the mouths of babes”.

It reminded me of good design practice, and how we need to build systems. They should be built so we can test them - all the time - easily. Meaningful tests, large scale tests, tests that in most environments people do not want to do.

When was the last time you arbitrarily powered down a rack of servers? A bunch of switches, your office, etc? Do the hairs on you neck stand on end just thinking about it?

Guess what? Such tests should not be a big deal. If they are - well - that tells you something about your system.

I often say, if you are not prepared to test your system for failure at the best possible time of day, with all your staff around - then it is not ready to tolerate failures that will inevitably happen at the worst of times!

A good example of positive testing was over at Facebook where they turned off an entire datacentre (10s of Megawatts) just to see how they coped.

Now THAT is testing. And this is what a lot of the other big players in the market do - all the time. This is now “normal” - and it should be for you too.

Because - you probably should test that stuff first…