You hear the terms Business Continuity, High Availability and Resilience all the time in our industry, however, I am no longer surprised that some companies just don’t get it. Over the years I have worked for blue chip global and SMB companies alike and it is interesting to see how their approach to continued functionality differs. Some companies talk the talk, some buy all the kit and some take a truly holistic view of a potential situation but only a few really pull it all together.
So, what do you need to know about keeping business systems running in the face of adversity? Here are a few pieces of information to help guide you through.
Understand the differences between Resilience, High Availability and Business Continuity
What is what? Well, in general terms, here are some guidelines:
– Resilience is where you apply more than one of a component to a system, i.e. multiple discs, PSUs, NICs, or fans. This provides a certain level of faith in the resilience of the hardware or solution.
– High Availability is more about having multiple servers; either physical or virtual, working together to provide full functionality should part fail.
– Business Continuity looks at the ‘what if’ situations of your systems becoming unavailable for a number of reasons. There can be a crossover between resilience and HA when using virtualisation technologies, but business continuity stands firm in its concept.
Resilience & High Availability
Many companies are sold technical solutions that are resilient or have claimed high availability. That is all well and good, just make sure everyone understands what degree of resilience you have. If you have sold or been sold a resilient system, you need to make sure that the person at the top signing the PO also understands what they are buying. All too often, when the business systems have gone bang, someone at the top is chucking rocks about and asking/ screaming what they paid all that money for? The system was meant to be resilient! Be clear about it, disc configurations and multiple NICs will not stop your applications failing. A full HA deployment will protect against application issues but will not stop a clean power failure or client switch issues. Do the senior management understand that? Set realistic expectations so nobody is disappointed.
Business Continuity
Technology by itself does not give you a ‘get out of jail free’ card as indicated above. Even companies who run a basic backup solution needs to think a little deeper. Take a small company that runs backups to a tape. All that data gets backed up and taken off site and backups have no errors. Then, FIRE! You are ok, you have backups. You do not however have an office, access to your equipment if it survived, a server, a replacement tape library, office space for the staff to work, the ability to reroute your ISP to a new location, the agreement/ licence numbers to be able to download the backup software again so you can restore your business solutions and so on… you see the problem?
True business continuity can snowball until you find yourself needing the most wonderfully complicated solution. Business continuity has to be looked at with a sensible pair of eyes, so where do you draw the line? Well only the business risk owners can really determine that.
One thing that companies fail to give consideration to is this, do your IT staff sit in the same office as your systems? Do you have more than one office/ site? If you have more than one site, consider putting your systems in one site and your IT staff in the other. If the building with your systems in catches fire, your IT staff with the knowledge to rebuild it are safe. If the building with your IT staff in it catches fire and there are casualties, then your systems will continue to function. Yes it is a rather cold hearted way of looking at things, but it is a valid point.
Think it through properly not just as a paper exercise. I recall one very large firm who had a wonderful business continuity plan, except for one small detail, whilst the data would be protected fully, none of its users outside of IT would have access to any of the systems. That to me makes the business continuity massively flawed. All your data is safe, but useless. I would not have wanted to be around the day the Cxx was screaming about his multi million dollar business continuity plan.
Try Surviving
The last point that needs to be raised is to actually try surviving. So many companies don’t and just put faith in the fact they have all this technology. Full business continuity testing should really be carried out every 6 months, run your business from your DR site for a week and then fail back. Prove it works, test it, try it, amend the plans if needed and make sure that all the IT staff understand what to do in a major issue. Who contacts who and when. You don’t want to be finding out on the fly. At times of real crisis it begins not with technology but with communication plans.
One excellent way of testing this is through the use of red dots! It is also a great team building exercise. With only a couple of people at the top knowing what is happening, plan a realistic business continuity test. Decide what disaster you are going to be faced with. Then when people walk in to the office in the morning, get a stranger to put a red dot on a number of randomly selected staff. (The stranger ensures the random part) These people are now not accessible to the business, remove smart phones etc. and send them off together for coffee. In reality they could be un-contactable, hospitalised or worse, you choose. Then implement your disaster. Study the pain points, ‘Only Janet knows the password to that server’, ‘who has the key to the safe?’ and ‘the DBA was the only person who knew anything about that database’. You will be astonished at how many single points of failure you will uncover.
The principles are all the same
Regardless of whether your company is large or small, the principles of business continuity are always the same. Look at where your systems are, what resilience and HA they have, look at who knows what about the systems. Dig out and uncover all the knowledge that is trapped inside the minds of others. IT staff are terrible for storing things in their head, remember the adage ‘if they were knocked down by a bus tomorrow’ and apply it to the business. Technology alone cannot protect properly!
Lastly, every company I have ever worked in I have suggested a systems-less day, not one single company has been brave enough to even try. Could you operate your business without systems?
Comments