How would your business cope if you had no internet for 4 days, or no office for 3 months?
As an IT service company, we write disaster recovery plans, test them and enact them as part of our core business so we take them seriously. But there is nothing like actually having to rely on the plan for a real live disaster to our own business to focus the mind on what really works.
A Small Problem
Ours was not a dramatic outage – no terrorist attack, or natural disaster – just something that can happen to any business any time: our main Internet connection went down, the fibre to our office was cut somewhere up the street. We had a backup link, but its much slower. Here are the things:
- We could not get any information about how long the link would be down – 4 hours? 4 days? Nobody could tell us.
- Whatever actions we take have consequences for staff, clients and the systems: do we dramatically enact the full plan, only to find the link back up within the day?
- The backup link was deliberately smaller – we don’t have budget to keep double our required internet link running all the time – so it was always intended as short term. Good for a few hours, but not for a few days.
The Plan
Yes, we have a lot of services running from our office (pure economics, its a lot cheaper than cloud). We’re an IT company, and have access to pretty good hardware and all the clever engineers we need. We also have a well defined DR plan, with an offsite DR centre and all the facility to bring up our systems on demand.
Our staff can and do work from home (or on the road) on a regular basis, so a few hours into the outage, we sent all the staff home so they would have better internet links. For the next two days, they continued to work from home with only a skeleton staff at the office (small numbers to avoid overwhelming the backup link).
The Cost
Despite good planning, and facilities, the outage cost us:
- missed client meetings
- engineers concentrating on DR instead of production.
- time taken to make the change to the DR centre, and for staff to go home and pick up their work from there.
- All the little things that didn’t work 100% and needed extra time to make them right.
Over the three days, I calculate we lost 30% of our productivity – could have been a lot worse of course, but still an unexpected loss.
A Much Bigger Disaster
About three months ago, there was a burst water main in a Sydney CBD street (common enough occurrence). The water flooded the basement of one an office tower, and took out all the electrical installations. The building was shut down for almost 3 months – yes, thats right: 3 months! The entire electrical infrastructure had to be replaced.
The Impact
- Every business in that building had to find long term ways to work without their offices.
- Insurance for most did not cover anything like the full cost of relocation, data recovery, alternative office space.
- Massive impact on production for many of the companies in that building
A Good News Story
One of our clients was in that building. In the scheme of things they count as one of the lucky ones, or perhaps one of the better prepared – its not really luck. We brought all their systems up at a DR location, and switched over their remote access so they could all work from home. So in their case, they could keep functioning due to good partners and forethought. Others were not so well served.
A Word Of Warning
What would be the effect on your business if you had no Internet for 4 days? Or no office for 3 months? The scenarios I have described are such everyday occurrences, they are not only likely to happen, your business is bound to be affected by something like this one day.
The moral of the story is: take DR seriously, its an essential for business survival.
Frances Russell is an expert on IT and Risk Management with 20 years experience managing technology, security and risk for businesses. Practical, real world experience working with organisations to improve their IT and reduce risk is backed up by solid academic qualifications and relevant industry certifications including ISACA: CISA CRISC CISM.
FooForce can work with you to minimise your IT risk. If you would like a chat about your backup and DR, feel free to call/email anytime.