Seven Tiers of Data Recovery

A fire broke out in the building that houses Land Victoria at 570 Bourke Street Melbourne taking out the sub-station and power and the land titles systems. We received the following message -

6 Nov 07 1.28pm "Please be advised that VIC Landata is currently unavailable".

I asked when the system will be up and running again and got this reply

"There has been a fire inside landata’s building, and consequently the whole building has lost power. Unfortunately there is no ETA. I recommend checking the ‘System Status’ screen for further updates. Otherwise, once the Landata message comes down, you will be able to process searches."

The update the following day

7 Nov 07 10.18am

"Please be aware that due to a fire in the office building that houses Landata yesterday they are currently experiencing major service disruptions.

We have been notified that VOTS connectivity has been restored meaning that titles are retrievable.

However, access to the imaging system remains down – so plans, instruments etc remain unavailable for the time being.

All Victorian Property Certificates will be delayed and whilst they can be ordered no orders will be processed until mid next week.

For Victorian Channel Partners we have also been advised that there is no over the counter service available.

We will keep you updated as soon as we know more."

What's the lesson in all this? There has been an unfortunate incident, the power has been taken out for over 12 hours and Land Victoria is out for 24 hours or so. And at the busiest time of the year, Victoria's land registration system is down. This is a major disruption and one that you would have thought avoidable.

I ran this past some technical guys who were astounded but not surprised. They were astounded because this breaches the basic protocol of disaster recovery systems put to the test. There is a good description in Wikipedia of the Seven Tiers of Disaster Recovery. After reading this entry I would have assumed that Land Victoria would have in place Tier 6: Zero or near-Zero data loss or Tier 7: Highly automated, business integrated solution for disaster recovery.

Definition: Tier 6 business continuity solutions maintain the highest levels of data currency. They are used by businesses with little or no tolerance for data loss and who need to restore data to applications rapidly. These solutions have no dependence on the applications or applications staffs to provide data consistency. Tier 6 solutions require some form of Disk Mirroring. There are various synchronous and asynchronous solutions available from the mainframe storage vendors. Each solution is somewhat different, offering different capabilities and providing different Recovery Point and Recovery Time objectives. Often some form of automated tape solution is also required. However, this can vary somewhat depending on the amount and type of data residing on tape.

In a nut shell Tier 6 assumes that your data is kept and mirrored in 2 geographically separate data centres. Data is being written and read to either or both data centres. The data is automatically synched. Each data centre is a mirror of the other. If one data centre goes offline, as it did at 570 Bourke Street, the service is unaffected as the load is then completely taken up at the other data centre.

A good analogy is if you are flying in a 747 and you lose the left engine, you can still fly and land on the right engine.

Google is a prime example of this to the extreme. It would be Tier 7(+). Google is rewriting the book on data centres.

Again I stress the above was an unfortunate incident with staff put at risk. It was shocking to hear people being treated for smoke inhalation or being trapped in lifts. But in a business environment I would have thought this is a case for a Tier 6 DRP (at a minimum). A Tier 6 solution equals 99.9% uptime with no loss of data or service. However, I am not in possession of the full facts or background or steps taken to get the system up and running with no loss of data. There are lessons in all of this.