Life Guarding Your Data
July 7, 2010 10 Comments
Last week as an adult leader of my son’s Boy Scout Troop, I accompanied a dozen fine young men to a week-long summer camp. While they were working on various merit badges, I took a training course that was available to both Scouts and adult leaders – the BSA Certified Life Guard course.
There’s a reason that BSA Certified Life Guards are so well respected in the life guard community. It’s a rigorous program that’s both physically and mentally challenging. It’s considered the cream of the crop for life guard programs.
As I look back at the course, I see that there are some principles that are common to life guarding and protecting your server’s data.
Prior to going to any camp outing, each Scout must turn in a health form that documents any medical conditions of the Scout. This allows the medical professionals to be better equipped with specific information should an emergency arise. For example, they know the medications he’s taking, conditions he has, etc.
A Server Run Book serves the same purpose in the IT world. It documents the settings on the server, the applications it supports, and the changes recently made to its configuration. Without a Run Book, diagnosing a problem is much more difficult, especially if multiple people administer the server.
Have a Plan
When things go wrong, it’s not the time to stop and think about your course of actions; it’s time to act. You need to know who’s going to do what and when. In Life Guarding it’s called a ERP, Emergency Response Plan, and it can be the difference between life and death.
The same is true for server issues. If your SQL Server goes down, you need to have a plan. You need to know who to call for what support. Know the contact information for the SAN administrator, your Windows administrator, and your boss. Know who is going to get the messaging out to the users. In short, know who will do what.
Know Your Tools
A Life Guard has many tools available to him to rescue drowning victims. He can use a pole to reach out the the person, a floating ring to toss the the victim, canoes or other boats to go get them, or a floatation tube to help in a personal rescue attempt. The Life Guard must be familiar with each of these devices before the need arises.
Likewise, a DBA has many tools at his disposal when recovering a downed server. Some are built into the SQL Server SKU, others are available through third parties. In any case, you should become fluent in their use before you need them in an emergency situation.
Keep in Shape
As a Life Guard -in-training, I had to swim, and swim, and swim. And then swim some more. And when the swimming was done, we started the training. The swimming was just a warm up.
As a DBA, it’s imperative that you keep in shape as well. Go to training. Go to conferences. Go to user group meetings. Learn the features of the new version. Know what tools are available. Incorporate the best practices into your environment. Read.
During the 30 hours of Life Guard training I received last week, almost all of it was in the form of practice. The instructor explained various techniques to us while standing on the dock and then minutes later we were in the water practicing those life saving techniques. It’s one thing to know in your head how to do something; it’s something else entirely to actually be able to do it when the pressure is on.
As DBAs we should test and practice as well. Regularly verify the integrity of your servers’ back ups. Perform some test restores to make sure that everything is happening as you wish. Don’t be caught off guard with a need to recover data only to find you can’t.
As I tell my students and clients – back ups are really all that important. It’s the ability to restore that’s important.
Know the Risks
Life Guarding is a inherently dangerous activity. You must be ready to risk your life to save another. Additionally there are legal implications to being a Life Guard.
As a Database Administrator, we have risks too. How much data can we afford to lose? How much does each minute of downtime cost our organization? The answer to these questions will help us to understand the risk and make recommendations.