Back-ups only help if they are actually working.
May 14, 2012 8:34 AM   Subscribe

This post was deleted for the following reason: If it's any comfort, this double post isn't as bad as Cars 2. -- cortex



 
If you're a sysadmin, and contemplating making tutting noises, consider that your time could be spent more profitably by testing your restore system.
posted by zamboni at 8:45 AM on May 14, 2012 [1 favorite]


I feel like I spent that two and a half minutes rather well. Thanks!
posted by Shutter at 8:47 AM on May 14, 2012


Previously.
posted by Nomyte at 8:48 AM on May 14, 2012


Dang! I searched!
posted by Kimberly at 8:50 AM on May 14, 2012


When I was up at Pixar for a meeting, the CIO told the story to me. Here's how I heard it.

Someone mistakenly was doing clean up and did an rm -rf in the main render instructions directory. The way Pixar animation works is that the creative tools are used to make the instructions that a very large (3,000+ node at the time) render farm uses to generate the actual video footage. These instructions are not that big in comparison to video footage, kind of like the script of a movie in a file as compared to the final video file.

As soon as the person realized what they had done, they called the IT group immediately. But by that point, there was nothing that could have been done. They went to the back-ups and found that they didn't have complete back-ups due to using an older method of back-up and also not checking the tapes to see if they were still readable. When magnetic tapes are used for back-up, they can be recorded over many, many times, and can end up in a state where either you can't read them back. Most modern back-up infrastructures now require verification of data recovery on separate hardware, but this was not the case at the time.

One of the producers had been working at home and had most all of the film as render instructions stored on her laptop. They were able to rebuild the data set for the most part and lose very little.

Needless to say, after that, the CIO was cleared to build the most foolproof back-up and archival system I had ever seen. Back-ups now occur at every level from the workstation to the various libraries and render farms. If one layer fails, there are other layers that can be accessed. Returning to my office, I advocated implementation of a similar back-up paradigm, but was turned down due to cost.

I ride bikes with the former CIO, and I'll ask him the next time we get out for more details.
posted by Argyle at 8:51 AM on May 14, 2012 [9 favorites]


« Older Shake & Fold   |   Build the Enterprise Newer »


This thread has been archived and is closed to new comments