/bin/rm -r -f *
May 20, 2012 8:49 PM Subscribe
Mythbusters' Tested Blog recently posted a special feature from the Toy Story 2 DVD, in which Pixar's Oren Jacob and Galyn Susman recounted how the files for the movie (just 10gb of data!) were almost lost due to both an erroneous Linux command and a bad backup. The folks at The Next Web: Media followed up with Mr. Jacob, and learned that the movie was actually tossed out and reworked from scratch again nine months prior to a release date that was set in stone, not by the computers, but by the filmmakers themselves: How Pixar’s Toy Story 2 was deleted twice, once by technology and again for its own good.
Offline backups, people. Offline, offsite backups. It's for your own good.
posted by schmod at 9:05 PM on May 20, 2012 [4 favorites]
posted by schmod at 9:05 PM on May 20, 2012 [4 favorites]
Call me a curmudgeon, but I wouldn't work for a company like that. 100 hour working weeks for months at a time. All for the sake of commerce. Not for me no thanks, I have a life.
posted by wilful at 9:08 PM on May 20, 2012 [11 favorites]
posted by wilful at 9:08 PM on May 20, 2012 [11 favorites]
I think a good service would be to snail-mail hard drives to a salt mine for $1 a year cold storage - pay by the drive, not the GB. Like a stem cell bank. Most available backup solutions are error-prone or costly, for big data. Drives are very cheap these days, throwaway commodities.
posted by stbalbach at 9:09 PM on May 20, 2012 [2 favorites]
posted by stbalbach at 9:09 PM on May 20, 2012 [2 favorites]
Tested backups. All the geographic and otherwise redundancy in the world isn't going to help you if it turns out your backups didn't actually work.
posted by kmz at 9:09 PM on May 20, 2012 [9 favorites]
posted by kmz at 9:09 PM on May 20, 2012 [9 favorites]
This is very surprising. I always thought big companies like that would have multiple back ups with some sort of alert to let you know if the back ups did not work. I would also think they would have some sort of protection built in so just anyone can't go in and start a such a simple command to erase something so important. BTW, I have watched TS2 with grandkids more times than I can count but it is still a real joy to watch.
posted by ElliotBoudin at 9:16 PM on May 20, 2012
posted by ElliotBoudin at 9:16 PM on May 20, 2012
Yeah, there's something about the first half of this story that feels like the geek version of Us Weekly's 'celebrities are just like us' feature.
posted by MCMikeNamara at 9:17 PM on May 20, 2012 [4 favorites]
posted by MCMikeNamara at 9:17 PM on May 20, 2012 [4 favorites]
Ah, 1998. When fast computers looked like fast computers, copy-on-write filesystems hadn't been invented yet, and models for an entire movie fit in only 10gb. (10gb hard drives were common at the time, so there's no way they were limited by storage. Probably that's just as much as they could afford to render.)
posted by miyabo at 9:24 PM on May 20, 2012
posted by miyabo at 9:24 PM on May 20, 2012
I always thought big companies like that would have multiple back ups with some sort of alert to let you know if the back ups did not work. I would also think they would have some sort of protection built in so just anyone can't go in and start a such a simple command to erase something so important.
In some ways, it's probably less true the geekier the employees! A number of my coworkers can barely find the computer's power switch. These folks regularly delete their own files by accident and have to get them restored from backup, so we'd hear about it within a couple days if the backups were corrupted.
The folks who would have access to the Toy Story 2 files, however, are the sort of savvy people who only delete their own files once in a blue moon. So unless IT proactively tested the backup procedures, there'd be no way to tell if they were actually working.
posted by Blue Jello Elf at 9:26 PM on May 20, 2012 [1 favorite]
In some ways, it's probably less true the geekier the employees! A number of my coworkers can barely find the computer's power switch. These folks regularly delete their own files by accident and have to get them restored from backup, so we'd hear about it within a couple days if the backups were corrupted.
The folks who would have access to the Toy Story 2 files, however, are the sort of savvy people who only delete their own files once in a blue moon. So unless IT proactively tested the backup procedures, there'd be no way to tell if they were actually working.
posted by Blue Jello Elf at 9:26 PM on May 20, 2012 [1 favorite]
I would also think they would have some sort of protection built in so just anyone can't go in and start a such a simple command to erase something so important.
As it says in the article, they could only move quickly if everyone could read and write files. There is such a thing now as rm -rf protection but it wasn't done back then (though they could have hacked it, but then the task the rm -rf was supposed to accomplish could not have been done either.)
Call me a curmudgeon, but I wouldn't work for a company like that. 100 hour working weeks for months at a time. All for the sake of commerce. Not for me no thanks, I have a life.
That's cool, but they were artists who wanted to see the movie happen, and they also didn't want the studio to go out of business, lose everyone's jobs and seriously damage the CG film industry that would have to provide the replacement jobs.
posted by michaelh at 9:53 PM on May 20, 2012 [8 favorites]
As it says in the article, they could only move quickly if everyone could read and write files. There is such a thing now as rm -rf protection but it wasn't done back then (though they could have hacked it, but then the task the rm -rf was supposed to accomplish could not have been done either.)
Call me a curmudgeon, but I wouldn't work for a company like that. 100 hour working weeks for months at a time. All for the sake of commerce. Not for me no thanks, I have a life.
That's cool, but they were artists who wanted to see the movie happen, and they also didn't want the studio to go out of business, lose everyone's jobs and seriously damage the CG film industry that would have to provide the replacement jobs.
posted by michaelh at 9:53 PM on May 20, 2012 [8 favorites]
I sure hope they got a nice chunk of the profit. Verbal thanks are necessary for something like that, but not sufficient.
posted by Malor at 9:56 PM on May 20, 2012 [2 favorites]
posted by Malor at 9:56 PM on May 20, 2012 [2 favorites]
I think a good service would be to snail-mail hard drives to a salt mine for $1 a year cold storage - pay by the drive, not the GB. Like a stem cell bank. Most available backup solutions are error-prone or costly, for big data. Drives are very cheap these days, throwaway commodities.
Tapes were made for this, and do an extremely good job of it. Hard drives were NOT designed for this.
Funny that the article states that in 1998 tape backups were how most companies protected their data, that is true however it is no different today, the majority of companies still rely on tape for data protection, because it works.
posted by Cosine at 10:00 PM on May 20, 2012 [1 favorite]
Tapes were made for this, and do an extremely good job of it. Hard drives were NOT designed for this.
Funny that the article states that in 1998 tape backups were how most companies protected their data, that is true however it is no different today, the majority of companies still rely on tape for data protection, because it works.
posted by Cosine at 10:00 PM on May 20, 2012 [1 favorite]
Same comment as wilful. It's sad to see a blog post perpetuating the "labor of love" cliche. If your employees are basically living at work, you have a profoundly dysfunctional and inhumane workplace, and there is nothing beautiful or noble about it. The sooner we stop writing about it as if it was, the closer we'll come to meaningful change.
Sure, lots of people are very attached to their Pixar products in intimately emotional ways, but that doesn't mean that Pixar employees owe us their lives in exchange for our money. This is especially relevant when you consider that extraordinary abuse of employees is endemic across the entire range of animation and gaming companies, all under the banner of "they do it out of love."
posted by Nomyte at 10:02 PM on May 20, 2012 [35 favorites]
Sure, lots of people are very attached to their Pixar products in intimately emotional ways, but that doesn't mean that Pixar employees owe us their lives in exchange for our money. This is especially relevant when you consider that extraordinary abuse of employees is endemic across the entire range of animation and gaming companies, all under the banner of "they do it out of love."
posted by Nomyte at 10:02 PM on May 20, 2012 [35 favorites]
I always thought big companies like that would have multiple back ups with some sort of alert to let you know if the back ups did not work. I would also think they would have some sort of protection built in so just anyone can't go in and start a such a simple command to erase something so important.
True story: On September 11th, 2001 Merrill Lynch's datacenter in the World Financial Center (Across the street from the World Trade Center) gets flooded and the servers go down. The towers fall and the entire area is considered a disaster zone. Among other things, all of the data associated with the bank's Credit Default SWAPS, Mortgage Backed Securities and Interest Rate Derivative desks are on those machines; no one knows what the bank's positions are or how much risk is on the books. No problem, there are backups, right? So where are the tapes? Oh, they are sitting on top of those same servers which are in a possibly structurally unstable, unpowered building, which happens to be currently surrounded by suspicious thick grey smoke.
The Governor of NY was called to have the National Guard escort bank systems administrators into the building. Armed with gasmasks and flashlights they retrieved the tapes.
posted by Stu-Pendous at 10:10 PM on May 20, 2012 [21 favorites]
True story: On September 11th, 2001 Merrill Lynch's datacenter in the World Financial Center (Across the street from the World Trade Center) gets flooded and the servers go down. The towers fall and the entire area is considered a disaster zone. Among other things, all of the data associated with the bank's Credit Default SWAPS, Mortgage Backed Securities and Interest Rate Derivative desks are on those machines; no one knows what the bank's positions are or how much risk is on the books. No problem, there are backups, right? So where are the tapes? Oh, they are sitting on top of those same servers which are in a possibly structurally unstable, unpowered building, which happens to be currently surrounded by suspicious thick grey smoke.
The Governor of NY was called to have the National Guard escort bank systems administrators into the building. Armed with gasmasks and flashlights they retrieved the tapes.
posted by Stu-Pendous at 10:10 PM on May 20, 2012 [21 favorites]
Yeah, always test your backups. I recently had a full time job as IT Manager of an extremely poorly run company. On my very first day, the primary server's disk got corrupted and it would not boot. It was only a modest 6Gb of data. They made rotating backups on external hard drives, taken offsite. Unfortunately nobody ever tested them. I located yesterday's backup and discovered it was formatted FAT32, so only 4Gb of the 6Gb backup image was recorded.
The server drive looked recoverable, it wasn't too bad, I could see the directories. But the owner insisted on calling his drinking buddy, the consultant that set up this useless backup system. I protested, this was what they hired me for, but no, I had to stand there and watch his buddy totally destroy the drive structure. Then he said he was late for an urgent appointment and left suddenly. I might have been able to recover this before he screwed it up further, but now it was irretrievable.
Result: $4000 spent on data recovery at Drivesavers and 5 days of downtime for 4 stores. 99.5% of the data was recovered.
posted by charlie don't surf at 10:14 PM on May 20, 2012 [4 favorites]
The server drive looked recoverable, it wasn't too bad, I could see the directories. But the owner insisted on calling his drinking buddy, the consultant that set up this useless backup system. I protested, this was what they hired me for, but no, I had to stand there and watch his buddy totally destroy the drive structure. Then he said he was late for an urgent appointment and left suddenly. I might have been able to recover this before he screwed it up further, but now it was irretrievable.
Result: $4000 spent on data recovery at Drivesavers and 5 days of downtime for 4 stores. 99.5% of the data was recovered.
posted by charlie don't surf at 10:14 PM on May 20, 2012 [4 favorites]
Tapes were made for this, and do an extremely good job of it. Hard drives were NOT designed for this.
Ok great mail your tapes in to the salt mine also, the salt mine is storage-media agnostic, it does not care what you store there. The salt mine only ensures it won't get stolen or burn for long term deep off-line cold storage. Just be sure it is a copy of a copy so you can access the data if you need it. This is catastrophic storage, last man standing.
posted by stbalbach at 10:45 PM on May 20, 2012
Ok great mail your tapes in to the salt mine also, the salt mine is storage-media agnostic, it does not care what you store there. The salt mine only ensures it won't get stolen or burn for long term deep off-line cold storage. Just be sure it is a copy of a copy so you can access the data if you need it. This is catastrophic storage, last man standing.
posted by stbalbach at 10:45 PM on May 20, 2012
back it up back it up back it up - cuz my daddy taught me good
posted by phaedon at 11:01 PM on May 20, 2012 [4 favorites]
posted by phaedon at 11:01 PM on May 20, 2012 [4 favorites]
Any time someone recommends the salt mines for storage I remember the salt mine that Lake Perigneur drained into after a drilling accident.
posted by saucysault at 11:05 PM on May 20, 2012 [4 favorites]
posted by saucysault at 11:05 PM on May 20, 2012 [4 favorites]
Tapes were made for this, and do an extremely good job of it. Hard drives were NOT designed for this.
Ok great mail your tapes in to the salt mine also, the salt mine is storage-media agnostic, it does not care what you store there. The salt mine only ensures it won't get stolen or burn for long term deep off-line cold storage. Just be sure it is a copy of a copy so you can access the data if you need it. This is catastrophic storage, last man standing.
Yeah, I handle data protection for a national financial services coop and am constantly stunned at the way I see other companies handle data protection. I keep a rotating cycle of ~20 backups on different sets of tapes in three different, geographically diverse, hardened locations and I consider that the MINIMUM possible effort (ie: the most that I can do with the budget I get).
posted by Cosine at 11:06 PM on May 20, 2012 [2 favorites]
Ok great mail your tapes in to the salt mine also, the salt mine is storage-media agnostic, it does not care what you store there. The salt mine only ensures it won't get stolen or burn for long term deep off-line cold storage. Just be sure it is a copy of a copy so you can access the data if you need it. This is catastrophic storage, last man standing.
Yeah, I handle data protection for a national financial services coop and am constantly stunned at the way I see other companies handle data protection. I keep a rotating cycle of ~20 backups on different sets of tapes in three different, geographically diverse, hardened locations and I consider that the MINIMUM possible effort (ie: the most that I can do with the budget I get).
posted by Cosine at 11:06 PM on May 20, 2012 [2 favorites]
Speaking of corrupted or missing data, thenextweb article badly needs the attention of a copy-editor. It's epically bad, almost as if the author submitted the equivalent of a test-render rather than the finished product by mistake.
posted by GeorgeBickham at 11:12 PM on May 20, 2012 [4 favorites]
posted by GeorgeBickham at 11:12 PM on May 20, 2012 [4 favorites]
Aside from issues over the failed backup, I want to know how they managed to get to 11 months before release before they discovered that their current draft "was not a good film". And moreover, if they're the kind of studio whose dedication to quality is so strong that they'd consider ditching their work and starting over, why didn't they consider that after the first restore from backup (which later turned out to be broken)? That, it seems, would have been the perfect time to watch the current draft of the movie and say, "You know, guys, even aside from the glitches, this really isn't the film we were hoping to make. Let's give up on trying to restore it and just make a better film."
posted by logopetria at 11:29 PM on May 20, 2012 [1 favorite]
posted by logopetria at 11:29 PM on May 20, 2012 [1 favorite]
Thenextweb is apparently too busy plagiarizing to have time for such niceties as copy editing.
posted by xil at 11:30 PM on May 20, 2012 [5 favorites]
posted by xil at 11:30 PM on May 20, 2012 [5 favorites]
This is especially relevant when you consider that extraordinary abuse of employees is endemic across the entire range of animation and gaming companies, all under the banner of "they do it out of love."
Depending on how the compensation works at a tech company, though, getting that product out the door can give you permanent financial security. It's not a certain thing, but I believe most people would be willing to work that hard for a year or two, in exchange for being able to pick and choose what they did with with the rest of their lives.
It often doesn't work out that way, but when you have a significant stake in the company's stock, one great product (be in a movie, or a program, or a website) can make you very wealthy, very quickly.
I have no idea if Pixar paid their employees appropriately for the work they did, but it's quite possible that many people involved (maybe even most) made a few million bucks for their labors. If that's how it played out, then I wouldn't call 100-hour weeks for 9 months abusive at all.
Considering the apparent love they have for the company, I suspect something like that probably happened.
This is quite different from many game companies, which really do work their employees to the bone and don't pay them anywhere near what they should.... and then you get ones like Activision, which make big promises, and then actively screw over their employees if the game actually does well. (See: Call of Duty series.) Where EA is just a bunch of inept assholes, Activision is actively evil.
posted by Malor at 12:08 AM on May 21, 2012 [1 favorite]
Depending on how the compensation works at a tech company, though, getting that product out the door can give you permanent financial security. It's not a certain thing, but I believe most people would be willing to work that hard for a year or two, in exchange for being able to pick and choose what they did with with the rest of their lives.
It often doesn't work out that way, but when you have a significant stake in the company's stock, one great product (be in a movie, or a program, or a website) can make you very wealthy, very quickly.
I have no idea if Pixar paid their employees appropriately for the work they did, but it's quite possible that many people involved (maybe even most) made a few million bucks for their labors. If that's how it played out, then I wouldn't call 100-hour weeks for 9 months abusive at all.
Considering the apparent love they have for the company, I suspect something like that probably happened.
This is quite different from many game companies, which really do work their employees to the bone and don't pay them anywhere near what they should.... and then you get ones like Activision, which make big promises, and then actively screw over their employees if the game actually does well. (See: Call of Duty series.) Where EA is just a bunch of inept assholes, Activision is actively evil.
posted by Malor at 12:08 AM on May 21, 2012 [1 favorite]
Aside from issues over the failed backup, I want to know how they managed to get to 11 months before release before they discovered that their current draft "was not a good film".
It happens. It's especially difficult for CG movies. I call it "the Jar Jar Effect." Animators get tunnel vision. They get entranced with some new technology like fabric simulation so they can make Jar Jar's ears flop around realistically. They're so anxious to develop some new effect that the lose the big picture. They watch some 1 second clip they're working on, they see it 5000 times over a week and they lose their ability to evaluate it. And then the other animators are all stuck in their little 1 second world, and nobody is looking at how they all work together until it's too late. The result is you have a state-of-the-art animation of a character that everyone hates but he has the most goddam amazing floppy ears.
posted by charlie don't surf at 12:22 AM on May 21, 2012 [21 favorites]
It happens. It's especially difficult for CG movies. I call it "the Jar Jar Effect." Animators get tunnel vision. They get entranced with some new technology like fabric simulation so they can make Jar Jar's ears flop around realistically. They're so anxious to develop some new effect that the lose the big picture. They watch some 1 second clip they're working on, they see it 5000 times over a week and they lose their ability to evaluate it. And then the other animators are all stuck in their little 1 second world, and nobody is looking at how they all work together until it's too late. The result is you have a state-of-the-art animation of a character that everyone hates but he has the most goddam amazing floppy ears.
posted by charlie don't surf at 12:22 AM on May 21, 2012 [21 favorites]
I always thought big companies like that would have multiple back ups with some sort of alert to let you know if the back ups did not work.
There are a lot of rather complicated ways in which things can "not work", which is why the only way to really know your backup worked is to actually do a restore and see if all the functionality on those servers can be restored.
In many small to midsize companies, the CFO oversees IT. How interested do you think s/he is in backup validation?
posted by benzenedream at 12:36 AM on May 21, 2012 [1 favorite]
There are a lot of rather complicated ways in which things can "not work", which is why the only way to really know your backup worked is to actually do a restore and see if all the functionality on those servers can be restored.
In many small to midsize companies, the CFO oversees IT. How interested do you think s/he is in backup validation?
posted by benzenedream at 12:36 AM on May 21, 2012 [1 favorite]
I think a good service would be to snail-mail hard drives to a salt mine for $1 a year cold storage - pay by the drive, not the GB. Like a stem cell bank. Most available backup solutions are error-prone or costly, for big data. Drives are very cheap these days, throwaway commodities. -- stbalbach
This is very surprising. I always thought big companies like that would have multiple back ups with some sort of alert to let you know if the back ups did not work -- ElliotBoudinHaving a backup system won't always stop data loss due to human error. Usually, there is going to be a lag between when you save your data, and when it shows up on the backup.
It sounds like they had a complete 'render' of the movie, generated from source files. This wouldn't take a lot of person time to recreate, it should be as easy as running a command. But if you lose the file, you could lose weeks of computer time, especially when the first Toy Story movie was made.
Also, working in a 'leading edge' tech company, which is what Pixar basically was a lot of the stuff is going to be hand made, by people working at the company. It's not always going to be bullet proof and it's even less likely to have the 'sharp edges' sanded down – that is, even if it works well it might not actively prevent you from fucking everything up with a bad command line.
Also, remember, this was the 90s. A lot of this was band new. My guess is a lot of backups happen automatically and transparently today, but might not have at the time.
posted by delmoi at 12:41 AM on May 21, 2012 [2 favorites]
It is worth remembering that this was 1998, which was almost 15 years ago, and the technologies and methodologies that make this trivial now didn't exist then. And, to a lesser extent, all of the people smugly married to their own preferred modern-day backup systems need to remember: like the man says, black swan events happen. All you are trying to do is minimise the likelihood of errant cygnets.
posted by DarlingBri at 12:42 AM on May 21, 2012 [2 favorites]
posted by DarlingBri at 12:42 AM on May 21, 2012 [2 favorites]
Thenextweb is apparently too busy plagiarizing to have time for such niceties as copy editing.
Oh, they're not plagiarizing, you silly head! They're very kindly offering a free, offsite backup service for your valuable content! Imagine how much trouble Pixar could've avoided if they'd had thenextweb to 'back up' their hard work!
posted by obiwanwasabi at 2:23 AM on May 21, 2012 [6 favorites]
Oh, they're not plagiarizing, you silly head! They're very kindly offering a free, offsite backup service for your valuable content! Imagine how much trouble Pixar could've avoided if they'd had thenextweb to 'back up' their hard work!
posted by obiwanwasabi at 2:23 AM on May 21, 2012 [6 favorites]
Regarding tunnel vision and losing sight of the bigger picture, I'm sure I've seen behind-the-scenes stuff where they storyboard things out in a much lower quality render - or did they not have the tech to do that in 1998?
posted by GallonOfAlan at 3:12 AM on May 21, 2012
posted by GallonOfAlan at 3:12 AM on May 21, 2012
I was the sysadmin for a small cross-platform unix dev shop right around this time. Our data needs were pretty much identical to theirs - lots of users requiring quick access to lots of files, on a network of workstations connected by NFS. I was the only sysadmin and network engineer/ architect.
1) We used a Network Appliance NFS server. This is a piece of hardware that exists only to serve files through NFS. Oh, it also offers snapshots so you can rebuild lost or damaged data without needing to go to your backups - so long as the space hasn't been filled by new junk, the old, deleted stuff remains in the snapshot, for just this sort of emergency.
2) We used a content versioning system called "CVS" - you check out the files you need from a virtual library, sometimes all of the files in a project at once (called a source tree) and work on them to your heart's content, and then check them back in, where they are merged with the library of files everyone is working on. You get big, loud notices when you're about to overwrite what someone else just checked in, and it lets you keep both copies until you've sorted it out, for just this sort of emergency.
3) I would do a random restore on a random set of files once a day. Actually, I didn't, the very nice backup software I was using did that for me. Then I shipped an entire backup to offsite storage once a month, and had them return a tape twice a year to make sure we could recover everything, for just this sort of emergency.
I was grinning at the article when they got to the part about where they decided there would be no witch hunt. You know why there was no witch hunt? Because the sysadmins were probably telling them from day one they had problems, and the execs were stuck in the "but we've always done it this way" rut, right up until it almost cost them the movie. I had to fight to get all of those measures in place. I looked like a superhero the first time someone deleted their home directory by mistake, let me tell you.
posted by Slap*Happy at 3:13 AM on May 21, 2012 [14 favorites]
1) We used a Network Appliance NFS server. This is a piece of hardware that exists only to serve files through NFS. Oh, it also offers snapshots so you can rebuild lost or damaged data without needing to go to your backups - so long as the space hasn't been filled by new junk, the old, deleted stuff remains in the snapshot, for just this sort of emergency.
2) We used a content versioning system called "CVS" - you check out the files you need from a virtual library, sometimes all of the files in a project at once (called a source tree) and work on them to your heart's content, and then check them back in, where they are merged with the library of files everyone is working on. You get big, loud notices when you're about to overwrite what someone else just checked in, and it lets you keep both copies until you've sorted it out, for just this sort of emergency.
3) I would do a random restore on a random set of files once a day. Actually, I didn't, the very nice backup software I was using did that for me. Then I shipped an entire backup to offsite storage once a month, and had them return a tape twice a year to make sure we could recover everything, for just this sort of emergency.
I was grinning at the article when they got to the part about where they decided there would be no witch hunt. You know why there was no witch hunt? Because the sysadmins were probably telling them from day one they had problems, and the execs were stuck in the "but we've always done it this way" rut, right up until it almost cost them the movie. I had to fight to get all of those measures in place. I looked like a superhero the first time someone deleted their home directory by mistake, let me tell you.
posted by Slap*Happy at 3:13 AM on May 21, 2012 [14 favorites]
Slap*Happy: You speak only truth in your last paragraph. I think this is the reality of the IT industry in every company. "Hey, guys, we're about to run into an iceberg." "No we're not! Just code the new features." "Uh, okay. Done. Iceberg's still there." "Geez, IT guys complain so much. Good job on the new feature, though. Now we need this one." "Uh, guys..." "What?" (crunch)
posted by sonic meat machine at 3:48 AM on May 21, 2012 [5 favorites]
posted by sonic meat machine at 3:48 AM on May 21, 2012 [5 favorites]
It is worth remembering that this was 1998, which was almost 15 years ago, and the technologies and methodologies that make this trivial now didn't exist then.
'98 wasn't the stone age, there were still tape back-up systems. I was doing testing of software for Solaris at the time and we used a tape back-up system to save and restore images of OS installs so that we could always test our software at an exact known state of the base system. And I know for certain that we were sending out tapes of our source code at least weekly.
posted by octothorpe at 4:02 AM on May 21, 2012
'98 wasn't the stone age, there were still tape back-up systems. I was doing testing of software for Solaris at the time and we used a tape back-up system to save and restore images of OS installs so that we could always test our software at an exact known state of the base system. And I know for certain that we were sending out tapes of our source code at least weekly.
posted by octothorpe at 4:02 AM on May 21, 2012
Use your left/right keys to browse stories at your peril
posted by oulipian at 4:07 AM on May 21, 2012
posted by oulipian at 4:07 AM on May 21, 2012
"In many small to midsize companies, the CFO oversees IT. How interested do you think s/he is in backup validation?"
It's a problem of framing, then. The CFO may not be interested in "backup validation", but they sure as hell ought to be interested in "risk management".
Also, to all those claiming that automated backup solutions didn't exist in 1998 - uhh, yeah they did. Tape robots, automated scheduling & tape handling, automatic tape library management etc all existed in the mid-90s, and before. The problems of, and solutions to, most large backup problems have been around since mainframe days.
posted by kcds at 4:15 AM on May 21, 2012 [1 favorite]
It's a problem of framing, then. The CFO may not be interested in "backup validation", but they sure as hell ought to be interested in "risk management".
Also, to all those claiming that automated backup solutions didn't exist in 1998 - uhh, yeah they did. Tape robots, automated scheduling & tape handling, automatic tape library management etc all existed in the mid-90s, and before. The problems of, and solutions to, most large backup problems have been around since mainframe days.
posted by kcds at 4:15 AM on May 21, 2012 [1 favorite]
Tape robots, automated scheduling & tape handling, automatic tape library management etc all existed in the mid-90s, and before.
I was working at a place with an Exabyte-compatible 8-tape changer in '94 - the tapes were in a removable rack, and once a month, we'd ship an entire rack off to Iron Mountain for offsite storage. In '96, the client where I worked at had a DLT robot tape library the size of a delivery truck, and it was about to be phased out for a new hexagonal robot library (the old robot could only go back and forth, up and down, the new one was an arm that could swivel as well... it could pack more tapes in less space, and change them more quickly.)
posted by Slap*Happy at 5:00 AM on May 21, 2012 [1 favorite]
I was working at a place with an Exabyte-compatible 8-tape changer in '94 - the tapes were in a removable rack, and once a month, we'd ship an entire rack off to Iron Mountain for offsite storage. In '96, the client where I worked at had a DLT robot tape library the size of a delivery truck, and it was about to be phased out for a new hexagonal robot library (the old robot could only go back and forth, up and down, the new one was an arm that could swivel as well... it could pack more tapes in less space, and change them more quickly.)
posted by Slap*Happy at 5:00 AM on May 21, 2012 [1 favorite]
I've lived this scenario at least three times across a few companies. The last two times I played the role of having the last remaining offsite copy of data. I'm a big believer in, as a user, taking ownership of your own data - and if you care about it keep your own copy.
Had one company's data disappear after a less than vigilant system administrator knowingly let a tape drive sit un-repaired for six months. His office and the server were in the same room and he was a chain smoker, which was probably what did the tape drive in. Then the RAID card ate the array.
You'd think this was a long time ago, but no, chain smoking indoors and co-ed server room offices still existed in 2003. So I brought in the project data I'd retained offsite and at least our group kept going (unlike the finance department that had to reconstruct six months of data).
Second instance, I'd quit a company and then returned a few years later as a consultant. The company had been sliding into oblivion in that time and their system administrator had been downsized. So in addition to the sad scene of walking into a office that had many empty cubes where my former office-mates had been I was also greeted by the wailing of the main storage server telling everyone that it was one drive failure away from end-of-life. But no-one was left with the skills (or money) to fix the problem.
So I asked the owner (who had walked me into to the office to look at his project) if I could link up to the dying storage server and grab a copy of the project files. He didn't mind, even if I wasn't doing it for his benefit. His company folded a few months later and I now have a copy of the projects to hand out to his orphaned customers and the consultants who picked up the remains.
posted by Exad at 5:11 AM on May 21, 2012 [1 favorite]
Had one company's data disappear after a less than vigilant system administrator knowingly let a tape drive sit un-repaired for six months. His office and the server were in the same room and he was a chain smoker, which was probably what did the tape drive in. Then the RAID card ate the array.
You'd think this was a long time ago, but no, chain smoking indoors and co-ed server room offices still existed in 2003. So I brought in the project data I'd retained offsite and at least our group kept going (unlike the finance department that had to reconstruct six months of data).
Second instance, I'd quit a company and then returned a few years later as a consultant. The company had been sliding into oblivion in that time and their system administrator had been downsized. So in addition to the sad scene of walking into a office that had many empty cubes where my former office-mates had been I was also greeted by the wailing of the main storage server telling everyone that it was one drive failure away from end-of-life. But no-one was left with the skills (or money) to fix the problem.
So I asked the owner (who had walked me into to the office to look at his project) if I could link up to the dying storage server and grab a copy of the project files. He didn't mind, even if I wasn't doing it for his benefit. His company folded a few months later and I now have a copy of the projects to hand out to his orphaned customers and the consultants who picked up the remains.
posted by Exad at 5:11 AM on May 21, 2012 [1 favorite]
I would do a random restore on a random set of files once a day
Which is the correct answer.
I don't do backups. I do restores as needed. Backups are a *step* in the file recovery process, not the end goal. So don't be proud about backing up -- be proud about restoring.
And, from the article.
Assigning micro-managed permissions would have eaten up administrative resources, especially in crunch time.
And deleting 90% of everybody's work doesn't?
I hear this all the time. 'You're being a roadblock.' 'Yes. Yes I am. Do you know why we set up road blocks? Because the road is *out*.'
I'd happily let them drive off the bridge, except then I'd need to clean up the mess. This is "cost of doing business" and if you choose not to do this, you better have another way of making that joe in accounting does nuke your core production files.
Or, you get to just rebuild everything -- which means you saved hours by spending tens-to-hundreds of hours.
(And never mind the very good comments up there about snapshotting file systems, multiple backups, source control -- you know, all the tools to make sure that even very active file systems are safe. Fuck, an rsnyc at midnight means you've only lost a day's work at worst.)
This is also a case where unix pedants screwed the pooch. Yes, modify is effectively delete, because if you replace the contents of a file with 0 bytes, then you've effectively deleted it. So, you are correct, the degenerate case does make modify sort of act like a delete.
But it isn't. It doesn't unlink the file. And, you know, I've never seen anybody accidentally write a find/xargs combo that accidentally copies /dev/null onto every file in the file system, but I have seen, repeatedly, someone accidentally try to unlink every file on the system.
So, unlink (which is what rm does) is different that modify, so perhaps we should HAVE A PERMISSION ON IT, YOU PEDANTICAL FUCKS, because if I took away unlink, J Random rm -f simply doesn't work, does it?
I need coffee
posted by eriko at 5:28 AM on May 21, 2012 [7 favorites]
Which is the correct answer.
I don't do backups. I do restores as needed. Backups are a *step* in the file recovery process, not the end goal. So don't be proud about backing up -- be proud about restoring.
And, from the article.
Assigning micro-managed permissions would have eaten up administrative resources, especially in crunch time.
And deleting 90% of everybody's work doesn't?
I hear this all the time. 'You're being a roadblock.' 'Yes. Yes I am. Do you know why we set up road blocks? Because the road is *out*.'
I'd happily let them drive off the bridge, except then I'd need to clean up the mess. This is "cost of doing business" and if you choose not to do this, you better have another way of making that joe in accounting does nuke your core production files.
Or, you get to just rebuild everything -- which means you saved hours by spending tens-to-hundreds of hours.
(And never mind the very good comments up there about snapshotting file systems, multiple backups, source control -- you know, all the tools to make sure that even very active file systems are safe. Fuck, an rsnyc at midnight means you've only lost a day's work at worst.)
This is also a case where unix pedants screwed the pooch. Yes, modify is effectively delete, because if you replace the contents of a file with 0 bytes, then you've effectively deleted it. So, you are correct, the degenerate case does make modify sort of act like a delete.
But it isn't. It doesn't unlink the file. And, you know, I've never seen anybody accidentally write a find/xargs combo that accidentally copies /dev/null onto every file in the file system, but I have seen, repeatedly, someone accidentally try to unlink every file on the system.
So, unlink (which is what rm does) is different that modify, so perhaps we should HAVE A PERMISSION ON IT, YOU PEDANTICAL FUCKS, because if I took away unlink, J Random rm -f simply doesn't work, does it?
I need coffee
posted by eriko at 5:28 AM on May 21, 2012 [7 favorites]
I like how Pixar only seems to have come to the conclusion about just how much its data was worth at the point where it was being driven about in somebody's Volvo.
posted by rongorongo at 5:53 AM on May 21, 2012 [2 favorites]
posted by rongorongo at 5:53 AM on May 21, 2012 [2 favorites]
Having experienced and witnessed a few catastrophic data-loss incidents, the initial emotional hit is brutal, but if the project is current, and still churning around in your mind, then once you get down to it the redo is not as hard as it seemed it would be, and usually comes out better.
I'd say something like 60% of the initial work is comprised of going down, and returning from, blind alleys. The hindsight, as long as it's still in sight, is a big help.
posted by StickyCarpet at 7:00 AM on May 21, 2012 [2 favorites]
I'd say something like 60% of the initial work is comprised of going down, and returning from, blind alleys. The hindsight, as long as it's still in sight, is a big help.
posted by StickyCarpet at 7:00 AM on May 21, 2012 [2 favorites]
It's nice that Pixar spent so much time and effort on the Toy Story 2 rewrite to cement their reputation as a company that wouldn't settle for less than a quality release...
...Only to follow this up in the end with Cars 2. Oh the bitter irony.
Don't get me wrong, my boy loves Cars and Cars 2, and I like the movies because they make him happy. But in terms of replay value? As an adult I can watch and rewatch any of the other Pixar movies, but Cars 2 is just... wrong. There's nothing there. It's OK at best. It has some fun parts. But it is entirely hollow. And I think the Pixar team knew this the entire time.
posted by caution live frogs at 7:57 AM on May 21, 2012 [3 favorites]
...Only to follow this up in the end with Cars 2. Oh the bitter irony.
Don't get me wrong, my boy loves Cars and Cars 2, and I like the movies because they make him happy. But in terms of replay value? As an adult I can watch and rewatch any of the other Pixar movies, but Cars 2 is just... wrong. There's nothing there. It's OK at best. It has some fun parts. But it is entirely hollow. And I think the Pixar team knew this the entire time.
posted by caution live frogs at 7:57 AM on May 21, 2012 [3 favorites]
I'd say something like 60% of the initial work is comprised of going down, and returning from, blind alleys. The hindsight, as long as it's still in sight, is a big help.
Very valid point. What I've learned is if you have a good team, if there's any way possible, you parallel efforts. "Okay, you get some disk online and get tapes spinning, you guys get the DR site starting, and me and her are going to clone the disk and see if some simple data recovery tricks don't get us back online. GO!"
I can ensure you that at least two of those three tasks won't be needed or help, but I can't tell you which two or three of them won't help, so you try, as much, to try everything.
However, the big rule -- the IRON CLAD BIG RULE -- is until you know the data is safe, you *DO NOT TOUCH THE ACTUAL DISKS PERIOD.* This is back to the $1000 dollar question, which is...
"...Is this data worth $1000 per disk to you?" If it is, you *do not touch those disks* and you call a professional data recovery service, who will likely be able to get all that data back for a fairly staggering amount of money, which, if worth more than $1K/disk, will be cheap.
If Pixar had tried its attempts at restoration onto new disks, then the old disks might have been recoverable. Once they threw the bad backup online and played with it for a week, the old state of the disk was so badly walked over that it was impossible to recreate.
So, learn that. If you're on a SAN or NAS, you might have the ability to do a block level clone. If so, you clone off the disk (or more properly, the device/LUN) and try your various tricks there. If they fail, you can clone again and try again -- the original damaged state is unharmed. Indeed, in this situation, the very first thing to do is unmount and remount read-only.
Once you get the data "back", you test the hell out of it, and you keep that original disk device/LUN/whatever untouched and read-only until you are confident that the data is good. Then, and only then, do you use them again -- either migrating the data back or just throwing them back into the pool.
Or, to say it simpler -- attempting to recover damaged data often loses more data, so don't do it with your only copy of the data.
And, be sure you have it all before you stop recovery. Note how Pixar spent a great deal of time in recovering, then worked with the data for a week before they realized how damaged the recovered data actually was. If they'd had the original device intact, they would have had more options. Instead, they went to an incidental offsite copy, then spent a great deal of time in verification.
Assuming the data was fine, esp. in the face of many missing files and version numbers wrong, cost Pixar a great deal of time -- a week of useless work *plus* the time on the 2nd recovery attempt.
Finally, if you're not sure -- call a professional data recovery service *first*.
posted by eriko at 8:04 AM on May 21, 2012 [1 favorite]
Very valid point. What I've learned is if you have a good team, if there's any way possible, you parallel efforts. "Okay, you get some disk online and get tapes spinning, you guys get the DR site starting, and me and her are going to clone the disk and see if some simple data recovery tricks don't get us back online. GO!"
I can ensure you that at least two of those three tasks won't be needed or help, but I can't tell you which two or three of them won't help, so you try, as much, to try everything.
However, the big rule -- the IRON CLAD BIG RULE -- is until you know the data is safe, you *DO NOT TOUCH THE ACTUAL DISKS PERIOD.* This is back to the $1000 dollar question, which is...
"...Is this data worth $1000 per disk to you?" If it is, you *do not touch those disks* and you call a professional data recovery service, who will likely be able to get all that data back for a fairly staggering amount of money, which, if worth more than $1K/disk, will be cheap.
If Pixar had tried its attempts at restoration onto new disks, then the old disks might have been recoverable. Once they threw the bad backup online and played with it for a week, the old state of the disk was so badly walked over that it was impossible to recreate.
So, learn that. If you're on a SAN or NAS, you might have the ability to do a block level clone. If so, you clone off the disk (or more properly, the device/LUN) and try your various tricks there. If they fail, you can clone again and try again -- the original damaged state is unharmed. Indeed, in this situation, the very first thing to do is unmount and remount read-only.
Once you get the data "back", you test the hell out of it, and you keep that original disk device/LUN/whatever untouched and read-only until you are confident that the data is good. Then, and only then, do you use them again -- either migrating the data back or just throwing them back into the pool.
Or, to say it simpler -- attempting to recover damaged data often loses more data, so don't do it with your only copy of the data.
And, be sure you have it all before you stop recovery. Note how Pixar spent a great deal of time in recovering, then worked with the data for a week before they realized how damaged the recovered data actually was. If they'd had the original device intact, they would have had more options. Instead, they went to an incidental offsite copy, then spent a great deal of time in verification.
Assuming the data was fine, esp. in the face of many missing files and version numbers wrong, cost Pixar a great deal of time -- a week of useless work *plus* the time on the 2nd recovery attempt.
Finally, if you're not sure -- call a professional data recovery service *first*.
posted by eriko at 8:04 AM on May 21, 2012 [1 favorite]
Call me a curmudgeon, but I wouldn't work for a company like that. 100 hour working weeks for months at a time. All for the sake of commerce. Not for me no thanks, I have a life.
wilful, my advice is: stay out of startups, and don't open your own business ever.
posted by IAmBroom at 8:09 AM on May 21, 2012 [1 favorite]
wilful, my advice is: stay out of startups, and don't open your own business ever.
posted by IAmBroom at 8:09 AM on May 21, 2012 [1 favorite]
Blue Jello Elf writes "So unless IT proactively tested the backup procedures, there'd be no way to tell if they were actually working."
That's part of a back up operator's job. Sadly just like users most companies are uninterested in dedicating resources to proper operations until they've been burned. I've seen it several times where I've said "You need to do this or some random disaster like theft/fire/flood/lightning strike is going to severely damage the company". Nothing changes and then some random event (usually user stupidity) severely damages the company. Being able to say I told you isn't much comfort at that point.
posted by Mitheral at 8:10 AM on May 21, 2012 [1 favorite]
That's part of a back up operator's job. Sadly just like users most companies are uninterested in dedicating resources to proper operations until they've been burned. I've seen it several times where I've said "You need to do this or some random disaster like theft/fire/flood/lightning strike is going to severely damage the company". Nothing changes and then some random event (usually user stupidity) severely damages the company. Being able to say I told you isn't much comfort at that point.
posted by Mitheral at 8:10 AM on May 21, 2012 [1 favorite]
Don't get me wrong, my boy loves Cars and Cars 2, and I like the movies because they make him happy. But in terms of replay value? As an adult I can watch and rewatch any of the other Pixar movies, but Cars 2 is just... wrong. There's nothing there. It's OK at best. It has some fun parts. But it is entirely hollow. And I think the Pixar team knew this the entire time.
They tried really hard to avoid being corrupted by Disney, and they've held it off longer than I think most companies would have, but there seem to be more signs of compromise every year.
posted by michaelh at 8:11 AM on May 21, 2012
They tried really hard to avoid being corrupted by Disney, and they've held it off longer than I think most companies would have, but there seem to be more signs of compromise every year.
posted by michaelh at 8:11 AM on May 21, 2012
Re: "Call me a curmudgeon..."
I was there, though I wasn't directly working on TS2 at the time, and I think I have a role in that story that... well... I'm not gonna tell because it's really about someone else I was working with. However:
Two things: First, if your work is not something that you enjoy and believe in to a very deep level, then I feel sorry for you.
Second: It's easy, when you're young and naive, to belive that more hours = more productivty. Nowadays, I strive for 6 hours of productivity a day, and if it takes 2-4 additional to support that, so be it. Back then, though, I thought that when I pulled 36 hours in front of the keyboard I was actually doing something.
posted by straw at 9:12 AM on May 21, 2012 [1 favorite]
I was there, though I wasn't directly working on TS2 at the time, and I think I have a role in that story that... well... I'm not gonna tell because it's really about someone else I was working with. However:
Two things: First, if your work is not something that you enjoy and believe in to a very deep level, then I feel sorry for you.
Second: It's easy, when you're young and naive, to belive that more hours = more productivty. Nowadays, I strive for 6 hours of productivity a day, and if it takes 2-4 additional to support that, so be it. Back then, though, I thought that when I pulled 36 hours in front of the keyboard I was actually doing something.
posted by straw at 9:12 AM on May 21, 2012 [1 favorite]
I don't care when this happened. Giving everyone root privileges is never and was never a good idea.
posted by Bonzai at 9:27 AM on May 21, 2012 [2 favorites]
posted by Bonzai at 9:27 AM on May 21, 2012 [2 favorites]
(The story of how Tested.com because Mythbusters' Tested is an interesting one in and of itself, though probably moreso on the Giantbomb side of the rise and fall of the Whiskey Media empire.)
Whoa. I totally had no idea about the Giant Bomb thing. So Gertsmann basically got pulled back into the corporation that screwed him in the first place.
posted by eyeballkid at 9:57 AM on May 21, 2012
Whoa. I totally had no idea about the Giant Bomb thing. So Gertsmann basically got pulled back into the corporation that screwed him in the first place.
posted by eyeballkid at 9:57 AM on May 21, 2012
Aside from issues over the failed backup, I want to know how they managed to get to 11 months before release before they discovered that their current draft "was not a good film".
Are you kidding? Some filmmakers get all the way past release and never figure it out.
posted by Gelatin at 10:09 AM on May 21, 2012
Are you kidding? Some filmmakers get all the way past release and never figure it out.
posted by Gelatin at 10:09 AM on May 21, 2012
If anybody has a REALLY long-gone drive they need recovered SpinRite is magic from the future. It's incredibly cheap (for the level of recovery it's capable of) and works as advertised.
I had a drive that was irradically spinning up and down. A complete shitshow of a failure. I hooked up a dummy box with the SpinRite live disc and three days later I was able to pull off (slowly) 100MB of critical documents.
I would've probably had to pay $500 to data recovery shop to get anything like those results.
posted by lattiboy at 10:22 AM on May 21, 2012 [3 favorites]
I had a drive that was irradically spinning up and down. A complete shitshow of a failure. I hooked up a dummy box with the SpinRite live disc and three days later I was able to pull off (slowly) 100MB of critical documents.
I would've probably had to pay $500 to data recovery shop to get anything like those results.
posted by lattiboy at 10:22 AM on May 21, 2012 [3 favorites]
caution live frogs: "
Don't get me wrong, my boy loves Cars and Cars 2, and I like the movies because they make him happy. But in terms of replay value? As an adult I can watch and rewatch any of the other Pixar movies, but Cars 2 is just... wrong. There's nothing there. It's OK at best. It has some fun parts. But it is entirely hollow. And I think the Pixar team knew this the entire time."
Hopefully Brave will show them getting back to basics. That movie's "The Prize" trailer seemed decent enough. I don't know what to think about Planes, though.
posted by zarq at 10:25 AM on May 21, 2012
Don't get me wrong, my boy loves Cars and Cars 2, and I like the movies because they make him happy. But in terms of replay value? As an adult I can watch and rewatch any of the other Pixar movies, but Cars 2 is just... wrong. There's nothing there. It's OK at best. It has some fun parts. But it is entirely hollow. And I think the Pixar team knew this the entire time."
Hopefully Brave will show them getting back to basics. That movie's "The Prize" trailer seemed decent enough. I don't know what to think about Planes, though.
posted by zarq at 10:25 AM on May 21, 2012
Bonzai, if it's what I think it was, it wasn't root privileges, it's that people in software development, as well as in animation and technical direction, had filesystems mounted that carried shared film assets. One of them was poking through his machine trying to free up space, got confused over what got copied to the local workstation vs what was checked in to the shared filesystem, and "rm -rf .."ed a bunch of it.
I believe that at the time asset management was done with tools over RCS, so users needed write access to the shared drives.
One sysadmin practice I try to do that I think would have slowed this down is to mount things in places that obviously look like an external mount (ie: /mnt/ts2assets), and then use symlinks to point things in the user space to those locations (~user/animation/ts2assets).
Now we can get back to talking about putting story first...
posted by straw at 11:12 AM on May 21, 2012
I believe that at the time asset management was done with tools over RCS, so users needed write access to the shared drives.
One sysadmin practice I try to do that I think would have slowed this down is to mount things in places that obviously look like an external mount (ie: /mnt/ts2assets), and then use symlinks to point things in the user space to those locations (~user/animation/ts2assets).
Now we can get back to talking about putting story first...
posted by straw at 11:12 AM on May 21, 2012
Regarding tunnel vision and losing sight of the bigger picture, I'm sure I've seen behind-the-scenes stuff where they storyboard things out in a much lower quality render - or did they not have the tech to do that in 1998?
Sure they did. Most animated films are storyboarded or done as quickie pencil tests or animatics. That's a little more complicated with CG since setting up a digital animatic means you've already started building the scene files.
But tunnel vision still happens. Some of it is workflow. Usually movies like Toy Story or a digitally rendered character like Jar Jar have the voice actors record the dialogue first. Then you design the models and fit the action (like moving lips) on top of the voice track. The animator hears the same 5 seconds of dialogue like a thousand times while he tweaks the lip synch, he tunes out the horrible patois Jar Jar accent. Sometimes tasks are split by specialty, like one guy does all the floppy ears and some other guy does the facial animation for lip sync. So they can all be looking at one tiny piece of the entire character. They don't see the problems because they're not looking at the bigger picture. Even the director only looks at rushes, they don't get pieced together until they're pretty far into production. That's usually when they discover the problems. There are plenty of stories of regular non-CG films that bombed in their first audience previews and were sent back for recutting or even filming new scenes.
posted by charlie don't surf at 11:24 AM on May 21, 2012
Sure they did. Most animated films are storyboarded or done as quickie pencil tests or animatics. That's a little more complicated with CG since setting up a digital animatic means you've already started building the scene files.
But tunnel vision still happens. Some of it is workflow. Usually movies like Toy Story or a digitally rendered character like Jar Jar have the voice actors record the dialogue first. Then you design the models and fit the action (like moving lips) on top of the voice track. The animator hears the same 5 seconds of dialogue like a thousand times while he tweaks the lip synch, he tunes out the horrible patois Jar Jar accent. Sometimes tasks are split by specialty, like one guy does all the floppy ears and some other guy does the facial animation for lip sync. So they can all be looking at one tiny piece of the entire character. They don't see the problems because they're not looking at the bigger picture. Even the director only looks at rushes, they don't get pieced together until they're pretty far into production. That's usually when they discover the problems. There are plenty of stories of regular non-CG films that bombed in their first audience previews and were sent back for recutting or even filming new scenes.
posted by charlie don't surf at 11:24 AM on May 21, 2012
Call me a curmudgeon, but I wouldn't work for a company like that. 100 hour working weeks for months at a time. All for the sake of commerce. Not for me no thanks, I have a life.
I would call you curmudgeon, but in my case it'd be a pretty clear case of pot v/s kettle, and really, I'm in a good mood today, so I'll just tell the following story and go back to work.
I hate my current employer. Like wishing fiery doom upon them, mouth of satan, ice-skating demons HATE. I find the work they do dodgy, at best. I don't think the whole mess is set up to benefit anyone, and I feel slightly dirty for working for them in the first place. However, it is a down economy and they do toss me a pittance for the use of my particular skill set, which does not have a very wide job market in the hills of $small_southern_USstate.
My job does have one other perk besides keeping me off of food stamps: it allows me to get up-close and personal with some really REALLY great art. I've gotten to touch some really nice stuff, and I mean other side of the velvet rope, hold hands with Michelangelo's David touch (okay, not really but I'm trying to be purposefully vague here). I love this art collection. I've made it very clear to all the jerks and fools and bean counters I work with that I am here to serve this art, not their egos and I will not mindlessly allow anyone to cause it harm. None of it was made by me, only some of it was made by artists I personally know, but it is a great collection and it deserves respect.
There are days when I am feeling down, and utterly tired by yet another request for a pretty painting of flowers for someone's office, or exhausted by having to explain yet again why that weird cubist painting is really very important and valuable, or beaten by questions about whether the dehumidifier in the gallery really needs to be fixed, but all I need to do snap myself out of that broken-down, angry line of thought is to go open one of the drawers in storage, and spend a few minutes intimately tracing the lines of a Jasper Johns drawing with my eyes, or stop in a hallway and admire the subtle play of light and shadow in a Henry Moore sketch or lean back in my office chair and take a moment to notice the wonderful expression of the subject in the portrait of Truman Capote by Andy Warhol and I'm back on track to spend massive amounts of time carefully reframing, cataloging, hanging and restoring work so it'll be around for future generations to do the exact same as me.
So I can imagine that if faced with the destruction of something I personally made, working for a company I adored (as many Pixar employees obviously do), I would put everything on hold to see that it survived, regardless of the cost in time, effort or sanity.
posted by 1f2frfbf at 11:36 AM on May 21, 2012
I would call you curmudgeon, but in my case it'd be a pretty clear case of pot v/s kettle, and really, I'm in a good mood today, so I'll just tell the following story and go back to work.
I hate my current employer. Like wishing fiery doom upon them, mouth of satan, ice-skating demons HATE. I find the work they do dodgy, at best. I don't think the whole mess is set up to benefit anyone, and I feel slightly dirty for working for them in the first place. However, it is a down economy and they do toss me a pittance for the use of my particular skill set, which does not have a very wide job market in the hills of $small_southern_USstate.
My job does have one other perk besides keeping me off of food stamps: it allows me to get up-close and personal with some really REALLY great art. I've gotten to touch some really nice stuff, and I mean other side of the velvet rope, hold hands with Michelangelo's David touch (okay, not really but I'm trying to be purposefully vague here). I love this art collection. I've made it very clear to all the jerks and fools and bean counters I work with that I am here to serve this art, not their egos and I will not mindlessly allow anyone to cause it harm. None of it was made by me, only some of it was made by artists I personally know, but it is a great collection and it deserves respect.
There are days when I am feeling down, and utterly tired by yet another request for a pretty painting of flowers for someone's office, or exhausted by having to explain yet again why that weird cubist painting is really very important and valuable, or beaten by questions about whether the dehumidifier in the gallery really needs to be fixed, but all I need to do snap myself out of that broken-down, angry line of thought is to go open one of the drawers in storage, and spend a few minutes intimately tracing the lines of a Jasper Johns drawing with my eyes, or stop in a hallway and admire the subtle play of light and shadow in a Henry Moore sketch or lean back in my office chair and take a moment to notice the wonderful expression of the subject in the portrait of Truman Capote by Andy Warhol and I'm back on track to spend massive amounts of time carefully reframing, cataloging, hanging and restoring work so it'll be around for future generations to do the exact same as me.
So I can imagine that if faced with the destruction of something I personally made, working for a company I adored (as many Pixar employees obviously do), I would put everything on hold to see that it survived, regardless of the cost in time, effort or sanity.
posted by 1f2frfbf at 11:36 AM on May 21, 2012
I ended up leaving a job I had for 10 years back in the late 90's, managing a computer department for a mid-sized business over something like this. We backed up our AS/400 minicomputer which hosted our time and billing system to digital backup tapes. I had hired an assistant to maintain the backups, and taught him how to check to make sure the backups were working, but the guy was a moron, and didn't know enough to ask for help on something he was obviously out of his depth on. So we had a major system malfunction, and had to resort to restoring from backups, but the backups the idiot was making were all bad. Luckily, I had done a complete system backup before doing a hardware upgrade two weeks before, but it still meant that two weeks of data had to be re-keyed. It was then that I realized that I didn't need the stress of maintaining a system that cost $10000 of lost productivity for every hour it was down.
posted by crunchland at 11:58 AM on May 21, 2012 [1 favorite]
posted by crunchland at 11:58 AM on May 21, 2012 [1 favorite]
I still have a REALLY hard time believing that the entirety of the show's files consisted of 10GB.
Granted, the models and sets for something like Toy Story 2 are much simpler by comparison, but consider that Titanic, which came out in 1997, had over 5 terabytes of storage in use. Even the original Jurassic Park used close to a terabyte back in the early 90s, when hard drive space was around $1 per megabyte.
If Toy Story 2's entire file directory accounted for just 10GB, it's beyond ludicrous that they didn't have multiple redundant backups all over the damn place.
posted by ShutterBun at 1:35 PM on May 21, 2012
Granted, the models and sets for something like Toy Story 2 are much simpler by comparison, but consider that Titanic, which came out in 1997, had over 5 terabytes of storage in use. Even the original Jurassic Park used close to a terabyte back in the early 90s, when hard drive space was around $1 per megabyte.
If Toy Story 2's entire file directory accounted for just 10GB, it's beyond ludicrous that they didn't have multiple redundant backups all over the damn place.
posted by ShutterBun at 1:35 PM on May 21, 2012
ShutterBun: "I still have a REALLY hard time believing that the entirety of the show's files consisted of 10GB."
Remember also that the project was incomplete when it was accidentally trashed. Oren Jacob believes this happened in 1998, and the movie was released at the end of 1999.
A comment on the TNW post may also be enlightening:
posted by zarq at 1:41 PM on May 21, 2012 [2 favorites]
Remember also that the project was incomplete when it was accidentally trashed. Oren Jacob believes this happened in 1998, and the movie was released at the end of 1999.
A comment on the TNW post may also be enlightening:
One major reason the whole project would fit into 10GB is the shader language in RenderMan.
RenderMan is based on procedural shaders which take up next to no space, Texture maps are the exception.
This makes for high resolution textures and actually improves the realism.
RenderMan was way ahead of it's time.
posted by zarq at 1:41 PM on May 21, 2012 [2 favorites]
This is new to some people? Do you people not watch the special features on the DVDs you buy for your kids?
posted by AzzaMcKazza at 4:27 PM on May 21, 2012
posted by AzzaMcKazza at 4:27 PM on May 21, 2012
I still have a REALLY hard time believing that the entirety of the show's files consisted of 10GB.
This is the instructions on how to render, and even more importantly, it's RenderMan, which is procedural. You'd tell it how to draw X, then tell it to move X over there, and it would figure out all the intermediate stages.
The actual rendered out images for the film are a whole different matter. That would have been huge. I recall Pixar using EMC Clarrion with RAID-3, because the fundamental I/O task turned out to be one very long write, followed by one very long read, and at sequential I/O when going out. Nothing is faster than RAID-3 doing full strip writes.
Aha, actual data on Toy Story here. 4.3M lines of code, rendered at 1536x922, 500GB (in 1995!) for the final movie, 34TB moved through RenderMan during the creation. I know Toy Story 3 and A Bug's Life were rendered at 2K, so I suspect TS2 was as well. A couple of postings have said that TS2 was rerendered for the Blu Ray release.
posted by eriko at 6:08 PM on May 21, 2012 [2 favorites]
This is the instructions on how to render, and even more importantly, it's RenderMan, which is procedural. You'd tell it how to draw X, then tell it to move X over there, and it would figure out all the intermediate stages.
The actual rendered out images for the film are a whole different matter. That would have been huge. I recall Pixar using EMC Clarrion with RAID-3, because the fundamental I/O task turned out to be one very long write, followed by one very long read, and at sequential I/O when going out. Nothing is faster than RAID-3 doing full strip writes.
Aha, actual data on Toy Story here. 4.3M lines of code, rendered at 1536x922, 500GB (in 1995!) for the final movie, 34TB moved through RenderMan during the creation. I know Toy Story 3 and A Bug's Life were rendered at 2K, so I suspect TS2 was as well. A couple of postings have said that TS2 was rerendered for the Blu Ray release.
posted by eriko at 6:08 PM on May 21, 2012 [2 favorites]
If Toy Story 2's entire file directory accounted for just 10GB, it's beyond ludicrous that they didn't have multiple redundant backups all over the damn place. --- You're looking with 21st century goggles. In 1995, a 5 gigabyte raid array cost $40,000.
posted by crunchland at 1:53 AM on May 22, 2012 [2 favorites]
posted by crunchland at 1:53 AM on May 22, 2012 [2 favorites]
A RAID array doesn't protect against someone getting overzealous with rm. 40K however is certainly less than payroll for a week. But tape would have been significantly cheaper; maybe a few thousand capital + $100 a week in tapes. Even in 1997 the podunk company I worked for with 6 employees had similar data utilization and I managed to keep it backed up. The real problem in Pixar's case is at that time cheap tape was fairly slow; with the employees working a 100hr a week (and probably not the same 100) you might have had a hard time scheduling a window.
posted by Mitheral at 5:39 AM on May 22, 2012
posted by Mitheral at 5:39 AM on May 22, 2012
In 1998 I had an Iomega Jaz drive which could store 1 GB on a reasonably fast removable disk for $100 a disk plus around $200 for the drive.
The tech existed. But businesses didn't yet have set-in-stone policies and legal requirements to back up every file they touched.
posted by miyabo at 4:25 PM on May 22, 2012
The tech existed. But businesses didn't yet have set-in-stone policies and legal requirements to back up every file they touched.
posted by miyabo at 4:25 PM on May 22, 2012
You're looking with 21st century goggles. In 1995, a 5 gigabyte raid array cost $40,000.
No, storage in the late '90s had gotten very large and very cheap and very fast, and the tapes at the time, DLT, were keeping pace. 10gb/tape, for the cheaper IIIxt, and it could sure as heck back up the 5gb on our NetApp Filer in just a couple hours on a relatively slow SCSI-2 connection. DLT4 was even faster, and ran on drives with a faster connection, and could hold 20gb per tape. I don't know how much the NetApp cost (I inherited it), but I can almost guarantee a kitted-out SGI Octane was more, and they were chucking those under everyone's desk at Pixar.
posted by Slap*Happy at 5:27 PM on May 22, 2012
No, storage in the late '90s had gotten very large and very cheap and very fast, and the tapes at the time, DLT, were keeping pace. 10gb/tape, for the cheaper IIIxt, and it could sure as heck back up the 5gb on our NetApp Filer in just a couple hours on a relatively slow SCSI-2 connection. DLT4 was even faster, and ran on drives with a faster connection, and could hold 20gb per tape. I don't know how much the NetApp cost (I inherited it), but I can almost guarantee a kitted-out SGI Octane was more, and they were chucking those under everyone's desk at Pixar.
posted by Slap*Happy at 5:27 PM on May 22, 2012
« Older Law & Order & Food | Ramblin' Jack Elliott at Old City Hall, Redding... Newer »
This thread has been archived and is closed to new comments
posted by kmz at 8:57 PM on May 20, 2012 [1 favorite]