Lessons From a Decade of IT Failures
October 27, 2015 5:21 PM Subscribe
To commemorate the last decade’s worth of failures, we organized and analyzed the data we’ve collected. We cannot claim—nor can anyone, really—to have a definitive, comprehensive database of debacles. Instead, from the incidents we have chronicled, we handpicked the most interesting and illustrative examples of big IT systems and projects gone awry and created the five interactives featured here. Each reveals different emerging patterns and lessons. Dive in to see what we’ve found. One big takeaway: While it’s impossible to say whether IT failures are more frequent now than in the past, it does seem that the aggregate consequences are worse.
Spectrum does some of the best reporting on the planet, assuming you want to know about software or electrical engineering.
There are, however, a bunch of broken links to "IT mercy rule" so there's the right link and it seems like a pretty good idea.
posted by GuyZero at 5:27 PM on October 27, 2015 [1 favorite]
There are, however, a bunch of broken links to "IT mercy rule" so there's the right link and it seems like a pretty good idea.
posted by GuyZero at 5:27 PM on October 27, 2015 [1 favorite]
can't we just blame society?
That is, in my experience is that there's always a client who wants something that's if not impossible, certainly magnitudes more complex than they imagine, and a sales agent who will pretend to consult with someone in IT and then nod and go, yeah, we can do that.
and so on.
posted by philip-random at 5:35 PM on October 27, 2015 [26 favorites]
That is, in my experience is that there's always a client who wants something that's if not impossible, certainly magnitudes more complex than they imagine, and a sales agent who will pretend to consult with someone in IT and then nod and go, yeah, we can do that.
and so on.
posted by philip-random at 5:35 PM on October 27, 2015 [26 favorites]
Wow, I'm pretty good at the IT Failure Blame Game. (8/10 correct)
posted by aubilenon at 5:36 PM on October 27, 2015 [1 favorite]
posted by aubilenon at 5:36 PM on October 27, 2015 [1 favorite]
can't we just blame society?
One consultant told me, "You can't really call yourself a mature project manager until you've led a project that was a complete failure." So there's that.
In my world, middle managers would make a great stand-in for society, so let's blame them. They're the ones who said they'd fund the project to completion but wouldn't spend any money on training. They're the ones who signed off on the requirements but didn't read or understand the documentation. They're the ones who appointed some dipshit with no imagination and no people skills (or any other apparent skills) to be the department lead on the project. They're the ones who led a pitchfork-waving rabble through the company to find a scapegoat when everything went to hell the first time. And they're the ones who hired a really competent engineer to lead the second kick at the same project, and then fired him when he told the truth. So yeah, fuck those guys.
(And I'm not even in IT.)
posted by sneebler at 5:56 PM on October 27, 2015 [37 favorites]
One consultant told me, "You can't really call yourself a mature project manager until you've led a project that was a complete failure." So there's that.
In my world, middle managers would make a great stand-in for society, so let's blame them. They're the ones who said they'd fund the project to completion but wouldn't spend any money on training. They're the ones who signed off on the requirements but didn't read or understand the documentation. They're the ones who appointed some dipshit with no imagination and no people skills (or any other apparent skills) to be the department lead on the project. They're the ones who led a pitchfork-waving rabble through the company to find a scapegoat when everything went to hell the first time. And they're the ones who hired a really competent engineer to lead the second kick at the same project, and then fired him when he told the truth. So yeah, fuck those guys.
(And I'm not even in IT.)
posted by sneebler at 5:56 PM on October 27, 2015 [37 favorites]
Eager to see the upcoming "Monuments to Failure" story. *doffs cap*
posted by destro at 5:58 PM on October 27, 2015
posted by destro at 5:58 PM on October 27, 2015
hehe, seems like an aggregate of the DailyWTF
posted by numaner at 6:00 PM on October 27, 2015 [2 favorites]
posted by numaner at 6:00 PM on October 27, 2015 [2 favorites]
Well, it's because IT is easy, so we don't need to spend any real time, money or rigor on it. Anybody can write code, and it always runs the first time.
Fast, correct, cheap. Everyone demands all three. That's why they get none of them.
posted by eriko at 6:07 PM on October 27, 2015 [28 favorites]
Fast, correct, cheap. Everyone demands all three. That's why they get none of them.
posted by eriko at 6:07 PM on October 27, 2015 [28 favorites]
Don't forget that someone with 10-20 years of experience is at best completely fungible with (when not strictly inferior to) someone with 0-2 years of experience. Especially if they have a nice (pe)degree and know AmazoFramework and HotNewLang.
posted by namespan at 6:28 PM on October 27, 2015 [15 favorites]
posted by namespan at 6:28 PM on October 27, 2015 [15 favorites]
I've been involved in some doozies over the years, including the memorable time that a PM asked "can we just swap .NET, Windows and SQL Server for J2EE, Linux and Oracle?"...
And only today I sat in on an initial requirements gathering meeting for a project that would basically require the team to solve the Traveling Salesman Problem. "But surely this scheduling and logistics stuff is what computers are good at?"
posted by 43rdAnd9th at 6:54 PM on October 27, 2015 [15 favorites]
And only today I sat in on an initial requirements gathering meeting for a project that would basically require the team to solve the Traveling Salesman Problem. "But surely this scheduling and logistics stuff is what computers are good at?"
posted by 43rdAnd9th at 6:54 PM on October 27, 2015 [15 favorites]
As a dev turned PM, this stuff is nightmare-fuel.
posted by blue_beetle at 7:03 PM on October 27, 2015 [1 favorite]
posted by blue_beetle at 7:03 PM on October 27, 2015 [1 favorite]
That is, in my experience is that there's always a client who wants something that's if not impossible, certainly magnitudes more complex than they imagine, and a sales agent who will pretend to consult with someone in IT and then nod and go, yeah, we can do that.
Winner, winner, chicken dinner. Sales guy does whatever they have to to land the contract. They get their bonus. The rest of the clusterfuck is for the nerds to figure out.
The other thing is that reputation both doesn't mean shit in the tender process and can't really be quantified anyway. The tender requirements are so complex and onerous (both rightly and wrongly so) that only the largest of companies can even think of bidding. So we get the same shit merry-go-round and more millions and billions of taxpayer dollars get shoveled into the money pit/toilet that is corporate IT welfare. Sometimes, when there's no competition either by accident or malicious design, we get clusterfucks like this one where infrastructure is so dramatically overbuilt you can't help but raise an eyebrow and think about the grease that was required for the palm.
posted by Talez at 7:14 PM on October 27, 2015 [12 favorites]
Winner, winner, chicken dinner. Sales guy does whatever they have to to land the contract. They get their bonus. The rest of the clusterfuck is for the nerds to figure out.
The other thing is that reputation both doesn't mean shit in the tender process and can't really be quantified anyway. The tender requirements are so complex and onerous (both rightly and wrongly so) that only the largest of companies can even think of bidding. So we get the same shit merry-go-round and more millions and billions of taxpayer dollars get shoveled into the money pit/toilet that is corporate IT welfare. Sometimes, when there's no competition either by accident or malicious design, we get clusterfucks like this one where infrastructure is so dramatically overbuilt you can't help but raise an eyebrow and think about the grease that was required for the palm.
posted by Talez at 7:14 PM on October 27, 2015 [12 favorites]
the shifting measure of complexity was practically screaming out that the problem itself was poorly understood, and success was never likely.
All of these IT failures are of course successes for the shareholders of the corporate IT contractors. That is the problem that public IT administrators are failing to understand.
posted by infinitewindow at 7:37 PM on October 27, 2015 [8 favorites]
All of these IT failures are of course successes for the shareholders of the corporate IT contractors. That is the problem that public IT administrators are failing to understand.
posted by infinitewindow at 7:37 PM on October 27, 2015 [8 favorites]
There's some sort of irony in the atrociousness of the UI of the "IT Failure Blame Game".
posted by asterix at 7:47 PM on October 27, 2015 [7 favorites]
posted by asterix at 7:47 PM on October 27, 2015 [7 favorites]
I chuckle to think of how much worse the "Impact of IT Systems Gone Wrong" chart would look if it included the aggregate wasted dollars in failed IT projects in publicly traded companies that never gets reported anywhere.
posted by Ickster at 8:14 PM on October 27, 2015 [5 favorites]
posted by Ickster at 8:14 PM on October 27, 2015 [5 favorites]
Famous last words on IT projects:
"Those guys in India can do it for 50% less"
"We can test it later"
"We can make up for lost time, later on."
"We saw the UI demo a month ago, why isn't the software ready?"
posted by storybored at 8:28 PM on October 27, 2015 [14 favorites]
"Those guys in India can do it for 50% less"
"We can test it later"
"We can make up for lost time, later on."
"We saw the UI demo a month ago, why isn't the software ready?"
posted by storybored at 8:28 PM on October 27, 2015 [14 favorites]
Oh, and my favorite:
"We don't think your estimates are right, so we cut the schedule by 20%"
posted by storybored at 8:29 PM on October 27, 2015 [19 favorites]
"We don't think your estimates are right, so we cut the schedule by 20%"
posted by storybored at 8:29 PM on October 27, 2015 [19 favorites]
I will definitely be sharing this link around the office. I've got about 15 hours of meetings this week reviewing a first pass at business requirements for a multi-million dollar system to replace a previous even more multi-million dollar system that died in procurement. I'm sure we'll get it right this time.
posted by sevenyearlurk at 8:34 PM on October 27, 2015 [4 favorites]
posted by sevenyearlurk at 8:34 PM on October 27, 2015 [4 favorites]
I will go read this, but right now even thinking about it gives me anxiety.
posted by fedward at 8:39 PM on October 27, 2015 [1 favorite]
posted by fedward at 8:39 PM on October 27, 2015 [1 favorite]
Back when I worked for a gaming company, the game post-mortems on Gama Sutra were fascinating reading. Some commonalities, to be sure, like "We didn't have the resources we needed when we needed them," or "We completely underestimated the difficulty of this facet of the project."
But then you'd get really weird shit, like, "The engineers and artists couldn't agree on a toolset, so we spent a good half-year building a bridge between their two separate formats" and the like.
All I know is that I wish I'd documented the email system from that old system a few years ago, where sending mass email involved dumping PL/SQL into a textarea so the scheduled job could pick up, process it, and pass the data to a third party. Would have been a DailyWTF article for sure.
Can't wait to see the post-mortem articles coming out of the ACA insurance exchange debacles. Software is hard enough to develop without idiots legislating the design and deadlines.
posted by fifteen schnitzengruben is my limit at 9:04 PM on October 27, 2015 [4 favorites]
But then you'd get really weird shit, like, "The engineers and artists couldn't agree on a toolset, so we spent a good half-year building a bridge between their two separate formats" and the like.
All I know is that I wish I'd documented the email system from that old system a few years ago, where sending mass email involved dumping PL/SQL into a textarea so the scheduled job could pick up, process it, and pass the data to a third party. Would have been a DailyWTF article for sure.
Can't wait to see the post-mortem articles coming out of the ACA insurance exchange debacles. Software is hard enough to develop without idiots legislating the design and deadlines.
posted by fifteen schnitzengruben is my limit at 9:04 PM on October 27, 2015 [4 favorites]
One of the biggest questions on my mind these days is whether the big tech companies (Apple, Facebook, Google, etc.) have found The Secret for making software projects successful (and are keeping it to themselves).... or they are just much better than average about hiding their failures.
posted by miyabo at 9:23 PM on October 27, 2015 [5 favorites]
posted by miyabo at 9:23 PM on October 27, 2015 [5 favorites]
Money seals a lot of lips, I suspect.
posted by Mrs. Davros at 9:25 PM on October 27, 2015
posted by Mrs. Davros at 9:25 PM on October 27, 2015
i was once directed to solve an np-hard mod/sim problem in 60 days. you can sell all the np-hard-automated-computing-tools you want. you just can't build em. #lawsuits
posted by j_curiouser at 9:30 PM on October 27, 2015 [4 favorites]
posted by j_curiouser at 9:30 PM on October 27, 2015 [4 favorites]
One of the biggest questions on my mind these days is whether the big tech companies (Apple, Facebook, Google, etc.) have found The Secret for making software projects successful (and are keeping it to themselves).... or they are just much better than average about hiding their failures.
Well, it's certainly true that we don't know what projects Apple or Google or Facebook are working on or when they'll be done until they see fit to tell us. Whereas with publicly funded projects, they're a matter of public record before anyone even starts working on them. But it's also true that the motivations are different when the company making the software loses money by going over budget, rather than getting more money when they go over budget.
posted by aubilenon at 9:41 PM on October 27, 2015 [4 favorites]
Well, it's certainly true that we don't know what projects Apple or Google or Facebook are working on or when they'll be done until they see fit to tell us. Whereas with publicly funded projects, they're a matter of public record before anyone even starts working on them. But it's also true that the motivations are different when the company making the software loses money by going over budget, rather than getting more money when they go over budget.
posted by aubilenon at 9:41 PM on October 27, 2015 [4 favorites]
"Believe it or not, the thing you're bitching at me about and thinking is my fault happens to be because despite my saying 'it's not going to work if you go with that infrastructure' you went with that infrastructure."
"So what, exactly, did you imagine would happen to your huge horizontal layout without any break points on mobile? The thing you claim to be good at? Again, what, exactly, is it you imagine I was going to do to make it work? I mean, even in vague terms."
"You know this package you have every client install is actually worse than useless if you provide no guidance on using it (it degrades performance considerably, which is barely worth it if it's being used properly)--which you can't do, because it happens at the end of the project when you've already checked out. Also, you don't know what it does, how it does it, and what the best practices for using it are, despite your protestations."
"What part of 'I am not knowledgeable about x, you'll need an x guy' did you not understand when you asked me a few weeks ago, and why are you now angry I don't know it, which you are now saying is critical for tomorrow morning? I do have an answer: good luck."
posted by maxwelton at 10:04 PM on October 27, 2015 [17 favorites]
"So what, exactly, did you imagine would happen to your huge horizontal layout without any break points on mobile? The thing you claim to be good at? Again, what, exactly, is it you imagine I was going to do to make it work? I mean, even in vague terms."
"You know this package you have every client install is actually worse than useless if you provide no guidance on using it (it degrades performance considerably, which is barely worth it if it's being used properly)--which you can't do, because it happens at the end of the project when you've already checked out. Also, you don't know what it does, how it does it, and what the best practices for using it are, despite your protestations."
"What part of 'I am not knowledgeable about x, you'll need an x guy' did you not understand when you asked me a few weeks ago, and why are you now angry I don't know it, which you are now saying is critical for tomorrow morning? I do have an answer: good luck."
posted by maxwelton at 10:04 PM on October 27, 2015 [17 favorites]
Also, on a related note, everyone is so used to using consumer software that is free and almost flawless at this point that they now have the same expectations for enterprise/niche software. Maybe a few years ago you could get away with selling an ERP system that needs to go down for maintenance every night and has a couple of operations that take 2 minutes to respond, but today your users are comparing it to Gmail and expect it to just work perfectly and instantly 24/7/365. It's certainly leading to a lot of employment for software engineers in the short term who are trying to make these legacy systems slick and friendly.
posted by miyabo at 10:23 PM on October 27, 2015 [9 favorites]
posted by miyabo at 10:23 PM on October 27, 2015 [9 favorites]
I've been worried about what to do with my legacy system at work since I've started working there. I just got off of work but now I'm going to read through this website to see if I can prevent future failures. Thanks for the post!
posted by RichAndCreamy at 11:11 PM on October 27, 2015
posted by RichAndCreamy at 11:11 PM on October 27, 2015
One of the biggest questions on my mind these days is whether the big tech companies (Apple, Facebook, Google, etc.) have found The Secret for making software projects successful (and are keeping it to themselves).... or they are just much better than average about hiding their failures.
I think a lot of it comes down to vertical integration. Google pays employees to build and maintain a system that it owns and profits from. In contrast Oracle pays employees to meet contractual obligations, no matter how dumb the contract or the people signing it.
posted by pwnguin at 11:20 PM on October 27, 2015 [6 favorites]
I think a lot of it comes down to vertical integration. Google pays employees to build and maintain a system that it owns and profits from. In contrast Oracle pays employees to meet contractual obligations, no matter how dumb the contract or the people signing it.
posted by pwnguin at 11:20 PM on October 27, 2015 [6 favorites]
One of the biggest questions on my mind these days is whether the big tech companies (Apple, Facebook, Google, etc.) have found The Secret
But this is survivorship bias at work. You have just named three survivors who have delivered successfully in the past, but they are few compared to the many dead whose last failure was fatal. And even then: Apple has had numerous failures (Apple III, Lisa, Newton, Copland, ...) that were only survivable because of the financial strength of other products.
posted by i_am_joe's_spleen at 11:54 PM on October 27, 2015 [5 favorites]
But this is survivorship bias at work. You have just named three survivors who have delivered successfully in the past, but they are few compared to the many dead whose last failure was fatal. And even then: Apple has had numerous failures (Apple III, Lisa, Newton, Copland, ...) that were only survivable because of the financial strength of other products.
posted by i_am_joe's_spleen at 11:54 PM on October 27, 2015 [5 favorites]
"Shhhh. You are not allowed to say things like 'In my opinion we should seriously consider [Ruby, Angular, Node, Drupal, or more-likely some random package heard about via a marketing conference]' unless you can explain what your new hotness brings to this project. Can you even explain what role you think it will play?"
"Oh, I totally agree, I should invest 60-80 hours of my own time learning the poorly documented API of this random service you sold this client on so that I can bill six hours over the next year with that knowledge. If you can name a single one of the previous five 'sure-fire next-big-thing everyone-will-be-using-it' services you asked me to become an expert at previously, I might consider this one."*
* That's not true, I'd never consider it under the terms they want , i.e., for free. I just say "I'd be happy to do that, but I'm going to bill the R&D time" and poof, the idea is gone.
posted by maxwelton at 12:31 AM on October 28, 2015 [1 favorite]
"Oh, I totally agree, I should invest 60-80 hours of my own time learning the poorly documented API of this random service you sold this client on so that I can bill six hours over the next year with that knowledge. If you can name a single one of the previous five 'sure-fire next-big-thing everyone-will-be-using-it' services you asked me to become an expert at previously, I might consider this one."*
* That's not true, I'd never consider it under the terms they want , i.e., for free. I just say "I'd be happy to do that, but I'm going to bill the R&D time" and poof, the idea is gone.
posted by maxwelton at 12:31 AM on October 28, 2015 [1 favorite]
Is there a name for the phenomenon where a company has a product which works and meets user needs just fine, and then they tinker with it over multiple iterations such that it gets further and further away from being as useful?
posted by biffa at 12:47 AM on October 28, 2015 [1 favorite]
posted by biffa at 12:47 AM on October 28, 2015 [1 favorite]
I've read plenty of postmortems, and contributed to enough of them to appreciate how difficult and time consuming they are to write. It would save plenty of effort if everyone used postmortem annotations in their code and design docs, like tattoos on the skin reading 'In case of autopsy, cut here first'.
Like
posted by Doroteo Arango II at 1:22 AM on October 28, 2015 [7 favorites]
Like
// POSTMORTEM: Quick fix to meet OKRs.
// POSTMORTEM: This is the most clever code I've written this week.
* @postmortem: This has not been touched in 8 years, the authors don't work here anymore, there is no
* documentation, no tests, and we lost the BUILD files. We've been using the same binary and changing
* the version numbers for ages. We are touching it today because the boss's boss's boss's boss just
* filed a Priority Zero feature request and the boss's boss's boss is going for promotion.posted by Doroteo Arango II at 1:22 AM on October 28, 2015 [7 favorites]
Many problems which are presented as IT problems are actually personality problems.
posted by Pope Guilty at 3:56 AM on October 28, 2015 [11 favorites]
posted by Pope Guilty at 3:56 AM on October 28, 2015 [11 favorites]
The "Staggering Cost Of Failure" link is delightful. 8 of the 10 largest failures in the U.S. are military projects. In the rest of the world, military projects comprise something like 10% of the failures, while the biggest failures come from things like health record systems, aviation, and infrastructure. American military project failures alone account for more wasted capital than the rest of the world's largest fuckups COMBINED. Tens of billions of dollars, and that's just from the ones that the press caught wind of. But yeah, we should totally keep shoveling money into that bottomless pit, because terrorism.
posted by Mayor West at 4:41 AM on October 28, 2015 [8 favorites]
posted by Mayor West at 4:41 AM on October 28, 2015 [8 favorites]
a PM asked "can we just swap .NET, Windows and SQL Server for J2EE, Linux and Oracle?"
...my soul just tried to leave my body.
posted by Mr. Bad Example at 4:47 AM on October 28, 2015 [8 favorites]
...my soul just tried to leave my body.
posted by Mr. Bad Example at 4:47 AM on October 28, 2015 [8 favorites]
If the project has a document repository (and, being public works, they often do), it's fascinating to dip into the status meetings (I mention meeting minutes because the retention schedule for that at my workplace is 'forever', but your mileage may vary -- maybe you retain minutes for +2 years and it's something else that lasts forever, so use those instead). There's an air of panic that settles into them as time goes on, and phrases like 'last-ditch' and 'there must be something that can be salvaged' start popping up for months before word gets out that there's trouble in paradise.
posted by Mogur at 5:21 AM on October 28, 2015 [4 favorites]
posted by Mogur at 5:21 AM on October 28, 2015 [4 favorites]
Metafilter: the grease that was required for the palm.
posted by oheso at 5:22 AM on October 28, 2015
posted by oheso at 5:22 AM on October 28, 2015
Is there a name for the phenomenon where a company has a product which works and meets user needs just fine, and then they tinker with it over multiple iterations such that it gets further and further away from being as useful?
Gold-plating
posted by oheso at 5:23 AM on October 28, 2015 [2 favorites]
Gold-plating
posted by oheso at 5:23 AM on October 28, 2015 [2 favorites]
The "Staggering Cost Of Failure" link is delightful. 8 of the 10 largest failures in the U.S. are military projects.
And yet somehow their cost doesn't even seem that much compared to other military expenditures. For example, the Air Force's Expeditionary Combat Support System, that was cancelled after 10 years and $1.03 billion dollars? That's only what, the cost of one B2 bomber? You can watch that much money go up in smoke on a runway in Guam in a 1 minute Youtube clip.
posted by L.P. Hatecraft at 5:31 AM on October 28, 2015 [2 favorites]
And yet somehow their cost doesn't even seem that much compared to other military expenditures. For example, the Air Force's Expeditionary Combat Support System, that was cancelled after 10 years and $1.03 billion dollars? That's only what, the cost of one B2 bomber? You can watch that much money go up in smoke on a runway in Guam in a 1 minute Youtube clip.
posted by L.P. Hatecraft at 5:31 AM on October 28, 2015 [2 favorites]
Is there a name for the phenomenon where a company has a product which works and meets user needs just fine, and then they tinker with it over multiple iterations such that it gets further and further away from being as useful?
Rock and Roll?
posted by Chitownfats at 6:17 AM on October 28, 2015
Rock and Roll?
posted by Chitownfats at 6:17 AM on October 28, 2015
One of the biggest questions on my mind these days is whether the big tech companies (Apple, Facebook, Google, etc.) have found The Secret for making software projects successful (and are keeping it to themselves).... or they are just much better than average about hiding their failures.
Probably the latter. It takes a lot to make a software project successful, and even when the software works, the business need being addressed may turn out to have flaws.
Also, software that 'works' doesn't always work ("iTunes").
posted by ZeusHumms at 8:24 AM on October 28, 2015
Probably the latter. It takes a lot to make a software project successful, and even when the software works, the business need being addressed may turn out to have flaws.
Also, software that 'works' doesn't always work ("iTunes").
posted by ZeusHumms at 8:24 AM on October 28, 2015
Is there a name for the phenomenon where a company has a product which works and meets user needs just fine, and then they tinker with it over multiple iterations such that it gets further and further away from being as useful?
the web
posted by j_curiouser at 9:52 AM on October 28, 2015
the web
posted by j_curiouser at 9:52 AM on October 28, 2015
The thing we've been struggling to accept as an industry and throughout business is that software requirements are a special kind of fractal. Superficially, they all appear to have equal depth, but upon closer observation, some of them reveal themselves to have multiple layers of complexity, some of which, in turn, reveal even more layers. Our processes from bidding to estimation and scheduling to development are all built on the assumption that it is possible to fully understand requirements until real progress has been made and the friction this causes has resulted in the emergence of ideas such as agile to try and deal with it, but I think fundamentally humans cannot accept uncertainty and we'll need an innovation in management or software before we can permanently resolve the "so the schedule went up the org chart and rando VP said 'are you serious? we have to have a deadline!' so can you just give me a date that we won't hold you to" impasse.
posted by feloniousmonk at 10:10 AM on October 28, 2015 [5 favorites]
posted by feloniousmonk at 10:10 AM on October 28, 2015 [5 favorites]
One of the biggest questions on my mind these days is whether the big tech companies (Apple, Facebook, Google, etc.) have found The Secret for making software projects successful (and are keeping it to themselves).... or they are just much better than average about hiding their failures.
So I'm pretty sure that every time Google cancels a product that there's a post about it right here on Metafilter and everybody says this is it, Google is nearly dead (Netcraft confirms it!) and no one has ever forgotten about READER GRAR, etc. Apple has lots of failures, both technical and market-wise but as long as the iPhone prints money like a mint the ship keeps sailing. iAd is basically a failure, many of Apple's cloud services spent ages being total shit, but given enough time they eventually shaped up. Most companies don't have the bankroll to let bad products take the time they need to fix themselves - Google and Apple both have the luxury of basically infinite money which is no small advantage.
There's a lot of organizational dynamics that contributes to the ultimate success or failure of a big IT system. I think that Google and Facebook have basically solved this problem. Software teams need to be like software systems - modularized, but not too much and not too little. Monolithic systems and monolithic teams only go so far. Teams run by separate leaders that communicate rarely and have unaligned incentives don't succeed.
I had the good fortune to see Mickey Dickerson give a talk about his work on the health insurance exchange and wow, the government teams building that thing were a mess when he got there. I mean, teams working on different parts of the system literally never met. No one triaged issues. Teams didn't even have internal meetings. Like they lacked simple agile development 101 stuff. When I heard him describe what he did it really seemed so clearly obvious, but it is often that way when really competent people describe what they did, they make it sound like it was the simplest thing in the world.
posted by GuyZero at 10:30 AM on October 28, 2015 [8 favorites]
So I'm pretty sure that every time Google cancels a product that there's a post about it right here on Metafilter and everybody says this is it, Google is nearly dead (Netcraft confirms it!) and no one has ever forgotten about READER GRAR, etc. Apple has lots of failures, both technical and market-wise but as long as the iPhone prints money like a mint the ship keeps sailing. iAd is basically a failure, many of Apple's cloud services spent ages being total shit, but given enough time they eventually shaped up. Most companies don't have the bankroll to let bad products take the time they need to fix themselves - Google and Apple both have the luxury of basically infinite money which is no small advantage.
There's a lot of organizational dynamics that contributes to the ultimate success or failure of a big IT system. I think that Google and Facebook have basically solved this problem. Software teams need to be like software systems - modularized, but not too much and not too little. Monolithic systems and monolithic teams only go so far. Teams run by separate leaders that communicate rarely and have unaligned incentives don't succeed.
I had the good fortune to see Mickey Dickerson give a talk about his work on the health insurance exchange and wow, the government teams building that thing were a mess when he got there. I mean, teams working on different parts of the system literally never met. No one triaged issues. Teams didn't even have internal meetings. Like they lacked simple agile development 101 stuff. When I heard him describe what he did it really seemed so clearly obvious, but it is often that way when really competent people describe what they did, they make it sound like it was the simplest thing in the world.
posted by GuyZero at 10:30 AM on October 28, 2015 [8 favorites]
One of the biggest questions on my mind these days is whether the big tech companies (Apple, Facebook, Google, etc.) have found The Secret for making software projects successful (and are keeping it to themselves).... or they are just much better than average about hiding their failures.
Nah, the big guys tank just as many projects as the rest of us. This fine gentleman went so far as to create his own abecedarium of failed Google projects.
posted by Mayor West at 11:06 AM on October 28, 2015 [1 favorite]
Nah, the big guys tank just as many projects as the rest of us. This fine gentleman went so far as to create his own abecedarium of failed Google projects.
posted by Mayor West at 11:06 AM on October 28, 2015 [1 favorite]
Cancelled Google projects is a totally different category of things than cancelled government IT projects - Google's projects ship and then are eventually turned off due to commercial reasons (or something). Cancelled government IT projects generally never get to the working stage are are killed like cancers before they consume any more of the host's resources.
posted by GuyZero at 11:45 AM on October 28, 2015 [3 favorites]
posted by GuyZero at 11:45 AM on October 28, 2015 [3 favorites]
abecedarium of failed Google projects
Most of the failure in the IEEE series is discussing 'completely failed': never delivered a product,never delivered value to society. Comparing that with the products Google has killed for a lack of profitability says nothing about their ability to manage projects or deliver quality IT.
It's notable that the IT failure blame game includes vastly different scales of failure. Gmail's a 2 hour outage and Bing's a 4 hour outage are held up as examples of failure. Meanwhile, projects in all likely hood had designed for nightly downtime / outages are presented alongside: Pennsylvania has a complete modernization failure, for a system that was probably designed to be offline nightly anyways, the US census scraps its handhelds for paper, and a Florida school district terminated a late project paying Deloitte $30 million and having “virtually nothing” usable they could rescue. These aren't discontinued products, they are complete failures. The fact that Google has killed 26 products people were using suggests they kinda know what they're doing when it comes to completing IT projects*.
This guy lays out a clear explanation of why school districts can fire Deloitte and see their own staff complete projects on time and on budget. Consultantcies like IBM, Deloitte, Oracle and Wipro have the wrong incentives. I remember hearing that construction projects are given a bonus for finishing early, and a penalty for finishing late. I wonder why we don't adopt a similar standard in gov't project / procurement. Too scared the initial bids would be too high?
posted by pwnguin at 11:58 AM on October 28, 2015 [3 favorites]
Most of the failure in the IEEE series is discussing 'completely failed': never delivered a product,never delivered value to society. Comparing that with the products Google has killed for a lack of profitability says nothing about their ability to manage projects or deliver quality IT.
It's notable that the IT failure blame game includes vastly different scales of failure. Gmail's a 2 hour outage and Bing's a 4 hour outage are held up as examples of failure. Meanwhile, projects in all likely hood had designed for nightly downtime / outages are presented alongside: Pennsylvania has a complete modernization failure, for a system that was probably designed to be offline nightly anyways, the US census scraps its handhelds for paper, and a Florida school district terminated a late project paying Deloitte $30 million and having “virtually nothing” usable they could rescue. These aren't discontinued products, they are complete failures. The fact that Google has killed 26 products people were using suggests they kinda know what they're doing when it comes to completing IT projects*.
This guy lays out a clear explanation of why school districts can fire Deloitte and see their own staff complete projects on time and on budget. Consultantcies like IBM, Deloitte, Oracle and Wipro have the wrong incentives. I remember hearing that construction projects are given a bonus for finishing early, and a penalty for finishing late. I wonder why we don't adopt a similar standard in gov't project / procurement. Too scared the initial bids would be too high?
posted by pwnguin at 11:58 AM on October 28, 2015 [3 favorites]
The discontinued google services do not, for the most part, represent IT or project management failures, any more than the cessation of sales of the Apple // represents an engineering failure. Some of them may have been a business failure, but it's plausible that, for instance, Reader fit some strategic niche in 2005 that was either accomplished by 2013, or was no longer even relevant eight years later.
posted by aubilenon at 12:06 PM on October 28, 2015 [1 favorite]
posted by aubilenon at 12:06 PM on October 28, 2015 [1 favorite]
"One of the biggest questions on my mind these days is whether the big tech companies (Apple, Facebook, Google, etc.) have found The Secret for making software projects successful (and are keeping it to themselves).... or they are just much better than average about hiding their failures."
It's a world of differences developping software when you control and understand the requirements. The fact that you can just ignore big parts of the problem and just address issues in a subsequent version if they're not critical also helps a lot, you can redefine success on the way (try that on a gov project...).
Take google search for example, the requirement is pretty straightforward 'I want to search the internet with keywords and I want it return menaningful features first'. The technology you have to build to make this work is not simple (huge scaling, performance issues, etc....) but the idea for the system was.
IMHO The usual IT project is the opposite, technically simple but with very complex, strict and misunderstood requirements. Your consultants are (maybe if you're lucky) Oracle/Java/SQL/.NET/... experts but they're not experts at your problem space/organisation most of the time so don't expect much insight from them.
posted by coust at 12:12 PM on October 28, 2015 [1 favorite]
It's a world of differences developping software when you control and understand the requirements. The fact that you can just ignore big parts of the problem and just address issues in a subsequent version if they're not critical also helps a lot, you can redefine success on the way (try that on a gov project...).
Take google search for example, the requirement is pretty straightforward 'I want to search the internet with keywords and I want it return menaningful features first'. The technology you have to build to make this work is not simple (huge scaling, performance issues, etc....) but the idea for the system was.
IMHO The usual IT project is the opposite, technically simple but with very complex, strict and misunderstood requirements. Your consultants are (maybe if you're lucky) Oracle/Java/SQL/.NET/... experts but they're not experts at your problem space/organisation most of the time so don't expect much insight from them.
posted by coust at 12:12 PM on October 28, 2015 [1 favorite]
The technology you have to build to make this work is not simple (huge scaling, performance issues, etc....) but the idea for the system was.
If by simple you mean "created by a pair of PhD students."
posted by pwnguin at 1:03 PM on October 28, 2015
If by simple you mean "created by a pair of PhD students."
posted by pwnguin at 1:03 PM on October 28, 2015
If by simple you mean "created by a pair of PhD students."
I was speaking more the initial requirement (what the system needs to do). But the idea behind pagerank (especially the 1st version) is simple, you still had to have it though. And then you had to make it work and scale it.
posted by coust at 1:24 PM on October 28, 2015
I was speaking more the initial requirement (what the system needs to do). But the idea behind pagerank (especially the 1st version) is simple, you still had to have it though. And then you had to make it work and scale it.
posted by coust at 1:24 PM on October 28, 2015
I might have been willing to believe that if any of the wikis or document systems I've ever encountered had a good search engine built into them.
posted by pwnguin at 1:43 PM on October 28, 2015
posted by pwnguin at 1:43 PM on October 28, 2015
coust: “Your consultants are (maybe if you're lucky) Oracle/Java/SQL/.NET/... experts but they're not experts at your problem space/organisation most of the time so don't expect much insight from them.”Consultants who can't elucidate requirements by meeting with the client and studying their existing business practices aren't worth their consulting fees. Yet the Big 4 suck up all the work and my partner and I are left scrapping for crumbs despite a proven track record of doing just that.
posted by ob1quixote at 1:52 PM on October 28, 2015 [1 favorite]
This guy lays out a clear explanation of why school districts can fire Deloitte and see their own staff complete projects on time and on budget. Consultantcies like IBM, Deloitte, Oracle and Wipro have the wrong incentives. I remember hearing that construction projects are given a bonus for finishing early, and a penalty for finishing late. I wonder why we don't adopt a similar standard in gov't project / procurement. Too scared the initial bids would be too high?
It's not like anyone being paid hourly wants to agree to penalty clauses, but academics studying this stuff usually advocate for smaller penalties if early, and bigger penalties if late. The thinking here is you don't want underbidding, but you also don't want excessive padding, either.
I had the good fortune to see Mickey Dickerson give a talk about his work on the health insurance exchange and wow, the government teams building that thing were a mess when he got there. I mean, teams working on different parts of the system literally never met. No one triaged issues. Teams didn't even have internal meetings. Like they lacked simple agile development 101 stuff. When I heard him describe what he did it really seemed so clearly obvious, but it is often that way when really competent people describe what they did, they make it sound like it was the simplest thing in the world.
For this and any other high profile software projects, be careful with public narratives. It's nice that the IEEE is trying to figure out what happened in these various projects, but I've found the stories are often quite different depending on who's doing the telling - even if all the storytellers were in the same room.
One of the biggest questions on my mind these days is whether the big tech companies (Apple, Facebook, Google, etc.) have found The Secret for making software projects successful (and are keeping it to themselves).... or they are just much better than average about hiding their failures.
Again - the public narrative is an unreliable narrator.
posted by NoRelationToLea at 1:58 PM on October 28, 2015 [2 favorites]
It's not like anyone being paid hourly wants to agree to penalty clauses, but academics studying this stuff usually advocate for smaller penalties if early, and bigger penalties if late. The thinking here is you don't want underbidding, but you also don't want excessive padding, either.
I had the good fortune to see Mickey Dickerson give a talk about his work on the health insurance exchange and wow, the government teams building that thing were a mess when he got there. I mean, teams working on different parts of the system literally never met. No one triaged issues. Teams didn't even have internal meetings. Like they lacked simple agile development 101 stuff. When I heard him describe what he did it really seemed so clearly obvious, but it is often that way when really competent people describe what they did, they make it sound like it was the simplest thing in the world.
For this and any other high profile software projects, be careful with public narratives. It's nice that the IEEE is trying to figure out what happened in these various projects, but I've found the stories are often quite different depending on who's doing the telling - even if all the storytellers were in the same room.
One of the biggest questions on my mind these days is whether the big tech companies (Apple, Facebook, Google, etc.) have found The Secret for making software projects successful (and are keeping it to themselves).... or they are just much better than average about hiding their failures.
Again - the public narrative is an unreliable narrator.
posted by NoRelationToLea at 1:58 PM on October 28, 2015 [2 favorites]
The Googles, Apples, and Microsofts of the world definitely don't have any secrets to success with software. This appearance of invincibility is a function of marketing, which isn't to say they don't actually write quality software, because of course they do, but rather that if you mention it to a group of developers who specialize in any one of those ecosystems, they'll laugh you out of the room at the suggestion that there's flawless and always-on-schedule software anywhere in the stack except maybe in the innermost hearts of the system. The idea that these companies write incredible software is both true to an extent but also a function of marketing, speaking more to prospective employees than the consumer market at large. These companies all have substantial developer evangelism organizations, although they go by different names in different places, whose mission is to make them look as good as possible to the average developer (or more realistically, development manager or VP of engineering, etc.). They don't deliver software radically differently from anyone else, but they do have access to a superior multi-disciplinary talent pool, effectively limitless resources, and the ability to put the best face on things by managing scope.
posted by feloniousmonk at 2:13 PM on October 28, 2015 [2 favorites]
posted by feloniousmonk at 2:13 PM on October 28, 2015 [2 favorites]
they do have access to a superior multi-disciplinary talent pool, effectively limitless resources, and the ability to put the best face on things by managing scope.
Also they don't have to micro-manage money. I never understand how a government IT project can waste a billion dollars with so many poorly paid developers.
posted by GuyZero at 2:25 PM on October 28, 2015
Also they don't have to micro-manage money. I never understand how a government IT project can waste a billion dollars with so many poorly paid developers.
posted by GuyZero at 2:25 PM on October 28, 2015
As an aside I would bet money that Tableau is being used to generate those visualization. Tableau is what happens to pivot charts when they grow up. Anytime you see data folded against itself in interesting ways, especially with color & size gradients, slider bars or other controls attached, it's Tableau doing the heavy lifting. (I have no relationship with Tableau other than as a happy user.)
posted by scalefree at 2:25 PM on October 28, 2015
posted by scalefree at 2:25 PM on October 28, 2015
Consultants who can't elucidate requirements by meeting with the client and studying their existing business practices aren't worth their consulting fees. Yet the Big 4 suck up all the work and my partner and I are left scrapping for crumbs despite a proven track record of doing just that.
Agreed, but when you go on those mega projects like "health record systems" it's colossal work. The requirements will be complex whoever good you are at elucidating.
I have a lot of college friends who worked for those big consulting firms and nobody had anything good to say about them (especially Oracle).
You'd naively think it should be simple to archive that kind of info and make it accessible, maybe the problem is that we try to apply too much structure that info upfront. If we where to just create a DB where each document/notes/results is added associated with a patient/practitioner and then there's a background layer that keeps running and structuring the data for later query/presentation (a la Google) it would let the project being used sooner and would results in quicker incremental adjustments. But it would be imperfect and even though that's how most of our systems end up being, I don't think that we'd accept that. Also, not being a health care practitioner I'm sure I'm missing key requirements, there's surely a lot more to that than what I just described (like no we don't want your freetyped name of the condition/medication we want you to select it from a list so that there's no ambiguity about what it is).
posted by coust at 2:52 PM on October 28, 2015
Agreed, but when you go on those mega projects like "health record systems" it's colossal work. The requirements will be complex whoever good you are at elucidating.
I have a lot of college friends who worked for those big consulting firms and nobody had anything good to say about them (especially Oracle).
You'd naively think it should be simple to archive that kind of info and make it accessible, maybe the problem is that we try to apply too much structure that info upfront. If we where to just create a DB where each document/notes/results is added associated with a patient/practitioner and then there's a background layer that keeps running and structuring the data for later query/presentation (a la Google) it would let the project being used sooner and would results in quicker incremental adjustments. But it would be imperfect and even though that's how most of our systems end up being, I don't think that we'd accept that. Also, not being a health care practitioner I'm sure I'm missing key requirements, there's surely a lot more to that than what I just described (like no we don't want your freetyped name of the condition/medication we want you to select it from a list so that there's no ambiguity about what it is).
posted by coust at 2:52 PM on October 28, 2015
I spend doing an estimate and then they say ok great we need it in half that time because we already have the release planned.
At this point I'm deeply skeptical that most players involved in asking for estimation even think of them as tools for understanding the nature of the project or the timeline on which it's likely to progress.
I think instead it's tacit knowledge that they're personal psychological and political levers with which to goad workers. "You said it would take this long!"
Just one of several reasons why estimates that don't meet the goals of those asking for them are pretty often discard and substituted with another form of pressure.
posted by namespan at 3:44 PM on October 28, 2015
At this point I'm deeply skeptical that most players involved in asking for estimation even think of them as tools for understanding the nature of the project or the timeline on which it's likely to progress.
I think instead it's tacit knowledge that they're personal psychological and political levers with which to goad workers. "You said it would take this long!"
Just one of several reasons why estimates that don't meet the goals of those asking for them are pretty often discard and substituted with another form of pressure.
posted by namespan at 3:44 PM on October 28, 2015
One of the biggest questions on my mind these days is whether the big tech companies (Apple, Facebook, Google, etc.) have found The Secret for making software projects successful (and are keeping it to themselves).... or they are just much better than average about hiding their failures.
So here's my theory on this:
There's a simple, two-step plan to maximize the chances of your software project working:
1. Hire good, smart, experienced people (developers, project managers, etc.) with a proven track record.
2. Let them do it however they want.
Google et. al. have a reputation for being a pleasant place for software developers to work. They also have money and are willing to spend it to get good people. So they can do 1) and their upper management knows to do 2).
In contrast, government contractors tend to be the opposite of Google in terms of pleasant working environment. While they do occasionally get good people (they have money, after all), they aren't good at identifying or keeping them. As a result, the quality of developers varies a lot. In addition, management doesn't trust them and will meddle a lot. So they end up botching 1) and 2) and disaster follows.
(By the way, methodologies like Agile are basically attempts to codify the practices of high-performing managers into something anyone can learn. Unfortunately, they're still really easy to screw up so the requirements still hold.)
posted by suetanvil at 6:40 PM on October 28, 2015 [6 favorites]
So here's my theory on this:
There's a simple, two-step plan to maximize the chances of your software project working:
1. Hire good, smart, experienced people (developers, project managers, etc.) with a proven track record.
2. Let them do it however they want.
Google et. al. have a reputation for being a pleasant place for software developers to work. They also have money and are willing to spend it to get good people. So they can do 1) and their upper management knows to do 2).
In contrast, government contractors tend to be the opposite of Google in terms of pleasant working environment. While they do occasionally get good people (they have money, after all), they aren't good at identifying or keeping them. As a result, the quality of developers varies a lot. In addition, management doesn't trust them and will meddle a lot. So they end up botching 1) and 2) and disaster follows.
(By the way, methodologies like Agile are basically attempts to codify the practices of high-performing managers into something anyone can learn. Unfortunately, they're still really easy to screw up so the requirements still hold.)
posted by suetanvil at 6:40 PM on October 28, 2015 [6 favorites]
The related 2005 article on Why Software Fails was painful to read. A recent project I was a lead on should have been canceled two months in but wasn't; their bullet list of common factors includes, by my self-interested count, seven problems with that project (not getting into my own failures, which I'm not qualified to assess except to acknowledge that it's likely I had them). Nonetheless we plowed through and finished, late and over budget, only to have somebody who wasn't involved decide to rebid the whole thing about a year after launch.
And even that wouldn't have been so bad (cf. "should have been canceled two months in") except when they rebid it they didn't involve me so I could say, "hey, here's a detailed list of all the points of failure with the last version that really need to be addressed for the new project to finish on time, within budget, and feature complete. So they bid it and signed a contract and told me about it only when they said they wanted me to walk the new developers through the existing site. Those were fun meetings. "Hey, here's all the stuff you didn't know when you agreed to take this project on."
I've left that company, but the new replacement just went live, taking twice as long (and presumably costing more than twice as much) to develop as planned.
posted by fedward at 8:46 PM on October 28, 2015 [2 favorites]
And even that wouldn't have been so bad (cf. "should have been canceled two months in") except when they rebid it they didn't involve me so I could say, "hey, here's a detailed list of all the points of failure with the last version that really need to be addressed for the new project to finish on time, within budget, and feature complete. So they bid it and signed a contract and told me about it only when they said they wanted me to walk the new developers through the existing site. Those were fun meetings. "Hey, here's all the stuff you didn't know when you agreed to take this project on."
I've left that company, but the new replacement just went live, taking twice as long (and presumably costing more than twice as much) to develop as planned.
posted by fedward at 8:46 PM on October 28, 2015 [2 favorites]
« Older Everyone sing along now. EVERYONE. | It sounds like you're living your best life! Newer »
This thread has been archived and is closed to new comments
posted by carter at 5:26 PM on October 27, 2015