"...And all for the want of a horseshoe nail."
February 9, 2009 9:21 AM Subscribe
Picture a three-guy trading floor. They would call a carrier,... [and] manually move trunks in and out of route by issuing SQL commands against the Veraz's Oracle database.... Let me write that out for you: One ass-hat residential customer with a 20yo telephone with four extra buttons did thirty million dollars in damages in less than one night. Anyways, that's how the company went bankrupt... and about 6000 or so people ... all got laid off.
From the little I understood, that doesn't sound like "some residential ass-hat". That sounds like corporate sabotage.
posted by DU at 9:31 AM on February 9, 2009 [4 favorites]
posted by DU at 9:31 AM on February 9, 2009 [4 favorites]
Yeah this was:
1. ?
2. ??
3. ???
4. FAIL
Poor design, management and operations.
posted by jkaczor at 9:34 AM on February 9, 2009 [1 favorite]
1. ?
2. ??
3. ???
4. FAIL
Poor design, management and operations.
posted by jkaczor at 9:34 AM on February 9, 2009 [1 favorite]
Yeah, no way is this just some random dude, their biggest night and all the attacker do is call-hangup-call-hangup? Who would do that? And who would be aware that it would destroy the company?
Probably a pissed off lineman.
posted by delmoi at 9:37 AM on February 9, 2009 [1 favorite]
Probably a pissed off lineman.
posted by delmoi at 9:37 AM on February 9, 2009 [1 favorite]
Anybody else see some parallels with this post from this weekend?
posted by wayofthedodo at 9:37 AM on February 9, 2009 [4 favorites]
posted by wayofthedodo at 9:37 AM on February 9, 2009 [4 favorites]
This story makes me feel empowered to wreck huge businesses. Anyone know where I can get a phone with a lot of buttons? Your time has come, Verizon, do you hear me, YOUR TIME HAS COME!
Evil laughter. Pets kitty.
posted by fuq at 9:38 AM on February 9, 2009 [5 favorites]
Evil laughter. Pets kitty.
posted by fuq at 9:38 AM on February 9, 2009 [5 favorites]
Survival of the fittest in the corporate world means sometimes companies that are crappily organized and have no contingency planning go out of business.
I picture the upper management hotshots who have been warned by lower level management about the risks, but decide to do nothing about it because it is THEY who are the asshats. Honestly...a $50 million make-or-break night with only three lowbies and no supervisor on site?
Result: 6000 people lose their jobs because of crappy management.
posted by darkstar at 9:39 AM on February 9, 2009 [1 favorite]
I picture the upper management hotshots who have been warned by lower level management about the risks, but decide to do nothing about it because it is THEY who are the asshats. Honestly...a $50 million make-or-break night with only three lowbies and no supervisor on site?
Result: 6000 people lose their jobs because of crappy management.
posted by darkstar at 9:39 AM on February 9, 2009 [1 favorite]
There are so many things wrong with this scenario it makes me scared that I exist on this planet.
1) Who would (besides the case of corporate sabotage, which still does not make me proud to be human) use a phone like that?
2) How bad is our telephone infrastructure that one person with a phone can cause a company to go bankrupt with a one person DOS.
3) How inept are the people working these lines?
Beyond that...I'm speechless. This was a failure on too many levels to be safe for a major communications field.
posted by Chan at 9:40 AM on February 9, 2009
1) Who would (besides the case of corporate sabotage, which still does not make me proud to be human) use a phone like that?
2) How bad is our telephone infrastructure that one person with a phone can cause a company to go bankrupt with a one person DOS.
3) How inept are the people working these lines?
Beyond that...I'm speechless. This was a failure on too many levels to be safe for a major communications field.
posted by Chan at 9:40 AM on February 9, 2009
I vote with ssF. Whoever was behind the DOS was being malicious, sure, but so much else went wrong and there were so many other single points of failure that it doesn't sound like a company Buffett or McDuck would want to buy into because of the superior management.
posted by jfuller at 9:40 AM on February 9, 2009
posted by jfuller at 9:40 AM on February 9, 2009
It would take me about 15 seconds of running a multi-million $$$ business via SQL commands before I at least slapped a VB GUI on that. I mean seriously.
posted by DU at 9:41 AM on February 9, 2009 [8 favorites]
posted by DU at 9:41 AM on February 9, 2009 [8 favorites]
If nothing else, the fact that this required manually entering a series of SQL statements - instead of, say, scripting it all and having a "Switch to Provider X" button - makes me astonished at their incompetence.
I mean, not that there aren't a dozen other things that are hilariously inept about these guys, but as someone who spends a nontrivial amount of time automating SQL updates, I find myself amazed at how deep the fail ran, from the high-level folks unwilling to spend the money to ensure their $50 million night went smoothly, to the low-level guys who took hours to think of shutting off the damn D button.
posted by Tomorrowful at 9:46 AM on February 9, 2009 [2 favorites]
I mean, not that there aren't a dozen other things that are hilariously inept about these guys, but as someone who spends a nontrivial amount of time automating SQL updates, I find myself amazed at how deep the fail ran, from the high-level folks unwilling to spend the money to ensure their $50 million night went smoothly, to the low-level guys who took hours to think of shutting off the damn D button.
posted by Tomorrowful at 9:46 AM on February 9, 2009 [2 favorites]
Your time has come, Verizon, do you hear me, YOUR TIME HAS COME!
Maybe that would teach 'em the difference between .002 dollars and .002 cents.
posted by FelliniBlank at 9:47 AM on February 9, 2009 [4 favorites]
Maybe that would teach 'em the difference between .002 dollars and .002 cents.
posted by FelliniBlank at 9:47 AM on February 9, 2009 [4 favorites]
You can't use a keyboard when it's GUI.
1) Good.
2) But why not?
posted by DU at 9:47 AM on February 9, 2009
1) Good.
2) But why not?
posted by DU at 9:47 AM on February 9, 2009
DU Because the keys are all sticky...
posted by cthuljew at 9:49 AM on February 9, 2009 [2 favorites]
posted by cthuljew at 9:49 AM on February 9, 2009 [2 favorites]
So that's a reddit comment, I guess?
posted by boo_radley at 9:49 AM on February 9, 2009
posted by boo_radley at 9:49 AM on February 9, 2009
Yeah, I'm incredibly unsympathetic. I mean, I have sympathy for the people who lost their jobs, but this sounds like one of the worst organized companies I've ever heard. I'm pretty unclear on exactly what the post is talking about, but even without understanding any of the acronyms he's throwing around, it's pretty fucking obvious to me that they should have invested in some decent infrastructure long before this ever happened. Sounds like the entire upper management was made of stupid, and they deserved to lose their pants.
posted by Caduceus at 9:50 AM on February 9, 2009
posted by Caduceus at 9:50 AM on February 9, 2009
Don't you watch CSI, DU?
posted by orthogonality at 9:50 AM on February 9, 2009
posted by orthogonality at 9:50 AM on February 9, 2009
This ass-hat calls the NOC over and over and over all night long. I mean, literally D-hangup-D-hangup-D-hangup. It was so bad that the NOC couldn't even make an outgoing call half the time and answering incoming calls from carriers was useless (you'd just get hung up on).
...and when I looked in the sand, there was only one set of footprints.
posted by quin at 9:50 AM on February 9, 2009 [12 favorites]
...and when I looked in the sand, there was only one set of footprints.
posted by quin at 9:50 AM on February 9, 2009 [12 favorites]
The CEO probably spent the entire Fiscal Year budget on putting a fireplace in his office and redesigning it. I mean, come on, people...PRIORITIES!!! 10grand for a VB GUI??? opfff
posted by spicynuts at 9:52 AM on February 9, 2009
posted by spicynuts at 9:52 AM on February 9, 2009
spicynuts I heard his nephew could do it for $300.00<sm
posted by boo_radley at 9:56 AM on February 9, 2009
posted by boo_radley at 9:56 AM on February 9, 2009
Of the many lessons you could take from this story the most important is this: If you have a company where a single night can make or break you, have the supervisor scheduled to be there no matter what happens.
posted by Bonzai at 9:57 AM on February 9, 2009 [1 favorite]
posted by Bonzai at 9:57 AM on February 9, 2009 [1 favorite]
I don't buy it. If they're not a local exchange carrier, then the D tone isn't going to do anything at the dial tone. If they're a calling-card provider, then why do they have "field techs"? Seriously, using in-band tones for signaling, even for something as simple as this, has been dead since the 80's (maybe with the exception of ACTS payphones, but that's just because there are a lot of them). And you're telling me a phone company NOC had only one phone line? Whatever.
posted by kiltedtaco at 10:02 AM on February 9, 2009 [4 favorites]
posted by kiltedtaco at 10:02 AM on February 9, 2009 [4 favorites]
Wow that's like reading about a train wreck. On a railroad built by morons and operated by crackheads, and held together by good luck and wishful thinking. So basically, not unexpected.
From the description of their operations, it pretty much deserved to fail, it was being run so poorly; if it hadn't been somebody performing a little sabotage with the D button, it would have been something else. The proximate cause might have been a DOS attack, but it's pretty clear -- if the description in the link is honest -- that the ultimate cause was underinvestment in infrastructure and procedural safeguards.
Also, the description of the problem's source as a 20-year old AT&T phone doesn't make sense, if the attack was being automated it was probably a modem being controlled by a computer. Wouldn't be hard to set that up, although given how specific and successful the attack was, it definitely sounds like either an inside job, or someone who was given inside information.
Still, an interesting anecdote. Makes you wonder how much stuff we take for granted every day is being held together so tenuously on the back end; none of the customers using this company's calling cards to phone home would have had any idea that their calls were being routed by a couple of guys typing SQL commands directly into an Oracle database and keeping their fingers crossed nothing went wrong.
posted by Kadin2048 at 10:03 AM on February 9, 2009 [1 favorite]
From the description of their operations, it pretty much deserved to fail, it was being run so poorly; if it hadn't been somebody performing a little sabotage with the D button, it would have been something else. The proximate cause might have been a DOS attack, but it's pretty clear -- if the description in the link is honest -- that the ultimate cause was underinvestment in infrastructure and procedural safeguards.
Also, the description of the problem's source as a 20-year old AT&T phone doesn't make sense, if the attack was being automated it was probably a modem being controlled by a computer. Wouldn't be hard to set that up, although given how specific and successful the attack was, it definitely sounds like either an inside job, or someone who was given inside information.
Still, an interesting anecdote. Makes you wonder how much stuff we take for granted every day is being held together so tenuously on the back end; none of the customers using this company's calling cards to phone home would have had any idea that their calls were being routed by a couple of guys typing SQL commands directly into an Oracle database and keeping their fingers crossed nothing went wrong.
posted by Kadin2048 at 10:03 AM on February 9, 2009 [1 favorite]
A+++ GOOD TRANSACTION!
posted by greensweater at 10:03 AM on February 9, 2009
posted by greensweater at 10:03 AM on February 9, 2009
And I'll also echo kiltedtaco's sentiments about the whole story being a bit fishy.
But hey, some of the best engineering "ghost stories" are a bit apocryphal and only get better with each retelling; they're more a way to scare young engineers out of doing stupid things than actual history.
posted by Kadin2048 at 10:05 AM on February 9, 2009
But hey, some of the best engineering "ghost stories" are a bit apocryphal and only get better with each retelling; they're more a way to scare young engineers out of doing stupid things than actual history.
posted by Kadin2048 at 10:05 AM on February 9, 2009
I have a few buttons on this old phone which must have some function that I don't know about. I'm gonna go press 'em a few times now. Let me know if any EU member suddenly goes tits-up.
posted by Spatch at 10:07 AM on February 9, 2009 [5 favorites]
posted by Spatch at 10:07 AM on February 9, 2009 [5 favorites]
Sounds like there's a lot of money to be made in phone cards...who knew?
posted by The Light Fantastic at 10:12 AM on February 9, 2009
posted by The Light Fantastic at 10:12 AM on February 9, 2009
Interesting factoid: My mom uses a calling card to call me, and half the time when I pick up I hear snippets of someone else's conversation before it connects me to my mom. Very weird. about 1 or 2 seconds worth.
posted by delmoi at 10:14 AM on February 9, 2009
posted by delmoi at 10:14 AM on February 9, 2009
Three guy trading floor? DOS by a guy with one phone? 3-1=2. Thats two lines still open for calling other carriers.. Doesn't add up.
posted by tomas316 at 10:15 AM on February 9, 2009
posted by tomas316 at 10:15 AM on February 9, 2009
Interesting factoid: My mom uses a calling card to call me, and half the time when I pick up I hear snippets of someone else's conversation before it connects me to my mom. Very weird. about 1 or 2 seconds worth.
You heard nothing, citizen.
posted by Pope Guilty at 10:24 AM on February 9, 2009 [4 favorites]
You heard nothing, citizen.
posted by Pope Guilty at 10:24 AM on February 9, 2009 [4 favorites]
6000 people work for a telecoms startup selling calling cards that is operated at it's most critical business period by two people and an Oracle console?
Say would you like to look at this bridge I have for sale.
posted by PenDevil at 10:27 AM on February 9, 2009 [2 favorites]
Say would you like to look at this bridge I have for sale.
posted by PenDevil at 10:27 AM on February 9, 2009 [2 favorites]
I mean, literally D-hangup-D-hangup-D-hangup
Deliberate DOS? More likely It was a fax or modem on autoredial that accidentally hit the "Dial NOC" command. I did similar stuff by accident when running a BBS many moons ago. Most modem programs would hang up immediately upon non-negotiation, and if you had redial set without a maximum # of retries, would continue calling relentlessly.
posted by benzenedream at 10:29 AM on February 9, 2009
Deliberate DOS? More likely It was a fax or modem on autoredial that accidentally hit the "Dial NOC" command. I did similar stuff by accident when running a BBS many moons ago. Most modem programs would hang up immediately upon non-negotiation, and if you had redial set without a maximum # of retries, would continue calling relentlessly.
posted by benzenedream at 10:29 AM on February 9, 2009
I like the (for me) nearly unreadable patois of programming slang and corporate lingo. Almost reminds me of something else.
posted by ford and the prefects at 10:30 AM on February 9, 2009
posted by ford and the prefects at 10:30 AM on February 9, 2009
kadin2048: oh good point about a modem. There were certainly a lot more modems around in 1999 than there were 16-button DTMF keypads. In fact maybe this was some honest mistake caused by a dialing script went wrong. If you accidentally added a D in front of your ISP's phone number, it might keep trying every 30 seconds until it got through. Or until your phone service provider went bankrupt.
Every single modem control program ever had a "dial this number until someone on the other end says BLOO-DOO-DOO-DOO-EE-EE-EE-EE-BSZLRLSZRHTSHTZHBS" function.
posted by aubilenon at 10:31 AM on February 9, 2009 [8 favorites]
Every single modem control program ever had a "dial this number until someone on the other end says BLOO-DOO-DOO-DOO-EE-EE-EE-EE-BSZLRLSZRHTSHTZHBS" function.
posted by aubilenon at 10:31 AM on February 9, 2009 [8 favorites]
Sure, you can try to crack NOC black ice. Your first warning will be the renal failure. That scares off the weaker crews - you know, the NSA, metatext police, Pythagorean jihadis. But if you're a real cowboy, we hit you with this neo-Cyrrilic torturegram, thing looks just like a Sudoku until -
Wait... is that a rotary phone? Holy shit, is that a D key? No- Noooooooo!
posted by kid ichorous at 10:34 AM on February 9, 2009 [37 favorites]
Wait... is that a rotary phone? Holy shit, is that a D key? No- Noooooooo!
posted by kid ichorous at 10:34 AM on February 9, 2009 [37 favorites]
Now I want a phone with a d button. And I don't know why.
posted by Astro Zombie at 10:34 AM on February 9, 2009 [2 favorites]
posted by Astro Zombie at 10:34 AM on February 9, 2009 [2 favorites]
6000 people work for a telecoms startup selling calling cards that is operated at it's most critical business period by two people and an Oracle console?
1) This is Florida.
2) This is during the dot-com era.
3) Stupidity in hiring practices of sales vs. support in the corporate world is the standard, not the exception.
It was like the perfect storm. I love the description of the updater process...throwing all the points of failure in there. I've worked at any number of businesses where you start to question the sanity of the programmer/software engineer that came before you after looking at the support network he's left behind.
posted by thanotopsis at 10:35 AM on February 9, 2009
1) This is Florida.
2) This is during the dot-com era.
3) Stupidity in hiring practices of sales vs. support in the corporate world is the standard, not the exception.
It was like the perfect storm. I love the description of the updater process...throwing all the points of failure in there. I've worked at any number of businesses where you start to question the sanity of the programmer/software engineer that came before you after looking at the support network he's left behind.
posted by thanotopsis at 10:35 AM on February 9, 2009
So it took awhile for this to sink in. Basically, this company's ENTIRE BUSINESS MODEL boiled down to a few guys negotiating the cheapest calling routes (so that they could make a profit on the rates they'd promised to consumers who bought their cards) and then rerouting calls along the agreed-upon paths by manually updating records in a massive database. Understandable perhaps if it's a startup with only a handful of employees, but at 6k employees . . . ? Holy crap, that's not just a massive fail on the management level, that's the corporate version of a ticking timed bomb.
posted by treepour at 10:44 AM on February 9, 2009 [2 favorites]
posted by treepour at 10:44 AM on February 9, 2009 [2 favorites]
I'm sure virtually all those 6000 were customer support call center employees. If this happened, I'd say either corporate sabotage or disgruntled employee. I imagine you can dial this D key from an ordinary modem.
posted by jeffburdges at 10:45 AM on February 9, 2009
posted by jeffburdges at 10:45 AM on February 9, 2009
There's something else here. I think the phrase for it is "systemic cultural failure".
All these idiots voluntarily went to work for this dodgy outfit and voluntarily (as in "free will") operated this wet-paper-bag-technology until it's faults blew itself up. No surprises there, and it's a little reminiscent of the current "Bankers are idiots" thing we are all sharing.
It's a culture that says "Make a buck by any means". It says "you are a worker? You are just a cog in a great big wheel. Shut up and do what you are told". It says "Business is a confidence game; you just gotta smile and keep selling shit to the rubes". Sound familiar?
posted by sporb at 10:49 AM on February 9, 2009 [7 favorites]
All these idiots voluntarily went to work for this dodgy outfit and voluntarily (as in "free will") operated this wet-paper-bag-technology until it's faults blew itself up. No surprises there, and it's a little reminiscent of the current "Bankers are idiots" thing we are all sharing.
It's a culture that says "Make a buck by any means". It says "you are a worker? You are just a cog in a great big wheel. Shut up and do what you are told". It says "Business is a confidence game; you just gotta smile and keep selling shit to the rubes". Sound familiar?
posted by sporb at 10:49 AM on February 9, 2009 [7 favorites]
Oh, phreak boxes, nostalgia for suburban ne'er-do-wells.
posted by filthy light thief at 10:52 AM on February 9, 2009
posted by filthy light thief at 10:52 AM on February 9, 2009
I'm not a telephony expert by any means, but this doesn't sound right at all.
posted by Optimus Chyme at 11:00 AM on February 9, 2009
posted by Optimus Chyme at 11:00 AM on February 9, 2009
So the lesson to be learned here, young people, is always mount a scratch monkey.
posted by mr_crash_davis mark II: Jazz Odyssey at 11:00 AM on February 9, 2009 [7 favorites]
posted by mr_crash_davis mark II: Jazz Odyssey at 11:00 AM on February 9, 2009 [7 favorites]
I've dealt with GUI-based interfaces into data apps which were so badly executed that the official-unofficial way to use them in the office was to invoke something like Admin -> Console -> Send SQL Command, at which time you were supposed to type in something from a multi-generation photocopied sheet with scribbled-over placeholders for whatever string you were trying to match on. So that much doesn't sound implausible.
On the other hand, a business folding with 6,000 jobs lost overnight is pretty dramatic, even in a city the size of Fort Lauderdale -- When Ford or GM cut off that many people it makes national headlines. There's got to be something in the local press about a sudden business collapse. So does anybody have access to newspaper records from Fort Lauderdale circa 2001?
posted by ardgedee at 11:00 AM on February 9, 2009 [2 favorites]
On the other hand, a business folding with 6,000 jobs lost overnight is pretty dramatic, even in a city the size of Fort Lauderdale -- When Ford or GM cut off that many people it makes national headlines. There's got to be something in the local press about a sudden business collapse. So does anybody have access to newspaper records from Fort Lauderdale circa 2001?
posted by ardgedee at 11:00 AM on February 9, 2009 [2 favorites]
1) This is Florida.
Pretty well sums it up, right there.
[Floor-DUH-ist]
posted by Devils Rancher at 11:08 AM on February 9, 2009
Pretty well sums it up, right there.
[Floor-DUH-ist]
posted by Devils Rancher at 11:08 AM on February 9, 2009
What were the other 5996 "employees" doing?
From the sounds of the company, they were in a heated competition to win either a Cadillac Eldorado or a set of steak knives.
posted by Shepherd at 11:11 AM on February 9, 2009 [8 favorites]
From the sounds of the company, they were in a heated competition to win either a Cadillac Eldorado or a set of steak knives.
posted by Shepherd at 11:11 AM on February 9, 2009 [8 favorites]
Metafilter: BLOO-DOO-DOO-DOO-EE-EE-EE-EE-BSZLRLSZRHTSHTZHBS
posted by Slap*Happy at 11:21 AM on February 9, 2009 [5 favorites]
posted by Slap*Happy at 11:21 AM on February 9, 2009 [5 favorites]
I don't know anything about the guy. I know they identified him immediately from the CDRs, but they didn't tell us anything specific about that part of it. Not even who it was other than that it was a residential customer somewhere.
His initials were R.F....
posted by Halloween Jack at 11:23 AM on February 9, 2009 [1 favorite]
His initials were R.F....
posted by Halloween Jack at 11:23 AM on February 9, 2009 [1 favorite]
Oh, phreak boxes, nostalgia for suburban ne'er-do-wells.
Wow, he even lists the Paisley Box, the plans for which consisted of taking a huge amount of drugs, then murdering some people in order to please Satan so he would make the box work. And the Blotto Box, which was a similar joke that would probably electrocute you if you tried to actually make it.
posted by DecemberBoy at 11:26 AM on February 9, 2009
Wow, he even lists the Paisley Box, the plans for which consisted of taking a huge amount of drugs, then murdering some people in order to please Satan so he would make the box work. And the Blotto Box, which was a similar joke that would probably electrocute you if you tried to actually make it.
posted by DecemberBoy at 11:26 AM on February 9, 2009
I do not believe this story is true.
That is all.
posted by Ynoxas at 11:32 AM on February 9, 2009 [1 favorite]
That is all.
posted by Ynoxas at 11:32 AM on February 9, 2009 [1 favorite]
Doesn't add up
Many details in this story do not add up.
His initials were R.F....
Or John G.
posted by mrgrimm at 11:34 AM on February 9, 2009
Many details in this story do not add up.
His initials were R.F....
Or John G.
posted by mrgrimm at 11:34 AM on February 9, 2009
What were the other 5996 "employees" doing?
Pressing D on their phones?
posted by ShadowCrash at 11:38 AM on February 9, 2009 [4 favorites]
Pressing D on their phones?
posted by ShadowCrash at 11:38 AM on February 9, 2009 [4 favorites]
Sporb: Calling people idiots makes people not listen to you, and you're never going to change the world if nobody likes you. I can understand your anger, but come on. Systematic cultural failure? Maybe, but you're not going to do anything about it by behaving like a jackass.
posted by seagull.apollo at 11:39 AM on February 9, 2009
posted by seagull.apollo at 11:39 AM on February 9, 2009
"It says sprocket not socket!"
posted by orme at 11:40 AM on February 9, 2009 [1 favorite]
posted by orme at 11:40 AM on February 9, 2009 [1 favorite]
Now I want a phone with a d button. And I don't know why.
Fuck that..I want a D button I can carry around and stick on people/things/corporations that piss me off. Stick it on, press it, ASSPLODE!!!!!!!!!! WOOOOOOOOOOOOO!!!
posted by spicynuts at 11:42 AM on February 9, 2009
Fuck that..I want a D button I can carry around and stick on people/things/corporations that piss me off. Stick it on, press it, ASSPLODE!!!!!!!!!! WOOOOOOOOOOOOO!!!
posted by spicynuts at 11:42 AM on February 9, 2009
I think this story was structurally interesting but I understood almost none of what actually happened. Were the three guys three actual guys or just a metaphor for three computers? What the hell is a D button? Can anyone re-write this story in English?
posted by jacquilynne at 11:44 AM on February 9, 2009
posted by jacquilynne at 11:44 AM on February 9, 2009
"Business is a confidence game; you just gotta smile and keep selling shit to the rubes". Sound familiar?
It is not a world of men, Machine. It is not a world of men.
posted by spicynuts at 11:44 AM on February 9, 2009
It is not a world of men, Machine. It is not a world of men.
posted by spicynuts at 11:44 AM on February 9, 2009
Reminds me of this story... This lawn supervisor was out on a sprinkler maintenance job, and he started working on a Findlay sprinkler head with a Langstrom seven-inch gangly wrench. Just then this little apprentice leaned over and said, ‘You can’t work on a Findlay sprinkler head with a Langstrom seven-inch wrench.’ Well, this infuriated the supervisor, so he went and got Volume 14 of the Kinsley manual, and he reads to him and says, ‘The Langstrom seven-inch wrench can be used with the Findlay sprocket.’ Just then the little apprentice leaned over and says, ‘It says sprocket, not socket!’...
posted by From Bklyn at 11:51 AM on February 9, 2009
posted by From Bklyn at 11:51 AM on February 9, 2009
The first comment on the page pretty much says it all:
"See, what I'm reading here is that a phone company was running a $50M/day operation on razor thin staff with security, scalability and continuation of business as a total non-consideration went under because they, a phone company, couldn't figure out how to keep members of the general public from calling directly into their NOC."
posted by Cool Papa Bell at 12:00 PM on February 9, 2009 [1 favorite]
"See, what I'm reading here is that a phone company was running a $50M/day operation on razor thin staff with security, scalability and continuation of business as a total non-consideration went under because they, a phone company, couldn't figure out how to keep members of the general public from calling directly into their NOC."
posted by Cool Papa Bell at 12:00 PM on February 9, 2009 [1 favorite]
Perhaps it was Supra Telecom?
Florida? check.
Bankruptcy? check.
allegations of mismanagement? check.
incompetent customer service? check.
Nigerians? check.
posted by joshwa at 12:05 PM on February 9, 2009 [2 favorites]
posted by joshwa at 12:05 PM on February 9, 2009 [2 favorites]
Okay to all you guys who don't understand what's going on:
Touch tone phones work by combing two different tones, one depending on the column and one depending on the row that the button you pressed is on. There's a standard set of frequencies for the four rows on your touch tone keypad, but there's also tones for four columns (even though virtually all keypads only have three). The four extra buttons your fourth column would have are called A, B, C, D.
So normally if you want to dial a phone number you have dial the whole number. However their network operations center had the phone number "D" instead of (probably actually in addition to) some regular 10 digit number. I guess this was convenient for line workers who had fancy telco handsets with all 16 buttons? But what it meant is that someone could easily "hack into" the NOC by dialing "D" on their phone, instead of having to find the actual telephone number. Or it meant that someone who misconfigured their modem was more likely to call it?
Anyway, their business relied on three humans sitting in a room talking on the phone with their various upstream service providers, finding out when rates go up and down, and renegotiating or changing providers as this happens. But because someone was dialing "D" all day the one (???) line to the NOC was busy all the time so they couldn't do this, so they wound up overpaying for some services by as much as half a cent a call. Which means they would have to have routed about sixteen billion long distance calls in order to go from an expected profit of $50M to a deficit of $30M so either I'm missing part of the story or the whole thing is either exaggerated or just plain false.
posted by aubilenon at 12:19 PM on February 9, 2009 [8 favorites]
Touch tone phones work by combing two different tones, one depending on the column and one depending on the row that the button you pressed is on. There's a standard set of frequencies for the four rows on your touch tone keypad, but there's also tones for four columns (even though virtually all keypads only have three). The four extra buttons your fourth column would have are called A, B, C, D.
So normally if you want to dial a phone number you have dial the whole number. However their network operations center had the phone number "D" instead of (probably actually in addition to) some regular 10 digit number. I guess this was convenient for line workers who had fancy telco handsets with all 16 buttons? But what it meant is that someone could easily "hack into" the NOC by dialing "D" on their phone, instead of having to find the actual telephone number. Or it meant that someone who misconfigured their modem was more likely to call it?
Anyway, their business relied on three humans sitting in a room talking on the phone with their various upstream service providers, finding out when rates go up and down, and renegotiating or changing providers as this happens. But because someone was dialing "D" all day the one (???) line to the NOC was busy all the time so they couldn't do this, so they wound up overpaying for some services by as much as half a cent a call. Which means they would have to have routed about sixteen billion long distance calls in order to go from an expected profit of $50M to a deficit of $30M so either I'm missing part of the story or the whole thing is either exaggerated or just plain false.
posted by aubilenon at 12:19 PM on February 9, 2009 [8 favorites]
What the hell is a D button?
Some older touch tone phones, in addition to the 0-9,*, and # keys, had additional ABCD keys. These keys were used on phone systems which had a private switch, most often military installations, to prioritize calls or other configurable functions. This little fly-by-night phone company, apparently, programmed its switch so that hitting the D key would route calls to the network operations center directly. The DOS was accomplished by someone (I don't buy that it was some random yahoo either, it had to be sabotage by someone who knew what he was doing) repeatedly hitting the D key and hanging up, tying up the line to the network operations center so that no other calls could go in or out, thus making it so they couldn't make or receive calls to upstream providers and use that information to enter proper routes on the switch so that calls were routed through the cheapest providers.
posted by DecemberBoy at 12:28 PM on February 9, 2009
Some older touch tone phones, in addition to the 0-9,*, and # keys, had additional ABCD keys. These keys were used on phone systems which had a private switch, most often military installations, to prioritize calls or other configurable functions. This little fly-by-night phone company, apparently, programmed its switch so that hitting the D key would route calls to the network operations center directly. The DOS was accomplished by someone (I don't buy that it was some random yahoo either, it had to be sabotage by someone who knew what he was doing) repeatedly hitting the D key and hanging up, tying up the line to the network operations center so that no other calls could go in or out, thus making it so they couldn't make or receive calls to upstream providers and use that information to enter proper routes on the switch so that calls were routed through the cheapest providers.
posted by DecemberBoy at 12:28 PM on February 9, 2009
Remember when we were talking about EVE Online and everyone was like "in real life one griefer couldn't do that kind of damage"?
This.
posted by Parasite Unseen at 12:28 PM on February 9, 2009 [10 favorites]
This.
posted by Parasite Unseen at 12:28 PM on February 9, 2009 [10 favorites]
More likely It was a fax or modem on autoredial that accidentally hit the "Dial NOC" command.
I had a nifty phone number on my business line, consisting of just two alternating numbers, that I had to change because I was getting near constant wrong number calls.
Turned out it was some guy with an auto-dialer, who had composed a particularly ornate version of Mary Had a Little Lamb, that he had to constantly demonstrate to his office mates. The opening note had a trill that corresponded to my number.
posted by StickyCarpet at 12:32 PM on February 9, 2009 [1 favorite]
I had a nifty phone number on my business line, consisting of just two alternating numbers, that I had to change because I was getting near constant wrong number calls.
Turned out it was some guy with an auto-dialer, who had composed a particularly ornate version of Mary Had a Little Lamb, that he had to constantly demonstrate to his office mates. The opening note had a trill that corresponded to my number.
posted by StickyCarpet at 12:32 PM on February 9, 2009 [1 favorite]
Also, I can believe that this company had infrastructure consisting of only three guys, a single cheap digital switch and one line in and out to/from the NOC. Some of these little calling card phone companies are extremely fly-by-night operations. What I don't believe is that they had 6,000 employees, presumably all salesmen and managers except for the three guys actually running the equipment.
posted by DecemberBoy at 12:32 PM on February 9, 2009
posted by DecemberBoy at 12:32 PM on February 9, 2009
> Perhaps it was Supra Telecom?
Possible, except...
" Supra had revenues of just $2.43 million in 2000, but that jumped to $18.69 million in 2001 and $74.12 million in the first nine months of [2002]...", which means it was fully operational-seeming a year after the OP's story ends.
Layoffs were in the hundreds rather than thousands.
They resold local lines rather than phone cards.
They shut down due to accumulated unpaid debts, mostly to BellSouth, rather than to long-distance carriers.
But aside from the date, scale, service provided, companies confronted, circumstances of failure and possibly location, the story holds up.
posted by ardgedee at 12:34 PM on February 9, 2009 [2 favorites]
Possible, except...
" Supra had revenues of just $2.43 million in 2000, but that jumped to $18.69 million in 2001 and $74.12 million in the first nine months of [2002]...", which means it was fully operational-seeming a year after the OP's story ends.
Layoffs were in the hundreds rather than thousands.
They resold local lines rather than phone cards.
They shut down due to accumulated unpaid debts, mostly to BellSouth, rather than to long-distance carriers.
But aside from the date, scale, service provided, companies confronted, circumstances of failure and possibly location, the story holds up.
posted by ardgedee at 12:34 PM on February 9, 2009 [2 favorites]
Could it have been a typo in a hand-coded modem script?
ATDD5551212
Doesn't seem all that unlikely.
posted by kc8nod at 12:42 PM on February 9, 2009 [1 favorite]
ATDD5551212
Doesn't seem all that unlikely.
posted by kc8nod at 12:42 PM on February 9, 2009 [1 favorite]
Unless we find that there was really a Ft Lauderdale telco with 6000 people that wasn't Supra, I'm going to chalk this up with the Neiman Marcus cookie story.
posted by crapmatic at 12:59 PM on February 9, 2009
posted by crapmatic at 12:59 PM on February 9, 2009
1) Who would...use a phone like that?
Old people. I used to work in an appliance store in central Virginia and we would have old folks come in from way out in the sticks and want all kinds of crazy stuff like:
- old style tub-wringer washing machines, which haven't been manufactured since the 1940's.
- "monitor-style" refrigerators, not made since freon was introduced in the 1930's.
These people had 50-60 year old appliances that would finally break down. When they went to the store they were convinced that they could get an appliance exactly like it, for exactly the same price they paid for it in 1929 or 1947 or whatever.
(you actually can still get tub-wringer washing machines but they're not cheap)
posted by smoothvirus at 1:16 PM on February 9, 2009
Old people. I used to work in an appliance store in central Virginia and we would have old folks come in from way out in the sticks and want all kinds of crazy stuff like:
- old style tub-wringer washing machines, which haven't been manufactured since the 1940's.
- "monitor-style" refrigerators, not made since freon was introduced in the 1930's.
These people had 50-60 year old appliances that would finally break down. When they went to the store they were convinced that they could get an appliance exactly like it, for exactly the same price they paid for it in 1929 or 1947 or whatever.
(you actually can still get tub-wringer washing machines but they're not cheap)
posted by smoothvirus at 1:16 PM on February 9, 2009
These people had 50-60 year old appliances that would finally break down. When they went to the store they were convinced that they could get an appliance exactly like it,
Completely off subject, but a couple of years ago I had to replace a washing machine that I brought with me from my mother's house, when we finally got rid of it, it was about 30 years old.
When we went to the appliance store, the guy who sold us our new one said "Haha, I wouldn't count on anything we have now lasting that long..."
To this day, I have to wonder if he thought I'd view that as a good thing? I mean, I kind of liked the fact that, while rudimentary, the damn thing just kept going for three decades between two houses. And I'm supposed to be happy that despite all our high technology nowadays, we can't build a washer that can outlast something from a third of a century ago?
posted by quin at 1:37 PM on February 9, 2009 [2 favorites]
Completely off subject, but a couple of years ago I had to replace a washing machine that I brought with me from my mother's house, when we finally got rid of it, it was about 30 years old.
When we went to the appliance store, the guy who sold us our new one said "Haha, I wouldn't count on anything we have now lasting that long..."
To this day, I have to wonder if he thought I'd view that as a good thing? I mean, I kind of liked the fact that, while rudimentary, the damn thing just kept going for three decades between two houses. And I'm supposed to be happy that despite all our high technology nowadays, we can't build a washer that can outlast something from a third of a century ago?
posted by quin at 1:37 PM on February 9, 2009 [2 favorites]
And I'm supposed to be happy that despite all our high technology nowadays, we can't build a washer that can outlast something from a third of a century ago?
The salesman is being kinda silly. Today's washers will last just as long. And if they don't? You get to make this marvelous trade-off in that, while today's washers might not be made of bomb-proof steel parts, they'll...
* Get your clothes cleaner
* Do it faster
* Be less likely to damage your clothes
* Use less energy
* Use less soap
* Use less water
Even if you had to replace a washer in 20 years instead of 30, all of the above will result in a net cost-savings over time.
posted by Cool Papa Bell at 2:30 PM on February 9, 2009
The salesman is being kinda silly. Today's washers will last just as long. And if they don't? You get to make this marvelous trade-off in that, while today's washers might not be made of bomb-proof steel parts, they'll...
* Get your clothes cleaner
* Do it faster
* Be less likely to damage your clothes
* Use less energy
* Use less soap
* Use less water
Even if you had to replace a washer in 20 years instead of 30, all of the above will result in a net cost-savings over time.
posted by Cool Papa Bell at 2:30 PM on February 9, 2009
Yeah, the 'use less energy' and 'use less water' were the big selling points on the new one, as was the fact that it basically doubled our capacity. But it was just the really odd matter-of-fact way he said it, like "Ho ho, those cars from the 70's sure were tanks huh! The car your driving today couldn't possibly last as long as those old ones!"
He seemed so cheerful about it. It was... odd.
Still, I've got nothing against the new one; it works great and sounds like a jet engine, I just liked the fact that, in an emergency, the old one would stop both a nuclear blast, and the resultant radiation.
posted by quin at 2:48 PM on February 9, 2009 [1 favorite]
He seemed so cheerful about it. It was... odd.
Still, I've got nothing against the new one; it works great and sounds like a jet engine, I just liked the fact that, in an emergency, the old one would stop both a nuclear blast, and the resultant radiation.
posted by quin at 2:48 PM on February 9, 2009 [1 favorite]
To this day, I have to wonder if he thought I'd view that as a good thing? I mean, I kind of liked the fact that, while rudimentary, the damn thing just kept going for three decades
I have been using my Mom's Electrolux vacuum, manufactured 1950 something, for like 12 years. It still goes. The power cord still retracts. Everything still works. You would have to drive over it with a semi to kill it and even then I think it would still TRY to vacuum.
posted by spicynuts at 2:51 PM on February 9, 2009
I have been using my Mom's Electrolux vacuum, manufactured 1950 something, for like 12 years. It still goes. The power cord still retracts. Everything still works. You would have to drive over it with a semi to kill it and even then I think it would still TRY to vacuum.
posted by spicynuts at 2:51 PM on February 9, 2009
Before out-of-band switching systems were invented that sent routing signals over a separate data channel, routing signals were sent in-band on the voice channel using various tones (including the infamous 2600Hz 'seize trunk' signal) that would be intercepted by the switch listening in on the line. The US military had a private phone network called AutoVon, and in addition to the usual set of signals they implemented ABCD codes which were used to indicate priority calls that should be given preference over standard traffic. Later on the ABCD tones were adapted to control some private PBX switches & Automated Call Distributor (ACD) systems that manage routing incoming calls to call centers & help desks. There would never have been a residential phone, no matter how old, that had ABCD keys. And for the terminally curious among you, CLLI codes are routing addresses used for telecom networks, somewhat analogous to IP addresses but for phone switches.
posted by scalefree at 3:08 PM on February 9, 2009 [1 favorite]
posted by scalefree at 3:08 PM on February 9, 2009 [1 favorite]
Yeah, but DecemberBoy has it right - the story's not talking about MFC or its US-equivalent (multi-frequency tones used for in-band signalling between exchanges during call establishment; interestingly, that particular meaning of MFC is not listed in Wikipedia - probably because it's European/Swedish), it's talking about good ol' DTMF as used from the phone to the STU (or modern equivalent) in the local exchange. Since DTMF is used to allow you to do phone banking, retrieve messages from your answering machines, etc., they are passed end-to-end.
DTMF was designed as a 4x4 matrix, allowing for 0-9, #, *, and A-D. In practice, the 4th column (A-D) is rarely, if ever, used.
I'm given to understand that, in the early-middling days of telco deregulation in the US when such dodgy telcos abounded, while the customer's line was still physically hooked up to the incumbent's switch, on lift-off it was automatically (or manually, by dialling an override code) switched through to $dodgy_telco's switch which provided its own dial tone. Which was programmed to regard "D" as "the number for the NOC". Hence, hilarity ensued.
posted by Pinback at 4:42 PM on February 9, 2009
DTMF was designed as a 4x4 matrix, allowing for 0-9, #, *, and A-D. In practice, the 4th column (A-D) is rarely, if ever, used.
I'm given to understand that, in the early-middling days of telco deregulation in the US when such dodgy telcos abounded, while the customer's line was still physically hooked up to the incumbent's switch, on lift-off it was automatically (or manually, by dialling an override code) switched through to $dodgy_telco's switch which provided its own dial tone. Which was programmed to regard "D" as "the number for the NOC". Hence, hilarity ensued.
posted by Pinback at 4:42 PM on February 9, 2009
here's what "D" sounds like:
no need for a phone. very easily done with a modem.
posted by spish at 5:04 PM on February 9, 2009
no need for a phone. very easily done with a modem.
posted by spish at 5:04 PM on February 9, 2009
we can't build a washer that can outlast something from a third of a century ago?
well, technically we can, but that's bad business
posted by mannequito at 5:54 PM on February 9, 2009
well, technically we can, but that's bad business
posted by mannequito at 5:54 PM on February 9, 2009
Bad business, not from the cynical standpoint of corporate "planned obsolescence", but from the standpoint of cost on the showroom floor. You could make a bulletproof widget, but people will buy the cheaper, non bulletproof ones.
That's why things are designed to be manufactured, and not maintained, and why ever-so-slightly-more-expensive green technologies have succumbed to cheaper dirtier ones for the last several hundred years.
Sad but true.
posted by Xoebe at 7:00 PM on February 9, 2009 [1 favorite]
That's why things are designed to be manufactured, and not maintained, and why ever-so-slightly-more-expensive green technologies have succumbed to cheaper dirtier ones for the last several hundred years.
Sad but true.
posted by Xoebe at 7:00 PM on February 9, 2009 [1 favorite]
Oh shoot, I forgot, God I love you phone phreaks. We used to have a pay phone in the foyer of my junior high school, that if you stuck an unfolded paper clip wire into the center hole in the mouthpiece and scratched the other end against the metal coin box, the phone would short and give you a free phone call.
posted by Xoebe at 7:03 PM on February 9, 2009
posted by Xoebe at 7:03 PM on February 9, 2009
Is there some sort of nerd version of Penthouse Forum? Because though I didn't really understand it when I read it (thanks to everybody who tried to explain it), I feel like this story comes from it.
"You'll never believe what happened to me..."
posted by MCMikeNamara at 7:50 PM on February 9, 2009 [1 favorite]
"You'll never believe what happened to me..."
posted by MCMikeNamara at 7:50 PM on February 9, 2009 [1 favorite]
We used to have a pay phone in the foyer of my junior high school, that if you stuck an unfolded paper clip wire into the center hole in the mouthpiece and scratched the other end against the metal coin box, the phone would short and give you a free phone call.
That would've been a ground-start phone that actually passed a current across the coins as you deposit them. Short the line & the phone thinks you've dropped a coin in the slot & seizes a trunk for you. They'd be a very rare beast today.
posted by scalefree at 8:50 PM on February 9, 2009 [1 favorite]
That would've been a ground-start phone that actually passed a current across the coins as you deposit them. Short the line & the phone thinks you've dropped a coin in the slot & seizes a trunk for you. They'd be a very rare beast today.
posted by scalefree at 8:50 PM on February 9, 2009 [1 favorite]
Metafilter: Some Sort of Nerd Version of Penthouse Forum
posted by benzenedream at 10:49 PM on February 9, 2009 [2 favorites]
posted by benzenedream at 10:49 PM on February 9, 2009 [2 favorites]
MCMikeNamara, perhaps you've heard of this thing called “the internet”?
posted by hattifattener at 11:32 PM on February 9, 2009
posted by hattifattener at 11:32 PM on February 9, 2009
We used to have a pay phone in the foyer of my junior high school, that if you stuck an unfolded paper clip wire into the center hole in the mouthpiece and scratched the other end against the metal coin box, the phone would short and give you a free phone call.
That would've been a ground-start phone that actually passed a current across the coins as you deposit them. Short the line & the phone thinks you've dropped a coin in the slot & seizes a trunk for you. They'd be a very rare beast today.
Yeah, the dude in WarGames does that trick, remember? They were next to impossible to find even then, the writers probably got the idea from phreak literature from the 70s. Today they're totally nonexistent, they wouldn't even work anymore. Hell, phones that use the ACTS tones (i.e. that you can red box from) are practically nonexistent. Pretty much all payphones you're likely to see nowadays are COCOTs.
posted by DecemberBoy at 5:38 AM on February 10, 2009
That would've been a ground-start phone that actually passed a current across the coins as you deposit them. Short the line & the phone thinks you've dropped a coin in the slot & seizes a trunk for you. They'd be a very rare beast today.
Yeah, the dude in WarGames does that trick, remember? They were next to impossible to find even then, the writers probably got the idea from phreak literature from the 70s. Today they're totally nonexistent, they wouldn't even work anymore. Hell, phones that use the ACTS tones (i.e. that you can red box from) are practically nonexistent. Pretty much all payphones you're likely to see nowadays are COCOTs.
posted by DecemberBoy at 5:38 AM on February 10, 2009
According to spec there are four valid coin deposit sequences for ACTS. A nickel is a single burst of 66ms; a dime is two bursts of 33ms; a quarter is 5 bursts of 33ms (all 1700 & 2200 Hz). What's the fourth sequence?
posted by scalefree at 7:30 AM on February 10, 2009
posted by scalefree at 7:30 AM on February 10, 2009
This sounds too much like a fabricated "what if..?" story to me. What if a small barely clinging to life phone company was manned only by three employees who did everything manually? What if they only had one line which could be blocked? What if someone connected an ancient phone to the system? What if they used it repeatedly on the busiest night of the year? What if the employees didn't know how to stop it until too late?
...and so on, it snowballs into supposedly bankrupting a company of 6000 employees. It's either really exaggerated or just made up.
posted by tachikoma_robot at 10:30 AM on February 10, 2009
...and so on, it snowballs into supposedly bankrupting a company of 6000 employees. It's either really exaggerated or just made up.
posted by tachikoma_robot at 10:30 AM on February 10, 2009
I'm curious if this is the same company (Veraz Networks) involved in a SEC probe last year? And, other than interesting reading, what made this hard yarn to swallow post-worthy on MetaFilter?
posted by FormlessOne at 6:35 PM on February 10, 2009
posted by FormlessOne at 6:35 PM on February 10, 2009
scalefree: This rather comprehensive-looking document from Bellcore (PDF, 4.3 MB) lists the fourth coin tone as being 600-700 milliseconds for a dollar coin.
posted by hattifattener at 8:14 PM on February 11, 2009 [1 favorite]
posted by hattifattener at 8:14 PM on February 11, 2009 [1 favorite]
Ha! Awesome. I've stumped members of the Legion of Doom with that one.
posted by scalefree at 7:40 AM on February 13, 2009
posted by scalefree at 7:40 AM on February 13, 2009
« Older "Journalists' children deserve to be fed" -- a... | High Five Escalator. Newer »
This thread has been archived and is closed to new comments
posted by stupidsexyFlanders at 9:31 AM on February 9, 2009 [28 favorites]