STOOPID %*@&ING COMPUTER!
January 2, 2009 2:15 AM Subscribe
Shouting in the datacenter - increased disk latency caused by shouting: Yelling at your computer* may cause an increase in disk latency and a decrease in performance.
This is so weird! loquacious and me were reading reddit at the same time!
posted by dirty lies at 2:36 AM on January 2, 2009
posted by dirty lies at 2:36 AM on January 2, 2009
This is the complete opposite: Using hard drives to make sound. Check the videos.
posted by dirty lies at 2:38 AM on January 2, 2009 [1 favorite]
posted by dirty lies at 2:38 AM on January 2, 2009 [1 favorite]
No, you need to kick it on that black 'X' we taped on the side. The Magic Switch.
posted by zengargoyle at 2:51 AM on January 2, 2009
posted by zengargoyle at 2:51 AM on January 2, 2009
So I yell at it because it's a piece of goddamn shit fucking stupid trashcan bastard, and thus make it more useless? Sounds like the circle of life to me.
posted by louche mustachio at 2:53 AM on January 2, 2009 [1 favorite]
posted by louche mustachio at 2:53 AM on January 2, 2009 [1 favorite]
Disk latency goes up if you leave the disk loose in the case too, instead of screwing it down to the rails.
posted by surlycat at 2:55 AM on January 2, 2009
posted by surlycat at 2:55 AM on January 2, 2009
Disk latency goes up if you leave the disk loose in the case too, instead of screwing it down to the rails.
That probably explains the truly crappy performance of the computers I built myself in college.
posted by dunkadunc at 3:09 AM on January 2, 2009
That probably explains the truly crappy performance of the computers I built myself in college.
posted by dunkadunc at 3:09 AM on January 2, 2009
No, you just built crappy computers.
posted by zengargoyle at 3:12 AM on January 2, 2009
posted by zengargoyle at 3:12 AM on January 2, 2009
Me? I call pish posh. HDDs are sealed. Air tight. This means no sound waves. I suspect the latency spike is caused by the gentleman pressing his face up against the sensitive disk array, but what do I know?
The lights are blinking because they're doing work.
posted by _aa_ at 3:36 AM on January 2, 2009
The lights are blinking because they're doing work.
posted by _aa_ at 3:36 AM on January 2, 2009
I am surprised this was on reddit. Where are the other 9 ways to increase your disk array latency?
posted by srboisvert at 3:39 AM on January 2, 2009 [6 favorites]
posted by srboisvert at 3:39 AM on January 2, 2009 [6 favorites]
The vibration can still go through the case, even if it's sealed.
posted by dunkadunc at 3:45 AM on January 2, 2009
posted by dunkadunc at 3:45 AM on January 2, 2009
It's not sealed.
posted by zengargoyle at 3:48 AM on January 2, 2009
posted by zengargoyle at 3:48 AM on January 2, 2009
Guess I should... Wikipedia it...
posted by zengargoyle at 4:22 AM on January 2, 2009 [1 favorite]
an HDD today is typically a sealed unit (except for a filtered vent hole to equalize air pressure)
The HDD's spindle system relies on air pressure inside the enclosure to support the heads at their proper flying height while the disk rotates. Hard disk drives require a certain range of air pressures in order to operate properly. The connection to the external environment and pressure occurs through a small hole in the enclosure (about 0.5 mm in diameter), usually with a carbon filter on the inside (the breather filter, see below). If the air pressure is too low, then there is not enough lift for the flying head, so the head gets too close to the disk, and there is a risk of head crashes and data loss. Specially manufactured sealed and pressurized disks are needed for reliable high-altitude operation, above about 3,000 m (10,000 feet).
posted by zengargoyle at 4:22 AM on January 2, 2009 [1 favorite]
This is so weird! loquacious and me were reading reddit at the same time!
Were you two holding hands too?
posted by special-k at 4:32 AM on January 2, 2009 [1 favorite]
Were you two holding hands too?
posted by special-k at 4:32 AM on January 2, 2009 [1 favorite]
I like the way all of the tags are in uppercase. Nicely done!
posted by twoleftfeet at 4:33 AM on January 2, 2009
posted by twoleftfeet at 4:33 AM on January 2, 2009
one time i had some friends of friends visiting & let them stay at my apartment. when they left, they gave me a bottle of wine & a small mexican god-like thing that had a tail with a hole at the end of it. a 'thanks for letting us stay here' gift. i showed it to my future ex & told him, 'if you blow into it's tail, it whistles.' he looked at me & said, 'if you blow up my ass i'd whistle, too.'
the video doesn't show them shouting *at* the computers, it shows them shouting *into* the computers. and i'm guessing that if you shouted into my hdd, my latency would increase, too.
just a theory.
posted by msconduct at 5:18 AM on January 2, 2009 [1 favorite]
the video doesn't show them shouting *at* the computers, it shows them shouting *into* the computers. and i'm guessing that if you shouted into my hdd, my latency would increase, too.
just a theory.
posted by msconduct at 5:18 AM on January 2, 2009 [1 favorite]
HDDs are sealed. Air tight. This means no sound waves.
This is true, and also why the inside of a submarine is a very quiet place.
posted by Drastic at 6:22 AM on January 2, 2009 [4 favorites]
This is true, and also why the inside of a submarine is a very quiet place.
posted by Drastic at 6:22 AM on January 2, 2009 [4 favorites]
MetaFilter: The lights are blinking because they're (not) doing work.
posted by The Bellman at 6:45 AM on January 2, 2009
posted by The Bellman at 6:45 AM on January 2, 2009
that if you shouted into my hdd, my latency would increase, too.
At a certain SPL, it definitely affects the drive -- 500msec latency is *bad*, and there was a huge spike.
However, air is a lousy way to carry vibration and couple it to other objects. I can a much easier way for this to be a problem -- the big compressor(s) in the AC unit. I know I can feel it in my feet when they kick in and out, and I'm thinking about my data center and where the big storage arrays are in relation to the AC unit.
Answer: Very close. I've also seen cases where disk latency has shot up for no reason whatsoever. What I'm trying to come up with is something that I can use to measure vibration at the rack, tray and drive level, and measure physical latency as well.
It may well be that this is a real problem -- not from guys yelling at racks, but from things shaking the floor -- and a rigid floor, with a AC unit and a rack rigidly mounted on it, would be a great way to couple that energy.
posted by eriko at 6:49 AM on January 2, 2009 [1 favorite]
At a certain SPL, it definitely affects the drive -- 500msec latency is *bad*, and there was a huge spike.
However, air is a lousy way to carry vibration and couple it to other objects. I can a much easier way for this to be a problem -- the big compressor(s) in the AC unit. I know I can feel it in my feet when they kick in and out, and I'm thinking about my data center and where the big storage arrays are in relation to the AC unit.
Answer: Very close. I've also seen cases where disk latency has shot up for no reason whatsoever. What I'm trying to come up with is something that I can use to measure vibration at the rack, tray and drive level, and measure physical latency as well.
It may well be that this is a real problem -- not from guys yelling at racks, but from things shaking the floor -- and a rigid floor, with a AC unit and a rack rigidly mounted on it, would be a great way to couple that energy.
posted by eriko at 6:49 AM on January 2, 2009 [1 favorite]
I'm wondering why we stay in the top 100 super-computers list and have scores and scores of servers and a metric butt-load of disks and robot arm tape storage and 'physical latency' is nothing I've heard about. They bitch about everything else..... AC, generators, unscheduled DWP work, backhoes, rats eating fiber.... never have I heard 'physical latency' as a problem. Maybe I'm in the wrong business.
posted by zengargoyle at 7:16 AM on January 2, 2009
posted by zengargoyle at 7:16 AM on January 2, 2009
I suspect eriko's on the money there--data centers aren't just noisy places. These data centers, they vibrate. A lot. (Noisy and cold and constantly shaking. I'm glad I don't work in data centers any more.)
The latency stuff isn't a problem to hear about for the most part precisely because of those metric butt-loads of disks. Drives fail all the time in those big hot-swappable arrays, and those arrays are used because you can just yank out the failed drive and slap in a new one whenever they start beeping frantically.
posted by Drastic at 7:36 AM on January 2, 2009
The latency stuff isn't a problem to hear about for the most part precisely because of those metric butt-loads of disks. Drives fail all the time in those big hot-swappable arrays, and those arrays are used because you can just yank out the failed drive and slap in a new one whenever they start beeping frantically.
posted by Drastic at 7:36 AM on January 2, 2009
In related news, watching a pot delays boiling indefinitely.
posted by dirigibleman at 7:37 AM on January 2, 2009
posted by dirigibleman at 7:37 AM on January 2, 2009
These data centers, they vibrate.
Only the Nimbus 2000 Data Centers vibrate
posted by ElvisJesus at 8:01 AM on January 2, 2009
Only the Nimbus 2000 Data Centers vibrate
posted by ElvisJesus at 8:01 AM on January 2, 2009
The latency stuff isn't a problem to hear about for the most part precisely because of those metric butt-loads of disks.
Or you just weren't looking for it -- note how short the latency problem happens -- or that you are seeing issues because of vibration, and you're blaming the wrong things. Or it's just so scattered that you've not been able to track it down and written it off as one of those things.
The interesting thing is that accelerometers have gotten cheap enough that we can easily get vibration data -- indeed, the first computer use I know of was in notebooks.1 I'm seriously looking at trying it. The hardware is easy. The question is can I get the data that I need out of the storage arrays quickly enough to be useful.
Alas, I'm not running Sun gear here -- thus, the SAN isn't going to have DTraceonboard. I'm going to have to wade into SNMP and see if I can't get the data I need (physical latency, by disk) out, and then correlate that with the accelerometer data.
Right now, this is all hand waving. First, I'm going to see if that data is reachable. If it is, expect something in Projects.
1) Apple, IIRC, was the first to use one in a consumer unit but they weren't looking for vibration. On the notebooks, they were looking for a sudden transition to near zero G acceleration. Why? Sitting on a desk, you'd measure a steady 1G acceleration, towards the center of the Earth -- Gravity, of course. The most likely reason for that acceleration to disappear? The notebook is falling, thus, it shows zero G. That means you just dropped the notebook. When the sensor sensed that, it sends a command to the drive telling it to get the heads off the disk platters -- making it far more likely that your data will survive the fall, even if your notebook doesn't.
Of course, developer quickly tapped that signal and did fun things with it. Yay, hacking!
posted by eriko at 8:03 AM on January 2, 2009 [2 favorites]
Or you just weren't looking for it -- note how short the latency problem happens -- or that you are seeing issues because of vibration, and you're blaming the wrong things. Or it's just so scattered that you've not been able to track it down and written it off as one of those things.
The interesting thing is that accelerometers have gotten cheap enough that we can easily get vibration data -- indeed, the first computer use I know of was in notebooks.1 I'm seriously looking at trying it. The hardware is easy. The question is can I get the data that I need out of the storage arrays quickly enough to be useful.
Alas, I'm not running Sun gear here -- thus, the SAN isn't going to have DTraceonboard. I'm going to have to wade into SNMP and see if I can't get the data I need (physical latency, by disk) out, and then correlate that with the accelerometer data.
Right now, this is all hand waving. First, I'm going to see if that data is reachable. If it is, expect something in Projects.
1) Apple, IIRC, was the first to use one in a consumer unit but they weren't looking for vibration. On the notebooks, they were looking for a sudden transition to near zero G acceleration. Why? Sitting on a desk, you'd measure a steady 1G acceleration, towards the center of the Earth -- Gravity, of course. The most likely reason for that acceleration to disappear? The notebook is falling, thus, it shows zero G. That means you just dropped the notebook. When the sensor sensed that, it sends a command to the drive telling it to get the heads off the disk platters -- making it far more likely that your data will survive the fall, even if your notebook doesn't.
Of course, developer quickly tapped that signal and did fun things with it. Yay, hacking!
posted by eriko at 8:03 AM on January 2, 2009 [2 favorites]
I could think of several theories:
1. Hoax
2. Vibration affecting disk/disk head dynamics--the vibration may affect the high speed spinning disk, changing the disk/head spacing or maybe some air pressure affect like zengargoyle mentioned
3. Vibration affecting ceramic capacitors (many people don't realize that they respond to vibration) which affects the disk power supplies, causing dropouts. I've never heard of this happening but I've always wondered if it could.
4. Vibration shaking loose the connectors to a poorly mounted disk as surlycat mentioned.
If it is real I'd guess at #2. Though they do their best to design disks so the 7200 RPM spinning disks with disk heads floating right above them aren't affected by your typical office vibrations, maybe they never accounted or tested for the frequencies involved with some guy shouting at the disks.
posted by eye of newt at 8:12 AM on January 2, 2009
1. Hoax
2. Vibration affecting disk/disk head dynamics--the vibration may affect the high speed spinning disk, changing the disk/head spacing or maybe some air pressure affect like zengargoyle mentioned
3. Vibration affecting ceramic capacitors (many people don't realize that they respond to vibration) which affects the disk power supplies, causing dropouts. I've never heard of this happening but I've always wondered if it could.
4. Vibration shaking loose the connectors to a poorly mounted disk as surlycat mentioned.
If it is real I'd guess at #2. Though they do their best to design disks so the 7200 RPM spinning disks with disk heads floating right above them aren't affected by your typical office vibrations, maybe they never accounted or tested for the frequencies involved with some guy shouting at the disks.
posted by eye of newt at 8:12 AM on January 2, 2009
It may well be that this is a real problem -- not from guys yelling at racks, but from things shaking the floor -- and a rigid floor, with a AC unit and a rack rigidly mounted on it, would be a great way to couple that energy.
Well.. sort of. I mean, it is all about resonances. Those guys have found a frequency of excitation that causes the drive problems -- something a little over 1kHz, probably. It is very unlikely that sufficient 1kHz vibrations could be coupled into a drive array through the floor!
posted by Chuckles at 8:27 AM on January 2, 2009 [1 favorite]
Well.. sort of. I mean, it is all about resonances. Those guys have found a frequency of excitation that causes the drive problems -- something a little over 1kHz, probably. It is very unlikely that sufficient 1kHz vibrations could be coupled into a drive array through the floor!
posted by Chuckles at 8:27 AM on January 2, 2009 [1 favorite]
I'm just glad that they're listening, that means there's still a chance I can get through to them.
posted by StickyCarpet at 8:33 AM on January 2, 2009 [3 favorites]
posted by StickyCarpet at 8:33 AM on January 2, 2009 [3 favorites]
3. Vibration affecting ceramic capacitors (many people don't realize that they respond to vibration) which affects the disk power supplies, causing dropouts. I've never heard of this happening but I've always wondered if it could.
Audiophiles have long considered the microphonics in electrolytic capacitors to be of great importance. Of course some of them also think that the material used for volume knobs is of great importance, but in the case of capacitor microphonics it might actually be true.
posted by Chuckles at 8:41 AM on January 2, 2009
Audiophiles have long considered the microphonics in electrolytic capacitors to be of great importance. Of course some of them also think that the material used for volume knobs is of great importance, but in the case of capacitor microphonics it might actually be true.
posted by Chuckles at 8:41 AM on January 2, 2009
This is why any experienced user knows to increase performance, you speak to your computer in calm, even tones, gently suggesting that failure to perform would lead to reprimand, followed by being thrown out the window.
posted by The Whelk at 9:22 AM on January 2, 2009 [1 favorite]
posted by The Whelk at 9:22 AM on January 2, 2009 [1 favorite]
Silly engineers, everyone knows the only way to make a computer behave is to casually place the scarred, dismembered corpse of one of its compatriots nearby. Pour encourager les autres.
posted by Skorgu at 9:38 AM on January 2, 2009 [3 favorites]
posted by Skorgu at 9:38 AM on January 2, 2009 [3 favorites]
Anti-vibration tables (and other more exotic gadgets) have been used for years in microscopy to protect the equipment from building vibrations.
posted by benzenedream at 10:38 AM on January 2, 2009
posted by benzenedream at 10:38 AM on January 2, 2009
No, you need to kick it on that black 'X' we taped on the side. The Magic Switch.
When I used to work in IT one of the default (forced change on next logon) passwords I handed out was propitiate.
posted by BrotherCaine at 10:44 AM on January 2, 2009
When I used to work in IT one of the default (forced change on next logon) passwords I handed out was propitiate.
posted by BrotherCaine at 10:44 AM on January 2, 2009
noised as flag.
posted by blue_beetle at 11:18 AM on January 2, 2009
posted by blue_beetle at 11:18 AM on January 2, 2009
Finally a valid cause for a New Years resolution: I shall not shout at my computer because I might hurt its feelings.
posted by Cranberry at 11:49 AM on January 2, 2009
posted by Cranberry at 11:49 AM on January 2, 2009
So, uh, the whispered apologies afterwards make the latency go back down, right?
I didn't mean it, baby! Boot up for daddy, please, sweetie. You can have the kids; I just want my backups.
posted by slimepuppy at 12:12 PM on January 2, 2009
I didn't mean it, baby! Boot up for daddy, please, sweetie. You can have the kids; I just want my backups.
posted by slimepuppy at 12:12 PM on January 2, 2009
The notebook is falling, thus, it shows zero G. That means you just dropped the notebook. When the sensor sensed that, it sends a command to the drive telling it to get the heads off the disk platters -- making it far more likely that your data will survive the fall, even if your notebook doesn't.
Does anybody know if the standard MacBooks have these accelerometers? I'm wondering if this might be related to something I'm puzzling over. My mom works for a school district, and a couple of weeks ago they pointed out to all their staff that there's a warning on the recent MacBooks in the manual:
Now they also said something else interesting:
I suspect that if the problem has developed recently, then it has to do with a change in the firmware/OS that determines how/when the heads are parked. But I haven't been able to figure out why the Pro models would be less susceptible than the plain MacBooks. Is it the accelerometer difference or might it be something else?
posted by weston at 12:14 PM on January 2, 2009
Does anybody know if the standard MacBooks have these accelerometers? I'm wondering if this might be related to something I'm puzzling over. My mom works for a school district, and a couple of weeks ago they pointed out to all their staff that there's a warning on the recent MacBooks in the manual:
Wait a few seconds until the sleep indicator light starts pulsing (indicating that the computer is in sleep and the hard disk has stopped spinning) before you move your Macbook. Moving your computer while the hard disk is spinning can damage it, causing loss of data or the inability to start up from the hard disk.
Now they also said something else interesting:
This only applies to those who received new Macbook laptops or use Macbooks produced since July 2007. It doesn't apply to Macbook Pros.
I suspect that if the problem has developed recently, then it has to do with a change in the firmware/OS that determines how/when the heads are parked. But I haven't been able to figure out why the Pro models would be less susceptible than the plain MacBooks. Is it the accelerometer difference or might it be something else?
posted by weston at 12:14 PM on January 2, 2009
I don't know anything about the validity of their testing procedures, but I would say that there could be design issues just related to the fact that server and rack practices were built and tested around before people were sticking 42 3.5" drives in a 4U enclosure.
That is a lot of spinning mass is a very small space, not to mention a lot of power and thermal load.
(My friend had problems switching their server farms to ESX virtualized nodes, because the hosting company had not anticipating having 40U of server space taken up by 10 16 core boxes, each with 16+ gigE ports. Of course it was just was matter of the hosting company building out a new rack, but they had to delay deployment as it was put together.)
posted by mrzarquon at 12:21 PM on January 2, 2009
That is a lot of spinning mass is a very small space, not to mention a lot of power and thermal load.
(My friend had problems switching their server farms to ESX virtualized nodes, because the hosting company had not anticipating having 40U of server space taken up by 10 16 core boxes, each with 16+ gigE ports. Of course it was just was matter of the hosting company building out a new rack, but they had to delay deployment as it was put together.)
posted by mrzarquon at 12:21 PM on January 2, 2009
weston-
that is the deep sleep mode on the machines. The contents of memory are being written to disk, to ensure that if the machine powers off completely while asleep, it can recover to where it left off once power is restored (ie, windows hibernation).
Since the drive is being written to, you should not try to swing the laptop around and throw it into your backpack immediately after putting the machine to sleep.
I think this is a 'problem' for both macbooks and macbook pros, but since the pros don't have shared video ram to be stored also (they have dedicated graphics cards) and they are less likely to be manhandled like a macbook is, the problem of bad sectors and disk damage from the machine being jostled during this hibernation stage manifests more frequently on the Macbooks.
posted by mrzarquon at 12:34 PM on January 2, 2009
that is the deep sleep mode on the machines. The contents of memory are being written to disk, to ensure that if the machine powers off completely while asleep, it can recover to where it left off once power is restored (ie, windows hibernation).
Since the drive is being written to, you should not try to swing the laptop around and throw it into your backpack immediately after putting the machine to sleep.
I think this is a 'problem' for both macbooks and macbook pros, but since the pros don't have shared video ram to be stored also (they have dedicated graphics cards) and they are less likely to be manhandled like a macbook is, the problem of bad sectors and disk damage from the machine being jostled during this hibernation stage manifests more frequently on the Macbooks.
posted by mrzarquon at 12:34 PM on January 2, 2009
I'm surprised this didn't end with a request to Purchasing to buy some quieter air conditioners. "We are losing $1eN every year to disk latency issues brought on by being in THE LOUDEST SERVER ROOM ON THE PLANET."
posted by DU at 6:57 PM on January 3, 2009
posted by DU at 6:57 PM on January 3, 2009
« Older Walter Monheit -- The Oldest Club Kid | JPG magazine folds Newer »
This thread has been archived and is closed to new comments
posted by loquacious at 2:15 AM on January 2, 2009 [1 favorite]