Legitimate Job Test or Something Wacky?
June 30, 2005 5:26 AM Subscribe
Legitimate Job Test or Something Wacky? H.J. Cummins of the Minneapolis Star Tribune writes about personality tests--never meant to screen job applicants--being used or misused by employers.
Test sample items:
"I see things or animals or people around me that others do not see."
"My soul sometimes leaves my body."
"I have a habit of counting things that are not important, such as bulbs on electric signs, and so forth."
I see dead people!
HR departments are always after something that will make their screening easier (read: less thought), so they can have junior level HR clerks read scripts to job applicants.
The last interview I had (and it was for a part time job at the HQ of a large book chain), I had to help the clerk read and interpret the questions--questions that I had myself used in some variants to hire people [describe a situation that did not turn out the way you expected--how did you respond?]
They met with me in person--in part because they wanted to meet someone who so fully aced their screen--but kept assuring me I was overqualified.
I have rarely met a personality screening test that is properly used.
posted by beelzbubba at 5:54 AM on June 30, 2005
HR departments are always after something that will make their screening easier (read: less thought), so they can have junior level HR clerks read scripts to job applicants.
The last interview I had (and it was for a part time job at the HQ of a large book chain), I had to help the clerk read and interpret the questions--questions that I had myself used in some variants to hire people [describe a situation that did not turn out the way you expected--how did you respond?]
They met with me in person--in part because they wanted to meet someone who so fully aced their screen--but kept assuring me I was overqualified.
I have rarely met a personality screening test that is properly used.
posted by beelzbubba at 5:54 AM on June 30, 2005
Malcolm Gladwell wrote a pretty extensive piece about personality tests about a year ago. Still worth a read...
posted by ph00dz at 6:02 AM on June 30, 2005
posted by ph00dz at 6:02 AM on June 30, 2005
I see stupid people. They're all around me. They don't know they're stupid.
I'd definitely have to tick the box next to that one.
posted by psmealey at 6:05 AM on June 30, 2005
I'd definitely have to tick the box next to that one.
posted by psmealey at 6:05 AM on June 30, 2005
Yeah, in many cases I'd assume the people directly involved with the application of these tests think they're as dumb as I do, but at your larger chain outfits the interview/hiring process is usually dictated from on high, so if some HR consultant can convince enough VPs that these tests will save them .01% on mental health payouts down the line, then it's payday for the HR consultant and the test publisher! The tests I've seen are usually the anti-stealing (or "shrinkage" as they call it in the industry) ones that are all about catching you out on internal inconsistencies. (i.e. you answered "never" to the first five "I steal stuff" questions, but "maybe" on the "I would rat out my co-workers for stealing" question).
posted by idontlikewords at 6:06 AM on June 30, 2005
posted by idontlikewords at 6:06 AM on June 30, 2005
A: "I believe that some people do not deserve the same rights as others."
B: True.
A: "I believe that those with more resources should exert power over those with less."
B: True.
A: "I believe that everything I do is correct because God says so."
B: True.
A: I'm sorry, sir; I'm afraid you are not qualified to be President of the United States.
posted by Faint of Butt at 6:07 AM on June 30, 2005
B: True.
A: "I believe that those with more resources should exert power over those with less."
B: True.
A: "I believe that everything I do is correct because God says so."
B: True.
A: I'm sorry, sir; I'm afraid you are not qualified to be President of the United States.
posted by Faint of Butt at 6:07 AM on June 30, 2005
Years ago, when I was fresh out of college, I applied for a job working at a temp agency (not a temp--as agency staff). I took a basic aptitude test that was apparently par for the course for employment at this agency--mostly really basic math, logic, literacy stuff. So I'm breezing along until I hit this question: rearrange the following words into logical order--and then mark as "true" or "false." The words were:
teaching no Love needs
This threw me off completely and I sat there for ten minutes trying to figure out the "true or false" part. There were a few other unscramble-the-sentence questions, but none of the others had the T/F component. I suspected it was a hidden psychological test and that they probably wanted me to be happy-crappy about it and say that of COURSE love (or Love, capital L--either to make it very clear that it wasn't "teaching needs no love"--which is true, unfortunately--or to further romanticize the damn question) needs no teaching--but I don't really believe that, and I really didn't want to lie about it. I left the question blank.
The company called and offered me the job (during which time I turned down another job and two interviews). The day before I was supposed to start, they called me and said I'm sorry, we aren't hiring you after all. They never gave me a reason and I always wondered if that question had anything to do with it--if it showed that I was psychologically unsound or something. Farfetched, I know, but people get stupid about Love.
posted by dlugoczaj at 6:12 AM on June 30, 2005 [1 favorite]
teaching no Love needs
This threw me off completely and I sat there for ten minutes trying to figure out the "true or false" part. There were a few other unscramble-the-sentence questions, but none of the others had the T/F component. I suspected it was a hidden psychological test and that they probably wanted me to be happy-crappy about it and say that of COURSE love (or Love, capital L--either to make it very clear that it wasn't "teaching needs no love"--which is true, unfortunately--or to further romanticize the damn question) needs no teaching--but I don't really believe that, and I really didn't want to lie about it. I left the question blank.
The company called and offered me the job (during which time I turned down another job and two interviews). The day before I was supposed to start, they called me and said I'm sorry, we aren't hiring you after all. They never gave me a reason and I always wondered if that question had anything to do with it--if it showed that I was psychologically unsound or something. Farfetched, I know, but people get stupid about Love.
posted by dlugoczaj at 6:12 AM on June 30, 2005 [1 favorite]
dlugoczaj, failure or unwillingness to answer that question indicates that you have a 49.3% probability of becoming a cannibalistic axe-murderer. Sorry.
posted by spazzm at 6:22 AM on June 30, 2005
posted by spazzm at 6:22 AM on June 30, 2005
The Malcolm Gladwell piece is online here and there's an interesting AskMeTa on these tests here.
posted by Slack-a-gogo at 6:24 AM on June 30, 2005
posted by Slack-a-gogo at 6:24 AM on June 30, 2005
To be hired, you must disavow your existential miasma.
posted by gorgor_balabala at 6:24 AM on June 30, 2005
posted by gorgor_balabala at 6:24 AM on June 30, 2005
And the correct answer would have been "no needs love teaching" and "true". No need to thank me.
posted by spazzm at 6:26 AM on June 30, 2005
posted by spazzm at 6:26 AM on June 30, 2005
spazzm--figured it had to be something like that. I'll have to warn my kids not to come around me if they see me with power tools or kitchen equipment, at least if they want a 53.8% chance of saving their wretched little hides.
posted by dlugoczaj at 6:39 AM on June 30, 2005
posted by dlugoczaj at 6:39 AM on June 30, 2005
dlugoczaj the fact you didn't answer the question showed you are a thoughtful person but you lack the sense to hide this fact. When the natives rub blue mud in their navels, always do the same. (Robert Heinlein)
posted by Goofyy at 6:41 AM on June 30, 2005
posted by Goofyy at 6:41 AM on June 30, 2005
Used properly, there's no reason not to use personality tests to screen job applicants. They're a source of information just like any other. You could just as soon complain that employers shouldn't be allowed to interview prospective employees because the way you come off in a half-hour conversation is not representative of who you are.
More frequently, the problem is that the people administering the tests don't really understand how the instruments work (or which ones do). For example, the MBTI is widely used in all sorts of applied settings, yet it's almost never used in academic personality psychology because, quite frankly, it's a crappy, unreliable test. You base decisions on the MBTI at your own risk if you're an employer.
On the other hand, there are lots of personality measures that are perfectly reliable and can be quite predictive of behavior. For example, there are lots of good measures of extraversion floating around. While level of extraversion alone shouldn't be used to give someone a job, it can be a pretty good indicator as to what kind of job a person is likely to enjoy. Someone who's 3 standard deviations below the mean on extraversion almost certainly isn't going to enjoy or do well at a sales job. Conversely, a raging extrovert is not going to want to be stuck in a corner office for 8 hours a day with no opportunity for interaction.
In clinical psychology, the use of questionnaires and other self-report measures has evolved to the point where actuarial diagnoses (i.e., formulaic applications of equations based on your scores) are, in almost every case they've been tested, far superior to psychologists' clinical judgments. That is, your chance of being accurately diagnosed are better if you fill out a bunch of measures and the clinician then uses an equation to come up with a diagnosis than if they interview you themselves and try to make a judgment on that basis. There's really no reason to think employers are any better at figuring out people than clinical psychologists, so the future in industry probably also belongs to actuarial hirings.
People don't like to think of themselves as a set of questionnaires, but used properly, those questionnaires are more informative then anything else you can get across to an interviewer in an hour...
posted by heavy water at 6:41 AM on June 30, 2005
More frequently, the problem is that the people administering the tests don't really understand how the instruments work (or which ones do). For example, the MBTI is widely used in all sorts of applied settings, yet it's almost never used in academic personality psychology because, quite frankly, it's a crappy, unreliable test. You base decisions on the MBTI at your own risk if you're an employer.
On the other hand, there are lots of personality measures that are perfectly reliable and can be quite predictive of behavior. For example, there are lots of good measures of extraversion floating around. While level of extraversion alone shouldn't be used to give someone a job, it can be a pretty good indicator as to what kind of job a person is likely to enjoy. Someone who's 3 standard deviations below the mean on extraversion almost certainly isn't going to enjoy or do well at a sales job. Conversely, a raging extrovert is not going to want to be stuck in a corner office for 8 hours a day with no opportunity for interaction.
In clinical psychology, the use of questionnaires and other self-report measures has evolved to the point where actuarial diagnoses (i.e., formulaic applications of equations based on your scores) are, in almost every case they've been tested, far superior to psychologists' clinical judgments. That is, your chance of being accurately diagnosed are better if you fill out a bunch of measures and the clinician then uses an equation to come up with a diagnosis than if they interview you themselves and try to make a judgment on that basis. There's really no reason to think employers are any better at figuring out people than clinical psychologists, so the future in industry probably also belongs to actuarial hirings.
People don't like to think of themselves as a set of questionnaires, but used properly, those questionnaires are more informative then anything else you can get across to an interviewer in an hour...
posted by heavy water at 6:41 AM on June 30, 2005
AFAIK these tests are all based on the MMPI, a personality test from the 30's that's still used a lot. The funny thing about this tests is that they force you to a high degree of introspection that is very unconfortable to some people (no wonder even the Scientologists start their recruitment with a test very much like MMPI).
The thing is - those tests work if you are playing the game of "spot the psychotic" in a large crowd. But applying them to regular good ole neurotics like you and me is not of much use. What happens then? Since recruiters got all that useless data in their hands, they end up misusing it because, hey, it's free data.
posted by falameufilho at 6:42 AM on June 30, 2005
The thing is - those tests work if you are playing the game of "spot the psychotic" in a large crowd. But applying them to regular good ole neurotics like you and me is not of much use. What happens then? Since recruiters got all that useless data in their hands, they end up misusing it because, hey, it's free data.
posted by falameufilho at 6:42 AM on June 30, 2005
heavy water: I'm not a professional in anything related to psychology (or really anything), but like most of us I've had to take several of these sorts of tests, and also like most of us I've completely bullshitted them. Sometimes it works, sometimes it doesn't. It just seems to me that these tests will only be as useful to an employer as the prospective employee is honest in the test.
What I mean is, somebody taking these tests for use by their psychologist is (hopefully) going to be completely upfront and honest with their responses, since their motivator is helping their psychologist understand their mental state. However, when someone takes the same test for a potential employer, their answers will probably be drastically different. For example, I recently had to take such a personality inventory that was supposedly tailored to the service industry (it was a restaurant job). Several questions along the lines of the following were on the test:
Sometimes you feel like telling people exactly what you think of them.
Strongly agree - agree - disgree - strongly disgree
Now obviously, since I post on MetaFilter, I strongly agree with this statement. However, were I to relate this to my prospective employer through the test, it would definitely count against me. So rather than answering honestly, I gamed it. As it turns out, I got the highest possible score on the test. So does this mean that I am ideally suited for a job in the service industry, or perhaps does it mean that I'm good at gaming tests like these?
Another example: Blockbuster uses the Unicru system for handling applications, which I gather is popular in the service industry. Part of the application system has a personality test. While I was employed at Blockbuster some years ago I tried to get a friend a job there. He took the test and his score designated him as "red"- do not hire under any circumstances. I convinced my boss to let him reapply, and after reviewing the test at a different store and re-taking it at mine, he got "green". He was subsequently hired.
So my question is, considering that some people can game these sorts of tests fairly easily, are they really helping employers hire the best possible people for the job, or will they just be hiring good test takers?
posted by baphomet at 7:12 AM on June 30, 2005
What I mean is, somebody taking these tests for use by their psychologist is (hopefully) going to be completely upfront and honest with their responses, since their motivator is helping their psychologist understand their mental state. However, when someone takes the same test for a potential employer, their answers will probably be drastically different. For example, I recently had to take such a personality inventory that was supposedly tailored to the service industry (it was a restaurant job). Several questions along the lines of the following were on the test:
Sometimes you feel like telling people exactly what you think of them.
Strongly agree - agree - disgree - strongly disgree
Now obviously, since I post on MetaFilter, I strongly agree with this statement. However, were I to relate this to my prospective employer through the test, it would definitely count against me. So rather than answering honestly, I gamed it. As it turns out, I got the highest possible score on the test. So does this mean that I am ideally suited for a job in the service industry, or perhaps does it mean that I'm good at gaming tests like these?
Another example: Blockbuster uses the Unicru system for handling applications, which I gather is popular in the service industry. Part of the application system has a personality test. While I was employed at Blockbuster some years ago I tried to get a friend a job there. He took the test and his score designated him as "red"- do not hire under any circumstances. I convinced my boss to let him reapply, and after reviewing the test at a different store and re-taking it at mine, he got "green". He was subsequently hired.
So my question is, considering that some people can game these sorts of tests fairly easily, are they really helping employers hire the best possible people for the job, or will they just be hiring good test takers?
posted by baphomet at 7:12 AM on June 30, 2005
So my question is, considering that some people can game these sorts of tests fairly easily, are they really helping employers hire the best possible people for the job, or will they just be hiring good test takers?
Sure, some tests can be gamed. But so can interviews. Clearly, most people don't answer personality tests for a job the way they'd answer them for fun. But most people also don't answer an interviewer's questions the way they answer their friend's. So the question isn't whether personality tests are immune to cheating; it's whether they're any more immune to cheating than interviews.
In fact, there are a number of ways you can make tests less susceptible to cheating (there's a whole branch of psychometric theory behind this). For one thing, you (and most people who deliberately try to test favorably) are probably unaware that many personality tests have 'lie' scales. A lie scale is one made up of questions that are positive but highly improbable. For example, items like "I've never told a lie in my life" or "I always admit it when I'm wrong". These are items that you might think are highly desirable and the employer is looking for, but in fact, answering too many of them positively sends up major red flags and is likely to get you disqualified. So right there, the employer has one pretty good way of determining whether you're being honest or not (whereas most people are terrible at determining whether they're being lied to verbally).
Another approach is to use so-called 'subtle' items. There are ways to design tests with questions that are almost opaque; i.e., you have no idea what the question is asking for. The MMPI, for example, has a lot of these scales. It's very difficult to lie on the MMPI, because it's not really obvious what many of the questions are asking and which answers are desirable.
That all said, I want to stress again that the biggest problem with the use of personality measures is not that there's anything wrong with the idea itself, but rather that the people who administer these things are typically not qualified to do so. Employers use a lot of measures that are unreliable, invalid, and completely inadequate for hiring purposes. My point isn't that current practices are just fine. It's that, in principle, a good questionnaire-based actuarial approach that integrates personality data with objective performance indicators is probably superior to an interview-based approach.
posted by heavy water at 7:34 AM on June 30, 2005
Sure, some tests can be gamed. But so can interviews. Clearly, most people don't answer personality tests for a job the way they'd answer them for fun. But most people also don't answer an interviewer's questions the way they answer their friend's. So the question isn't whether personality tests are immune to cheating; it's whether they're any more immune to cheating than interviews.
In fact, there are a number of ways you can make tests less susceptible to cheating (there's a whole branch of psychometric theory behind this). For one thing, you (and most people who deliberately try to test favorably) are probably unaware that many personality tests have 'lie' scales. A lie scale is one made up of questions that are positive but highly improbable. For example, items like "I've never told a lie in my life" or "I always admit it when I'm wrong". These are items that you might think are highly desirable and the employer is looking for, but in fact, answering too many of them positively sends up major red flags and is likely to get you disqualified. So right there, the employer has one pretty good way of determining whether you're being honest or not (whereas most people are terrible at determining whether they're being lied to verbally).
Another approach is to use so-called 'subtle' items. There are ways to design tests with questions that are almost opaque; i.e., you have no idea what the question is asking for. The MMPI, for example, has a lot of these scales. It's very difficult to lie on the MMPI, because it's not really obvious what many of the questions are asking and which answers are desirable.
That all said, I want to stress again that the biggest problem with the use of personality measures is not that there's anything wrong with the idea itself, but rather that the people who administer these things are typically not qualified to do so. Employers use a lot of measures that are unreliable, invalid, and completely inadequate for hiring purposes. My point isn't that current practices are just fine. It's that, in principle, a good questionnaire-based actuarial approach that integrates personality data with objective performance indicators is probably superior to an interview-based approach.
posted by heavy water at 7:34 AM on June 30, 2005
i have a habit of counting things or animals or people that sometimes leave my invisible body.
posted by quonsar at 8:00 AM on June 30, 2005
posted by quonsar at 8:00 AM on June 30, 2005
Can you give an example of a so-called 'subtle' item, heavy water? I don't see how the answer to a question whose intent is opaque can be at all meaningful, since all you have is the answer given, not the interpretation the person asked gave it (and in order to answer a question you have to give it some interpretation). If I ask you "jumble wozzle ricky socks?" and you strongly agree, wtf have I learned? I haven't even learned that you tend to strongly agree with questions you don't understand because maybe it made sense to you!
posted by kenko at 8:38 AM on June 30, 2005
posted by kenko at 8:38 AM on June 30, 2005
I wanted to second what heavy water said on one point: the quality of administration is key. For example, permitting note taking or not enforcing time limits can enable a test-taker to game a test which is well-designed to prevent gaming if administered per the rules.
As I said in the linke AskMe thread, the most important thing for a job applicant to do is to be very pollyanish -- any time you characterize OTHER people and situations negatively it is taken as either a sign of projection (i.e., YOU have that negative quality) or paranoia/misanthropy.
posted by MattD at 8:48 AM on June 30, 2005
As I said in the linke AskMe thread, the most important thing for a job applicant to do is to be very pollyanish -- any time you characterize OTHER people and situations negatively it is taken as either a sign of projection (i.e., YOU have that negative quality) or paranoia/misanthropy.
posted by MattD at 8:48 AM on June 30, 2005
Funny, when I took the MMPI-2 I was told I'd scored as mentally ill, a misfit and troublemaker, probably gay, and very consistently honest. (I often wish I could have copied my answers and the scoring results; it'd be great on a home page.) What I can't figure is how my answers would have shown which line of work I was best suited for.
posted by davy at 8:54 AM on June 30, 2005
posted by davy at 8:54 AM on June 30, 2005
A: "I believe that some people do not deserve the same rights as others."
B: True.
A: "I believe that those with more resources should exert power over those with less."
B: True.
A: "I believe that everything I do is correct because God says so."
B: True.
A: I'm sorry, sir; I'm afraid you are not qualified to be President of the United States.
posted by Faint of Butt at 6:07 AM PST on June 30 [!]>
No, but you are an excellent candidate for matriculation into Patrick Henry College!
posted by beelzbubba at 9:24 AM on June 30, 2005
B: True.
A: "I believe that those with more resources should exert power over those with less."
B: True.
A: "I believe that everything I do is correct because God says so."
B: True.
A: I'm sorry, sir; I'm afraid you are not qualified to be President of the United States.
posted by Faint of Butt at 6:07 AM PST on June 30 [!]>
No, but you are an excellent candidate for matriculation into Patrick Henry College!
posted by beelzbubba at 9:24 AM on June 30, 2005
I once had an HR weasel/rep force the entire IT department to take one of those "enneagram" personality tests. Several of us protested that we didn't want such an invasion of privacy in our HR files. She assured us, "it is just for your own knowledge, I won't be collecting the worksheets. Plus, it is mandatory."
I didn't believe her about the information staying private, and so filled in mine to show what I intended it to show, that I am a loyal busy-bee worker. Of course, her HR weasel-nature was proven by her walking around looking over everyone's shoulders, taking notes the entire time we filled in our personality test forms.
A month or so later, the layoffs began.
No wonder I unreservedly despise HR people and departments. Lying weasels to the last, they do not deserve any attempt at honesty or straight dealing, since they will always misuse it to their own ends.
posted by Invoke at 9:27 AM on June 30, 2005
I didn't believe her about the information staying private, and so filled in mine to show what I intended it to show, that I am a loyal busy-bee worker. Of course, her HR weasel-nature was proven by her walking around looking over everyone's shoulders, taking notes the entire time we filled in our personality test forms.
A month or so later, the layoffs began.
No wonder I unreservedly despise HR people and departments. Lying weasels to the last, they do not deserve any attempt at honesty or straight dealing, since they will always misuse it to their own ends.
posted by Invoke at 9:27 AM on June 30, 2005
I got burned by one of these many years ago. I was applying for a job at an office supply store and they had a psych test you were required to take.
I was 17 or so, and didn't drink. Never had. So I was seriously unprepared for a line of questions like:
"Did you ever hide your drinking from your parents?"
"When your parents caught you drinking, did they get upset?"
And several others all based on the assumption that _everybody_ experimented with alcohol.
I didn't get the job. And my opinion of psych tests ever since has been "By idiots, for idiots."
Aside: I failed my first written driving test for the same reason. I skipped the chapter on alcohol and drunk driving laws in the handbook, because I didn't care how many beers and/or fortified wines I could have before being legally drunk. Guess what chapter the majority of questions were drawn from?
posted by bitmage at 9:27 AM on June 30, 2005
I was 17 or so, and didn't drink. Never had. So I was seriously unprepared for a line of questions like:
"Did you ever hide your drinking from your parents?"
"When your parents caught you drinking, did they get upset?"
And several others all based on the assumption that _everybody_ experimented with alcohol.
I didn't get the job. And my opinion of psych tests ever since has been "By idiots, for idiots."
Aside: I failed my first written driving test for the same reason. I skipped the chapter on alcohol and drunk driving laws in the handbook, because I didn't care how many beers and/or fortified wines I could have before being legally drunk. Guess what chapter the majority of questions were drawn from?
posted by bitmage at 9:27 AM on June 30, 2005
dlugoczaj, they wanted you to agree that "Love needs no teaching". Which by the way I'd answer is false.
posted by davy at 9:35 AM on June 30, 2005
posted by davy at 9:35 AM on June 30, 2005
That fact that people can game a test does not necessarily invalidate it. The ability to game a test can, itself, be a desirable quality.
To take baphomet's example,
Sometimes you feel like telling people exactly what you think of them.
Strongly agree - agree - disagree - strongly disagree
If you know to disagree on the test, you will also know not to tell people exactly what you think of them when on the job.
A certain amount of dishonesty is expected and similar types of people tend to be dishonest in similar, predictable ways.
That said, I don't believe that HR departments are capable of using these tests effectively.
posted by Zetetics at 9:36 AM on June 30, 2005
To take baphomet's example,
Sometimes you feel like telling people exactly what you think of them.
Strongly agree - agree - disagree - strongly disagree
If you know to disagree on the test, you will also know not to tell people exactly what you think of them when on the job.
A certain amount of dishonesty is expected and similar types of people tend to be dishonest in similar, predictable ways.
That said, I don't believe that HR departments are capable of using these tests effectively.
posted by Zetetics at 9:36 AM on June 30, 2005
"I'm sorry, sir. We can't hire you because your personality test showed that your ideal Hobbit Husband would be Merry. We're looking for a Sam or maybe a Pippin. Better luck next time!"
posted by robocop is bleeding at 10:00 AM on June 30, 2005
posted by robocop is bleeding at 10:00 AM on June 30, 2005
Bitmage, I had a similar experience on a driving test. Oy!
posted by JHarris at 10:14 AM on June 30, 2005
posted by JHarris at 10:14 AM on June 30, 2005
Um, davy, I got that--and I absolutely believe that it's false, but thought I'd get screwed by the test admin as a morbid doomsayer if I did, and I figured that lying about it was even worse (apparently, I should have lied; how could they tell?). Turned out that when I got home and told my parents about the question, my father immediately jumped to the good little do-bee answer and then realized that actually, I was right to say it was false. As a social worker for 30 years, I think he'd darn well better realize that love needs teaching; he spent his life cleaning up the messes of people who weren't taught.
posted by dlugoczaj at 10:23 AM on June 30, 2005
posted by dlugoczaj at 10:23 AM on June 30, 2005
Can you give an example of a so-called 'subtle' item, heavy water? I don't see how the answer to a question whose intent is opaque can be at all meaningful, since all you have is the answer given, not the interpretation the person asked gave it (and in order to answer a question you have to give it some interpretation).
People who score personality tests rarely (or at least, shouldn't) look at individual items. The question is what the total score for a scale is; individual items aren't reliable measure of anything because there's plenty of error associated with measuring something just once. So it doesn't really matter how someone interprets a particular question. Almost everyone misinterprets questions here and there; that's why a good measure has several questions testing each dimension of interest.
The way empirically-keyed measures are developed is that the researcher basically has two clearly-distinguished populations answer a large database of questions (e.g., several hundred or thousand), and the items that best discriminate between the two groups are then put on the final measure. This often results in completely non-intuitive items. For example, the MMPI item "I'm independent from family rule" discriminates well between psychopaths and non-psychopaths. But it actually discriminates in the opposite direction from what you'd expect: psychopaths typically answer FALSE to it. Now, you can certainly come up with sensible explanations after the fact, but the point is that if you were trying to fake non-psychopath, you'd probably be more likely to think you should answer false. At the very least, the item doesn't seem like much of an indicator of psychopathy. That's true of a lot of the MMPI dimensions: they have items that are virtually impossible to 'figure out', yet they consistently discriminate between the intended populations.
Empirical keying isn't, of course, practical for all measures. The point is just that if you were going to develop a measure that was non-intuitive, or accounted for people's tendency to lie, there are ways to do so. Again, that doesn't mean HR departments get it right--I'm sure the vast majority of people who administer these things aren't even familiar with the idea of subtle items or lie scales. But the larger point is that questionnaire measures are not bogus instruments made up to make people's lives difficult. Used properly, they're better indicators of all sorts of outcomes than unstructured interviews are.
posted by heavy water at 11:04 AM on June 30, 2005
People who score personality tests rarely (or at least, shouldn't) look at individual items. The question is what the total score for a scale is; individual items aren't reliable measure of anything because there's plenty of error associated with measuring something just once. So it doesn't really matter how someone interprets a particular question. Almost everyone misinterprets questions here and there; that's why a good measure has several questions testing each dimension of interest.
The way empirically-keyed measures are developed is that the researcher basically has two clearly-distinguished populations answer a large database of questions (e.g., several hundred or thousand), and the items that best discriminate between the two groups are then put on the final measure. This often results in completely non-intuitive items. For example, the MMPI item "I'm independent from family rule" discriminates well between psychopaths and non-psychopaths. But it actually discriminates in the opposite direction from what you'd expect: psychopaths typically answer FALSE to it. Now, you can certainly come up with sensible explanations after the fact, but the point is that if you were trying to fake non-psychopath, you'd probably be more likely to think you should answer false. At the very least, the item doesn't seem like much of an indicator of psychopathy. That's true of a lot of the MMPI dimensions: they have items that are virtually impossible to 'figure out', yet they consistently discriminate between the intended populations.
Empirical keying isn't, of course, practical for all measures. The point is just that if you were going to develop a measure that was non-intuitive, or accounted for people's tendency to lie, there are ways to do so. Again, that doesn't mean HR departments get it right--I'm sure the vast majority of people who administer these things aren't even familiar with the idea of subtle items or lie scales. But the larger point is that questionnaire measures are not bogus instruments made up to make people's lives difficult. Used properly, they're better indicators of all sorts of outcomes than unstructured interviews are.
posted by heavy water at 11:04 AM on June 30, 2005
Oddly enough, I had lunch today with a co-worker who had gotten involved in college with a study of schizophrenia. They had paid her cash to take a survey on the proviso they could contact her again later. They managed to track her down again last week and the questions they asked her were eerily similar to ones in the article.
posted by tommasz at 12:15 PM on June 30, 2005
posted by tommasz at 12:15 PM on June 30, 2005
Come to think of it, I think the study of how to game these sort of silly hurdles would make a great website. It could solicit examples and real-world stories of peoples' approaches to these things.
My approach is basically, "put yourself in the mindset of what you are emulating, and never allow yourself to rationalize 'everyone does x'." For example, the correct answer to "have you stolen office supplies from any previous employer?" is "no".
Why "no"? Isn't that question a lie-detector such as those heavy water mentions? It possibly should be, but those scoring your test are too stupid to use it that way. Assume those who would give you such a test are idiots, and answer in the way they obviously want.
If possible - and this is gold - try to get them to show you your scores afterwards. It helps refine your test-manipulation powers. I've successfully gotten one hiring manager to do that, and it helped a lot.
posted by Invoke at 1:49 PM on June 30, 2005
My approach is basically, "put yourself in the mindset of what you are emulating, and never allow yourself to rationalize 'everyone does x'." For example, the correct answer to "have you stolen office supplies from any previous employer?" is "no".
Why "no"? Isn't that question a lie-detector such as those heavy water mentions? It possibly should be, but those scoring your test are too stupid to use it that way. Assume those who would give you such a test are idiots, and answer in the way they obviously want.
If possible - and this is gold - try to get them to show you your scores afterwards. It helps refine your test-manipulation powers. I've successfully gotten one hiring manager to do that, and it helped a lot.
posted by Invoke at 1:49 PM on June 30, 2005
dlugoczaj, I was just picking on you so that I could show off. Someday I'll be as mighty as Quonsar himself!
posted by davy at 8:04 PM on June 30, 2005
posted by davy at 8:04 PM on June 30, 2005
I see no reason to use a MMPI or other psychological tests on potential job seekers unless they are in the helping fields (eg: nursing/doctors/police) where that kind of information might be pertinent. The potential for abuse and misinterpretation of the results is too high.
posted by squeak at 12:17 AM on July 1, 2005
posted by squeak at 12:17 AM on July 1, 2005
Robert MacNamara in The Fog of War recalls a test given to would be Ford execs just after the war in which you were asked whether you would hypothetically prefer to work as a florist or as a coalminer. He had worked as a florist while going to school.
He put down coal miner.
He got the job.
Lesson 12- Treat tests with the respect they deserve.
posted by IndigoJones at 8:16 AM on July 1, 2005
He put down coal miner.
He got the job.
Lesson 12- Treat tests with the respect they deserve.
posted by IndigoJones at 8:16 AM on July 1, 2005
I took the MMPI about 15 years ago. It was a 600 question test as I remember (perhaps it was mainly testing endurance ?)
For most of the test, I was able to keep my mind on answering simply and truthfully, and letting the conclusions fall where they may.
One question boggled my mind though - answer true or false to the statement "I used to like to play drop-the-handkerchief".
After useless minutes of overthinking it, I left it blank. One of life's mysteries.
posted by AuntLisa at 11:45 AM on July 1, 2005
For most of the test, I was able to keep my mind on answering simply and truthfully, and letting the conclusions fall where they may.
One question boggled my mind though - answer true or false to the statement "I used to like to play drop-the-handkerchief".
After useless minutes of overthinking it, I left it blank. One of life's mysteries.
posted by AuntLisa at 11:45 AM on July 1, 2005
"I was 17 or so, and didn't drink. Never had. So I was seriously unprepared for a line of questions like:
'Did you ever hide your drinking from your parents?'
'When your parents caught you drinking, did they get upset?'"
Yes, I had a similar experience. When I was 20, I took a test to work at a record store, and it asked if I had ever smoked pot, smoked cigarettes, or been drunk. I said no to all, which was completely true. I didn't get the job and was later told by a friend who worked there that those answers indicated dishonesty. Needless to say, I was angry.
The girl who got hired ended up marrying the manager, as it turned out.
posted by litlnemo at 2:09 PM on July 1, 2005
'Did you ever hide your drinking from your parents?'
'When your parents caught you drinking, did they get upset?'"
Yes, I had a similar experience. When I was 20, I took a test to work at a record store, and it asked if I had ever smoked pot, smoked cigarettes, or been drunk. I said no to all, which was completely true. I didn't get the job and was later told by a friend who worked there that those answers indicated dishonesty. Needless to say, I was angry.
The girl who got hired ended up marrying the manager, as it turned out.
posted by litlnemo at 2:09 PM on July 1, 2005
"Have you ever used sugar... or PCP?"
--Mitch Hedberg
posted by idontlikewords at 12:02 PM on July 6, 2005
--Mitch Hedberg
posted by idontlikewords at 12:02 PM on July 6, 2005
« Older AIPAC rules Ok? | Then and Now: Truth and spin on Iraq Newer »
This thread has been archived and is closed to new comments
posted by Pollomacho at 5:37 AM on June 30, 2005