Play games, teach computers
May 27, 2008 4:45 PM Subscribe
Each year, people around the world spend billions of hours playing computer games. What if all this time and energy could be channeled into useful work? What if people playing computer games could, without consciously doing so, simultaneously solve large-scale problems?
GWAP is Luis van Ahn's answer [PDF, HTML cache] to these questions, a collection of easy and engaging games that make computers smarter.
GWAP is Luis van Ahn's answer [PDF, HTML cache] to these questions, a collection of easy and engaging games that make computers smarter.
Are you trying to tell me that collecting every single hidden orb is not "useful work"? Nice try, Stalin!
posted by turgid dahlia at 5:22 PM on May 27, 2008 [1 favorite]
posted by turgid dahlia at 5:22 PM on May 27, 2008 [1 favorite]
I don't understand. The point of GWAP's games is to process computer-unfriendly tasks like image identification and linguistic analysis using input from many thousands of people, yes? So what's the point of sticking those tools behind a registration wall? Don't these games work better the more people participate? I don't see how discouraging participation in this way works. And an open system would be resistant to purposefully-incorrect input since most of the games rely on two players agreeing as much as possible.
Or maybe I'm just slightly peeved that I was going to post this weeks ago but the site's registration was closed indefinitely at the time...
posted by Rhaomi at 5:26 PM on May 27, 2008 [5 favorites]
Or maybe I'm just slightly peeved that I was going to post this weeks ago but the site's registration was closed indefinitely at the time...
posted by Rhaomi at 5:26 PM on May 27, 2008 [5 favorites]
Nitpicking aside, this is really fun stuff. I wonder how effective it will be at training the relevant software.
posted by Rhaomi at 5:40 PM on May 27, 2008
posted by Rhaomi at 5:40 PM on May 27, 2008
So what's the point of sticking those tools behind a registration wall? Don't these games work better the more people participate? I don't see how discouraging participation in this way works.
I don't know, but I bet it has to do with this being developed at a university research center. Yes. Registration walls aren't much of a discouragement anymore. An annoyance maybe, but not really discouraging. Anyway, it's free, and the games are fun, so it's worth it. I'm maxing out at 1400 on Verbosity right now.
posted by carsonb at 5:57 PM on May 27, 2008
I don't know, but I bet it has to do with this being developed at a university research center. Yes. Registration walls aren't much of a discouragement anymore. An annoyance maybe, but not really discouraging. Anyway, it's free, and the games are fun, so it's worth it. I'm maxing out at 1400 on Verbosity right now.
posted by carsonb at 5:57 PM on May 27, 2008
"Everyone benefits." Are the results of this site's game play public domain? Not that I don't want smarter search engines, but it's going to feel a bit icky if in a year we learn "GWAP was acquired by Google (the 'don't do all that much harm unless absolutely necessary for new jets, etc.' company)."
posted by maxwelton at 6:01 PM on May 27, 2008
posted by maxwelton at 6:01 PM on May 27, 2008
This sort of reminds me of the Phillip K Dick story where the hero's solutions to a daily newspaper puzzle are used to predict enemy military attacks.
posted by canoehead at 6:05 PM on May 27, 2008
posted by canoehead at 6:05 PM on May 27, 2008
Registration walls aren't much of a discouragement anymore.
Unless the front page is really compelling, they discourage me.
posted by DU at 6:05 PM on May 27, 2008
Unless the front page is really compelling, they discourage me.
posted by DU at 6:05 PM on May 27, 2008
maxwelton: ""Everyone benefits." Are the results of this site's game play public domain? Not that I don't want smarter search engines, but it's going to feel a bit icky if in a year we learn "GWAP was acquired by Google (the 'don't do all that much harm unless absolutely necessary for new jets, etc.' company).""
Google already has a tool similar to the ESP game. I wouldn't be surprised if they cut a deal with GWAP to benefit from their user data, or replicated the games themselves.
posted by Rhaomi at 6:09 PM on May 27, 2008
Google already has a tool similar to the ESP game. I wouldn't be surprised if they cut a deal with GWAP to benefit from their user data, or replicated the games themselves.
posted by Rhaomi at 6:09 PM on May 27, 2008
This is good. It's really slick, doesn't feel like a research project, I like it. The partner aspect really makes it.
posted by Khalad at 6:15 PM on May 27, 2008
posted by Khalad at 6:15 PM on May 27, 2008
Like maxwelton said, will they freely distribute the database of info they get from the machine learning algorithms they learn? If not, I don't think they can make the claim that "everyone benefits". I like Google Image Labeler, but they say that you're helping Google's Image Search, not "everyone".
posted by demiurge at 6:16 PM on May 27, 2008
posted by demiurge at 6:16 PM on May 27, 2008
GWAP is a great idea, but unfortunately Verbosity is a deeply disappointing game. Here's why:
(1) Our partner is anonymous. This means all the negative and positive effects of social interaction are gone. We have no history with the person, we can't keep friends lists (or idiot lists), and one of the most vitally important drivers of this kind of game, rivalry, is diminished. Verbosity is precisely the sort of intellectual peacock-tail displaying contest that I very, very much want to be involved in. It's FUN. :)
(2) When we're matching well with a person, we want to continue. The game designers should want us to continue too, since we're giving good results. However, the game is set up to allow as many or few matches as you can get in four minutes, come hell or high water. It should reset the time allowed every time a match is obtained.
(3) The scoring system is opaque. Clearly, the best result is obtained from an instant guess from one clue. But everything, pretty much, scores 200 points. Given the level of fineness a score in the hundreds allows, it could easily be adjusted for time, guess quality, clue quality and ...
(4) Word difficulty. "cat" "thunder" "sustenance" "foot" "yellow". See a problem? The system is designed to numerically rate the difficulty of the identification of any given word. Higher-scoring players need to get words that are more difficult.
(5) Word repetition. Some words repeat a lot. There are over 267,000 words in the English language without even getting outside of Scrabble validity. Why am I seeing "thunder" every third game? Individual players' scores for words can be kept, which in itself would provide a source of useful information.
(6) Immediate feedback. Once both players pass, then and there the word should be displayed. This gives the guesser a great deal of needed insight into the mindset of the cluegiver.
(7) More and greater variety of "it is ..." "it is normally near ..." introductory phrases. It should be easy enough to create such phrases with a variation on the same game.
I've thought about making up a card game based on Verbosity. Haven't got to the stage of actually printing up cards yet, but here's the idea:
MATERIALS:
- 10 cards each printed with a different introductory phrase. "IT IS" "NORMALLY NEAR" "OPPOSITE OF" ... etc. You could generate dozens of these.
- Dictionary.
- Method of randomly choosing a word in the dictionary. The ideal is a program running on a PDA or mobile phone that chooses the word itself, but at the least, a method of choosing a random page, a random column, and a random entry in the column is required. 10-sided dice will work well if the dictionary's near enough to 100 or 1000 pages. For 500 pages, roll a 6-sided die, subtract 1, multiply by 100, and add percentile dice. For columns, roll a dice, flip a coin, or just alternate. For the specific word, a 20-sided die will probably do, or two 20-sided dice (skews probability towards the middle of the page, but this doesn't much matter), or a 20-sided dice and a top/bottom coinflip.
- Scorepad, or chips for scoring. Players score individually.
- Clock, egg-timer, or similar device. Experiment with times; 1min seems good.
METHOD:
- Each player starts on some number of points. 10 should work.
- Each round, a guesser and a cluegiver are appointed. The simplest way to do this is to rotate around the table with each player getting a chance to be guesser once, and each other player getting a chance to be cluegiver for the guesser.
- The guesser receives the 10 cards. (Or a random 10 out of the larger set.)
- The cluegiver randomly generates a word, out of sight of the guesser so as not to provide a clue by its position in the dictionary, looks it up and reviews the meaning, then hands the dictionary to a third player, indicating the word. The other players, if any, also get to see it.
- The guesser hands a card of his/her choice to the cluegiver, face up, and starts the clock.
- The cluegiver reviews the introductory phrase, and says one or two English words. These words must be found in the dictionary, and must logically (if not grammatically) follow the introductory phrase. (Thus "IT IS religious forever" is a valid clue for "eternity", but "IT IS eight long" is not.) Players will need to decide for themselves if word length or word structure or analogous words (eg, "LOOKS LIKE entity") are allowed. I suggest one of the cards be specifically "WORD LENGTH").
- Words containing the word, grammatical variations and conjugations, plurals, etc are not allowed as clues.
- Other players act as judges to enforce the rules. If the other players agree that the clue is invalid, the cluegiver and guesser are forced to pass.
- The cluegiver and guesser must make at least one attempt per card, providing one clue, and making one guess. After this, they may pass.
- After each guess the guesser may guess again, hand another card to the cluegiver, or pass.
- If the guesser passes, he/she loses one point. The cluegiver compulsorily passes as well, but does not lose points.
- If the cluegiver passes, he/she loses one point. The guesser compulsorily passes as well, but does not lose points.
- If the time for the word runs out, both players compulsorily pass, but do not lose points.
- If the word is guessed correctly, both players receive one point each.
- If the guesser still has cards, the cluegiver randomly generates another word, and the game continues. As before, the guesser resets the timer when handing over his/her first card.
- When the guesser runs out of cards to hand the cluegiver, the turn is over.
- After all players have been guesser and cluegiver in all combinations, the game ends, and the person(s) on highest points wins.
posted by aeschenkarnos at 6:46 PM on May 27, 2008 [4 favorites]
(1) Our partner is anonymous. This means all the negative and positive effects of social interaction are gone. We have no history with the person, we can't keep friends lists (or idiot lists), and one of the most vitally important drivers of this kind of game, rivalry, is diminished. Verbosity is precisely the sort of intellectual peacock-tail displaying contest that I very, very much want to be involved in. It's FUN. :)
(2) When we're matching well with a person, we want to continue. The game designers should want us to continue too, since we're giving good results. However, the game is set up to allow as many or few matches as you can get in four minutes, come hell or high water. It should reset the time allowed every time a match is obtained.
(3) The scoring system is opaque. Clearly, the best result is obtained from an instant guess from one clue. But everything, pretty much, scores 200 points. Given the level of fineness a score in the hundreds allows, it could easily be adjusted for time, guess quality, clue quality and ...
(4) Word difficulty. "cat" "thunder" "sustenance" "foot" "yellow". See a problem? The system is designed to numerically rate the difficulty of the identification of any given word. Higher-scoring players need to get words that are more difficult.
(5) Word repetition. Some words repeat a lot. There are over 267,000 words in the English language without even getting outside of Scrabble validity. Why am I seeing "thunder" every third game? Individual players' scores for words can be kept, which in itself would provide a source of useful information.
(6) Immediate feedback. Once both players pass, then and there the word should be displayed. This gives the guesser a great deal of needed insight into the mindset of the cluegiver.
(7) More and greater variety of "it is ..." "it is normally near ..." introductory phrases. It should be easy enough to create such phrases with a variation on the same game.
I've thought about making up a card game based on Verbosity. Haven't got to the stage of actually printing up cards yet, but here's the idea:
MATERIALS:
- 10 cards each printed with a different introductory phrase. "IT IS" "NORMALLY NEAR" "OPPOSITE OF" ... etc. You could generate dozens of these.
- Dictionary.
- Method of randomly choosing a word in the dictionary. The ideal is a program running on a PDA or mobile phone that chooses the word itself, but at the least, a method of choosing a random page, a random column, and a random entry in the column is required. 10-sided dice will work well if the dictionary's near enough to 100 or 1000 pages. For 500 pages, roll a 6-sided die, subtract 1, multiply by 100, and add percentile dice. For columns, roll a dice, flip a coin, or just alternate. For the specific word, a 20-sided die will probably do, or two 20-sided dice (skews probability towards the middle of the page, but this doesn't much matter), or a 20-sided dice and a top/bottom coinflip.
- Scorepad, or chips for scoring. Players score individually.
- Clock, egg-timer, or similar device. Experiment with times; 1min seems good.
METHOD:
- Each player starts on some number of points. 10 should work.
- Each round, a guesser and a cluegiver are appointed. The simplest way to do this is to rotate around the table with each player getting a chance to be guesser once, and each other player getting a chance to be cluegiver for the guesser.
- The guesser receives the 10 cards. (Or a random 10 out of the larger set.)
- The cluegiver randomly generates a word, out of sight of the guesser so as not to provide a clue by its position in the dictionary, looks it up and reviews the meaning, then hands the dictionary to a third player, indicating the word. The other players, if any, also get to see it.
- The guesser hands a card of his/her choice to the cluegiver, face up, and starts the clock.
- The cluegiver reviews the introductory phrase, and says one or two English words. These words must be found in the dictionary, and must logically (if not grammatically) follow the introductory phrase. (Thus "IT IS religious forever" is a valid clue for "eternity", but "IT IS eight long" is not.) Players will need to decide for themselves if word length or word structure or analogous words (eg, "LOOKS LIKE entity") are allowed. I suggest one of the cards be specifically "WORD LENGTH").
- Words containing the word, grammatical variations and conjugations, plurals, etc are not allowed as clues.
- Other players act as judges to enforce the rules. If the other players agree that the clue is invalid, the cluegiver and guesser are forced to pass.
- The cluegiver and guesser must make at least one attempt per card, providing one clue, and making one guess. After this, they may pass.
- After each guess the guesser may guess again, hand another card to the cluegiver, or pass.
- If the guesser passes, he/she loses one point. The cluegiver compulsorily passes as well, but does not lose points.
- If the cluegiver passes, he/she loses one point. The guesser compulsorily passes as well, but does not lose points.
- If the time for the word runs out, both players compulsorily pass, but do not lose points.
- If the word is guessed correctly, both players receive one point each.
- If the guesser still has cards, the cluegiver randomly generates another word, and the game continues. As before, the guesser resets the timer when handing over his/her first card.
- When the guesser runs out of cards to hand the cluegiver, the turn is over.
- After all players have been guesser and cluegiver in all combinations, the game ends, and the person(s) on highest points wins.
posted by aeschenkarnos at 6:46 PM on May 27, 2008 [4 favorites]
Google already has a tool similar to the ESP game. I wouldn't be surprised if they cut a deal with GWAP to benefit from their user data, or replicated the games themselves.
Luis von Ahn was (I believe) working for Google at the time he developed it. The "Google Image Labeller" is the "ESP game".
posted by aeschenkarnos at 6:48 PM on May 27, 2008
Luis von Ahn was (I believe) working for Google at the time he developed it. The "Google Image Labeller" is the "ESP game".
posted by aeschenkarnos at 6:48 PM on May 27, 2008
Fuck, I was hoping for a good excuse to play Halo 3 until my thumbs bleed.
posted by jimmythefish at 7:04 PM on May 27, 2008
posted by jimmythefish at 7:04 PM on May 27, 2008
I don't know, but I bet it has to do with this being developed at a university research center. Yes. Registration walls aren't much of a discouragement anymore.
Are you kidding? They certainly discourage me. 99 times out of 100 I won't bother with something if I have to give an email address or otherwise waste a bunch of time with something. The whole point of the web is immediacy and registration destroys that, along with providing someone with the opportunity to spam you.
And some real data: I once had a website the allowed comments. I required users to pick a username and password (and all they had to do was type their password in twice, no email or any multi-step registration whatsoever) and I got a few comments. When I allowed people to comment anonymously, I got 10x as many comments with the same number of readers.
Even the simplest registration is a huge detriment to people participating.
posted by delmoi at 7:15 PM on May 27, 2008
Are you kidding? They certainly discourage me. 99 times out of 100 I won't bother with something if I have to give an email address or otherwise waste a bunch of time with something. The whole point of the web is immediacy and registration destroys that, along with providing someone with the opportunity to spam you.
And some real data: I once had a website the allowed comments. I required users to pick a username and password (and all they had to do was type their password in twice, no email or any multi-step registration whatsoever) and I got a few comments. When I allowed people to comment anonymously, I got 10x as many comments with the same number of readers.
Even the simplest registration is a huge detriment to people participating.
posted by delmoi at 7:15 PM on May 27, 2008
Some further thoughts on my card game idea: The cluegiver can be deprived of the definition (which will probably help) by having a third player randomly generate the word, look it up, and write it down. Alternatively, the cluegiver can be forbidden to use any words in the definition.
The game could use some kind of economic model for "buying" clues (rather than handing over cards) and giving higher scores for more difficult words. However I can't see an easy way yet to prevent a player from working out what the word is, then "farming" guesses to raise the value of it.
Maybe change the structure of the game: the cluegiver generates the word, takes the top phrase card off the deck, and gives the clue. Then all players except the cluegiver take turns to guess, and contribute a point for each wrong guess, and the bounty goes to the successful guesser. (Although that may swing things a bit much; maybe just give the successful guesser double his/her own point pile back.) After each guesser has had an unsuccessful guess, draw a new phrase card. (But this doesn't incentivize the cluegiver to give good clues ... maybe give the cluegiver points equal to the number of cards left in the pile? Draw maybe 5 cards from the pile, and go on to another word if no-one gets it after 5 clues?)
This looks like an interesting class of game to experiment with, IMO.
posted by aeschenkarnos at 7:20 PM on May 27, 2008
The game could use some kind of economic model for "buying" clues (rather than handing over cards) and giving higher scores for more difficult words. However I can't see an easy way yet to prevent a player from working out what the word is, then "farming" guesses to raise the value of it.
Maybe change the structure of the game: the cluegiver generates the word, takes the top phrase card off the deck, and gives the clue. Then all players except the cluegiver take turns to guess, and contribute a point for each wrong guess, and the bounty goes to the successful guesser. (Although that may swing things a bit much; maybe just give the successful guesser double his/her own point pile back.) After each guesser has had an unsuccessful guess, draw a new phrase card. (But this doesn't incentivize the cluegiver to give good clues ... maybe give the cluegiver points equal to the number of cards left in the pile? Draw maybe 5 cards from the pile, and go on to another word if no-one gets it after 5 clues?)
This looks like an interesting class of game to experiment with, IMO.
posted by aeschenkarnos at 7:20 PM on May 27, 2008
delmoi Even the simplest registration is a huge detriment to people participating.
Damn right. I detest having to register, open an email, activate an account, etc to post a comment at some site that gives me absolutely nothing except the opportunity to post the comment. Chances are I'm going to forget the password anyway, and once whatever prompted me to register is over and done with, I may not ever come back.
Kitten-captchas etc should stop the first wave of spammers, occasional comment deletion should stop the second, and IP blocking should stop the third. After that, maybe time to change the blog software, put in delays etc to make automated commenting harder.
posted by aeschenkarnos at 7:24 PM on May 27, 2008
Damn right. I detest having to register, open an email, activate an account, etc to post a comment at some site that gives me absolutely nothing except the opportunity to post the comment. Chances are I'm going to forget the password anyway, and once whatever prompted me to register is over and done with, I may not ever come back.
Kitten-captchas etc should stop the first wave of spammers, occasional comment deletion should stop the second, and IP blocking should stop the third. After that, maybe time to change the blog software, put in delays etc to make automated commenting harder.
posted by aeschenkarnos at 7:24 PM on May 27, 2008
Great site. Truly "fun" for a few minutes.
Agree on registration. WTF von Ahn? I fail to see how it helps.
Every one of aeschenkarnos' suggestions in the first post should be incorporated.
posted by Ynoxas at 8:38 PM on May 27, 2008
Agree on registration. WTF von Ahn? I fail to see how it helps.
Every one of aeschenkarnos' suggestions in the first post should be incorporated.
posted by Ynoxas at 8:38 PM on May 27, 2008
Aeschenkarnos' suggestions 1,2, and 6 might make the game more fun, but would destroy the validity of the results for use in machine learning. The point of the fixed time period, anonymity, and lack of feedback is to prevent collusion between players. If two players realize that by answering "banana" to everything they'll have perfect esp, they'll get a great score, but the results will be worthless.
It doesn't have to even be that extreme to taint the results: if you know that your partner has a tendency to relate everything to food, that will help your guesses, but will make the results worse, since you're answering based on your partner and not based on the presented content.
Registration is presumably to prevent collusion as well; the ability to effortlessly create sockpuppets (not that registration makes things much better) would be bad for results.
posted by Pyry at 10:23 PM on May 27, 2008
It doesn't have to even be that extreme to taint the results: if you know that your partner has a tendency to relate everything to food, that will help your guesses, but will make the results worse, since you're answering based on your partner and not based on the presented content.
Registration is presumably to prevent collusion as well; the ability to effortlessly create sockpuppets (not that registration makes things much better) would be bad for results.
posted by Pyry at 10:23 PM on May 27, 2008
Aeschenkarnos' suggestions 1,2, and 6 might make the game more fun, but would destroy the validity of the results for use in machine learning.
Your point is well-taken Pyry, but the game not being much fun does more to reduce the size and quality of the data set than collusion would. Also, collusion along the lines of banana-banana is relatively easy to detect, and as von Ahn explains in his video, results get confirmed against each other a lot. You'd have to banana-ize for weeks straight (or get a goon swarm to do so, which I admit wouldn't be hard) to substantially affect the results.
posted by aeschenkarnos at 1:59 AM on May 28, 2008
Your point is well-taken Pyry, but the game not being much fun does more to reduce the size and quality of the data set than collusion would. Also, collusion along the lines of banana-banana is relatively easy to detect, and as von Ahn explains in his video, results get confirmed against each other a lot. You'd have to banana-ize for weeks straight (or get a goon swarm to do so, which I admit wouldn't be hard) to substantially affect the results.
posted by aeschenkarnos at 1:59 AM on May 28, 2008
Shall we play a game?
posted by dasheekeejones at 3:50 AM on May 28, 2008
posted by dasheekeejones at 3:50 AM on May 28, 2008
this ESP game is immensely frustrating. first off i get paired with some dude who actually guess velocaraptor (misspelled and all) with me and that kind of stuff all the way through, then i play with someone who makes one terrible, terrible guess every round, then forgets about it for 30seconds before clicking pass or guessing one obscure word in the picture.
would be more interesting if i could see what my partner guessed, and even more so if i could berate them afterwards for having such an undeveloped creative mind. maybe it's a computer after all and this is a turing test training programme >:|
posted by Dillonlikescookies at 4:39 AM on May 28, 2008
would be more interesting if i could see what my partner guessed, and even more so if i could berate them afterwards for having such an undeveloped creative mind. maybe it's a computer after all and this is a turing test training programme >:|
posted by Dillonlikescookies at 4:39 AM on May 28, 2008
You can pretty much tell when they pair you with recordings of previous guessers. The timing's off. They often guess before you've finished a clue that they couldn't possibly have guessed that from (eg, word is "field", your clue is "magnet aura", their guess is "pasture"). They often appear to not really respond as such, just randomly guess stuff until they suddenly get it right. I think this accounts for a lot of the word repetition, too.
posted by aeschenkarnos at 5:42 AM on May 28, 2008
posted by aeschenkarnos at 5:42 AM on May 28, 2008
aeschenkarnos: According to Luis von Ahn's CV, he never worked for Google. According to his Wikipedia page, ESP was licensed to Google, and became Google Image Labeler.
I imagine the registration requirement, as others have pointed out, is meant to help the validity of the data collected. I agree that it is somewhat problematic from a user's perspective, though.
posted by zpaine at 9:21 AM on May 28, 2008
I imagine the registration requirement, as others have pointed out, is meant to help the validity of the data collected. I agree that it is somewhat problematic from a user's perspective, though.
posted by zpaine at 9:21 AM on May 28, 2008
Zpaine: my mistake, noted. :)
I imagine the registration requirement, as others have pointed out, is meant to help the validity of the data collected. I agree that it is somewhat problematic from a user's perspective, though.
Username to identify you, so it and you can keep track of your actions; password to reserve your username, and stop others from interfering with it; email address in case you lose your password and need it reset. That's all that's needed. I think that might be the setup as it stands, actually.
I wonder if this is in anyway trademarked or copyrighted or whatever ... ie, if someone were inclined to think they could do the same thing but better, could they do so without legal problems? :)
posted by aeschenkarnos at 4:22 PM on May 28, 2008
I imagine the registration requirement, as others have pointed out, is meant to help the validity of the data collected. I agree that it is somewhat problematic from a user's perspective, though.
Username to identify you, so it and you can keep track of your actions; password to reserve your username, and stop others from interfering with it; email address in case you lose your password and need it reset. That's all that's needed. I think that might be the setup as it stands, actually.
I wonder if this is in anyway trademarked or copyrighted or whatever ... ie, if someone were inclined to think they could do the same thing but better, could they do so without legal problems? :)
posted by aeschenkarnos at 4:22 PM on May 28, 2008
« Older Smoke and mirrors | Dr. Ernst Stuhlinger, 1914-2008 Newer »
This thread has been archived and is closed to new comments
reCAPTCHA on MeFi
posted by carsonb at 4:59 PM on May 27, 2008