Faceted hierarchy as killer app
December 5, 2004 5:26 PM Subscribe
Recently we've all been thinking about flat (or better, faceted) hierarchy web apps that organize email, photos, bookmarks, and general knowledge. The common threads are metadata (tags, categories, labels) that enrich relationships within and hence searchability of large collections. But besides marketroid hype (buzzwords, snark) and a computer that plays Twenty Questions what else can we do and study using faceted data structures: searchable culture references in The Simpsons, library science, computer filesystems, A.I. development, models for human memory and cognition?
Personally I think the whole metadata thing is a bit overblown. This summer I worked for IBM as an intern, and around 90% of the projects involved metadata in some way. Some of them were pretty useful, but for the majority the metadata didnt really accomplish all that much. The problem with metadata is that it has to come from somewhere. Unless we all magically become really good librarians, we're going to be as bad at organizing things with metadata as we were without metadata. But, it's a loss because we can't blame the lack of metadata any more. It's an interesting field, but in a few years the word "ontology" will fade to obscurity again and we'll be talking about "push vs pull" again or something.
This response could be the result of me just spending a few days cleaning up the metadata of my music collection. People I illegally download music from should really do a better job of organizing things!
Also, dude, way too many links. You need a pretty complex ontology to describe how the various links you gave actually relate to each other.
posted by JZig at 5:50 PM on December 5, 2004
This response could be the result of me just spending a few days cleaning up the metadata of my music collection. People I illegally download music from should really do a better job of organizing things!
Also, dude, way too many links. You need a pretty complex ontology to describe how the various links you gave actually relate to each other.
posted by JZig at 5:50 PM on December 5, 2004
Can you wait about six hours? I'm just now finishing up a major article on this very topic and its importance to the next generation(s) of Web services, riffing mostly on Ben Cerveny's presentation to Design Engaged a few weeks ago, called "The Growth of the Soil."
But, in brief, the answer is just about anything, and with surprising, even disturbing richness.
posted by Adam Greenfield at 5:50 PM on December 5, 2004
But, in brief, the answer is just about anything, and with surprising, even disturbing richness.
posted by Adam Greenfield at 5:50 PM on December 5, 2004
Oh yeah, and I predict that the next wave in spam will be Metadata spam. Similar to how blogspam is the new and exciting thing, I'd bet $10 that in a few years we'll be talking about how hard it is to weed out all the penis enlargement ad labels people attach to things when we search for relationships.
posted by JZig at 5:55 PM on December 5, 2004
posted by JZig at 5:55 PM on December 5, 2004
JZig, I totally, totally disagree. Consider the role of collaborative, bottom-up "folksonomies" (it's Thomas Vander Wal's word) in helping a community of strangers make sense of wildly disparate and heterogeneous data streams on del.icii.ous, or, for that matter, the success of Flickr.
Nobody needed to explain faceted classification to the users, in either case, and nobody had to develop elaborate taxonomies or controlled vocabularies. The taxonomies, in either case, accreted out of the tags people applied to their own data, and frequently enough that of others as well. The result, for me, is an astonishing degree of insight into how other people conceive of the things they see and share - the differences in level of granularity, in perspective, in the relative importance of one aspect over another.
I really think metadata will be determinative to the character of a lot of what's coming, I think Ben did a great job of imagining how, and I'm struggling to convey the elegance of his insight in the piece I'm (supposed to be) working on at this very moment.
posted by Adam Greenfield at 5:56 PM on December 5, 2004
Nobody needed to explain faceted classification to the users, in either case, and nobody had to develop elaborate taxonomies or controlled vocabularies. The taxonomies, in either case, accreted out of the tags people applied to their own data, and frequently enough that of others as well. The result, for me, is an astonishing degree of insight into how other people conceive of the things they see and share - the differences in level of granularity, in perspective, in the relative importance of one aspect over another.
I really think metadata will be determinative to the character of a lot of what's coming, I think Ben did a great job of imagining how, and I'm struggling to convey the elegance of his insight in the piece I'm (supposed to be) working on at this very moment.
posted by Adam Greenfield at 5:56 PM on December 5, 2004
JZig: You're right about metadata spam, though. That metadata can be gamed should be apparent to anyone. (Why do you think search engines ignore META tags?)
But, yeah, back to that article.
posted by Adam Greenfield at 6:01 PM on December 5, 2004
But, yeah, back to that article.
posted by Adam Greenfield at 6:01 PM on December 5, 2004
Jzig - concerning metadata spam: a fascinating and timely observation.
posted by fatllama at 6:01 PM on December 5, 2004
posted by fatllama at 6:01 PM on December 5, 2004
Meh. It will get better. jimfl's link was pretty interesting. I've heard of other methods of using the Bayesian filter for content filtering, and each time, well.. I was not that enthusiastic.
That being said, it is still necessary to find an automated way to do it. Having worked with a number of systems that rely solely on user input I'm strongly convinced that we have to take any amount of thinking out of the content labelling or else it just will not work.
People are lazy, prone to cut corners, or just plain get things wrong. Even dynamically-created metadata is based upon content within the article, report, site etc. XML offers some possibility of creating a formulaic approach to this sort of problem but it's lightyears away from wide use due to the fact that most people cannot understand it or realize any way that it could help in this particular instance.
But, I'm still optimistic. Software has no way to go except improve and I, for one, welcome our new software-based overlords.
posted by purephase at 6:02 PM on December 5, 2004
That being said, it is still necessary to find an automated way to do it. Having worked with a number of systems that rely solely on user input I'm strongly convinced that we have to take any amount of thinking out of the content labelling or else it just will not work.
People are lazy, prone to cut corners, or just plain get things wrong. Even dynamically-created metadata is based upon content within the article, report, site etc. XML offers some possibility of creating a formulaic approach to this sort of problem but it's lightyears away from wide use due to the fact that most people cannot understand it or realize any way that it could help in this particular instance.
But, I'm still optimistic. Software has no way to go except improve and I, for one, welcome our new software-based overlords.
posted by purephase at 6:02 PM on December 5, 2004
Actually, I agree with you Adam. I'm more used to the more explicit taxonomies that are in vogue in AI and business. The basic idea behind these systems is that if you get the taxonomy right, everything else is trivial. This is basically false. However, the loosely structured systems like flickr seem to be a much better conception of how we, as humans, actually build relations. I'm definitely interested in the field (I hope to do some work on it when I enter grad school), but I think some people have unrealistically high expectations of it. In particular, I really want to see how the semantic web stuff works out. The only experience I have with semantic web specifically is Haystack which failed to impress me (you basically have to hard code everything it says you dont have to hard code). That said, I'm largely just being contrarian, I've had too many of my friends rave to me about how semantic technologies will solve ever problem ever.
posted by JZig at 6:11 PM on December 5, 2004
posted by JZig at 6:11 PM on December 5, 2004
Isn't the hierarchy just a kind of metadata anyway? I mean, when you organize your email into folders, the location is metadata. It's hard to shoehorn multiple pieces of metadata into the location metaphor, granted, but it's not as if changing the dominant metaphor means that for the first time, metadata will be widely employed.
posted by kenko at 6:18 PM on December 5, 2004
posted by kenko at 6:18 PM on December 5, 2004
The tyranny of metadata and the tyrants who wish for our enslavement.
Users should not have to deal with metadata. That is why Berners-Lee said the web is a failure, as users have to deal with metadata such as "http" and "www" .. his vision was to have a human language interface, which is why Google is so popular, type in a few words and arrive at your destination.
posted by stbalbach at 7:13 PM on December 5, 2004
Users should not have to deal with metadata. That is why Berners-Lee said the web is a failure, as users have to deal with metadata such as "http" and "www" .. his vision was to have a human language interface, which is why Google is so popular, type in a few words and arrive at your destination.
posted by stbalbach at 7:13 PM on December 5, 2004
Huh?
posted by Adam Greenfield at 7:22 PM on December 5, 2004
posted by Adam Greenfield at 7:22 PM on December 5, 2004
Seems to me that running a Bayesian filter on the comments people have written about an RSS item would be the best bet. To use Mefi as an example, items whose conversations contain a dozen [this is good]s would be flagged "interesting" and posts with lots of "double post!" comments would go away (assuming your filter is so trained, of course).
I don't know if it would be able to categorize news/business/etc., but you could improve the ordering of items in an aggregator over what we have today.
If only there were a reliable way to get to the conversation from an RSS/Atom stream!
posted by jewzilla at 7:48 PM on December 5, 2004
I don't know if it would be able to categorize news/business/etc., but you could improve the ordering of items in an aggregator over what we have today.
If only there were a reliable way to get to the conversation from an RSS/Atom stream!
posted by jewzilla at 7:48 PM on December 5, 2004
kenko - Yep. I find, however, that all the interesting cases exist between the two obvious extremes:
posted by fatllama at 8:01 PM on December 5, 2004
- Rigid parent/child hierarchy - Each node can be connected to at most two other nodes (and no fewer than one other node), all arrows are directed, all arrows imply the same relationship.
- Messy brainstorm - No restrictions on node connectivity, directed and non-directed arrows, no a priori universal designation of what relationship an arrow implies, etc.
posted by fatllama at 8:01 PM on December 5, 2004
Nice post, fatllama.
(as purephase mentions) I find that I am far too lazy to (personally) fully utilize metadata to an extent that it becomes useful (in regards to my own data). Instead of "tagging" data accordingly, I simply dump it into separate piles (in various locales). I try to be organized (for example: on flickr/del.icio.us), but the result is generally not worth the effort required (At least for me). My system makes sense to me and is streamlined to my workflow but it's pretty useless to others. That said, I am much more "organized" than I was a decade ago (datawise) so perhaps there will come a time when software will meet me halfway.
posted by shoepal at 8:24 PM on December 5, 2004
(as purephase mentions) I find that I am far too lazy to (personally) fully utilize metadata to an extent that it becomes useful (in regards to my own data). Instead of "tagging" data accordingly, I simply dump it into separate piles (in various locales). I try to be organized (for example: on flickr/del.icio.us), but the result is generally not worth the effort required (At least for me). My system makes sense to me and is streamlined to my workflow but it's pretty useless to others. That said, I am much more "organized" than I was a decade ago (datawise) so perhaps there will come a time when software will meet me halfway.
posted by shoepal at 8:24 PM on December 5, 2004
I find discussions of categorization and taxonomy totally fascinating, btw, ever since taking a class on the concepts of resemblance and family resemblance in which we had to read, among other things, Goethe on the Urpflanze, Thomas Galton on composite photography, and Wittgenstein on seeing-as. Sadly I never wrote the paper for it. It seems like an area with a lot of potential for very interesting cross-disciplinary study. (Seemingly unconnected: the smartest man alive, J. Z. Smith, despite being ostensibly a student of religion, reads biological taxonomical journals.)
I think we've learned that there can be an awful lot of content in connectivity
I think we've learned that there can be an awful lot of content in connectivity
"And so the chorus points to a secret law" one feels like saying to Frazer's collection of facts. I can represent this law, this idea, by means of an evolutionary hypothesis, or also, analogously to the schema of a plant, by means of the schema of a religious ceremony, but also by means of the arrangement of its factual content alone, in a 'perspicuous' representation.posted by kenko at 8:32 PM on December 5, 2004
What a great post! I need to synthesize all this.
Thank you fatllama!
posted by xammerboy at 8:37 PM on December 5, 2004
Thank you fatllama!
posted by xammerboy at 8:37 PM on December 5, 2004
WWBT?
posted by Cryptical Envelopment at 8:47 PM on December 5, 2004
posted by Cryptical Envelopment at 8:47 PM on December 5, 2004
Great links, thanks, fatllama! I need to synthesise too ...
posted by carter at 9:10 PM on December 5, 2004
posted by carter at 9:10 PM on December 5, 2004
Users should not have to deal with metadata. That is why Berners-Lee said the web is a failure, as users have to deal with metadata such as "http" and "www" .. his vision was to have a human language interface, which is why Google is so popular, type in a few words and arrive at your destination.
posted by stbalbach at 10:13 PM EST on December 5
WHAT?
Tim Berners-Lee is a major proponent of the semantic web, which is all about metadata!
http:// and www.. etc.--protocols and standards--are all major parts of that sort of thing.
posted by Firas at 10:25 PM on December 5, 2004 [1 favorite]
posted by stbalbach at 10:13 PM EST on December 5
WHAT?
Tim Berners-Lee is a major proponent of the semantic web, which is all about metadata!
http:// and www.. etc.--protocols and standards--are all major parts of that sort of thing.
posted by Firas at 10:25 PM on December 5, 2004 [1 favorite]
Yeah but he's not a proponant of users having to interface with metadata.
posted by stbalbach at 11:35 PM on December 5, 2004
posted by stbalbach at 11:35 PM on December 5, 2004
It's what emerges from the welter of metadata that is important. In other words, the forest, from the trees.
posted by stavrosthewonderchicken at 1:02 AM on December 6, 2004
posted by stavrosthewonderchicken at 1:02 AM on December 6, 2004
Managing metadata seems an interesting challenge. A rich system becomes richer with better metadata and it seems clear that improving metadata quality gets complicated as the burden of entry and management is pushed to the scope of the user interface.
A weird thought: One way to encourage metadata contributions could be to have the entry more resemble play than work. Perhaps if metadata entry were tied explicitly to a customization result that was interesting - a theme or something visual. I dunno.
At last week's W3C Advisory Committee, Mr. Berners-Lee was emphasizing the importance of metadata (i.e. semantic web) and its place as a machine-generated system - whenever he was asked about it he kept coming back to the point that users shouldn't have to see the base XML - they should benefit and contribute via gentler handshakes.
The problem with metadata is that it has to come from somewhere.
True, but metadata, like social networking, is becoming a feature of participatory systems without the users having to do anything. For example there's Movable Type feeds, TypePad feeds, Blogger feeds, Wordpress feeds, Yahoo! News feeds, Yahoo! Calendar feeds, Flickr feeds, MSN Music feeds, iTunes feeds, etc, etc.
posted by massless at 3:31 AM on December 6, 2004
A weird thought: One way to encourage metadata contributions could be to have the entry more resemble play than work. Perhaps if metadata entry were tied explicitly to a customization result that was interesting - a theme or something visual. I dunno.
At last week's W3C Advisory Committee, Mr. Berners-Lee was emphasizing the importance of metadata (i.e. semantic web) and its place as a machine-generated system - whenever he was asked about it he kept coming back to the point that users shouldn't have to see the base XML - they should benefit and contribute via gentler handshakes.
The problem with metadata is that it has to come from somewhere.
True, but metadata, like social networking, is becoming a feature of participatory systems without the users having to do anything. For example there's Movable Type feeds, TypePad feeds, Blogger feeds, Wordpress feeds, Yahoo! News feeds, Yahoo! Calendar feeds, Flickr feeds, MSN Music feeds, iTunes feeds, etc, etc.
posted by massless at 3:31 AM on December 6, 2004
.. which is why Google is so popular, type in a few words and arrive at your destination.
Even Google ultimately fails with this approach since it relies on user input (the search bar). A user must still subscribe to an RSS or Atom feed, and a user must still enter www. for any number of websites (a problem that any good webserver operator should fix - but most don't).
I think the general idea is that the current paradigm of a passive internet (or application) is what is at fault. I'm reminded of a number of good web application design books that remind us that a user is the number one problem with any interface, design, application, data entry etc.
Imagine a browser (or another application) that adapts to your needs. You read a story and it fills out the page with associated links to non-biased (or competing) sites containing further information on what your looking at. Take it even further to use adaptive forms in web/standalone applications (Firefox has a neat way of handling it in my opinion) but standardization would go a long way to help.
Anyway, the possibilities are there and I would imagine that there are a number of companies working on this issue at this moment. I know I am.
posted by purephase at 4:42 AM on December 6, 2004
Even Google ultimately fails with this approach since it relies on user input (the search bar). A user must still subscribe to an RSS or Atom feed, and a user must still enter www. for any number of websites (a problem that any good webserver operator should fix - but most don't).
I think the general idea is that the current paradigm of a passive internet (or application) is what is at fault. I'm reminded of a number of good web application design books that remind us that a user is the number one problem with any interface, design, application, data entry etc.
Imagine a browser (or another application) that adapts to your needs. You read a story and it fills out the page with associated links to non-biased (or competing) sites containing further information on what your looking at. Take it even further to use adaptive forms in web/standalone applications (Firefox has a neat way of handling it in my opinion) but standardization would go a long way to help.
Anyway, the possibilities are there and I would imagine that there are a number of companies working on this issue at this moment. I know I am.
posted by purephase at 4:42 AM on December 6, 2004
stalbach: Hard to see if maybe your sarcasm tags got stripped out, but I don't quite get how metadata is "tyranny." Not inherently, at least. Metadata is -- it's something we do naturally, given the opportunity. And as Adam pointed out, if people are left to their own devices with it, some interesting results can emerge. Furthermore, it's really the only way to capture people's idea of the value of what they're linking to.
Googleism and Del.isio.us-ness represent two divergent paths on the road to discerning value. Of the two, Google will continue to be more successful in terms of capturing the public imagination and harvesting the fruits of its market share, because it creates the impression of greater comprehensiveness.
But of the two, del.icio.us is the one that actually manages to capture people's own estimation of value. Google, while its partisans claim otherwise, does nothing of the sort: It merely records linking. Without the "metadata" of the link (e.g., whether the fact of linking is meant to express something bad or good about the target -- consider the googlewhack), there is no actual value implicit in the fact that it exists. (Well, aside from the fact that it was "noticeable" enough to get linked in the first place. Which isn't actually saying very much.)
Where it starts to get really interesting is when you apply googlesque searching techniques to a crawl of the del.icio.us repository.....
posted by lodurr at 5:43 AM on December 6, 2004
Googleism and Del.isio.us-ness represent two divergent paths on the road to discerning value. Of the two, Google will continue to be more successful in terms of capturing the public imagination and harvesting the fruits of its market share, because it creates the impression of greater comprehensiveness.
But of the two, del.icio.us is the one that actually manages to capture people's own estimation of value. Google, while its partisans claim otherwise, does nothing of the sort: It merely records linking. Without the "metadata" of the link (e.g., whether the fact of linking is meant to express something bad or good about the target -- consider the googlewhack), there is no actual value implicit in the fact that it exists. (Well, aside from the fact that it was "noticeable" enough to get linked in the first place. Which isn't actually saying very much.)
Where it starts to get really interesting is when you apply googlesque searching techniques to a crawl of the del.icio.us repository.....
posted by lodurr at 5:43 AM on December 6, 2004
Purephase: But what you describe is ultimately a very passive experience. You're trading the application-level passivity of the web for human-level passivity.
In an active-browser scenario, such as the one you describe, we end up taking what the browser gives us. Even if we reject its choices, we're following its decision trees. And even if it alters it's "tastes" based on our input, it's still all ultimately driven by the browser's facile algorithms, which have had a few hundred million fewer years to develop than ours.
Humans have been acting on the information resrouce that is the world -- forever. We'll be better than software at finding what we want, given a well-organized data field, for a long, long time. Success in that area comes in the way of small enhancements, such as helping us recast the data field in different ways. Again, del.icio.us provides a good example of a way to do that with little algorithmic effort. The "googlebrowsers" that people were playing around with a few years ago were another interesting way. We're just at the beginning of this, and AFAICS, browsers that automatically surf for links similar to what we're reading might be more likely to get in the way of progress than assist it -- in that they would tend to provide, as google does, the false impression of accuracy and comprehensiveness. (80:20 on the scale of people's perceived needs, as it were.)
The problem is that market-successful implementations tend to define people's future expectations, and tend to define what people will expend money and effort developing in the future. Open Source ameliorates some of that problem, of course, but it continually remains to be seen (since it's all becoming, not static being) how much impact it will have.
posted by lodurr at 5:57 AM on December 6, 2004
In an active-browser scenario, such as the one you describe, we end up taking what the browser gives us. Even if we reject its choices, we're following its decision trees. And even if it alters it's "tastes" based on our input, it's still all ultimately driven by the browser's facile algorithms, which have had a few hundred million fewer years to develop than ours.
Humans have been acting on the information resrouce that is the world -- forever. We'll be better than software at finding what we want, given a well-organized data field, for a long, long time. Success in that area comes in the way of small enhancements, such as helping us recast the data field in different ways. Again, del.icio.us provides a good example of a way to do that with little algorithmic effort. The "googlebrowsers" that people were playing around with a few years ago were another interesting way. We're just at the beginning of this, and AFAICS, browsers that automatically surf for links similar to what we're reading might be more likely to get in the way of progress than assist it -- in that they would tend to provide, as google does, the false impression of accuracy and comprehensiveness. (80:20 on the scale of people's perceived needs, as it were.)
The problem is that market-successful implementations tend to define people's future expectations, and tend to define what people will expend money and effort developing in the future. Open Source ameliorates some of that problem, of course, but it continually remains to be seen (since it's all becoming, not static being) how much impact it will have.
posted by lodurr at 5:57 AM on December 6, 2004
fatllama: I agree that all the recent buzz is a good thing, but the way you word the first sentence of your post implies that there's a logical connection between freetags and faceting which I don't think is there right now. Not that freetag systems aren't or can't be faceted, just that it's not explicit.
Which, as Chris starts off on, isn't necessarily a bad thing. While sure, there are ways to create assistive tagging interfaces, there's metric ton-loads of really obvious implicit metadata that's available within the content, the collections, and yes the tags that can be sliced and diced by a number of readily available (libraries out there, no Ph.D required) ways. Lots lower hanging.
Some interesting stuff I've been looking at recently:
* Went to a Tufte lecture that got me thinking again on processing of multi-variate data. While he dismisses the screen, it loses by... at least two magnitudes from a resolution perspective, the capabilities you gain in motion and acceleration across luminance, chrominance and shape I think hasn't been really explored on the web and can do a lot to offset data density lost in pure resolution.
* a mini back-and forth w/ a friend. His latest post, infornography, references Namesys's ReiserFS whitepaper, which among other things, describes transparently merging faceted hierarchies
* a recent discussion on taxonomies with a friend got me pointed to Ivan Krsul's Software Vulnerability Ph.D Thesis which spends the first few chapters discussing taxonomies and classification systems (definitely a toread in my copious free time (or more likely, the next time I'm procrastinating on real work, like now)
For those w/ an interest, clicking through Citeseer will lead to an almost infinite amount of good stuff. To throw some more links onto the fire, here are a couple loose articles n things I've found useful when relating the new-fangled 'emergent' km/kr stuff to more formal models:
* What are the differences between a vocabulary, a taxonomy, a thesaurus, an ontology, and a meta-model? - short summary
* Ontology, Metadata, and Semiotics - a look at the web with a semiotics bent
* Metadata? Thesauri? Taxonomies? Topic Maps! - IS/LS view of subject
* Faceted Classification - all about formal FC
* Two Models of Facets - fascinating (imo) discussion
Here's a question to toss out. How does one pull predicates from freetags?
posted by lhl at 6:06 AM on December 6, 2004 [1 favorite]
Which, as Chris starts off on, isn't necessarily a bad thing. While sure, there are ways to create assistive tagging interfaces, there's metric ton-loads of really obvious implicit metadata that's available within the content, the collections, and yes the tags that can be sliced and diced by a number of readily available (libraries out there, no Ph.D required) ways. Lots lower hanging.
Some interesting stuff I've been looking at recently:
* Went to a Tufte lecture that got me thinking again on processing of multi-variate data. While he dismisses the screen, it loses by... at least two magnitudes from a resolution perspective, the capabilities you gain in motion and acceleration across luminance, chrominance and shape I think hasn't been really explored on the web and can do a lot to offset data density lost in pure resolution.
* a mini back-and forth w/ a friend. His latest post, infornography, references Namesys's ReiserFS whitepaper, which among other things, describes transparently merging faceted hierarchies
* a recent discussion on taxonomies with a friend got me pointed to Ivan Krsul's Software Vulnerability Ph.D Thesis which spends the first few chapters discussing taxonomies and classification systems (definitely a toread in my copious free time (or more likely, the next time I'm procrastinating on real work, like now)
For those w/ an interest, clicking through Citeseer will lead to an almost infinite amount of good stuff. To throw some more links onto the fire, here are a couple loose articles n things I've found useful when relating the new-fangled 'emergent' km/kr stuff to more formal models:
* What are the differences between a vocabulary, a taxonomy, a thesaurus, an ontology, and a meta-model? - short summary
* Ontology, Metadata, and Semiotics - a look at the web with a semiotics bent
* Metadata? Thesauri? Taxonomies? Topic Maps! - IS/LS view of subject
* Faceted Classification - all about formal FC
* Two Models of Facets - fascinating (imo) discussion
Here's a question to toss out. How does one pull predicates from freetags?
posted by lhl at 6:06 AM on December 6, 2004 [1 favorite]
Taxonomies don't work. Not only because people are lazy, as has been mentioned previously, but because they pigeon-hole content into a preconceived notion of the data's topography.
For example, the sales people here at Vivísimo use "the Michael Jackson test," because so often that query yields a cluster labeled, "child molestation." It's an appropriate category, but what taxonomy would ever include a label like that?
On the other hand, taxonomists may add categories that really aren't appropriate. Maybe ever person in the taxonomy has a "biography" sub-category, but not every person in the taxonomy has a sub-category. I run into this kind of problem with my Drupal site all the time. Some categories are very big--suggesting they should be further broken down into sub-categories--while others are tiny, or even empty. It doesn't reflect reality, it reflects my preconceived notions of reality.
You need an automated solution. Companies are spending millions of dollars developing taxonomies that reflect the taxonomist's ideas about reality, but not necessarily reality itself. They are also expensive to maintain, and inherently lag behind (it will take a while to add a new category for a breaking news story, won't it?). An automated solution eliminates the problem of laziness, is always up-to-date, and gives you a map of what the data really is, rather than what an editor thinks it should be. It also strikes a balance, allowing content the freedom to be what they are, and only imposing hierarchy at the last minute, for presentation's sake.
"Clustering" is exactly that, that's what Vivísimo does. It's a little niche we more or less own.
posted by jefgodesky at 6:36 AM on December 6, 2004
For example, the sales people here at Vivísimo use "the Michael Jackson test," because so often that query yields a cluster labeled, "child molestation." It's an appropriate category, but what taxonomy would ever include a label like that?
On the other hand, taxonomists may add categories that really aren't appropriate. Maybe ever person in the taxonomy has a "biography" sub-category, but not every person in the taxonomy has a sub-category. I run into this kind of problem with my Drupal site all the time. Some categories are very big--suggesting they should be further broken down into sub-categories--while others are tiny, or even empty. It doesn't reflect reality, it reflects my preconceived notions of reality.
You need an automated solution. Companies are spending millions of dollars developing taxonomies that reflect the taxonomist's ideas about reality, but not necessarily reality itself. They are also expensive to maintain, and inherently lag behind (it will take a while to add a new category for a breaking news story, won't it?). An automated solution eliminates the problem of laziness, is always up-to-date, and gives you a map of what the data really is, rather than what an editor thinks it should be. It also strikes a balance, allowing content the freedom to be what they are, and only imposing hierarchy at the last minute, for presentation's sake.
"Clustering" is exactly that, that's what Vivísimo does. It's a little niche we more or less own.
posted by jefgodesky at 6:36 AM on December 6, 2004
Hey, thanks all, for all the contention and knowledgeable POVs, it's certainly given me a lot to think about (and may, sigh, force a rewrite, but them's the breaks).
And massless, as long as you're kissing ass, remember, it's Sir Tim.
posted by Adam Greenfield at 8:02 AM on December 6, 2004
And massless, as long as you're kissing ass, remember, it's Sir Tim.
posted by Adam Greenfield at 8:02 AM on December 6, 2004
Taxonomies don't work.
Frozen taxonomies don't work. Dynamic taxonomies do. Every day. All over the place. Even in your own life.
posted by lodurr at 8:15 AM on December 6, 2004
Frozen taxonomies don't work. Dynamic taxonomies do. Every day. All over the place. Even in your own life.
posted by lodurr at 8:15 AM on December 6, 2004
There are three types of flat descriptive environments, personal, narrow social (narrow folksonomy), and broad social (broad folksonomy). Gmail is a personal self metadata tagging tool, Flickr is mostly a narrow folksonomy as one person tags the elements and shares them out (and one's contacts can also add metadata), and del.icio.us is broad as anybody can tag anything.
The predicates are easier to find in the broad context as one can use a variety of sources to get a broad and deep categorization of the item. The power laws can apply in these situations, but the spike is as usable as is the long tail. The nice thing is finding information can work no matter what discipline or cultural terminology is applied to the item.
The narrow folksonomy works well with in communities and for individuals to keep found information found. As others in the narrow folksonomy add what they believe to be relevant metadata the ability for others with the same vocabulary can find the items.
The personal tagging works well to Keep found things found and to associate items as they see a need. One e-mail can be for a client, a research project, and two very different articles.
To summarize: People call items different things depending on culture, discipline, and/or language. The folksonomy seems to be a way to find information based on what a person calls it. The network effect provides for more tagging of the information, which can be leveraged by those who have naming conventions that are divergent from the norm. The power law curve benefits the enculturated, but the tail of the curve also works for those out of the norm.
posted by vanderwal at 8:41 AM on December 6, 2004
The predicates are easier to find in the broad context as one can use a variety of sources to get a broad and deep categorization of the item. The power laws can apply in these situations, but the spike is as usable as is the long tail. The nice thing is finding information can work no matter what discipline or cultural terminology is applied to the item.
The narrow folksonomy works well with in communities and for individuals to keep found information found. As others in the narrow folksonomy add what they believe to be relevant metadata the ability for others with the same vocabulary can find the items.
The personal tagging works well to Keep found things found and to associate items as they see a need. One e-mail can be for a client, a research project, and two very different articles.
To summarize: People call items different things depending on culture, discipline, and/or language. The folksonomy seems to be a way to find information based on what a person calls it. The network effect provides for more tagging of the information, which can be leveraged by those who have naming conventions that are divergent from the norm. The power law curve benefits the enculturated, but the tail of the curve also works for those out of the norm.
posted by vanderwal at 8:41 AM on December 6, 2004
Frozen taxonomies don't work. Dynamic taxonomies do. Every day. All over the place. Even in your own life.
Dynamic taxonomies work because they use great expenditures of time, effort and money to approximate (usually badly) an automated solution like clustering. They work only as much as they approximate that functionality.
posted by jefgodesky at 8:49 AM on December 6, 2004
Dynamic taxonomies work because they use great expenditures of time, effort and money to approximate (usually badly) an automated solution like clustering. They work only as much as they approximate that functionality.
posted by jefgodesky at 8:49 AM on December 6, 2004
You're not exactly an unbiased observer, jefgodesky.
posted by Adam Greenfield at 10:18 AM on December 6, 2004
posted by Adam Greenfield at 10:18 AM on December 6, 2004
... work because they use great expenditures of time, effort and money ...
Probably in specific implementations, sure. But I had more of a general principle in mind. Like the taxonomies that I'm continually creating and modifying in my own mind. Those happen without any additional effort or expense, and are fully automated. They work pretty well, and don't have to approximate the functionality of anything else.
posted by lodurr at 11:11 AM on December 6, 2004
Probably in specific implementations, sure. But I had more of a general principle in mind. Like the taxonomies that I'm continually creating and modifying in my own mind. Those happen without any additional effort or expense, and are fully automated. They work pretty well, and don't have to approximate the functionality of anything else.
posted by lodurr at 11:11 AM on December 6, 2004
You're not exactly an unbiased observer, jefgodesky.
Never said I was. Actually I explicitly said I wasn't. But the subject matter is something I spend my days mired in, so I probably have a unique perspective on the matter.
But I had more of a general principle in mind. Like the taxonomies that I'm continually creating and modifying in my own mind. Those happen without any additional effort or expense, and are fully automated. They work pretty well, and don't have to approximate the functionality of anything else.
Oh, now this is interesting. What is the role of taxonomy in human cognition? I honestly don't know, I'd like to see some good information on that. But even that suffers from the fact that your brain cannot hold and impartially categorize all the data in it--your taxonomy reflects your preconceived notions. That it's continually, instantly updated gives it the same up-to-the-minute nature as an automated solution, but you still lack the impartiality of the machine to draw up an accurate map of the contextual topography.
posted by jefgodesky at 11:34 AM on December 6, 2004
Never said I was. Actually I explicitly said I wasn't. But the subject matter is something I spend my days mired in, so I probably have a unique perspective on the matter.
But I had more of a general principle in mind. Like the taxonomies that I'm continually creating and modifying in my own mind. Those happen without any additional effort or expense, and are fully automated. They work pretty well, and don't have to approximate the functionality of anything else.
Oh, now this is interesting. What is the role of taxonomy in human cognition? I honestly don't know, I'd like to see some good information on that. But even that suffers from the fact that your brain cannot hold and impartially categorize all the data in it--your taxonomy reflects your preconceived notions. That it's continually, instantly updated gives it the same up-to-the-minute nature as an automated solution, but you still lack the impartiality of the machine to draw up an accurate map of the contextual topography.
posted by jefgodesky at 11:34 AM on December 6, 2004
Well, "the impartiality of the machine" would be a bit of a misnomer, since the machine's decisions are not made on an impartial basis -- they're merely routinized and thus have the appearance of impartiality. And of course anyone's internally created taxonomies reflect their preconceived notions -- how could it be otherwise? Why would we want it to be?
I'm a bit resistent to terms like "folksonomy" not in least part because I think they trivialize something that's actually pretty profound and basic. After all, an anointed "taxonomy" really differs from a "folksonomy" only insofar as it's blessed by authority. Put another way: Today's "taxonomy" is probably yesterday's "folksonomy" (at least in part).
posted by lodurr at 11:54 AM on December 6, 2004
I'm a bit resistent to terms like "folksonomy" not in least part because I think they trivialize something that's actually pretty profound and basic. After all, an anointed "taxonomy" really differs from a "folksonomy" only insofar as it's blessed by authority. Put another way: Today's "taxonomy" is probably yesterday's "folksonomy" (at least in part).
posted by lodurr at 11:54 AM on December 6, 2004
That's actually very much my point, lodurr--it can't be otherwise. But an empty category can give emphasis to a distinction that does not really exist (e.g., races among Homo sapiens sapiens), just as a large category can obscure important distinctions (e.g., the differences between Christian conservatives and neoconservatives are often obscured because they're both "right-wing"). So obviously, this is not desirable; how can we ever be open to differing points of view when we mentally file them under such artificial headings?
The human brain is a very good taxonomist, though, capable of adapting the taxonomy very quickly and easily. But the earlier in the taxonomy a mistake is made, the more difficult it is to correct. Whatever notion we start with is very difficult to break, regardless of what contrary evidence there is. We'll simply begin stretching the taxonomy to fit more and more wildly divergent data, before we begin to consider that we may have gone wrong with the first few nodes. This happens in formal computer taxonomies all the time; when it happens with your dynamic mental model of the world, we call it "prejuidice," or sometimes "being closed-minded."
posted by jefgodesky at 12:04 PM on December 6, 2004
The human brain is a very good taxonomist, though, capable of adapting the taxonomy very quickly and easily. But the earlier in the taxonomy a mistake is made, the more difficult it is to correct. Whatever notion we start with is very difficult to break, regardless of what contrary evidence there is. We'll simply begin stretching the taxonomy to fit more and more wildly divergent data, before we begin to consider that we may have gone wrong with the first few nodes. This happens in formal computer taxonomies all the time; when it happens with your dynamic mental model of the world, we call it "prejuidice," or sometimes "being closed-minded."
posted by jefgodesky at 12:04 PM on December 6, 2004
lodurr - Today's *taxonomy* is probably yesterday's "folksonomy" (at least in part), is close, but it relies on hierarchy. The last 15 years or more I have watched people from outside a discipline or culture have problems finding information because the vocabulary was usable for them or the structure was unknown. This has happened within the same organization with different departments having communication difficulties as well as professional and client relationships. I have watched the exact same thing happen on the web in the nine years I have been professionally involved with the web. I have built systems that have incorporated systems that had crosswalks between the formal taxonomies for cross-discipline information sites so that expertise from many disciplines could be leveraged. In this manner taxonomies have failed people who are not from the indoctrinated group.
Web search began to change some of this. While many sites have a top-down approach to structure (many built by trying to incorporate bottom-up understanding) the rest of the web, which is external to that site is still bottom-up in structure. Google really made this apparent through its use of external links and the vocabulary in those links being used to augment the site's vocabulary. One could find a site about football even if they were calling it soccer and the site never used the word soccer. This is just a simple example as there are emergent names bubbling up all the time and cultural interests are changing as more of the world gets access to the web. Things are more complex than taxonomies let on.
The key is capturing what regular people call things. Capturing the vocabulary as it is applied. The simplification of the metadata input that del.icio.us and Flickr have provided have helped open the door for finding related information based on what people actually call something. This approach augmented with a thesaurus, which is difficult for terms with multiple definitions and for emergent vocabularies could provide results for people that are an improvement over what is currently done through the formal systems.
posted by vanderwal at 1:22 PM on December 6, 2004
Web search began to change some of this. While many sites have a top-down approach to structure (many built by trying to incorporate bottom-up understanding) the rest of the web, which is external to that site is still bottom-up in structure. Google really made this apparent through its use of external links and the vocabulary in those links being used to augment the site's vocabulary. One could find a site about football even if they were calling it soccer and the site never used the word soccer. This is just a simple example as there are emergent names bubbling up all the time and cultural interests are changing as more of the world gets access to the web. Things are more complex than taxonomies let on.
The key is capturing what regular people call things. Capturing the vocabulary as it is applied. The simplification of the metadata input that del.icio.us and Flickr have provided have helped open the door for finding related information based on what people actually call something. This approach augmented with a thesaurus, which is difficult for terms with multiple definitions and for emergent vocabularies could provide results for people that are an improvement over what is currently done through the formal systems.
posted by vanderwal at 1:22 PM on December 6, 2004
Whenever I try to organize or sort anything (like files in my home directory, or books on my shelf, or names in my address book, or ...) in any sort of multi-dimensional way I run into two problems with any organization scheme I try. The problems are sort of abstract and I can't talk about them. I suspect some math guys somewhere have already formally expressed them (Godel perhaps). However, I believe both problems are adequately expressed in the following phrase: "The filing system is blah, blah, blah, blah, blah, blah, blah, ..."
posted by wobh at 1:23 PM on December 6, 2004
posted by wobh at 1:23 PM on December 6, 2004
wobh, it sounds as if you may be running headfirst into the primary problem with faceted classification in the real world, which is that it's difficult for physical objects to occupy multiple locations simultaneously. ; . )
posted by Adam Greenfield at 2:43 PM on December 6, 2004
posted by Adam Greenfield at 2:43 PM on December 6, 2004
"A weird thought: One way to encourage metadata contributions could be to have the entry more resemble play than work."
massless, I think you're looking for the ESP game.
posted by revgeorge at 4:09 PM on December 6, 2004
massless, I think you're looking for the ESP game.
posted by revgeorge at 4:09 PM on December 6, 2004
wobh, the formal structures are failing you and the digital structures were quasi built on physical structures (hierarchy of folders). Softlinks, alias, and shortcuts are too messy and do not cut it either. The sorting patterns of music and books can be associative and contextual, grouped in various ways some are faceted by time, topic, genre. While others are purely associative by your our own patterns and traditions (I always listen to this two CDs back-to-back, or when we refer to this Lakoff book we always think of the quote from Microserfs so I keep them together. The join between these two may come down to one word, which we could store and retrieve if our systems for organizing allow for this.
Apple's next OS iteration should allow for this in Spotlight, if not fully, it should partially.
Should everything devolve down to flat categorical systems? No, not at all. The sciences and law have their well structured and taught hierarchal structures that work for these disciplines.
posted by vanderwal at 8:34 PM on December 6, 2004
Apple's next OS iteration should allow for this in Spotlight, if not fully, it should partially.
Should everything devolve down to flat categorical systems? No, not at all. The sciences and law have their well structured and taught hierarchal structures that work for these disciplines.
posted by vanderwal at 8:34 PM on December 6, 2004
« Older (Miao Miao Miao) | Early Celebrity. Newer »
This thread has been archived and is closed to new comments
posted by euphorb at 5:31 PM on December 5, 2004