Aella, an independent researcher designs a giant survey about kinks

October 9, 2023 6:34 AM

A lot about how to design a survey to attract large numbers of people and hopefully get honest anwers.
posted by Nancy Lebovitz (20 comments total)

I'm very curious as to what the userbase of this site thinks of this. I've mostly encountered her work and discussions of it via people who are not fans of the rationalist community finding fault with rationalist adjacent people which is obviously prejudicial.

With regards to her specific example of vore fetishists as a demographic that's hard to reach, I'm curious - is there a term of art for sexual fetishists which are, I imagine, entirely based in fantasy? I imagine monstergirl fetishists are probably similar, in that the object of the attraction doesn't exist outside of art and fiction.
posted by Kikujiro's Summer at 7:04 AM on October 9, 2023

Now I'm curious about that, too, and wish you hadn't brought it up so soon. Rationalists like Aella generally speaking, but she seems to me to be off on her own path which doesn't involve the issues which cause a lot of mefites to dislike rationalists.
posted by Nancy Lebovitz at 7:44 AM on October 9, 2023

aella's not a researcher by any stretch, and a genuinely terrible person.
posted by jubal at 8:17 AM on October 9, 2023 [17 favorites]

“Independent researcher?” Sure, Jan.
Anyone with a keyboard and a lot of free time can do independent research.
posted by Ideefixe at 8:23 AM on October 9, 2023 [1 favorite]

Hmm. Kink and furry communities are peppered with people wanting to "study" them, to the extent that it's stock advice in many circles to ignore the stream of supposed psychologists and anthropologists parachuting in and demanding survey responses. There's a mixed history of how this data has been handled, too, from shoddy anonymization, to exploiting community goodwill to generate flashy-but-denigrating headlines. So this sort of thing raises hackles whenever I see it: it's disconcerting to have a large contingent of researchers view you and your friends like lab rats.
posted by Spinda at 8:31 AM on October 9, 2023 [6 favorites]

What does "rationalist community" mean in this case? My understanding of the term Rationalist is as an epistemological theory, the opposite of Empiricist, which doesn't seem to fit here.
posted by I-Write-Essays at 9:04 AM on October 9, 2023

From previous MeFi threads I am also suspicious of anything that tries to paint LessWrong as mostly positive, so my eyes narrowed a bit during this paragraph:

I’ve been saturated in the world of rationalists — an intellectual community formed around the group blog LessWrong — for eight years or so, and from watching them viciously tear apart other studies, I’d gained a sense of mistakes not to make: Don’t control for too many or too few things, do not trust your intuitions on how stats should work, don’t avoid reporting things that contradict your results, don’t check for new ways to investigate data just because it didn’t match your expectations, and report the null findings. I got odds-ratio-pilled and used those over p-values when I could; I published raw data; I shared my code; and I tried to keep my analysis and interpretation separate.

With recent threads and articles about dishonesty among dishonesty researchers, I've been wondering how anyone ever gets good data on anything, and maybe the answer is they don't. So the blog post otherwise looked relevant to my interests.

The milkshake duck backstory that jubal linked above paints a grimmer picture of just how much Aella might be aligned with the worst parts of LessWrong - lots of "just asking questions" there. Maybe this is not going to be something I want to quote after all!

Anyway I am interested to see what the rest of you dig up on this; these threads are typically eye-opening. So far my biggest realization for today is that maybe my kink is bad science and as the algorithms get better my chumbox feed is going to start promising Graphic Plots! • Twisted Statistics! • You won't believe what this data actually means! • What happened to that unbelievable science result? • They were a Stanford bigshot in the 90's, where are they now?
posted by pulposus at 9:05 AM on October 9, 2023 [4 favorites]

One Weird Old Trick That The IRB Doesn't Want You To Know!
posted by Halloween Jack at 9:20 AM on October 9, 2023 [15 favorites]

What does "rationalist community" mean in this case?

My understanding of the term is the same as yours, but that community seems to mean "we are super smart people who can justify our prejudices using reason, so there."
posted by joannemerriam at 9:20 AM on October 9, 2023 [8 favorites]

Roko's basilisk is my kink.
posted by star gentle uterus at 9:43 AM on October 9, 2023

Hmm. Kink and furry communities are peppered with people wanting to "study" them, to the extent that it's stock advice in many circles to ignore the stream of supposed psychologists and anthropologists parachuting in and demanding survey responses. There's a mixed history of how this data has been handled, too, from shoddy anonymization, to exploiting community goodwill to generate flashy-but-denigrating headlines. So this sort of thing raises hackles whenever I see it: it's disconcerting to have a large contingent of researchers view you and your friends like lab rats.

This, though my experience comes from fandom, asexual communities, and nonbinary trans communities. This looks to me like SurveyFail pt II in the making, except that unlike Ogas and Gaddam, she's not.... trained in any form of ethics... and doesn't even have the fig leaf of the pretense of IRB oversight...

It turns out that, when you get data, you have to figure out how to interpret it. Since I’m a homeschooler without a college degree, I started wrangling smart people to tell me how the hell a correlation worked, what p-hacking was, and how to do factor analysis. I’m a slow but thorough learner, and eventually got to the point where I was pretty solid with simple statistics.

But my datasets were getting huge, because my survey design was getting better at seducing people into taking them. My surveys, originally getting 10,000 responses, recently started to explode into the hundreds of thousands. You can’t process huge data in Google Sheets, so starting in 2021 I begrudgingly learned how to program in Python.

You can't see how widely my lips are peeled tight over my teeth through the screen, so I'll try to communicate the sheer level of my "oh fuck no" in words. Ye gods. This is the Tearoom Trade all over again--and that was an equally 'well-meaning' study, too. Say what you want about organized psychology, at least there's been more concentrated knowledge about what you absolutely should not do with research subjects there, and far more in the way of explicit training about research ethics, than any other field I've interacted with. They've made all the mistakes already, so we don't have to make them again! Unless, I suppose, you're coming in with a bright but self-taught person's excitement paired with naivety over the consequences.

The thing is, everything she's laying out is a known problem in the field of anyone trying to study sexuality--including marginalized sexuality!--right down to the problems with the genital measurement contraptions. The sampling problems are astounding, and the problems with people being willing to talk to you are worse. It's really hard to get people to talk honestly about sex, and it's even harder to get people to trust you without worrying about the repercussions of sharing for their communities--especially right now, and especially for trans folks. All of those are real problems, and her approaches to those problems aren't the worst I've ever seen--not even close, and not even if restricted to the sample set of people doing informal research within their own sexuality-based communities.

But holy crap she has no safeguards on her data protection or what anyone might do with that data. She has no one to enforce consequences if she does anything legally prohibited with that information. She doesn't even have a fucking corporate body overseeing her that can be sued (ala Facebook's reprehensible mood-altering feed studies). She's just some person drawing whatever conclusions, and doing whatever she wants, with that data, which is as secure or not secure as her own personal protocols. Yikes, yikes, yikes.

Even if I completely ignore the other information people are bringing here about what kind of person she is or isn't, and I give her the greatest possible benefit of the doubt, I would be telling any communities of mine she approached in the strongest possible terms to back the fuck away from her research until and unless there are institutional safeguards appended. Whatever conclusions she draws--and I agree, they sound interesting!--you cannot rely on her to treat your data according to normative international standards for safety and informed consent. Yikes.
posted by sciatrix at 10:07 AM on October 9, 2023 [22 favorites]

aella's not a researcher by any stretch, and a genuinely terrible person.

Yikes, you weren't kidding around jubal.

I'm guessing she was an avid reader of /r/gendercritical because she wanted to ask "rational questions" about trans people.

This is disturbing shit and it's troubling how many cis people are probably reading it thinking it's totally okay and acceptable.
posted by fight or flight at 10:39 AM on October 9, 2023

“Independent researcher?” Sure, Jan.
Anyone with a keyboard and a lot of free time can do independent research.

This is not literally the entire point of another substack sletter currently on the front page of the Blue or anything...

Ethics incredibly important, cannot be overstated, but also, I've been reading that particular newsletter since it started and I'm lowkey on board with Adam's general ethos of 'anyone with a keyboard and a lot of free time can do independent research and that's a good thing'. This is an interesting counterpoint.
posted by ngaiotonga at 11:55 AM on October 9, 2023 [2 favorites]

dont give rationalists a platform.
posted by AlbertCalavicci at 12:46 PM on October 9, 2023 [1 favorite]

I’m not familiar with this person. I DID have cause recently to sit on an ethics committee reviewing a different kink researcher’s project. That researcher clearly had struggled with some of these questions, came up with some similar and some very different ways to address them, and ultimately after some discussion and revisions I felt good about releasing that survey out into the world of the specific kink it was about. This as described here, I would not in a million years approve as is for the reasons sciatrix identified among others. Fucking yikes. Definitely a lot of really interesting methodological stuff to think and talk about but don’t actually do the survey like this!!!!

(I WOULD enjoy the process of listening to my ethics committee debate whether an “are you kinkier than SpongeBob” quiz result was appropriate compensation. But I don’t know if any of them are prepared to look me in the eyes yet after I had to explain what Fetlife was and why I felt the somewhat spicy (honestly, not that spicy) recruitment advertisements were appropriate to post there.)
posted by Stacey at 12:56 PM on October 9, 2023 [4 favorites]

I think some people have romantic ideas about the 18th and 19th century gentleman researchers on their country estates and forget the bad things about that era and those people and their work.

Also Aella seems to be an edge-lady.
posted by i_am_joe's_spleen at 1:42 PM on October 9, 2023

I think some people have romantic ideas about the 18th and 19th century gentleman researchers on their country estates and forget the bad things about that era and those people and their work.

I realize this isn't what is meant, but I was momentarily entertained by the image of 18th and 19th century gentleman researchers earnestly canvassing their servants on what gets them off and why.
posted by AdamCSnider at 3:06 PM on October 9, 2023 [3 favorites]

Now imagine it in fact not being earnest but with all the dubious motivation in the world and independent wealth and no restraints other than the local parson's not very inspiring sermon last Sunday.
posted by i_am_joe's_spleen at 3:16 PM on October 9, 2023

Aella's piece reads as full r/iamverysmart, and the link to her terribleness won't load, but I don't know enough about the details here to really parse whether what she's doing makes any sense. But, two observations:

1) like a lot of other profoundly un-kinky people, I find kink absolutely cringeworthy, and if she were to approach me and ask me to participate in her survey, I'd go all the way to the "look at that behind you!" and then run away when she turns around level, all in order to avoid having to answer questions about it—so how does she get anything like a representative sample of people?

2) I'm a lifelong hardcore atheist, and there's nothing worse than "rationalist communities", which are without exception in my experience chock-full of Dunning-Kruger level idiots and also libertarians. I'd honestly spend my day in one of those feed-the-homeless churches, even though I loathe everything about hippies.
posted by outgrown_hobnail at 3:56 PM on October 9, 2023

Mod note: A few deleted. Please keep the conversation on track and re-consider the space you may be taking up in threads. Also, the best way to have an unfavorable situation handled is to write to us via Contact form or utilize the flag feature.
posted by travelingthyme (staff) at 4:22 PM on October 9, 2023 [1 favorite]