Bad, but not THAT bad.

February 19, 2013 1:56 AM

Last month, we saw that Google Flu Trends was predicting doom for the U.S. Turns out, Google was wrong.

Estimates of flu activity in the US made by Google's algorithms overshot the percent of the population with flu at its peak by about 66%.

Google Flu may have been fooled by its own hype. Experts believe part of the problem was public fear influencing search behavior in an unusual way. Norovirus - an illness many mistakenly call "stomach flu" - may have also contributed to Google's snaFlu.

posted by OHSnap (34 comments total)

I think their estimate completely undershoot the incidence in my area. It's been a brutal season...
posted by anonymisc at 2:04 AM on February 19, 2013

He comments, "I suspect that passive monitoring of social media will always yield more data than systems that rely on people to actively respond to surveys, like Flu Near You."

I wonder if this would perhaps be a better way to elect public officials as well. A system that relies on people to actually cast votes seems so old-fashioned.
posted by three blind mice at 2:20 AM on February 19, 2013

I wonder if this would perhaps be a better way to elect public officials as well. A system that relies on people to actually cast votes seems so old-fashioned.

Only if nobody knew about it; otherwise the gaming of the system would start so quickly you'd see it yesterday.
posted by solarion at 3:00 AM on February 19, 2013

Er, no. There's a profound assumption that traditional sources of flu data represent ground truth, and any disagreement from that data must represent error. What's actually happening is that there is a source of error and we're not sure where it is. Traditional monitoring is dependent on physician reports, which is dependent on physician visits, which in a down economy are expensive. Not that Google doesn't have its own issues (increased severity, low cost of uninfected friends/family searching) either.

But all this crowing is silly.

Anecdotally, this seemed to be a particularly rough year with the flu.
posted by effugas at 3:10 AM on February 19, 2013

The way I heard it in Switzerland (from my partner) was simply that this year, they choose quite well which vaccine to use, and put a very good check against it. I'd have to ask him the details (was it about here, or the US or Europe) to know anything more.
posted by Goofyy at 3:12 AM on February 19, 2013

There's a profound assumption that traditional sources of flu data represent ground truth, and any disagreement from that data must represent error.

Except Google says right on the FAQ page: "Google Flu Trends models are validated using historic flu surveillance data." So Google is in fact saying that CDC data is the standard against which it judges itself.

Further, you argue that physician visits are expensive in a down economy, which threw off Google's model in 2012-13. Then how come Google's model worked so well 2009-11, which were also down economy years?

If in years past Google's model has been validated against CDC data, and Google has been lauded in the past by various sources for how well its model fit that data, then the only logical conclusion is that something happened this year to fool Google's model.
posted by OHSnap at 3:28 AM on February 19, 2013 [2 favorites]

OHSnap,

Again. The two signals have been similar in previous years, and are different now. If it turned out Google's model was actually *superior* in some years, it would look like this as well.

What we know is that this year's data is different, not at all that it is wrong.
posted by effugas at 3:49 AM on February 19, 2013

I was wondering if there was some positive feedback at work here, where awareness of the Google Trends flu model was prompting more googling to find out about the model, rather than looking for help with flu symptoms.
posted by carter at 4:17 AM on February 19, 2013 [1 favorite]

So what you're saying is, your google flu is failing you?
posted by Brodiggitty at 4:41 AM on February 19, 2013 [6 favorites]

One (probably minor) possibility is that people are actually, you know, reading up on the flu and doing something about it. Washing hands, minimizing exposure etc.

Be ironic if the Google searches were giving people useful information that reduces the spread of the disease.

If that is the case, it's a pretty simple fix - just drop the page rank of the useful stuff and front load with amusing Youtube videos. People will stop being sensible and the trends will match the projections again. Win!

(Sorry - just feeling a little cynical today)
posted by YAMWAK at 4:47 AM on February 19, 2013

Oh god, I just spent the last 7 days in flu jail tending to my 10-yr-old, who had a 104+ fever for at least five of those days. The last time he was that sick with the flu it was H1N1. This was pretty brutal. (I will say, though, that constantly washing my hands and basting myself in a thick coating of Purell at all times seems to have staved it off for me, despite the close and extended contact with Patient Zero.)
posted by mothershock at 4:57 AM on February 19, 2013

...a 104+ fever for at least five of those days...

As a parent of multiple kids at a wide range of ages, I'm now pretty relaxed when it comes to illness. But I think I still would have brought this kid to the emergency room by day 3.
posted by DU at 5:05 AM on February 19, 2013

Oh god, I just spent the last 7 days in flu jail tending to my 10-yr-old [...]

Says "mothershock".
posted by pracowity at 5:07 AM on February 19, 2013

DU: Oh yeah, I was more than a little concerned, especially since with H1N1 he ended up in the hospital. I was in touch with an ER pediatrician through the course of the week, who advised me that the high fever, though terrifying, was typical for what she's seen with kids and this flu this season, and that if the fever was responsive to Tylenol/Motrin, it was okay to just monitor and wait it out. What *would* necessitate a trip to the ER would be a change in mental status in the absence of fever, a stiff neck, petechiae, or if the fever didn't resolve after a few more days. Luckily it did resolve, and it didn't turn into pneumonia or anything else, and (disappointingly for him) he is now back at school. But yes: the super high fever was frightening.
posted by mothershock at 5:18 AM on February 19, 2013 [1 favorite]

I'm not sure how it was nation-wide, but I know in my area, a TON of people (including myself and my husband) caught something that was Not The Flu. Fevers, chills, fatigue, nausea, weakness, but we were both tested and it was not flu. Whatever it was, I couldn't walk for more than a week. I was vaccinated too, although I guess it could have been one of the less common strains.

I am not a scientist: does a standard flu test look for all strains, or just the CDC-designated most common ones?
posted by specialagentwebb at 5:21 AM on February 19, 2013

Quantum effects in flu observation? Measuring it changes it.
posted by blue_beetle at 5:36 AM on February 19, 2013

Maybe Google forgot that the first third of its data is sponsored ads?
posted by Thorzdad at 5:57 AM on February 19, 2013 [1 favorite]

There may be a technical difference between the flu and "flu-like symptoms", but not when you're the one with said symptoms.
posted by tommasz at 5:58 AM on February 19, 2013 [1 favorite]

That's the bad rap with credible prediction making. It causes enough people to immediately respond with preventions, then the result changes, and then the prediction is challenged as wrong.
posted by Brian B. at 6:11 AM on February 19, 2013

Quantum effects in flu observation? Measuring it changes it.
posted by blue_beetle at 8:36 AM on February 19 [+] [!]

I was thinking more like the "Decline Effect". [Trigger warning: RadioLab and Jonah Lehrer references]
posted by aught at 7:01 AM on February 19, 2013

I got the flu (or something flu-like, I wasn't tested) on January 18th and I'm only just starting to feel human again. And it's making its way around my office. Very unpleasant. I don't think I've ever had a fever over 102 as an adult before.
posted by elsietheeel at 8:01 AM on February 19, 2013

Then how come Google's model worked so well 2009-11

It worked "well" in those years compared to the CDC data; again, there's the assumption that the CDC data represents objective truth, which may or may not be correct. Google is using the CDC data to validate their model, but that doesn't necessarily mean that the CDC data is objectively correct.

This is a fairly hard problem in data modeling and prediction, when there's no way to directly measure the variable you're trying to model.
posted by Kadin2048 at 8:14 AM on February 19, 2013

On the one hand you have Google, a smart advertising company that for the past few years has been putting some part time resources on analyzing query trends to predict flu. On the other hand you have CDC, a 65 year old organization of scientists and doctors who has hundreds of people studying the flu every single year, trying to understand epidemic behavior. Collecting data from a variety of sources, mostly doctors and nurses on the ground looking at actual patients. Who is more likely to be correct?

I would love to understand how Google filters out their own positive feedback loop. This year there was a lot of online discussion about Google Flu Trends, and I have to assume more people searched Google for flu info because they heard Google said there was a lot of flu this year. But they're smart enough to understand that kind of bias at Google, I just wonder how they control for it.
posted by Nelson at 8:31 AM on February 19, 2013

specialagentwebb,

Oversimplification but:
You can find lots of information on flu testing on the CDC website but summarized the answer to your question is many hospitals use some sort of PCR test to detect the presence of influenza RNA. All of these tests will test for influenza A (the bad one, subtypes like H1N1 fall under this) and some test for influenza B. There is a third kind of influenza, creatively enough named influenza C, but it generally isn't as common or as serious and isn't tested for or vaccinated against.
Other hospitals may just use a (less accurate) rapidtest for a quick result, although the automated PCRs are getting really fast. Generally the CDC tells providers not to wait for a confirmatory flu test if a pt at risk for flu has the obvious symptoms and to go ahead and use their clinical judgement to start treatment (usually with an antiviral like tamiflu).
So the short answer to your question is it depends what test your provider ordered.

There is some evidence to suggest that (as mentioned in the OP) a high prevalence of norovirus or similar "stomach bug" infections may have made a lot of people sick who didn't have flu, although (at least in NYC) there was a lot more flu this year than last year.
posted by Wretch729 at 8:38 AM on February 19, 2013

Eh, I'm guessing it is mostly due to increased awareness. The CDC & all the news carried warnings/predictions about a severe flu season. So, chances are everybody who had (or knew someone who had) the slightest of symptoms (fever, headache, cold, aches etc, which are all very common) would do a related search online to see if it might be the flu. This probably ended up throwing off Googles algorithms.
posted by asra at 9:14 AM on February 19, 2013

This was a difficult read for me because sometimes doom casts a narrow shadow. Just last week I attended a life celebration party for a very good friend, the brother cited in my first MeFi post regarding Maureen Walsh. It started with the flu, then progressed to pneumonia, then ARDS and kidney failure, which led to his being paralyzed by the docs in hopes of stabilizing him. Yet, the next day he coded twice and was taken off life support on the subsequent day. Approximately ten days from start to finish. I've got the wrong kind of Blues just now.

Though I have generally poo-poo'ed flu shots in the past, I'll be getting one next year. And, as an aside, Maureen and I hugged and cried at the party. I asked her about her fifteen minutes of fame. She said, "I didn't know what it meant to go viral before, but I sure do now." Very pleased by all the support she received at that moment, some of which came from MetaFilter.
posted by CincyBlues at 9:36 AM on February 19, 2013 [1 favorite]

Well I caught the flu this year, which I haven't had for many many many years. I had been vaccinated 4 days before, dammit. I wasn't tested, just diagnosed. Still tired with a productive cough. I know here in NM, at least 89 people have died.

When I went to get vaccinated, a lot of places were running low, so I suspect more people got vaccinated than usual. That might have contributed to the lower numbers.

On preview, Cincyblues, I am so sorry about your loss. *HUG*
posted by annsunny at 9:41 AM on February 19, 2013

This is a perfect example of how a model can never contain itself. Any model that's part of the system it's modeling will change that system, rendering the model less and less accurate, depending on how much influence it has.

In other words, if only a few people knew about Google's flu trends, and they kept the results to themselves, then they would be fairly accurate. But the more widely known they become, the less predictive they become, as the modeled system changes because of the model's influence.

The model must then be updated to include its own effects on the thing that's modeled, which will have different effects on the target, which means it must be modified again, and again.... it becomes recursive.

This is part of why economics is such a terrible science, and why government attempts to manipulate economies are such a very, very bad idea, over the long term. Smart traders will be able to predict what governments will do, and will jump in front of the trades and make themselves very wealthy, at society's expense.
posted by Malor at 9:43 AM on February 19, 2013

Oh, cripes, after CincyBlues' comment, that sounds so tone-deaf.

I would absolutely not have gone anywhere near economics if I'd realized. Sorry, Cincy.
posted by Malor at 9:44 AM on February 19, 2013

Thanks, annsunny. Hugs right back.

Oh, cripes, after CincyBlues' comment, that sounds so tone-deaf.

That's okay, Malor. Macro and micro. Having done some grad work focusing on American economic history, I think what you have to say is relevant--perhaps even more so than my comment. Personally, I do tend to favor government intervention in the economy. I've had more than one surreptitious conversation in the past few weeks about just how my friend's spouse is going to pay for all that treatment--ICU, ventilators, etc... . There is some insurance, but...single payer, anyone?
posted by CincyBlues at 10:06 AM on February 19, 2013

I think the issue is that Google is located in the Silicon Valley and the engineers who might have tuned the algorithm were, along with most of the rest of the population, lying in darkened bedrooms nearly comatose except when they were trying to hack up a lung.

When you've got whatever has been running around the valley this year, dragging yourself into work to share good news about the flu season is probably low on the priority list.
posted by Tell Me No Lies at 10:25 AM on February 19, 2013

On the one hand you have Google, a smart advertising company that for the past few years has been putting some part time resources on analyzing query trends to predict flu. On the other hand you have CDC, a 65 year old organization of scientists and doctors who has hundreds of people studying the flu every single year, trying to understand epidemic behavior. Collecting data from a variety of sources, mostly doctors and nurses on the ground looking at actual patients. Who is more likely to be correct?

You think the answer is obvious. It isn't. Google's sample set is vastly larger than the CDC's and suffers much less selection bias, Google pretty much led a revolution in big data analysis, and 65 years of experience actually reduces the ability to change models (for much the same reason Nielsen's ratings don't reflect actual TV watching patterns -- to change the metrics is to invalidate differentials over time).

Put simply, you can be scientific, or you can think Google's data is obviously bunk, but you cannot do both.
posted by effugas at 6:20 PM on February 19, 2013

I don't know if I had The Flu, but whatever it was, it was viral and knocked me on the ass 3x over. Alternating weeks of exhaustion, mental fog, and bedrest; and being mobile and capable of working.

Didn't visit the doctor because (a) ain't nothing he can do and (b) fuck leaving my bed for that hassle.

Anecdotally, several "flus" ripped through my community, and many people reported them as gawdawful.
posted by five fresh fish at 11:36 PM on February 19, 2013

Also, I am increasingly convinced that we Westerners need to adopt Japan's facemask courtesy when ill. If sickos won't stay home, they can at least try to not spread their germs.
posted by five fresh fish at 11:38 PM on February 19, 2013 [1 favorite]