And how effective is this program?

February 17, 2006 6:53 AM

The White House Office of Management and Budget has published a rundown on all federal programs called ExpectMore.gov, rating them as either "Performing" or "Not Peforming." (See Previous discussion on programs slated for budget cuts , in which some wondered about the analysis behind the cuts -- this is it.) 72% of all programs are rated Performing (of which 15% are Effective, 29% are Moderately Effective, and 28% are Adequate. 28% are rated Not Performing, but of these only 4% are Ineffective -- the other 24% are rated Results Not Demonstrated because of a lack of established goals or lack of collected data. Under each rating you can find all the programs in the category and drill down to details on the assessment For example, leading the "Ineffective" list is Even Start, a DOEducation program that is supposed to "the cycle of poverty and illiteracy for low-income families." But OMB says that three major studies of the program failed to show it was having any impact. Ditto AmeriCorps NCCC and the Oil Technology Program (it hasn't led to any new reserves being found). In case you are wondering, both the Dept. of Homeland Security's Domestic Icebreaking Program and its Biological Countemeasures program are considered Effective.
posted by beagle (27 comments total)

Another recognizable name on the list of ineffective programs: Amtrak. It's no surprise. Amtrak's more expensive than flying (often by a factor of 2 or more) and slower.
posted by majick at 7:02 AM on February 17, 2006

Keyword search for "Iraq" turns up 1 hit. Hmm.
posted by docgonzo at 7:08 AM on February 17, 2006

Curious. I wonder how one defines 'effective'.
posted by verb at 7:17 AM on February 17, 2006

Amtrak's more expensive than flying (often by a factor of 2 or more) and slower.

Actually that depends on how you time your trip. From start to finish the train is sometimes quicker thanks to airport checkin requirments, baggage claims, delays, out of city airport locations etc...
posted by srboisvert at 7:21 AM on February 17, 2006

Well, I don't think a picture of an Amtrak train's next to it in the dictionary, Verb. We've gone by train a couple of times to New Orleans - it's a good ride, but as Majick says it's slow. And you pay a premium. But it's a good way to start a vacation because you're not crammed into a flying Greyhound bus. There's a LOT more leg and seat room on a train, and the food in the Dining Car's better.

I'd like to see a better train infrastructure here in the US, but the economics just aren't there. Even with government subsidies.
posted by JB71 at 7:24 AM on February 17, 2006

Curious. I wonder how one defines 'effective'.
Em, the link to the definition is in the post, but here it is again.
posted by beagle at 7:26 AM on February 17, 2006

JB71, oh, I agree. There are definitely some real duds in that 'ineffective' list. There's just some curious stuff as well. Under 'effective,' the Army's Land Forces Operations are listed. One line item. Why is it effective? "The Army continuously refines its approach to training to improve the output and maximize its return on investment as it prepares for combat."

Okay, then. Effective it is!
posted by verb at 7:28 AM on February 17, 2006

From start to finish the train is sometimes quicker...

I can fly to Seattle in two hours, or take Amtrak and be there in two days. I've never had to wait two days to check in at the airport.

It might, possibly, maybe, be faster to take a train from here to Sacramento or something, but most folks just drive.

I'm given to understand the time/distance economy is a little different out on the east coast, but Amtrak is seriously worthless in the rest of the world.
posted by majick at 7:31 AM on February 17, 2006

Flying is often cheaper, yes, but the airlines have also received subsidies and pension plan buyouts and whatnot. And several have filed for bankruptcy protection due to having to cut fares below operational costs, or are just out of it with unclear futures, etc.

Meanwhile, FEMA disaster response is seen as adequate.

The OMB has had something of a rep in the past for being held captive by the Office of the Presidency, despite being established and still funded by Congress. It's why Congress established the Congressional Budget Office (CBO). I don't know what has been the agency's rep as of the past few years, but this list doesn't reassure me.
posted by raysmj at 7:39 AM on February 17, 2006

ExpectMore(corruptionandincompetencetocomplainabout).org might be a better domain name.
posted by scaryblackdeath at 7:45 AM on February 17, 2006

For Americorps, I think you really want this. It's ranked "Adequate", not "Ineffective." A detailed breakdown is here.
posted by lodurr at 7:47 AM on February 17, 2006

On a related note, one has to wonder who sets the criteria and makes the evaluations.
posted by lodurr at 7:48 AM on February 17, 2006

This list is more than a bit misleading. It'd be much more useful if the OMB provided a ROI (return on investment) index for these programs. Then we'd get to see how much resources these programs are consuming versus their effectiveness. Also, there needs to be a measure of risk incorporated into the index. It's quite silly that all the various DoD programs are rated as effective when everybody knows that military projects that stay on schedule and on budget are mythical creatures that only exist on paper. If the OMB incorporated allocation and risk into this picture we'd get a very, very different picture of the program situation.
posted by nixerman at 7:53 AM on February 17, 2006

On a related note, one has to wonder who sets the criteria and makes the evaluations.

it's http://www.whitehouse.gov/omb/

See that "whitehouse" in there? You know, the guys who won't let NASA scientists talk in public about global warming, the ones who were all "WMD" a few years ago - those guys? You can believe what you want, but I've gotta run down to the store and get another bag of salt.
posted by Kirth Gerson at 7:54 AM on February 17, 2006

Thanks for the correction on AmeriCorps, lodurr.
posted by beagle at 8:00 AM on February 17, 2006

Rating them as either "Performing" or "Not Peforming."

Since it's the Shitehouse doing the evaluations I expected every program to be rated, "Doing a heck of a job."

The "No program left behind" program seems to be failing.
posted by three blind mice at 8:21 AM on February 17, 2006

Golly, Kirth, you don't think that our Executive would lie to us, do you? <g/>

nixerman, I'm not sure how a ROI index would help. Actually, I think it would be worse. After all, you'd have to make some kind of a decision about how to map achievement of goals onto ROI, and ROI is really only comprehensible as a dollar value. How do you measure the ROI on, say, better primary education? Or, more problematic still, on something that's intended to do something inherently nebulous like build a sense of community or enhance a sense of safety in urban neighborhoods.

I'm not suggesting we shouldn't evaluate for effectiveness, but I am suggesting that the effort is inherently prone to abuse, most likely by those who set and evaluate against the criteria.
posted by lodurr at 8:24 AM on February 17, 2006

The most curious part of this website is why the OMB didn't add budget expenditures to the list. Seeing the return on investment quantified in some way would be nice, but the first cut would be to see how much is invested, and to index that with the effectiveness rankings. If the Administration was planning on making this website into a PR tool, why wouldn't they add those numbers to the mix? They'd gain the ability to say, "800 Billion in ineffective programs! 2 Trillion in programs without clear goals! We are are working to fix this, and what we can't fix, we'll cut!"

But they're not. So you gotta wonder why.
posted by suckerpunch at 8:29 AM on February 17, 2006

lodurr, the ROI index would simply take into account the total percentage of the federal budget that each program consumes. Program X may be 'Moderately Effective' and Program Y may be 'Adequate' but if X consumes 400 billion and Y consumes 400 million than relatively Y is a better investment. The system used by the OMB is biased towards military programs because it fails to take into account the enormous budgets and risks involved in most military programs. (And yeah, the military is generally great at being 'effective' it just generally takes 10x as long and 100x more money.)
posted by nixerman at 8:34 AM on February 17, 2006

Suckerpunch: Click on any program, then click the "Details and Current Status"link at the bottom of the individual program page -- you get complete assessment and budget history. Also, here's a spreadsheet listing all the programs, how they were rated, and showing whether and how much their budget went up or down
posted by beagle at 8:36 AM on February 17, 2006

Federal Election Laws - Complaince and Enforcement: Results not demonstrated.

Not too terribly shocked, I'm afraid.
posted by contessa at 8:40 AM on February 17, 2006

Thanks, beagle - my eyes slid right over that listing in the details - I was more concerned with the questions below (or, more accurately, I was wondering how you could effectively evaluate all gov't programs using only 25 questions). And double thanks for the spreadsheet.
posted by suckerpunch at 8:44 AM on February 17, 2006

Whoa. Stop. Americorps NCCC is one of many Americorps programs. NCCC is the only one deemed ineffective. For instance, Americorps VISTA and Americorps are doing well. For instance, this link says "adequate."

The author might want to ask to clarify this in the FPP.

My quickie rumor understanding is that the inefficiency has to do with NCCC having higher costs than the other programs, since they travel around the country doing good, and incur a lot of travel costs. I've also heard that the management just isn't good at measuring outcomes. It sucks hardcore that Bush's proposed budget is going to cut NCCC. They did some amazing work after Katrina (mobile volunteer corps == good for disasters).
posted by Skwirl at 8:59 AM on February 17, 2006

Skwirl, requested to Matt via email, but it was already corrected earlier in the thread. My bad. Adequate is not stellar however, in this scheme of things.
posted by beagle at 9:30 AM on February 17, 2006

It'd be much more useful if the OMB provided a ROI (return on investment) index for these programs.

For many (most?) government activities, ROI would be impossible to calculate, a completely inappropriate metric, or both.

This site, with its mealy-mouthed pseudo-financial non-definitions of "effective" and "not performing," and the slick flag-waving webdesign, looks very much like just another poltical tool.

The only thing that surprises me is that anyone would consider it to be anything else.
posted by Western Infidels at 9:59 AM on February 17, 2006

Somehow I don't think reports like this were included with the $2.6B propaganda budget. This is a bonus.
posted by Balisong at 10:22 AM on February 17, 2006

Oh, look. Someone posted an adverstisement for GWB! Greeeaaaaaattttt.
posted by Mental Wimp at 11:24 AM on February 17, 2006