This morning, I posted a long piece on PPP's opaque, ad hoc methodology. If you haven't read it, I hope you'll find it worth the time. PPP's director, Tom Jensen, had two unconvincing responses.
First, Jensen said that he was comfortable with his methodology and that it works. But he offered no justification for his controversial methodological choices. And the reason is simple: There might not be a justification.
I can't find anyone who will defend them. I tried. In writing this piece, I contacted more than a dozen highly regarded survey methodologists, political scientists, and pollsters. Most only raised concerns and expressed astonishment. If I were one of those reporters who used unnamed sources, there would have been some pretty incredible one-liners in here.
I even asked PPP to provide someone who would defend its approach, but Jensen could not do so, saying that they're not out to "make friends" with AAPOR (American Association of Public Opinion Research). Well, fair enough. But I don't really think there are survey methodologists who would start bashing a scientifically defensible practice because they're not friends with the pollster.
Jensen says he doesn't care about doing things the way everyone else does. But "everyone else" is doing things for a reason: rigorous, scientific research about how to conduct accurate polling. So if there's no one who thinks this stuff is a good idea, why are they doing it? Why can't they explain why it is a good idea, other than simply reiterating that their results are accurate? Why hasn't Jensen published some great book critiquing the modern survey research enterprise?
Second, Jensen posted our full email exchange. I'm totally fine with this, since, well, my original piece accurately represents the exchange. Unfortunately, Jensen suggested that I hadn't included his explanations when, in fact, I did. And when asked for a specific example of something that I should have posted, Jensen refrained from offering a reply. And if PPP thinks there was some important detail in our exchange, they should have just said so. But there's not one.
The real reason Jensen posted the exchange: To make it look like he was really transparent. But Jensen only explained his methodology after I provided data that truly required an explanation. And there's a reason PPP didn't want to mention this until now: it's a bad practice. Selectively using the '08 election isn't too much different from weighting to a desired result; it's not too much different than weighting by party ID; and it's particularly egregious to incorporate it into race weighting, rather than weight simulatenously.
And even after Jensen offered an explanation, he was still bullshitting me--to use the technical term. He said that they only used the '08 ballot because he feared Republicans would criticize him for showing an O+7 electorate in Ohio. But Jensen had deleted the question from his survey, so no one would know there was an O+7 electorate. There are other examples in the original piece. Ultimately, explanation under duress does not absolve PPP from refraining to disclose the use of a powerful weighting tool for four years. That's not transparency.