Poll Flippery

I think the more precise analogy would be (various versions of) the placebo effect. If the mere asking of the followup induces the flip, then its uninteresting.

I'm not sure that this is the same as embedding bias as described above. Obviously, what questions you ask (and how you ask them) can make a huge difference in the response, but the specific phenomenon is whether the mere fact of adding a detail in a follow up causes flipping and under what conditions. For example, the flip rate might be (partially) a measure of the uncertainy or ambivalence people felt about their answer. (I.e., it could be akin to asking, "Are you *sure*? REALLY sure?!" and getting people to change their answer or at least withdraw it.)

I'm less certain that this is a real problem and even less certain that knowing the average would tell me much. The interesting fact is whether people just flip regardless of the content of the follow up. Or, perhaps, whether you would get different answers if you structured it as a follow up as opposed to embedding all the information into the original question. If neither of these are true, then I don't see that there's a problem.

Political scientists have been working on this questions basically since modern survey techniques were invented. What's interesting to me is that even a fairly wonkish political blogger like Kevin Drum could imagine that, 50 years later, the entire field of politics might not have thought much about this issue. Sure, he grants that "maybe someone has already done this research," but that's pretty mild. Would you ever say, "maybe someone in physics has thought about measurement error in their instruments?" Of course not; you would probably assume (correctly) that there entire sub-disciplines that deal with these issues. Funny how people imagine otherwise in political science. It's a lot different from the few bar graphs that occasionally show up here or on Yglesias!

Anyway, snark aside, here's a fairly canonical paper from 30 years ago. If you want something more sophisticated and recent (say, with lots of simulated data), just scholar.google.com it.

Mass Political Attitudes and the Survey Response; Christopher H. Achen (American Political Science Review, 1975).
http://people.virginia.edu/~naj5n/Comp/achen.apsr.1975.pdf

That paper focusses on random response variation (not variation due to changing contexts), and tackles most of the questions in the second-to-last paragraph of KD's post in fairly thorough (and, at the time, novel) detail.

As mentioned there is a fairly wide set of literature on the subject of instability within survey responses. You could start with Zaller's book "The Nature and Origins of Mass Opinion" for a model explaining such instability, while also branching out into framing literature.

http://books.google.com/books?id=83yNzu6toisC&dq=John+Zaller&printsec=fr...

The gist of Zaller's model is that individuals receive arguments from elite discourse concerning policies which is then filtered through their ideological predispositions and values. Those with stronger predispositions are able to defend against persuasive attempts that are at odds with prior values; those who do not have strong predispositions will end up taking in a broader variance of considerations concerning an issue. When we turn to the survey response situation, the individual samples from these considerations and constructs an opinion on the spot. Individuals, particularly those with weaker predispositions and thus a higher variance of considerations, will be especially attuned to cues contained within the question or the survey. [Perhaps the canonical example is asking one set of respondents about a rally by the KKK but framing it either in terms of civil liberties or danger to civil order; response rates in support can drastically change depending on that frame because different considerations are brought to mind.] Framing studies really have worked off this basic model, stressing the role of cues within the interview process in driving response rates. [This model has been challenged recently by Sniderman and Theriault who argue that those studies insuffiiently reflect political reality. Take the example provided: 1/2 of the survey would receive one frame and the other half the other. Sniderman and Theriault point out that this isn't quite how political discourse actually occurs, that we need to use competitive framing examples in our studies. This in turn might decrease the propensity for different frames to blow people around as if they were simply in the wind.]

So, in the example provided (at least by this model, there are others...) the additional information in the second question is bringing to mind considerations that these people might have had in long-term memory but were not accessible (because insufficiently primed by the first question). At least that is one possible example. There is a lot of research on this.

For a really pessimistic view on public opinion polls see: "The Illusion of Public Opinion" by Bishop (http://books.google.com/books?id=fpzrs-bSD3gC&printsec=frontcover&dq=ill...). For a view about why micro-level instability (like the stuff above) might not matter considering the possibility of a rational *aggregate* public opinion see "The Rational Public" by Benjamin Page and Robert Shapiro (http://books.google.com/books?id=kjPryRO9puoC&dq=rational+public&client=...). For a wider list of readings current as of 2004 see MIT Open Course Ware: http://ocw.mit.edu/OcwWeb/Political-Science/17-951Spring2004/Readings/in...

I don't see the flipping in that poll as being the result of a lack of information; there is plenty of room for being offended at being required to buy insurance (in current conditions) compared to being required to buy insurance with gov't subsidies or other cost discounts. In other words, the followup wasn't simply clarifying or slightly tweaking the conditions of the original question - it was adding a very significant piece of information.