Public opinion pollsters have a well-known ability to give academic survey researchers migraines, namely because when a poll uses the calling cards of statistical social science, replete with "margins of error," "confidence intervals," and "weights," they are generally given the sheen of legitimacy, or worse, "truth," by the public, regardless of the research's actual quality.  This isn't to say that there aren't some outstanding pollsters out there - there are.  However, the general public's unfamiliarity with statistical and survey methodologies makes it difficult for the average poll reader to gauge the reliability of any given survey - which brings us to the Zogby poll on the public's perception of higher education on which we commented yesterday.  With the information that we've been provided by the Zogby organization (through their press office), it's difficult to ascertain the legitimacy or reliability of their results.  However, from what we have seen, their report is certainly problematic and raises a number of questions.

First and foremost, it should be noted that the three questions that Zogby asked about higher education came as a part of a much larger survey that the organization conducts for their monthly newsletter (the higher ed questions are 110, 112, and 113).  Now, when researchers release their research results, they are expected to also release their data and methodologies so that others may examine the outcomes as part of the peer-review process.  When asked to provide a copy of the survey instrument for examination, the representative from Zogby stated that they would not be able to release it until the complete survey results were presented in August.  While we understand the proprietary issues involved, in our eyes, the release of survey results without the attendant supporting documentation is highly irresponsible on the part of the Zogby organization.  We are basically being asked to take their word for it that the poll results are legitimate without being able to examine how the data was gathered and analyzed.  If a professional academic engaged in such behavior, they'd justly be harassed out of the room.

Why would we want to see the complete survey if we're only concerned with three of the questions?  Because it's well-known to researchers that not only the phrasing of a question, but where a question appears in a survey and what questions appear around it can bias respondents to answer in a particular manner.  In fact, partisan polling outfits count on this in shaping their poll results.  This isn't to say that the Zogby poll is partisan, but until we see the complete survey instrument, we'll have no way of knowing how legitimate the responses to these particular questions are.

Another quite large problem with this survey is their internet sampling techniques, into which the Columbia Journalism Review has previously looked.  While claiming that their sample is representative, we should note that it is not a random sample, which is part of the gold standard for producing a truly representative sample.  The problem with internet surveys is self-selection bias.  In the case of the Zogby Interactive poll, respondents volunteer to take their polls and are selected on the basis of their opting-in.  As it is, using this non-probability based sampling technique should force Zogby to make some caveats as to how representative their sample is.  Indeed, if we examine some of their demographic data, we can see that their sample isn't representative of the general population: looking at the breakdown by income groups, for example, we see that almost one-quarter of their respondents (the largest of any category) has an income of over $100K, compared to the Census Bureau's figure of 12% of the population with that level of income.  The sample is clearly not representative.

There's a number of other issues which bothered us (and apparently some of the people who commented on the issue at InsideHigherEd).  The Zogby poll doesn't include any demographic data on educational attainment or whether respondents have children in college.  It's therefore conceivable that a good number of poll respondents have no experience with higher education on which to base their perceptions.  Likewise, the phrasing of the questions leaves something to be desired.  For example, the question on the quality of higher education reads, "Do you think the quality of a college education today compared to the quality of a college education 25 years ago is much better, somewhat better, somewhat worse, much worse, or is about the same?"  While we all may know some sixth- or seventh-year seniors, how many people have experiences with higher education that span 25 years upon which they may make a comparison?  Similarly, the vagueness of terms like "problem" and "political bias" problematizes the question of political bias in college classrooms, rendering it near meaningless.