When Samples Don't Tell the Story

A few days ago I expressed the wish that a Sister Survey similar to those conducted by the distinguish sister-sociologist, Marie Augusta Neal, in the years following Vatican II, could be done now to find attitudes of sisters independently in the midst of the investigation crisis.

Meanwhile, word comes that the estimable periodical, U.S. Catholic, is surveying sisters about the investigation and separately inviting readers to post their views of the probe. The general reader survey is described as a "poll." The first wave of responses showed 55 percent sharply critical of the Vatican's initiative, but after a priest filed a entry on his blog highly in favor of the process, together with a link to the magazine's site, the results tilted heavily in the other direction, 80 percent approving the probe.

Be careful what you believe. The methods and results can be terribly misleading. I worked for a brief time at the Gallup Poll. An occupuational hazzard there and in similar operations is that the public often placed their methods of scientific random sampling on a par with undisciplined surveys. To qualify as a valid poll means to use the scientific procedure that usually leaves a very small margin of error.

By contrast, efforts to gain opinion by making no attempt to sample a cross section of the public but instead just issues a general invitation has little to no credibility as a measure of attitudes. When the local television news asks viewers to call in to say whether or not they favor installing new swings at the playground, the result will indicate something about who was watching and who was motivated to vote, but it won't capture what the public at large thinks.

U.S. Catholic will come up with something interesting in its survey but without a colossal coincidence the sampling won't represent sisters at large. And obviously the "poll" of readers is subject to factors like the priest's advocacy that swing interest groups one way or another.

Scientific surveys have their flaws, of course. They're limited by the quality of the questions and the ubiquitous lack of context. But they do accurately reflect what they look for.

Collecting opinions and thoughts of people on a non-scientific basis has its own value. Much can be derived by giving subjects a chance to nuance their ideas and feelings. It's just that these projects shouldn't be confused with the actual polling process.

In other words, unscientific samplings have real benefits, but they cannot carry the weight of the rigorous Sister Surveys that Sister Neal guided. That would provide a sweeping portrait of what the array of sisters are thinking in the midst of an inquisition they did not choose.