Skewed Survey's: or Lies, Damn Lies, and Statistics

MSNBC has put up a web page entitled "About our Live Votes and surveys: How 1,000 people can be more representative than 200,000"
http://www.msnbc.msn.com/id/3704453/

This is an important addition to the discussion about information literacy. It is a concise and informative article about how polls, surveys, and online votes can differ greatly in results even on the same topic and with the same questions. It is also gratifying to see a major media outlet not only to be so circumspect about how they present information, but to also be open about it with the public. Some of the more interesting statements include:

One week in the middle of the Clinton-Lewinsky scandal, more than 200,000 people took part in an MSNBC Live Vote that asked whether President Clinton should leave office. Seventy-three percent said yes. That same week, an NBC News-Wall Street Journal poll found that only 34 percent of about 2,000 people who were surveyed thought so.


To explain the vast gap in the numbers in this and other similar cases, it is necessary to look at the difference in the two kinds of surveys.

While a poll of 100 people will be more accurate than a poll of 10, studies have shown that accuracy begins to improve less at about 500 people and increases only a minor amount beyond 1,000 people.

Random selection of those polled is necessary to ensure a broad representation of the population at large.

To begin with, the people who respond choose to do so — they are not randomly selected and asked to participate, but instead make the choice to read a story about a certain topic and then vote on a related question. There is thus no guarantee that the votes would reflect anything close to a statistical sample...

This is a good and brief explanation about statistical sampling and reliability that I think would be useful for everybody to review. It is a good reminder of things we tend to forget. To many of us these may seem obvious, however it is easy when reading an article, book or web site for us to just accept the statistics offered without considering the way they were collected and the context in which they are delivered. With plethora of information providers both ethical and less ethical it is now more important than ever to check the sources and verify information with separate and independent resources.

So what is the point with Wikipedia and Britannica?

In an article in the Chronicle of Higher Education Jimmy Wales "founder" of Wikipedia warned students about Wikipedia. He goes on to say "For God sake, you're in college; don't cite the encyclopedia,".

When deciding to use Wikipedia or Britannica the first question is which can you get to? If I am at work at the library I will use Britannica. However if I am off campus and have to go through some kind of proxy verification or other hassle I will use Wikipedia. I would not use the free version of Britannica because of the excessive amount of advertising and the information provided is barely as much as you would find in a good biographical dictionary.

The problem is that this is not why I posted on Tuesday, December 5, 2006 The Tao of Searching: Wikipedia Britannica and missing the point. Or why even the Journal Nature missed the point. The point is: Encyclopedias are not a research destination. Encyclopedias are a starting point or a roadmap for research. Over the last few years web gurus have been using the term sticky (see Make Things Sticky) to describe a good web site. They don’t want users to just visit a page and then leave the site. They want them stay on the site. One problem with Wikipedia and Britannica is that they have great designers that adopted this philosophy. They too offer up more and more choices and links to keep the user on the site. The problem is when you are doing research you do not want to get stuck in an encyclopedia.

At Cornell University they put together a brief introduction on how to do research called “The Seven Steps of the Research Process”. Step two is: Look up your keywords in the indexes to subject encyclopedias. Read articles in these encyclopedias to set the context for your research. Note any relevant items in the bibliographies at the end of the encyclopedia articles.

This is how to use encyclopedias. If you will notice they specifically mentioned “subject encyclopedias”. Subject encyclopedias focus on one discipline or area of knowledge and therefore usually will cover a topic to a greater depth than a regular encyclopedia. It is also likely to be more authoritative in that particular field than a general encyclopedia. The quick and dirty way to find them at the library, is to go to the catalog and do a keyword search for “encyclopedia psychology” (or geology, religion or whatever your topic is). Then look for books with the word encyclopedia and your subject in the title Once you have a couple of call numbers, go to that section of the library and look at the encyclopedias you picked, also look at the books nearby. They will be on the same topic and there may be another book that will work better for you. Or you could just ASK A LIBRARIAN.

If you are on the web you can go to Google and type in “encyclopedia philosophy” (or whatever your topic is) and you should get a lot of results. When looking at the results you will want to look at the URL to see if it is a .com .org or an .edu site. A .com will probably be trying to sell you something. You will also want to read the description. Then visit a few of the sites and see which ones will work best for you. Better yet would be to go to The Open Directory Project (ODP) and type in encyclopedia and look at their collection. Or you could just ASK A LIBRARIAN.

Ultimately the only books that people claim inerrant are the Qur'an, Bible, and The Book of Mormon. Therefore, we may assume that everything else may have some errors and problems. Britannica has fewer errors by a wide margin. It is written and edited to professional academic standards, and it has years of experience in publishing encyclopedias. Unfortunately only a small portion is free, unless you have access through an institution like a library. This may involve proxy access or passwords or IP verification that may not work so well at your location or with your device. Of course you can buy access but an experienced researcher can probably find the equivalent amount and quality of information they need using the web, maybe even through Wikipedia.

Wikipedia is accessible from almost any type of web access device. It is free and is easy to use, and you can get sucked into editing entries on some of your favorite topics. It is surprising how enjoyable it is to contribute to or edit Wikipedia. On the downside any idiot like me can contribute or edit Wikipedia. It does have an unacceptable number of errors to be considered authoritative or equivalent to Britannica.

Despite the characteristics described here, the point again is that “the encyclopedia is not a research destination. Encyclopedias are a starting point or a roadmap for research.” It’s the things you discover along the way, the sense of accomplishment for finishing an assignment, or mastering a new area of knowledge, that is the destination.

Next maybe we will talk about how to pick out a book and then maybe how to evaluate information on web sites.

R Philip Reynolds