HOW TO EVALUATE SOURCES

 

We are bombarded with contradictory information every day.  This is particularly true with respect to media reports concerning matters that involve our participation as citizens.  Even when observers and analysts agree on what happened during a particular time at a particular place, they frequently disagree about how to interpret those events or the social conditions that preceded or followed those events.

 

The question arises: How does one make sense of this seemingly chaotic jumble of conflicting information and interpretation?  Whom should one believe?  How can one arrive at an independent and reliable judgment of what happened and what it means?

 

Here are some suggestions about how to evaluate the information we receive from all the sources that constantly beg for our attention.

 

General Reminders

 

1.   Keep an open mind.  Never assume that what you are reading or hearing is all you need to know about a subject.  Be willing to challenge your own preconceptions, even your most basic values.

2.   Keep asking questions.  Even when you think you know everything about a topic, be willing to entertain new questions.  Keep looking for new sources from different perspectives.

 

Specific Recommendations

 

1.   Gather as much information as you can.  The more controversial the topic, the more information will be available and the more information you will have to sift through to draw a conclusion you can have confidence in.

 

2.   Whenever possible, find out about the author’s (or source’s) background.  Has this person written or spoken about this subject before?  Does this source have a reputation as an “objective” observer?  Does the source regularly state the perspective from which the information is being interpreted?  In other words, does the source provide any clues about his or her potential biases?

 

3.   Is the information anecdotal or systematic?  In other words, is the information based on reports that have not been substantiated by others (anecdotes), or has the information been gathered in a systematic way, so that the information represents the impressions and observations of a carefully selected range of observers or respondents?

 

4.   Scholarly works are most reliable because they are based on a systematic, preconceived search for information and a disciplined interpretation of findings.  (“Disciplined” means that the researcher/analyst is skilled in methods developed and refined by other members of a community of scholars; the methods used to find and interpret information are not idiosyncratic.)  Of course, scholars disagree among themselves about the reliability of some information and the validity of some interpretations.  Nevertheless, scholarly research is the most reliable source of information and analysis.

 

5.   Is there a reason to suspect a particular bias by the source?  Government documents (no matter which government), for example, are good sources of statements of official policy, but they are notoriously weak sources of reliable information.  In some cases, however, govern­ment documents can be reliable, especially if the agency that produces them has already established a record of reliable data reporting (e.g., the Bureau of Labor Statistics).  Other reliable agencies are those that do not participate in promulgating and executing policy (e.g., the Internal Revenue Service and Census Bureau).  By these standards, State Department, CIA, FBI, and Defense Department “white papers” or reports on particular conditions and events should be regarded with extreme skepticism.  These sources are mostly interested in defending a previously stated policy or in making themselves look good to the public.  Likewise, announcements and reports by corporations and industries that need to protect their public image should be regarded as containing an inherent bias.

 

6.   Ideological biases may also skew data collection and are often the bases for analysis.  “Ideology” refers to a set of beliefs and values that forms a worldview.  In particular, ideologies favor the interests of one or a few social groups (or classes) over all others.  In other words, ideologies are most often justifications for particular power relationships, either the one in existence or some alternative.  Ideological biases are almost unavoidable, since every agency that collects information and interprets it has goals it wants to attain.  Decide whether the goals served are the same as yours.  (But remember to keep an open mind even about your values and goals.)  Sources with obvious ideological biases are often useful, no matter what the bias, but YOU should be aware of the bias.  Bias determines the purposes for which the information was collected and the slant on the interpretation the information has received. 

 

7.   Look for “third party” reports, whether produced by the third party itself or included in media reports.  “Third parties” are observers who have no conceivable commitment to a particular resolution of a conflict but are interested in an issue for reasons distinct from those of the opponents or combatants in a conflict.  (That is why government reports are so unreliable: governments always have a stake in outcomes.)  Primary among such groups are human rights organizations, churches and church personnel (at least, during the last 30 years), non-governmental relief organizations, etc.  Such groups sometimes make mistakes (e.g., Amnesty International’s false information about Iraqi atrocities before and during the Gulf War), but they are more likely than any other source to have both a person (or group) at the scene and a system of corroborating and evaluating eyewitness accounts.

 

8.   Beware the privileged voice!  The voice of privilege speaks from the position of power and has a vested interest in maintaining the status quo.  The status quo is obviously the source of privilege.  The privileged clearly have little or no interest in changing the conditions that have given them privilege and power.  They are not likely to be able to understand or appreciate the circumstances of the disadvantaged, and they are certainly not able to understand arguments for social change.  On the other hand, when privileged voices speak out in favor of social change that clearly does not serve the powerful, they should be listened to very carefully (but with the general skepticism afforded to all sources).

 

9.  Look for unusual sources and interpretations.  This is a continuation of the previous point.  Dominant or ordinary interpretations get to be that way because they are acceptable to the powerful.  Time, Newsweek, U.S. News & World Report, etc.—even the New York Times, Washington Post, and Wall Street Journal—are all sources of elite interpretation and opinion (because they are themselves major corporations or subsidiaries of corporations).  They are useful, but they select and interpret the news they report from a particular perspective that generally serves the most powerful sector of society. 

 

10. Look for sources that say something different from what you would expect from such a source.  For example, CIA officers and agents who challenge official policy are inherently more reliable than those who follow agency policy because they have had to do so much soul-searching and have taken significant risks to themselves and their careers by doing so.  The same is true for whistle-blowers in general.  They might be wrong on some facts and biased in some interpretations, but the risk they have taken to say what they have to say makes it somewhat more credible than official versions.  Former opponents of official policies who now support those policies should also be given careful attention—unless, of course, there is evidence that they have been bought off.

 

11. Greater weight must go, of course, to the preponderance of empirical (observable) evidence in a case.  In the same vein, interpretations and argumentation must be internally consistent (that is, logical) as well as faithful to empirical evidence.  In other words, a theory or interpretation that does not have observable evidence to substantiate it should be at least temporarily rejected.  I say “at least temporarily” because later empirical evidence may emerge to provide some support for the theory.  But the basic question for any interpretation of anything is this: Is there evidence for this conclusion?  Secondly, is this conclusion based on a logical argument?

 

Final Considerations

 

1.  Remember that even science never proves anything.  The scientific method merely shows what is not proved.  In social science, the evidence is even less certain and precise because the subject of social science is humanity, a phenomenon that can think and can therefore change its destiny!

2.  Do not confuse passionate concern with bias.  The people who have thought hardest and longest about an issue are likely to have strong opinions.  You need to be able to apply the above list of recommendations to your evaluation of the validity of those opinions, regardless of the passion with which they are presented.  Passionate advocacy of a particular position and “objectivity” are not necessarily contradictory.  A person (or source) may be passionate about an issue precisely because that person has analyzed it very carefully and arrived at a conclusion that is consistent with all available evidence.  For example, Martin Luther King, Jr., was a careful and precise student of matters relating to the conditions of African-Americans and, more generally, the poor (white as well as black).  He was passionate about his advocacy of their rights (he died for them!), but his passion never overwhelmed his ability to see their condition clearly and insightfully.