ON THE VALUE OF COMPARATIVE ANALYSIS in EVALUATING SOURCES

 

Many students think they have done their job when they gather evidence or present analyses that support the position they held when they began. But adequate research and analysis on a topic requires that both the "facts" and the analysis be evaluated for their validity and reliability. Validity is the extent to which the evidence actually applies to the topic under consideration and the extent to which the analysis abides by rules of logic and evidence. Reliability is the extent to which other trustworthy observers studying the same phenomenon would get the same results. In other words, one needs to evaluate whether the evidence is based on a method of research that would satisfy someone who is experienced in doing research. Furthermore, one needs to evaluate whether the analysis provided is conjecture or solidly based on the available evidence and whether it deals with potentially discrediting information and interpretations.

There are various ways to evaluate the sources one finds. Most of them, however, require lengthy study, which undergraduate students cannot have done. How, for example, can a student readily tell what the ideological biases of reporters and social scientists are? Writers on controversial topics rarely spell out their ideological biases. Most do not recognize their own errors in logic and data analysis. And most writers who get published present convincing arguments. They know how to present what they have to say in a way that will be persuasive to readers who know little about the topic and do not have the training to evaluate arguments systematically.

If you are thoroughly familiar with the usual ideological biases that dominate discussions of particular topics, you are already ahead in the process of evaluation. One way to determine that is by knowing which publications and organizations support particular ideological tendencies. But most students do not know the ideological slants of even the most dominant establishment news sources.

There is one method by which a student can make a fairly good evaluation of the validity and reliability of information and analysis. That method involves determining the context in which the information and analysis are presented.

Context can be determined in several ways, but it is important to keep in mind that the writer, especially of journalistic accounts, will rarely provide context explicitly; you will have to figure out what it is in most cases. The major form of context is revealed by the way the topic is framed. What is the author trying to prove? Why is the author presenting this information and providing this interpretation?

Nevertheless, writers can provide context in a very straightforward manner simply by putting their findings in an appropriate comparative framework. Examples of comparative frameworks are comparisons with other groups within a society, comparisons between countries, or even comparisons with "ideal" conditions. For example, to evaluate information about women, it is reasonable to compare their circumstances to those of men. When trying to determine the usefulness of an analysis of minorities in a particular country, compare the conditions of minorities in that country with the conditions of minorities in other countries. If one wanted to know how badly off one group is in a society, one might compare how that group fares on various dimensions with a hypothetical condition of equality. Or one might see how a group’s current condition compares with the condition of the group at some previous time.

If writers do not present their analysis in some appropriate and meaningful comparative context, it is likely that they are trying to fool the reader. Short of that, they simply do not know how to perform data analysis or how to collect meaningful information. In either case, they are unreliable sources.

Let us say that your topic is "women in Cuba," and you want to know how well or badly Cuban women are doing. Perhaps your additional goal is to evaluate whether the Cuban government is sincere in its professed dedication to gender equality, or better, whether government policies have actually narrowed the gap between women and men and whether they have improved the manner in which men treat women.

The first issue is relatively easy to address. You need to collect information about various aspects of women’s lives: economic independence (including access to professional careers), health services, educational attainment, anti-discrimination legislation, the enforcement of anti-discrimination legislation, the incidence of rape and sexual harassment, representation in governing bodies from local to national, etc. But in order to make sense of that information you would have to compare that information with other information about women’s conditions. To determine the meaning of the information about Cuban women you would naturally compare their conditions with the conditions of women in other countries. You could compare Cuban women with women in other Latin American countries, or in other countries with similar levels of industrialization or economic prosperity. Or, in order to make some comparison with conditions in industrially advanced or more prosperous countries you would choose a comparison country (or countries) in Europe or North America. You might also compare the status of women in contemporary Cuba with some point in Cuba’s past to see whether the current condition of Cuban women represents an improvement or a decline.

A source that does not contain these kinds of comparison cannot tell you what you need to know to make a satisfactory judgment about the condition of women in Cuba. A source that describes the current condition of women in Cuba as "oppressed" or "exploited" or simply terrible—or, on the contrary, as "equitable" or secure and trouble-free—without offering some sort of comparison is not trustworthy. Furthermore, the points of comparison must be reasonable, that is, the justification or "the point" of such a comparison should be readily clear or explained. If, for example, a source says that prostitution in Cuba is "widespread and growing," you need to know how Cuban prostitution compares with prostitution in other countries and in Cuba’s past. You would also want to know whether the causes of prostitution in Cuba are the same as the causes of prostitution elsewhere or in Cuba’s past, and whether Cuba criminalizes prostitution to the extent other countries do. In short, you would need to know how prostitution in Cuba compares with prostitution in other countries or in Cuba’s history.

The second part of the question—whether Cuba’s policies have contributed to greater gender equality and more respect for women—is harder to answer. You would first have to determine the standing of women relative to men on a variety of dimensions—in terms of the economy, politics, health, family responsibilities, etc.—and compare that with a time in the past. You would also have to determine a way to link any changes you find to government policy. For example, are there laws that protect women from discrimination in the economy, politics and household? Are those laws enforced? Are educational and medical resources distributed in an equitable manner between men and women, i.e., do women receive a fair share of national resources? Have government policies contributed to an increase or decrease in the probability that women occupy positions of professional and administrative responsibility? Has the government used its propaganda powers to encourage gender equality or maintain male dominance? You would also have to determine whether the improvement or decline in women’s standing was the result of government policies or despite them—the result, for example, of deep-seated cultural factors that have proved difficult for government policy to counter.

A source that does not address at least some of these issues is almost certainly directed by ideological concerns, even if the author is not aware of it. At the very least, it is problematic and perhaps incompetent, unable to protect itself from the insidious invasion of unconscious bias. A good methodology for gaining information recognizes its value orientation and seeks to make that clear to readers. You should be especially wary of a source that pretends to be value-neutral or without an ideological perspective, especially when the topic is one that is in any way related to any government’s policies, including the policies of one’s own government. That applies not only to journalistic sources but to "social science research" as well. Unfortunately, "social science" is not immune to ideological infections. I would argue, in fact, that all social science necessarily contains an ideological element.

Even when a source offers comparisons along the lines suggested above, one must be careful about invidious comparisons—that is, those intended to cast a bad light by not taking into consideration appropriate and reasonable qualifications. It is hardly reasonable, for example, to condemn El Salvador as a nation for its horrible human rights record compared to Costa Rica without noting the differential influence of the United States in each country. Costa Rica might be congratulated for keeping the U.S. at bay (relatively speaking) and for making the extraordinarily wise decision to abolish its military in 1948. But it is clear to all with the remotest knowledge of these two countries that the difference between the two nations with respect to human rights is the result of the different level of U.S. intervention in each country, and that is a factor over which neither Costa Rica nor El Salvador have had much control. Similarly, a condemnation of the Cuban government for censorship needs to take into consideration the particular circumstances of Cuba under constant threat of annihilation by the United States compared to other governments with which its censorship is compared. Censorship is always bad, but it does make a difference why and how it is practiced. All governments (the United States included) practice censorship in some form, and censorship occurs for reasons other than government policy (e.g., because of corporate sponsorship or ownership of media). Comparisons must be qualified by recognition of peculiar conditions.

Ideological biases are not necessarily "bad," but they should be recognized. Researchers, commentators, and analysts may have very good reasons for their perspectives. Nevertheless, the honest and trustworthy author tells the reader in clear terms what that perspective is. The author who does not do so might inadvertently provide useful information, but it requires the diligence of the reader to determine what that is. The sensible reader is skeptical about perspectives that favor any government’s proclamations and policies, since all governments have vested interests in particular interpretations of existing conditions.

The best way to determine the reliability of a source is to consider the comparisons made by an author. Does the author make appropriate comparisons, with peculiarities noted? Is some standard provided to which the author’s criticisms of a condition can be compared? Is there a clearly defined context in which the author’s information and analysis can be interpreted? Is the author clear about her/his own ideological position? Does the provided evidence really substantiate the author’s conclusions? Or does the author depend upon unmentioned assumptions to reach conclusions?

Of course, there is another important factor in evaluating a source: is the author telling the truth? That is, even when comparisons are made, are the comparisons based on the best available data? Have the authors compared their own findings to research by other observers, particularly those with different ideological perspectives? Can they provide a satisfying explanation for the differences between their perceptions and those observers with whom they disagree? Does an author give an adequate and accurate account of conflicting perceptions and analyses?

The most difficult task of any serious student today is the evaluation of information and analysis. For an undergraduate student, who necessarily lacks training and experience in a discipline of study, the most important aspect of that task is to keep an open mind and to be skeptical of all sources of information. As undergraduates—and as citizens of an imperfect democracy—your primary responsibility is to learn how to assemble and evaluate information and valid interpretations of that information. It is not to draw hard and fast conclusions. You do need to develop a value orientation: what do you think are the most important factors in a morally sound and meaningful life?

But that is a topic for a different essay. The issue here is how to gather and evaluate sources. Although there are many important aspects of that process, the simplest and most crucial is to determine the context—ideological and otherwise—in which the information and analysis are presented. And the most fundamental practice in that process is to ask what kinds of comparisons, if any, are made.