Cumulation from Proper Specification:
Theory, Logic, Research Design, and ‘Nice’ Laws
Harvey Starr
Department of Political Science
University of South Carolina
starr-harvey@sc.edu
Paper prepared for the panel, “Control Variables, Specification, and Knowledge Accumulation”
at the Annual Meeting of the Peace Science Society (International), November 2004, Houston.
INTRODUCTION
Jim Ray (2003a, 2003b) in two recent papers has presented a broad set of guidelines concerning
how we should go about doing multivariate analysis. The first paper, a published version of his
Peace Science Society Presidential Address (2003a), focused on the issue of control variables;
the second paper, presented at the 2003 Peace science meeting (2003b), used Stuart Bremer’s
classic study of dangerous dyads (1992) as the starting point for a critique of how some
scholars go about doing multivariate analyses. The arguments that Ray presents in these papers
can be easily translated to, or subsumed under, the broader enterprise of “cumulation.” That is,
how can we account for the lack of cumulation of knowledge in certain research areas, and what
lessons concerning the logic of inquiry and research design could we apply to increase both the
additive and integrative cumulation of knowledge in these areas?
In both papers Ray was concerned with knowledge accumulation in the specific area of
international conflict. Most and Starr (1989) similarly used the area of international conflict to
illustrate a set of research design problems that had hindered cumulation in the study of
international relations. Through the use of the “research triad” of logic, theory and method. Most
and Starr, like Ray, took a “critical” approach to how scholars were designing their research.
Most and Starr were concerned with the lock-step use of “standard” approaches, and an often
unthinking application of the basic social science statistical cookbook. One example, initially
presented in Most and Starr, is the problematic application of standard methods to causal
relationships based on necessity.
Similarly, when Ray begins his critique of the use of control
variables, he notes that he is dealing with the “norms or customs” regarding the general use of
control variables in multivariate analyses (2003a:1). Thus, I see the basic enterprise in Ray’s
papers and the Most and Starr volume as the same, and I am in general agreement with the thrust
of Ray’s arguments. The purpose of this paper is to show how his points are supported by
earlier, complementary lines of argument, work that did not specifically look at the issues of
control or the specific design of multivariate studies.
APPLYING THE RESEARCH TRIAD: CONTROL, THEORY AND NICE LAWS
Ray begins with a basic point that he uses to explain the increased (and indiscriminate) use of
control variables. He takes one route to argue the primacy of theory in the selection and use of
control variables, and I will take another. The basic point Ray develops (2003a:2-4) is that over
the past 15-20 years researchers have moved beyond searching for “ideal” or “best fit” general
models. Instead, they have developed models “to evaluate the impact of one key
factor”(2003a:3), seeing if the impact of that factor can survive to addition of control variables.
The latter trend is clearly within the spirit of Most and Starr’s idea of “nice laws”– or
“sometimes true ‘domain specific’ laws” (1989:98). Most and Starr remind us that while we
should aim for generality, the "right type of law" is one which is clearly specified; that the
relationships among variables that it proposes will work only under specified conditions.
Most
and Starr question whether social scientists will ever generate important "universal" laws. They
note, however (1989: 117) that:
...it may be useful to recognize that there could very well be laws that are in some
sense "good," "domain specific," or "nice" even though the relationships they
imply are not necessarily very general empirically... it may be more productive to
think of laws each of which is always true under certain conditions (or within
certain domains) but which is only "sometimes true" empirically because those
conditions do not always hold in the empirical world.
The more recent strategy that Ray discusses, looks to see under what conditions a key
factor doesn’t hold, through the use of control variables. In the spirit of additive cumulation, such
studies provides new information through the incorporation of new variables. For example, Ray
(2003b:5), discussing Bremer’s work on dangerous dyads, stresses: “Again, the addition of one
explanatory factor to the multivariate model originally constructed by Bremer (1992) has several
substantial impacts on several of the other relationships that are analyzed. The inclusion of this
one additional factor results in a fundamental revision of the rank orderings of the variables by
the weight of their apparent causal impact.”
The important question is whether or not studies of the impact of some key variable can
be part of a process of integrative cumulation. The answer rests with theory, and the place of
theory in the research process. Ray supports the “key factor” approach, using a theoretical
critique of the ideal/best fit approach (2003a:4):
We have no theory, or theoretical approach that will tell us what are the best six,
seven, or eight predictor variables to put into a multivariate model aimed at
accounting for interstate war or conflict... we have no sound theoretical basis for
saying “these are the seven most important variables on which to base any
explanation of interstate wars. It is these seven and only these seven...”
What Ray is pointing out is that researchers need theory– they need it first and they need it to be
specified. In order for the theory to be specified, that is to understand which factors/variables/etc
are important when, under what conditions, and in what form, researchers require a model, and a
model which strongly incorporates some theoretical process (e.g.,see Bremer and Cusack 1996;
see also Starr 1996). In order for theory to be specified, then, researchers require a procedure by
which they can build process models. Lave and March (1975) provide such procedures for
“conceptual modeling.” These procedure include thinking in terms of process as one of their
three “rules of thumb.” Most and Starr (1989) rely heavily on the Lave and March model
building process, including their own variations such as the use of stylized facts for quick and
dirty tests of the implications of the theory/model. The theoretical specification of a model or
theory requires an understanding of nice laws. That is, the proposed relationships or effects of a
model are seen to hold only under specified conditions– which derive from the process within the
model-theory-story. The theoretical specification of a model must also, then, take into account
the possibility of “substitutability” as developed by Most and Starr (1989; see also Cioffi-Revilla
and Starr 1995) . As the flip side of nice laws, substitutability refers to the existence of a set of
alternative modes of response by which decision makers could deal with the same situation.
These behaviors have also been called "alternative modes of redundancy" (Cioffi-Revilla and
Starr 1995, 456-57).
Theory, nice laws, and substitutability will all be used below tin
commenting on Ray’s “guidelines” for the use of control variables.
The primacy of theory is a central theme in Most and Starr , among many others (see also
Starr 2002). Understanding the form of the relationship generated by a theory, as well as the
cases needed to test or evaluate the theory, as well as understanding the context in which the
theory should be expected to hold (or, in which one could expect to find evidence of the
implications of the theory), are all called for in Most and Starr. Understanding the broader
concepts that are of concern in the theory, as well as understanding what phenomena are really
under investigation and why (e.g. Most and Starr 1989:ch.4 on conceptualizing war) are also
necessary. The context involves nice laws. The broader understanding of concepts and process
will involve substitutability. All of these activities must take place for the proper specification of
the theory, which is needed for integrative cumulation.
Cumulation is a social process that occurs within a community of scholars. One can
argue that it has occurred in areas such as the democratic peace, power transition theory, and the
role of territory in conflict. Integrative cumulation, for Zinnes (1976) occurs “when earlier
studies are ‘crucial’ to the conceptual and theoretical components of the subsequent study’s
research design” (Most and Starr 1989:7). For example, Russett and Starr (2000) argue that it
was only after the actual phenomenon of the democratic peace was empirically established – that
two “democracies” do not fight “wars” against each other– that intensive work on theories as to
why this should be occurred. Such theoretical work involved intensive conceptual discussions of
democracy and war, as well as their possible relationships. As theories were generated and
evaluated, they were examined in a variety of different historical and regional contexts. The
implications of these theories, which relied on process, were subsequently tested on a variety of
different behaviors (dependent variables), both conflictual and cooperative. It was then, after a
variety of theories proposed certain types of behaviors based on the nature and presence of
democracy, that researchers expanded their dyadic investigations to monadic studies of
democracies (with Ray being one of the first to argue strongly for such monadic effects).
Thus, integrative cumulation has been found, but only after hard theorizing, specification
of theories, the use of nice laws and substitutability, and substantial additive cumulation have
taken place. It has not come through the use of wholesale, general, global models including
some set of “best fit” variables.
ILLUSTRATIONS WITH RAY’S GUIDELINES
Ray (2003a) offers a set of five “guidelines” that he believes could “improve the quality of
evidence produced by multivariate analyses, as well as the credibility and intelligibility of that
evidence.” Let me provide some brief comments applying my points to each of these guidelines.
Guideline #1 states: “do not control for intervening variables.” Discussing an article that
simply adds control variables “...that previous research has shown to have an impact on war and
dispute involvement” (2003a:5), Ray argues that research design should not treat intervening
variables as control variables. The point of my arguments in the present discussion is that if one
wants to move from additive cumulation to integrative cumulation, then researchers must
recognize exactly what types of variables they are dealing with, and then build a better
model/theory. If the variables are truly intervening, then the model must be more closely
specified to indicate in what ways the new variables are part of the story, and what that means for
the implications of the model, and under what conditions the model will hold.
Similar comments apply to the next three guidelines as well. Specifically, Guideline #2
states: “distinguish between complementary and competing explanatory factors.” Ray (2003a:7)
notes that in many cases, control variables are simply “alternative causes,” and that “rival
explanations” are simply alternative causes. One major point developed in Most and Starr, and
deriving from nice laws, is the position that very few models or theories are actually
“contending;” the key question, they propose is rather “how can both be true?” That is, under
what conditions does each hold.? Thus, clearer specification and the logic of nice laws helps to
support Ray’s guideline. One must also investigate whether “alternative” causes are truly
“alternative.” That is, does substitutability come into play here? Are the alternatives simply
substitutable, second order factors by which the behavioral units involved satisfy the first order
factors of opportunity or willingness? (see Cioffi-Revilla and Starr 1995). As noted, all of these
points apply to Guidelines #3: “do not introduce factors as control variables merely on the
grounds that they have an impact on the dependent variable,” and #4 as well: “do not control for
variables that are related to each other or the key explanatory factor by definition.” Especially
regarding #4, substitutability can be used here to distinguish between truly substitutable factors
or merely multiple measures of the same factor. Again, for #3 and #4 taking nice laws and
substitutability into account drives the researcher to more clearly and fully specified theory.
Ray (2003b) discusses the complexity of multivariate analyses more fully. However,
Most and Starr reach many of the same conclusions as Ray by discussing the use of quasi-experiments, and looking at different factors as “treatments” which are either present or absent,
and the relationship of the treatment to the occurrence of some dependent variable (see Most and
Starr 1980 and Siverson and Starr 1991 for the use of these ideas in the study of the diffusion of
conflict and war). Ray (2003b:11) notes , “Basically, my argument here is that multivariate
models of interstate conflict should be simplified.”
This can be done in many cases using the
“treatment” strategy. Nice laws would also ask for simplifying analyses by looking at how
relationships stand up to different conditions– using methods as simple as contingency tables!
(See also Ray 2003b:5).
While appearing in his discussion of guideline #4, the following
example provided by Ray (2003a:18) illustrates these points:
Running two different analyses, one with contiguity as the control variable for
geographic proximity and another with distance between capital cities as a control
variable, is a perfectly legitimate strategy. Selecting the model with the “best”
results from the point of view of the analyst is also justified., especially if the
analyst can come up with some plausible theoretical conjecture that might account
for the differences in the two analyses. This strategy is clearly preferable, in my
view, to one involving the inclusion of both contiguity and distance as control
variables in the same model or analysis.
Guideline #5 states: “control for possible differences between across space and over time
relationships.” Within the spirit of nice laws, #5 looks at the most frequently used, and perhaps
most important of those conditions that would vary, and affect the applicability of our
models/theories. Time and space are key elements in many studies that look to see “under what
conditions” a model or theory might hold. Starr (2003) argues that, generally, time has taken
precedence over spatial factors (and for a variety of understandable reasons). Most of our
research has stressed the temporal dimension or temporal context. This is easily seen in the
design of our research where temporal patterns are central, the use of time series data and designs
are standard, and the use of time to delineate of our units of analysis is standard. Scholars
routinely divide time into historical eras or different international “systems” with a temporal
boundary. We do so because we think that theoretically (either explicitly or implicitly) some
conditions have changed so much, or on some important dimension, that these changed
conditions should alter the way that humans behave or how our theories should work (or cease to
work). Changes in technology, especially, as well as the changes in geopolitical structure that
often follow wars, also affect crucial space-time relationships (e.g. Boulding’s loss-of-strength
gradient). Simply put, guideline #5 is extraordinarily important, especially from a nice law
perspective.
A FEW LAST WORDS
As with Most and Starr, Ray has tried to highlight problems that come with an unthinking
acceptance and use of the “norms or customary practices” (2003a:28) found in the research
designs of IR scholars. He has illustrated these problem, for example, with his five guidelines for
the use of control variables. In the last lines of his article he concludes (2003a:28):
Were these guidelines to be adopted in future research efforts, the results of
empirical evaluations of multivariate models might be more readily interpretable
and understood. Ideally, research on international conflict would then be more
cumulative, as well as more productive of valuable insights into those processes
leading to interstate conflict and war.
These lines capture the spirit and substance of the enterprise set forth in Most and Starr.
Let me also provide the last five lines of that work (1989:190):
As Ben Most once noted (1986:15) it may be very “useful to think from time to
time about how we work. While there’s probably not too much that any one of us
can do to make ourselves more intelligent, we certainly can learn to reason and
research more efficiently or with greater effectiveness.”
REFERENCES
Achen, Christopher. 2002. “Toward a New Political Methodology: Microfoundations and ART.”
Annual Review of Political Science 5: 423-450.
Bennett, D. Scott, and Timothy Nordstrom. 2000. “Foreign Policy Substitutability and Internal
Economic Problems in Enduring Rivalries.” Journal of Conflict Resolution 44: 33-61.
Braumoeller, , Bear F., and Gary Goertz. 2003. “The Statistical Methodology of Necessary
Conditions.” In Gary Goertz and Harvey Starr (eds.) Necessary Conditions: Theory,
Methodology, and Applications. Lanham, MD: Rowman & Littlefield.
Bremer, Stuart A. 1992. “Dangerous Dyads: Conditions Affecting the Likelihood of Interstate
War, 1816-1965.” Journal of Conflict Resolution 36:309-341.
Bremer and Cusack, eds.. 1996. The Process of War. Philadelphia: Gordon and Breach.
Chernoff, Fred. 2004. “The Study of Democratic Peace and Progress in International Relations.”
International Studies Review 6: 49-77.
Cioffi-Revilla, Claudio, and Harvey Starr. 1995. "Opportunity, Willingness, and Political
Uncertainty: Theoretical Foundations of Politics." Journal of Theoretical Politics 7: 447-476
Goertz, Gary. 2003. “Cause, Correlation, and Necessary Conditions.” In Gary Goertz and Harvey
Starr (eds.) Necessary Conditions: Theory, Methodology, and Applications. Lanham, MD:
Rowman & Littlefield.
Goertz, Gary, and Harvey Starr, eds. 2003a. Necessary Conditions: Theory, Methodology, and
Applications. Lanham, MD: Rowman & Littlefield.
Goertz, Gary, and Harvey Starr. 2003b. “Introduction: Necessary Condition Logics, Research
Design, and Theory.” In Gary Goertz and Harvey Starr (eds.) Necessary Conditions:
Theory, Methodology, and Applications. Lanham, MD: Rowman & Littlefield.
Lave, Charles A., and James G. March. 1975. An Introduction to Models in the Social Sciences.
New York: Harper and Row.
McGinnis Michael D. 2000. “Policy Substitutability in Complex Humanitarian Emergencies: A
Model of Individual Choice and International Response.” Journal of Conflict Resolution
44: 62-89.
Morgan, T. Clifton, and Glenn Palmer. 2000. “A Model of Foreign Policy Substitutability.”
Journal of Conflict Resolution 44: 11-32.
Most, Benjamin A. 1986. “Getting Started on Political Research.” Paper presented at the annual
meeting of the Midwest Political Science Association, Chicago, April.
Most, Benjamin A., and Harvey Starr. 1989. Inquiry, Logic and International Politics. Columbia,
SC: University of South Carolina Press.
Palmer, Glenn, ed. 2000. Special Issue on “Substitutability in Foreign Policy: Applications and
Advances.” Journal of Conflict Resolution 44 (1).
Ray, James Lee. 2003a. “Explaining Interstate Conflict and War: What Should be Controlled
For?” Conflict Management and Peace Science 20: 1-31.
Ray, James Lee. 2003b. “Constructing Multivariate Analyses (of Dangerous Dyads).” Paper
presented at the 2003 Peace Science Society (International) meeting, Ann Arbor,
November.
Regan, Patrick M. 2000. “Substituting Policies During U.S. Interventions in Internal Conflicts.”
Journal of Conflict Resolution 44: 90-106.
Russett, Bruce, and Harvey Starr. 2000. "From Democratic Peace to Kantian Peace: Democracy
and Conflict in the International System." In Manus I. Midlarsky (ed.) Handbook of War
Studies II. Ann Arbor, MI: University of Michigan Press.
Siverson, Randolph M., and Harvey Starr. 1991. The Diffusion of War: A Study of Opportunity
and Willingness. Ann Arbor: University of Michigan Press.
Starr, Harvey. 1996. "Advancing the Scientific Study of War: A Commentary." In Stuart A.
Bremer and Thomas R. Cusack (eds.) The Process of War: Advancing the Scientific Study
of War. Philadelphia: Gordon and Breach.
Starr, Harvey. 2003. “The Power of Place and the Future of Spatial Analysis in the Study of
Conflict.” Conflict Management and Peace Science 20: 1-20.
Zinnes, Dina A. 1976. “The Problem of Cumulation.” In James N. Rosenau (ed.) In Search of
Global Patterns. New York: Free Press.