Is the title specific enough to differentiate it from
other related topics?
Do subtitles, if present, provide important information
regarding the research?
Are the main variables expressed in the title?
Are the terms in the title easily understood by most
Does the title avoid any reference to the study’s
Overall, is this a good title? Why or why not?
2. Ethical Evaluation
Are the steps the researcher took to honor ethical
responsibilities to individuals clear?
Are they appropriate? Are they enough?
If there were any findings (based on your readings of
tables or other means of data presentation) that refuted the researcher’s
hypothesis, did he address these findings?
If any results were unexpected, did the researcher
discuss any explanations for the unexpected effects?
the researcher adequately acknowledge the limitations of the research?
has the researcher adequately fulfilled his ethical obligations?
3. Literature Review
the material presented in the literature review relevant to your research
the special problem area identified in the first paragraph or two of the
the researcher establish the importance of the research problem?
the researcher been appropriately selective in deciding what studies to include
in the literature review?
the research cited recent?
the literature review critical?
the researcher clear as to what is research, theory and opinion?
do you think this is an adequate literature review? Why or why not?
and Measurement (5.5 pts)
the conceptualization suitably specific?
the definitions productive?
many different dimensions are being measured at once?
the various dimensions sufficient?
the actual questions (or a sample of them) provided?
the response format clear, or, when not already clear, does the researcher
provide information on the response format?
Is there any information on restrictions in respondents’ responses?
the researcher is using a published instrument, does he or she cite sources
where additional information can be found?
the researcher avoided overstating the preciseness of the measurement?
the researcher provide some measure of reliability? What type of reliability is established? Do the measures indicate adequate reliability
for your purposes?
the research provide some measure of validity?
What measures of validity are presented and are they adequate for your
is the measurement appropriate and adequate given the research purpose?
5. Sample Strategy (3
the research goal lend itself to generalization? Is the broad sampling method
appropriate for the research goal?
the researcher provide information regarding the study population? The sample?
the exact sampling method (e.g. simple random, purposive) specified? Remember, it is not sufficient for a
researcher to simply state that a sample was selected ‘randomly.’
the sample size sufficient, given the research goals, the degree of accuracy
the researcher desires, and the nature of the population studied? Given the nature of the research, is the
sample size sufficient?
the researcher uses a probability sample, does he or she generalize the
findings to the appropriate population?
If the researcher uses a non-probability sample, does he or she refrain
from generalizing to a wider population?
is the sampling appropriate?
should include ONE of the following sections (6-10) (4 pts):
you identify a treatment variable that indicates that an experiment is the
method of observation?
many groups were studied?
If there were two or more groups, did the researcher
use random assignments
If the researcher did not use random assignment, did the researcher present evidence that
the groups were similar regarding key variables at the beginning of the study?
the treatment and any pre- or posttests described in sufficient detail that
If so, is the deception within the parameters of the
Have the participants been debriefed so they know the
true nature of the study (and can enact their right to privacy by declining to
participate after the fact?)
on the description of treatment and experimental procedure, do you see any red
flags regarding ethical issues?
the researcher use assistants?
If so, did the researcher state that they were properly
If so, did the researcher specify any special measures
to make sure that the assistants administered the treatment properly?
the setting natural or artificial (in a laboratory)?
If it’s in a laboratory, does the researcher recognize that
external validity may be weak?
If it’s in a natural setting, does the researcher
recognize that there may be some differences in the environments of the various
Overall, do you think the experimental design is sound?
the research topic worded appropriately for survey research?
the researcher specifically state which type of survey method was used?
the survey questions adequately address the topic?
the survey questions constructed correctly?
the researcher provide any information about the response rate? Did the
researcher provide any information about follow-up mailings or other ways of
increasing response rate? What are the implications of the response rate?
the researcher explain how he or she guaranteed anonymity or confidentiality?
is the survey methodology effective and appropriate?
8. Field Research
the research describe the selected site? Does the research provide some
explanation as to how that site was chosen?
the researchers explain how they addressed gatekeepers?
the researcher address how he developed field relations? If conflict arose, did the researcher make
any comment about how personal or research problems in the field were
the researcher adequately protect the identity of the respondents? Did the
researcher address other ethical considerations?
the researcher describe, at least in passing, his method of note taking? Does the method seem adequate?
the analysis, does the researcher present general patterns of behavior and
support those patterns with data such as quoted comments? Does the researcher
use quotes selectively?
the researcher make any mention of issues of validity and/or reliability?
is the research adequate?
9. Unobtrusive Measures
is the researcher’s research purpose or hypothesis? Is content analysis an appropriate method of
are the researcher’s units of analysis?
What are the units of observation (if they are different than the units
the researcher studying a population or a sample of these units? If the researcher is studying a sample, is it
a probability sample? If so, was it
correctly drawn? If the researcher is
not studying a population or a probability sample, is he or she appropriately
cautious about the nature of any conclusions?
the researcher identify the characteristics and level of content being
analyzed? Does the researcher explain
how material is coded, especially for issues of latent content?
the researcher do any type of pretest with other coders to test for
reliability? Where they any tests for
the conclusions consistent with the units of analysis?
the results clearly presented and the conclusions appropriate?
is the method of observation done appropriately?
is the purpose of the evaluation presented?
the nature of the program described in detail?
the goals presented and can the goals that the author presents be evaluated?
type of observation method is used? Is it appropriate, given the real-life
restrictions of evaluation research?
a control group used? If so, how has the researcher tried to show that it is
equivalent to the experimental group? If
not, does the researcher adequately explain its omission?
are people selected for program participation? Does this affect the
interpretation of findings, and, if so, does the researcher discuss this?
the results clearly explained?
does the article address the other areas of evaluation discussed in earlier
should include ONE of the following sections (3 pts):
the results section a cohesive essay?
the researcher connect the results to any general research questions or goals?
the perspective of the results presentation appropriate? Does it match the
the writer presented enough examples to support the conclusions? Do the examples make the readers ‘believe’
the researcher’s points?
you have reason to believe that the presence of the researcher influenced the
actions or statements of other group members? If this is possible, has the
researcher addressed it in the research?
in field research (although this may be an issue to a lesser degree in other
forms of qualitative data gathering), does the researcher discuss how he or she
interacted with subjects in the field, what problems arose, and how the
researcher addressed them?
the results section a cohesive essay with the important findings highlighted?
the essay, does the researcher tie the results to the research hypotheses or
goals stated in the introduction?
there are tables or graphs, are they clearly presented?
the researcher present any descriptive statistics?
the statistics appropriate for the level of measurement?
the conclusions the researcher draws appropriate for the statistical
the discussion section, does the researcher briefly summarize the research
purposes, methodologies, and key findings (in a non-statistical manner)?
the researcher acknowledge any methodological or statistical weaknesses?
the implications of the research or suggestions for future research discussed?
is the results section adequate?
is the discussion section adequate?