Note: Requirements Below
Evaluation should only include
- ALL sections 1-5
- 1 section from 6-10 (based on design in your article), and
- 1 section from 11-12 (based on analysis in your article).
HW1: EVALUATION OF A PEER-REVIEWED ARTICLE (25 points):
Due no later than 11:55p ET on Sunday of Week 3 – submit using
link in Sakai AND to turnitin.com
PURPOSE: The purpose of exercise is to conduct a detailed,
critical evaluation of the research design, methods and analysis of a study
written up and published in a peer-reviewed journal. Students will be
using Wolfer’s (2007) critical questions for evaluating written research to
evaluate an article; these questions are provided below. Tips on Finding
a Peer Reviewed Journal Article may be found under Lessons>Course
Is the title specific enough to differentiate it from other
Do subtitles, if present, provide important information
regarding the research?
Are the main variables expressed in the title?
Are the terms in the title easily understood by most people?
Does the title avoid any reference to the study’s results?
Overall, is this a good title? Why or why not?
Evaluation (2.5 pts)
Are the steps the researcher took to honor ethical responsibilities
to individuals clear? Are they appropriate? Are they enough?
If there were any findings (based on your readings of tables or
other means of data presentation) that refuted the researcher’s hypothesis, did
he address these findings?
If any results were unexpected, did the researcher discuss any
explanations for the unexpected effects?
10.Did the researcher
adequately acknowledge the limitations of the research?
11.Overall, has the
researcher adequately fulfilled his ethical obligations?
3. Literature Review
12.Is the material
presented in the literature review relevant to your research interests?
13.Is the special
problem area identified in the first paragraph or two of the report?
14.Does the researcher
establish the importance of the research problem?
15.Has the researcher
been appropriately selective in deciding what studies to include in the
16.Is the research
17.Is the literature
18.Is the researcher
clear as to what is research, theory and opinion?
19.Overall, do you think
this is an adequate literature review? Why or why not?
Operationalization and Measurement (5.5 pts)
conceptualization suitably specific?
21.Are the definitions
22.How many different
dimensions are being measured at once?
23.Are the various
24.Are the actual
questions (or a sample of them) provided?
25.Is the response
format clear, or, when not already clear, does the researcher provide
information on the response format? Is there any information on restrictions
in respondents’ responses?
26.If the researcher
is using a published instrument, does he or she cite sources where additional
information can be found?
27.Has the researcher
avoided overstating the preciseness of the measurement?
28.Does the researcher
provide some measure of reliability? What type of reliability is
established? Do the measures indicate adequate reliability for your
29.Does the research
provide some measure of validity? What measures of validity are presented
and are they adequate for your purposes?
30.Overall, is the
measurement appropriate and adequate given the research purpose?
5. Sample Strategy
31.Does the research
goal lend itself to generalization? Is the broad sampling method appropriate
for the research goal?
32.Does the researcher
provide information regarding the study population? The sample?
33.Is the exact
sampling method (e.g. simple random, purposive) specified? Remember, it
is not sufficient for a researcher to simply state that a sample was selected
34.Is the sample size
sufficient, given the research goals, the degree of accuracy the researcher
desires, and the nature of the population studied? Given the nature of
the research, is the sample size sufficient?
35.If the researcher
uses a probability sample, does he or she generalize the findings to the
appropriate population? If the researcher uses a non-probability sample,
does he or she refrain from generalizing to a wider population?
36.Overall, is the
should include ONE of the following sections (6-10) (4 pts):
37.Can you identify a
treatment variable that indicates that an experiment is the method of
38.How many groups
If there were two or more groups, did the researcher use random
If the researcher did not use random assignment, did the
researcher present evidence that the groups were similar regarding key
variables at the beginning of the study?
39.Is the treatment
and any pre- or posttests described in sufficient detail that facilitates
If so, is the deception within the parameters of the research
Have the participants been debriefed so they know the true nature
of the study (and can enact their right to privacy by declining to participate
after the fact?)
41.Based on the
description of treatment and experimental procedure, do you see any red flags
regarding ethical issues?
42.Did the researcher
If so, did the researcher state that they were properly trained?
If so, did the researcher specify any special measures to make
sure that the assistants administered the treatment properly?
43.Is the setting
natural or artificial (in a laboratory)?
If it’s in a laboratory, does the researcher recognize that
external validity may be weak?
If it’s in a natural setting, does the researcher recognize that
there may be some differences in the environments of the various groups?
Overall, do you think the experimental design is sound?
44.Is the research
topic worded appropriately for survey research?
45.Did the researcher
specifically state which type of survey method was used?
46.Do the survey
questions adequately address the topic?
47.Are the survey
questions constructed correctly?
48.Did the researcher
provide any information about the response rate? Did the researcher provide any
information about follow-up mailings or other ways of increasing response rate?
What are the implications of the response rate?
49.Did the researcher
explain how he or she guaranteed anonymity or confidentiality?
50.Overall, is the
survey methodology effective and appropriate?
8. Field Research
51.Does the research
describe the selected site? Does the research provide some explanation as to
how that site was chosen?
52.Did the researchers
explain how they addressed gatekeepers?
53.Did the researcher
address how he developed field relations? If conflict arose, did the
researcher make any comment about how personal or research problems in the
field were addressed?
54.Did the researcher
adequately protect the identity of the respondents? Did the researcher address
other ethical considerations?
55.Did the researcher
describe, at least in passing, his method of note taking? Does the method
56.In the analysis,
does the researcher present general patterns of behavior and support those
patterns with data such as quoted comments? Does the researcher use quotes
57.Does the researcher
make any mention of issues of validity and/or reliability?
58.Overall, is the
59.What is the
researcher’s research purpose or hypothesis? Is content analysis an
appropriate method of observation?
60.What are the
researcher’s units of analysis? What are the units of observation (if
they are different than the units of analysis)?
61.Is the researcher
studying a population or a sample of these units? If the researcher is
studying a sample, is it a probability sample? If so, was it correctly
drawn? If the researcher is not studying a population or a probability
sample, is he or she appropriately cautious about the nature of any
62.Does the researcher
identify the characteristics and level of content being analyzed? Does
the researcher explain how material is coded, especially for issues of latent
63.Did the researcher
do any type of pretest with other coders to test for reliability? Where
they any tests for validity?
64.Are the conclusions
consistent with the units of analysis?
65.Are the results
clearly presented and the conclusions appropriate?
66.Generally, is the
method of observation done appropriately?
67.What is the purpose
of the evaluation presented?
68.Is the nature of
the program described in detail?
69.Are the goals
presented and can the goals that the author presents be evaluated?
70.What type of
observation method is used? Is it appropriate, given the real-life restrictions
of evaluation research?
71.Is a control group
used? If so, how has the researcher tried to show that it is equivalent to the
experimental group? If not, does the researcher adequately explain its
72.How are people
selected for program participation? Does this affect the interpretation of
findings, and, if so, does the researcher discuss this?
73.Are the results
74.How does the
article address the other areas of evaluation discussed in earlier chapters?
should include ONE of the following sections (3 pts):
75.Is the results
section a cohesive essay?
76.Does the researcher
connect the results to any general research questions or goals?
77.Is the perspective
of the results presentation appropriate? Does it match the research technique?
78.Has the writer
presented enough examples to support the conclusions? Do the examples
make the readers ‘believe’ the researcher’s points?
79.Do you have reason
to believe that the presence of the researcher influenced the actions or
statements of other group members? If this is possible, has the researcher
addressed it in the research?
80.Especially in field
research (although this may be an issue to a lesser degree in other forms of
qualitative data gathering), does the researcher discuss how he or she
interacted with subjects in the field, what problems arose, and how the
researcher addressed them?
81.Is the results
section a cohesive essay with the important findings highlighted?
82.In the essay, does
the researcher tie the results to the research hypotheses or goals stated in
83.If there are tables
or graphs, are they clearly presented?
84.Does the researcher
present any descriptive statistics?
85.Are the statistics
appropriate for the level of measurement?
86.Are the conclusions
the researcher draws appropriate for the statistical information?
87.In the discussion
section, does the researcher briefly summarize the research purposes,
methodologies, and key findings (in a non-statistical manner)?
88.Does the researcher
acknowledge any methodological or statistical weaknesses?
implications of the research or suggestions for future research discussed?
90.Overall, is the
results section adequate?
91.Overall, is the
discussion section adequate?