Running Head: LITERATURE REVIEW
1
Literature Review: Comparing and Contrasting Articles
Name
Institutional Affiliation
LITERATURE REVIEW
2
In a well-researched qualitative study conducted by Schildkraut & Hernandez (2014) to
examine the effectiveness of strict gun control legislations, it was concluded that gun control
legislation in the United States is not effective in preventing shootings in schools. The study
conducted by Schildkraut & Hernandez (2014) is in contrast with the study conducted by Spitzer
(2014). According to Spitzer (2014), strict gun laws on ownership are essential because it helps
to minimize public safety and health policy goals that were designed to minimize the
unwarranted ownership of the guns. Spitzer (2014) believes that strict gun laws minimize illegal
possession hence resulting in effective eradication of careless shootings.
Both Schildkraut & Hernandez (2014) and Spitzer (2014) agree to the fact that the topic
of strict gun policies should be approached carefully and with caution to avoid infringement of
human rights of others. Even though Spitzer (2014) argues that enactment of strict gun control
laws helps in curbing illegal possession, he shares similar sentiments with Schildkraut &
Hernandez (2014) that gun legislation should be enacted to ease the minds of the public and
further research should be conducted to ascertain the perceptions and attitudes of the public
towards strict guns laws. By doing so, policymakers will be in a position to understand its
reliability and effectiveness.
In a study conducted by Luca, Malhotra, and Poliquin (2017) to examine the impacts of
handgun waiting on the rates of gun violence, it was concluded that increasing the waiting
periods for guns to be delivered to those who purchase helps to reduce the cases of gun violence
in the society. The findings of this study are in agreements with the findings of a study conducted
by Smith-Walter et al (2016). According to Smith-Walter et al (2016), restricting the delivery
periods of a gun purchased helps to minimize the gun violence in the society. Smith-Walter et al
LITERATURE REVIEW
3
state that NRA and Brandy Campaign should be centered on sensitizing the gun sellers to
increase the delivery period.
In a well-researched qualitative study conducted by Taxman (2016) to examine whether
the tri-vector models help to reduce the gun violence in the United States, it was concluded that
tri-vector offers potential solutions to minimizing gun shootings. According to Taxman (2016),
tri-vector models help one to apprehend and understand the complexities revolving the mental
and psychological conditions of the shooters in American. Taxman recommends that there is a
need for the development of programs that increase emotional attachment, empathy, and improve
the mental satisfaction and happiness of individuals past shooters. By doing so, the chances of
past shooters committing such crimes in the future are minimal.
Similar to Taxman (2016) findings that being mentally and psychologically attached to
past gun shooters help to reduce future gun violent acts, the study conducted by Hickey (2013)
revealed that limiting people with mental illness to access guns is important because it minimizes
unconscious shootings that happen because of mental illness. According to Hickey (2013), a
mentally ill person should first be counseled and encouraged to adopt with normal life before
being given with gun license. Counseling of mentally ill persons is relatively important because
it changes their mental perspective towards gun violence. Taxman (2016) and Hickey (2013)
shares one sentiment that mental aspects of an individual contribute to gun violence, therefore,
gun sellers should first try to understand the past mental conditions a person before giving a
license to own a gun.
Wu (2018) conducted a qualitative study by to examine whether cueing and framing has
an impact on youths’ attitudes towards gun rights and gun control. Wu concluded that majority
LITERATURE REVIEW
4
of the youths and especially female do not believe that arming and training staff help to protect
schools from gun shootings. The students who are politically inclined with Republic affiliations
believe that arming of school guards help to reduce gun violence whereas those who subscribe to
Democrats ideologies believe that arming and training of staff escalate the situations.
Similar to Wu (2018) findings, the study conducted by Zgonjanin (2014) to examine the
feasibility of the NRA's proposed actions that support the arming and training of school
personnel in order to counter school shooting incidents revealed that arming of school personnel
is not effective because it affects qualified immunity to such personnel. According to Zgonjanin,
arming personnel to quell gun violence complicates the immunity regarding perils committed in
the course of duty. Zgonjanin states that arming of personnel exposes them to liability for actions
committed by third parties despite the fact that the duty of the personnel is beyond the scope of
their official duties.
In a well-researched qualitative study conducted by Yamane (2014) to examine the
sociological facts surrounding the US gun culture, it was found that misinformed stigma has
been placed on the ownership of the guns. According to Yamane (2014), most of the people do
not own a gun in order to commit crime or shootings but rather they own guns for leisure and
others own it for game hunting. Yamane states that gun ownership should not only be seen in
crime shootings perspective but rather it should be viewed in the angle of participation in nature.
According to Yamane (2014), "gun culture 2.0" should not only be seen in the lenses of
criminology perspective and owners' desire to inflict pain on others but should be seen from
people's desires to leisure, prestige, and other sociological perspectives. Yamane believes that
changing the people's mindset on the ownership of gun requires demystification of the wide-held
LITERATURE REVIEW
5
notions and misconceptions that people in possession of the guns only intends to commit the
crimes. Yamane states that public health reasons for gun controls are totally skewed and only
designed to make people hate gun ownership. The gun controls are skewed because it does not
consider the social desire for ownership.
In a qualitative research conducted by Rajan et al (2016) examine whether the funding
towards gun violence prevention has any impacts on safety and health of Americans, it was
found that amendments such as Dickey Amendments that supports the reduction of funds for gun
research prevention are contributing to an increase in gun violence. Rajan et al (2016) argue that
the Congress should remain active and pass laws that support increasing of funding channeled
towards gun research prevention. According to Rajan et al, the current state of gun violence
prevention research is understudied and underfunded. The authors believe that underfunding is a
drawback in ending gun violence because it denies an opportunity for human security personnel
to mount and sustain the security surveillance. In addition, underfunding and understudy denies
an opportunity for security personnel to pinpoint causes and potential solutions to gun violence.
LITERATURE REVIEW
6
References
Hickey, J. D. (2013). Gun Prohibitions For People With Mental Illness—What Should the Policy
Be?. DEV. MENTAL HEALTH L., 32, 1-2.
Luca, M., Malhotra, D., &Poliquin, C. (2017). Handgun waiting periods reduce gun deaths.
Proceedings of the National Academy of Sciences, 201619896.
Rajan, S., Branas, C. C., Hargarten, S., &Allegrante, J. P. (2018). Funding for Gun Violence
Research Is Key to the Health and Safety of the Nation. American Journal Of Public
Health, 108(2), 194-195. doi:10.2105/AJPH.2017.304235
Schildkraut, J., & Hernandez, T. (2014). Laws That Bit The Bullet: A Review of Legislative
Responses to School Shootings. American Journal Of Criminal Justice, 39(2), 358-374.
doi:10.1007/s12103-013-9214-6
Smith-Walter, A., Peterson, H. L., Jones, M. D., & Nicole Reynolds Marshall, A. (2016). Gun
Stories: How Evidence Shapes Firearm Policy in the United States. Politics & Policy,
44(6), 1053-1088. doi:10.1111/polp.12187
Spitzer, R. J. (2014). New York State and the New York SAFE Act: a case study in strict gun
laws. Alb. L. Rev., 78, 749.
Taxman, J., (2016). Gun Violence in America- A Tri-Vector Model. J. Appl. Psychoanal. Studies
2016, 13: 113-123.
Wu, S., (2018). The Effects of Cueing and Framing on Youth Attitudes towards Gun Control and
Gun rights. Social Sciences, 72(2), 29. Published 15th, February, 2018.
LITERATURE REVIEW
7
Yamane, D. (n.d). The sociology of U.S. gun culture. Sociology Compass, 11(7),
Zgonjanin, M. (2014). When Victims become Responsible: Deputizing School Personnel and
Destruction of Qualified Immunity [notes]. Journal Of Law & Education, (3), 455.
Research Design:
•Experimental Design
•Non-Experimental Design
Research Design
• A research design is a plan of action for executing a research
project:
– Plan for how to test hypothesis
– Usually involves the following:
• When you need to collect data: Before and after comparison
– You take a measurement – then modify something – then take another measurement
– These are called pre and post tests
• How many groups do you need to compare
– Usually: Side by side comparison
– There are 2 groups, but only one receives the treatment
– These are called treatment and control groups
– Ultimately the research design a guide to collecting, analyzing, and
interpreting results
– You must have observable data
• Original or secondary
Research Design
• The choice of research design is affected by:
– The purpose of the research
• Exploratory (one-shot case study) Prediction (aggregate random sample data)
– Resources like time, money, and skill or ethical concerns
• Should you experiment with prison conditions?
– Randomly assign people to max or min security?
• Should you experiment with effects of syphilis/HIV?
• Every research design has positive and negative attributes
– Talking about internal and external validity here
– Tradeoffs
• You should match a design with your requirements
– Not one size fits all
Research Design
• All research designs attempt to:
– Establish a causal relationship between two or more
variables (internal validity)
• The different designs do this with varying degrees of success
– Demonstrate that the results are generally true
(External Validity)
• Generalization
– Eliminate as many alternative explanations as possible
• Biggest problem with being able to make causal statements
• We want variation to be a result of IV and only IV
Research Design
• The goal of scientific research is to find causal relationships. Causal
relationships meet three characteristics:
• Causal is more than correlation
– One event leads to or produces a change in something else
– This is the gold standard and it can be difficult to achieve
1. Covariation: the alleged cause varies with the supposed effect.
2. Time order: the cause precedes the effect in time.
3. Elimination of alternative explanations to isolate causation to one
factor.
• The best research designs incorporate all three
New Twist on Old Themes
• Validity
– Formerly: accurate measure of core idea (UCR
measure of rape)
– Now: Internal Validity
• Manipulation of X is what is really causing Y
• Controlled experiments are great at this
– Now: External Validity
• Generalizable
– Classical Experiments are great but lack this
» Conditions are often times artificial.
» Experimental conditions do not mirror reality.
– Do laboratory conditions hold in real world?
» Example: Gibson and Judicial Advertisements
» Example: Sommers and Juror race characteristics
Terms
• Experimental or Treatment Group
– Receives IV/Treatment
• Control Group
– Does not receive IV/Treatment
• Independent variable = X
– Treatment, shock, intervention
• Dependent Variable = O
• Pre-test: measure of DV before IV
• Post-test: measure of DV after IV
Experimental Design
• The classical randomized experiment has five basic characteristics:
1.
2.
3.
4.
5.
At least one experimental group that will have exposure to the
treatment and one control group that will not.
Randomly assigned individuals to each group, avoiding selfselection.
Controlled administration of the treatment, including the
circumstances under which the experimental group is exposed.
Measurement of a dependent variable before and after the
treatment with a pre-test and post-test. Any difference between the
pre and post-tests can be attributed to the experimental effect of
exposure to the treatment.
Controls the environment of the experiment (time, location, and
other physical aspects).
Experimental Design
• Experimental research designs are especially
good for isolating causal factors.
– Excellent internal validity
• Experimentation allows a researcher to make
causal inferences with great confidence in the
design through control over exposure to an
experimental treatment.
• Experiments often suffer from weak external
validity.
Experimental Design
• We will discuss the following experimental
designs:
• Classic
• Post-Test
• Repeated Measurement
• Multiple group
• Field experiments
Classic Experiment
• R O(t) X O(t)
• R O(c) O(c)
• Intention to vote – negative ad – intention to vote
• Intention to vote – neutral ad – intention to vote
• High Internal Validity – Lower External Validity
Experimental Design
• Post-test
– Shares the characteristics of the classical randomized experiment
except that no pre-test is used because the sample is truly random
and sufficiently large that one can assume that the control and
experimental group(s) are equivalent.
– R X O(t)
– R
O(c)
– Negative ad – intention to vote
– Neutral ad – intention to vote
– High Internal Validity (lower than classic) and lower external validity
• Random assignment improves chance that groups are similar, but without that
pretest you can’t be quite as sure.
Experimental Design
• Repeated-measurement/Interrupted Time
Series
– Adds additional pre-tests, post-tests, or both in an
effort to measure longer-term effects of
experimental treatments to the classic example.
– R OOOOOOOOOOXOOOOOOOOOOO
– R OOOOOOOOOOOOOOOOOOOOOO
– Juvenile crime – Scared Straight – juvenile crime
– Juvenile crime – No Scared Straight – juvenile
crime
Experimental Design
• Multiple group
– Is a modification of the classic example where
more than one experimental group is created to
compare the effects of different treatments.
– R OX1O
– R OX2O
– R OX3O
– R OX4O
Experimental Design
• Field experiment
– Study performed outside laboratory
– An experiment in a natural setting in which the investigator may or
may not have control over group membership but does have control
over one or more independent variables.
– Effects of having a criminal record on employment
• Matched individuals were identical except one applicant had a criminal record
• Outcome measure is whether or not applicant was called back
• Those with criminal records were half as likely to be called back
– Minneapolis Domestic Violence Experiment
– Randomly select residents to receive negative ads and then measure
who voted
• Random sample of 1,400 FL residents in mayoral election
• Treatment group received negative ads
• Official voting records was the DV
Quasi-Experimental Design
• Nonexperimental designs are characterized by
at least one of the following:
– Presence of a single group
– Lack of control over the assignment of subjects to
groups
– Lack of control over the application of the
independent variable
– Inability to measure the dependent variable
before and after exposure to the independent
variable occurs
Quasi-Experimental Designs
•
•
•
•
•
•
Case Studies
Cross section designs
Longitudinal designs
Trend analysis
Panel studies
Intervention analyses
Quasi-Experimental Design
• Case Studies
• Similar to Field Study but researcher does not control
shock
– Shock is observed by researcher
• Foot patrol v. Car patrol
– Compare 2 high crime areas
– Police policies control shock
• New welfare programs
– State and federal government control the shock
• Individual mandate before ACA (compare TX to MA)
– State legislators and governors control shock
Quasi-Experimental Design
• One shot case study
–
–
–
–
X–0
No Pretest and no comparison groups
No random assignment
We do something and make inferences based on
outcome measures
– This is a very weak design but it is used all the time
– Example: Teacher gives instruction and we assume it
causes students to learn
– Example: Officer sensitivity training – excessive force
claims
Quasi-Experimental Design
• One-Group pretest-posttest
– OXO
– Lacks comparison group and random assignment
– Improvement over one shot case study because we know if something
changed from time 1 to time 2
• But we still can’t isolate the independent variable as being the main cause
– Example: Test on day one – student takes class – final exam
• Final exam should be better
– Example: Anti-gang program in school. Measure violence at beginning
and end of year.
– Example: Excessive force claims – training – excessive force claims
Quasi-Experimental Design
• Static Group Comparison
• X-0
–0
• Addition of second group but still no random
assignment
– We don’t know how different groups are before experiment
• Teacher teaching 2 of same classes uses new technique
in one
• Lack of pretest raises possibility that the 2 groups are
different to begin with so it is not only the IV that
causes change in DV
Quasi-Experimental Design
• Cross Sectional designs (survey, aggregate analysis)
– Characterized by measurements of the independent and
dependent variables at approximately the same time.
– No control over assignment or administration of
independent variable
– Data analysis, rather than a treatment, is necessary for
making causal inferences.
– Many subjects – one point in time
– Low internal validity but higher external
– Survey – did you see negative ad?
» Did you vote?
Quasi-Experimental Design
• Longitudinal design
– Allow for the measurement of variables at
different points in time.
– Can model change across time, examine the time
order of a causal relationship, and estimate age,
cohort, and period affects.
– Time Series
– Panel and Cohort
Quasi-Experimental Design
• Panel studies
– Cross-sectional designs that include a time
element.
– The design relies on measurement of the same
units of analysis at different points in time—
creating waves of data for analysis over time.
– Example:
• Analysis of DARE programs – use same students at
same schools during elementary school
Quasi-Experimental Design
• Intervention analysis
• Measurements of a dependent variable both
before and after the introduction of an
independent variable that is observed but not
controlled by the researcher.
• Ex: Presidential Debates
• OOOOOOOOXOOOOOOOOO
Quasi-Experimental Design
• Non-experimental designs are generally
characterized as having less internal reliability
but better external validity than experimental
designs.
• There is always a tradeoff when moving from
one design to another.
Threats to Internal Validity
• Primary Question: Are changes in the independent variable indeed
responsible for the observed variation in the dependent variable
• This is a problem that mainly impacts longitudinal (long-term)
studies
• High internal validity means we have evidence of a strong causal
relationship
– X causes Y
• Think of these as alternative reasons for a change in the DV
– Things other than X that change Y
• The following things lower internal validity because they are all reasons
we might see a change in the DV that is not attributable to the IV
Threats to Internal Validity
• History (group)
– Large scale event that influences most or all of the
participants in your study
– Unanticipated event occurs during course of study
– Changes in the real world
– Example: New anti-depression drug: Pretest: Administered
August 2001 Posttest: Administered October 2001: Result
– people became more depressed
– Example: Pretest: % of underage sale of alcohol –
Intervention (police issue citation) – Posttest: % of
underage sales
• Threat: highly publicized death of person who was underage
Threats to Internal Validity
• Maturation (individual)
– Events that affect the individual
– Got sick, bored, death in family, lost insurance, had a
baby, lost job during the course of the study
– Example: New anti-depression drug: Pretest –
treatment and someone lost job – posttest
– Example: New anti-depression drug: Pretest –
treatment and someone gets large inheritance –
posttest
Threats to Internal Validity
• Mortality
– Subjects drop of out study for variety of reasons
– This become problematic if the students that leave are
systematically different than those that remain
• Example: DARE evaluations from middle school through
high school
• Students move, drop out, refuse to participate etc.
• This is a problem if those that are no longer in the study are
different from those who remain
• Those that drop out may be the ones most likely to not be
affected by DARE which makes program more effective
Threats to Internal Validity
• Instrumentation
– Hawthorn Effect
• People know they are being studied and alter their behavior
• Sometimes the pretest will make people aware of what you are
studying
• Implementation
– Researcher unintentionally influences respondents
• Usually not deliberate
• Researches may give subtle cues as to how they want person to
respond (tone of voice, facial expressions, etc)
• Double-blind
Summary
• Experimental vs. Quasi-Experimental
• Must carefully specify design in order to make
conclusions
• Validity and Reliability
The Survey: Asking People
Questions
Chapter 6
Introduction to Political Research
Critical Assumption
• Goal of Survey: People reveal true preferences
– People pick choice that is closest to their true attitude
• Assumption: People have crystallized opinions
– Based on the rational-choice model of behavior
– Well thought out, defined, fixed, and actual opinions
• They have weighed the costs and benefits of each issue and have
selected alternative that provides them with maximum benefit.
– People know about topics, issues, subjects asked about
– People have single peaked preferences
• They have one and only one choice that gives them maximum
utility
Introduction to Political Research
Critical Assumption
• If this is true we should see consistent answers:
– When people are asked same question at different times:
• Only about 50% give the same answer with repeated questioning
• In open ended questions – 50%-75% give conflicting views on the
same exact policy
– They see good and bad in welfare, food stamps, ACA, foreign policy, etc.
• People should not be swayed by variations in word
choice
– Welfare v. financial assistance to low wage earner
• 20% more likely to support assistance to low wage earners
– ACA v. Obamacare
• 48% favor Obamacare vs. 70% favoring ACA
– Terminating pregnancy v. abortion
Introduction to Political Research
Critical Assumption
• People won’t be affected by question order
– Religion first then abortion
• 15%-20% less likely to offer support
– If you ask about life of mother first
• Support increases for overall ability to have abortion
– Soviet reporters before US reporters
• 37% in favor if asked before allowing US
• 73% in favor if asked about Soviet reporters after
– Policy questions after knowledge questions
• More likely to decline to answer
Introduction to Political Research
Question Wording Effects
• Obamacare vs. Affordable Care Act
Introduction to Political Research
Zaller and Feldman
• 2 Primary theories
– On-line Theory
• We have crystalized opinions
• Opinions are stable but not completely static
– Constantly updating our opinions with new information
• Have consistent view though maybe not detailed
• Example:
• People have consistent views on Voter ID laws though
they may not know the rules of their own state or the
recent decisions regarding the legality of the laws
Introduction to Political Research
Zaller and Feldman
• Doorstep Theory
– Formulate opinions off the top of our heads
• People have conflicting opinions
• Answer based on what is most salient to them at the time
– We have a mix of emotions that are made salient by what
is immediately present to us
• Leads to response instability and different answers at different
times
– People are not rational actors
– What really matters when I’m asked to offer an opinion
(survey, voting booth) I base that decision on what is most
relevant and salient at the time.
– Opinions are unstable and constantly fluctuating
Introduction to Political Research
Social Surveys
• Surveys are some of the most widely used datagathering technique in the social sciences and other
fields
– But expensive
• Available option is omnibus survey
• In a social survey, a large number of people (called
respondents) hear and give an answer to the exact
same questions
– They may answer many questions about past behaviors,
experiences, opinions and characteristics
– Respondents are revealing their true opinions to
interviewers
Introduction to Political Research
Types of Information Gained from
Surveys
• Attitudes/Beliefs/Opinions
– Govt shutdown
– Obamacare
– Contraception mandate
– Presidential Approval
– Elections
– Building a Wall
– Jailing people that get abortions
– Senate and Supreme Court nomination
Introduction to Political Research
Types of Information Gained from
Surveys
• Individual Behavior
– People
• Drug use, where get information (TV, newspapers, etc),
healthcare, smoking, drinking, voting
• Characteristics of Individuals
– People on food stamps, those receiving welfare,
those who have been incarcerated,
unemployment, victims of crime
Introduction to Political Research
Surveys and Variables
• You can use one survey to measure many variables and
test multiple hypotheses at the same time
• Survey Questions are variables
– Use answers to test hypotheses
• Ask about demographic characteristics and presidential vote
• Ask about adequacy of care coverage and support for Obamacare
• Which is why it is critical you know what information
you want before you introduce survey into field
– If I don’t ask the question, I can’t test the hypothesis
• I may want to know if respondent has son or daughter who is gay
and how that impacts support for same-sex marriage
Introduction to Political Research
Steps in Conducting a Survey
Introduction to Political Research
Conducting a Survey
• Start Up Stage: In this stage, you address the following
three questions,
– Who will be the respondents of your survey?
• Random sample of people, potential voters, people in halfway houses,
students, etc…
– What information do you want to learn from them? and
• Becomes the basis for your questions (based on your hypotheses)
– Type of and design of survey?
• If self-administered: Questionnaire = fixed collection of questions in a
social survey that respondents answer.
– Mail, Online surveys
– Physical layout should be clear and easy to follow
• If you have interviewers: Interview schedule = a questionnaire
specifically designed for an interviewer asking respondents the
questions.
– Telephone or personal interviews
Introduction to Political Research
Conducting a Survey
• Execution stage
– Locate the sampled respondents
• Call them, drive to house, etc.
– Provide respondents with information about the
survey and instructions on how to complete it.
– Record information
• Data Analysis Stage
– This stage will be discussed in later chapters
Introduction to Political Research
Principles of Good Question Writing
• Avoid Confusion (be clear!)
– Do you go to doctor regularly?
– What is your Income?
• Avoid jargon, slang and abbreviations
– DOMA, Polarized, ABA, ACA
• Avoid emotional language (neutral language)
– Socialist Obamacare
– Doctor who murders babies vs. abortion doctor
• Avoid Prestige bias (prestige bias – associating
a statement with a prestigious person or
group)
Introduction to Political Research
Good Question Writing
• Avoid double-barreled questions
– Support marriage and civil unions?
– Support lifetime caps and keeping children on insurance?
• Avoid leading questions
– Bush/McCain example (2000)
• Would you be more likely or less likely to vote for John
McCain if you knew that he fathered an illegitimate
black child?
• Don’t you think it is your patriotic duty to not burn the
flag?
• Avoid overlapping or unbalance response categories
– People are 10-20% more likely to agree with something if
you only offer them an opportunity to agree.
Introduction to Political Research
Types of Questions
• Threatening Questions
– Questions about sensitive topics
• Using drugs, unconventional sexual behavior, health questions (venereal
disease), income, evading taxes
– Easier after rapport established
– Face to face more successful
• Skip or Contingency Questions
– If answer yes to one question, ask another
– If no, skip to another section
• Open v. Closed Ended Questions
• Open-ended question format = Survey questions that allow respondents to
give any answer.
• Closed-ended question format = Survey questions in which respondents must
chose among fixed answer choices.
– Predetermined answer categories
Introduction to Political Research
Introduction to Political Research
Open v. Closed Ended Questions
Open Ended Questions
Advantages
Disadvantages
Permit an unlimited number of possible answers
Different respondents give different degrees of detail
Respondents can answer in detail and can quality/clarify
responses
Responses may be irrelevant or buried in useless detail
Unanticipated findings can be discovered
Comparisons and statistical analysis become very
difficult
Permit adequate answers to complex issues
Coding is difficult
Permit creativity, self-expression and richness of detail
Articulate and highly literate respondents have an
advantage
Reveal a respondent’s logic, thinking process and frame
of reference
Questions may be too general for respondents who lose
direction
Responses are written verbatim, which is difficult for
interviewers
Great amount of respondent time, thought and effort is
necessary
Respondents can be intimidated by questions
Answers take up a lot of space in questionnaires
Introduction to Political Research
Open v. Closed Ended Questions
Closed Ended Questions
Advantages
Disadvantages
Easier and quicker for respondents to answer
Can suggest ideas that the respondent would not
otherwise have
Answers of different respondents are easier to compare
Respondents with no opinion or no knowledge an answer
anyway
Answers are easier to code and statistically analyze
Respondents can be frustrated because their desired
answer is not a choice
Response choices can clarify questions meaning for
respondents
It is confusing if many (e.g., 20) response choices are
offered
Respondents are more likely to answer about sensitive
topics
Misinterpretation of a question can go unnoticed
There are fewer irrelevant or confused answers to
questions
Distinctions between respondent answers may be blurred
Less articulate or less literate respondents are not at a
disadvantage
Clerical mistakes or marking the wrong response is possible
Replication is easier
They force respondents to give simplistic responses to
complex issues
They force people to make choices they would not make in
Introduction to Political
Research
the real
world
Writing Good Closed Format Response
Choices
• Answer choices should have three features:
– Mutually exclusive
• Response categories do not overlap
• Likert
– Exhaustive
• Each respondent has a choice
• To be safe – don’t know option
– Balanced
• Offer the favorable or unfavorable choices equally
– Reduces bias
Introduction to Political Research
Caution….
• Social Desirability Bias = A tendency for survey
respondents to answer in a way that conforms to
social expectations or makes they look good
rather than to answer honestly.
• Can result in overestimating some things:
– Did you vote?
– Do you support efforts to reduce discrimination?
• Can result in underestimating other things:
– Are you going to vote for Trump?
– Are you using illegal drugs?
Introduction to Political Research
Caution
• Interviewer bias
– Race and gender of interviewer effect
– Support for Affirmative Action increases if someone perceives
that they are speaking with minorities
– Support for abortion rights increases if women are asking the
questions
• Length of Survey
– People become tired, bored, and impatient
– Hang up the phone
– Will just start answering questions without really thinking about
them
– Phone interviews must be shorter (20 minutes) vs. in person
interviews (can be an hour or longer)
Introduction to Political Research
Types of Surveys
Mail and Self Administered Surveys
Advantages
Disadvantages
Give questionnaires directly to respondents or mail
them
Respondents to not always complete or return them
Least expensive
Mail surveys have a low response rate
Can be conducted by a single researcher
Some are returned in 2 weeks, other trickle in
Can cover a wide geographical area
Conditions can not be controlled for mail surveys
Respondents can complete the questionnaire when it
is convenient and check personal records if necessary
Questions can not be clarified and probing is not an
option
Anonymity
Someone other than the targeted individual may
complete
Avoids interviewer bias
Respondents may not complete them in a single
sitting
Effective
Format limits the kinds of questions that can be used
Response rates may be high for educated target
population that has a strong interest in the topic or
survey organization
Questions requiring visual-aids or open-ended
questions are difficult
Introduction to Political Research
Types of Surveys
Web Surveys
Advantages
Disadvantages
Fast
Coverage
Inexpensive
Privacy and confidentiality
Flexible design options
Design Issues
Can include visual aids
Introduction to Political Research
Types of Surveys
Telephone Surveys
Advantages
Disadvantages
95% of the population be reached by
phone
Higher cost and limited interview length
Response rates can reach 90%
Impossible to reach some respondents
Flexible with most of the strengths of a
face-to-face interview, but half the cost
Call may come at an inconvenient time for
respondent
Reduces anonymity
Introduces potential interviewer bias
Introduction to Political Research
Types of Surveys
Face-to-Face Surveys
Advantages
Disadvantages
Highest response rate
High cost
Permits the longest questionnaires
Training, travel and supervision costs and
efforts are high
Observe surroundings and use nonverbal
communication and visual aids
Interviewer bias is the greatest
Well-trained interviewers can ask all types
of questions, can ask complex questions
and use extensive probes
Introduction to Political Research
Types of Surveys
Introduction to Political Research
Introduction to Political Research
Interviews
• Role of an interviewer
– Structured conversation
– They may simulate conversation, but everything
they say is scripted
– There should be no variation in how one interview
is conducted compared to the next
– There should be no variation from interviewer to
another
– The same questions should be asked to different
people
Introduction to Political Research
Interviews
• Stages of an interview
– Intro – purpose and confidentiality
– Middle – ask substantive questions
– Exit – Thank the respondent and let them know when
results will be available
• Training interviewers
– Receive extensive training before being placed in the field
– They must know how to use computer programs
– Probe = A neutral request made by an interviewer to
clarify an ambiguous answer, complete an incomplete
answer, or obtain a relevant response.
Introduction to Political Research
Time Dimension
• Cross-Sectional
– Single point in time – snapshot
– Surveys
• Longitudinal
– Time series
– Panel – same people – DARE Evaluation
– Cohort – different people with same characteristics
• People born in 1980, people that began employment at
same time
Introduction to Political Research
Sampling
• Random Digit Dialing = Computer based random
sampling of telephone numbers.
– Not using a telephone book as sampling frame
• Will miss unregistered numbers which in some areas can be
40%
– Using all possible numbers in an area
• Area code – exchange – 4 numbers
• Within Household Samples = Random sampling
from within households
– Ask question to determine respondent.
Introduction to Political Research
Public Support
For
Against
No
Opinion
2005 May 2-5
74%
23%
03%
2004 May 2-4
71%
26%
03%
2003 Oct 6-8
64%
32%
04%
2003 May 19-21
70%
28%
02%
2003 May 5-7
74%
24%
02%
2002 October 10
70%
25%
05%
2002 May 6-9
72%
25%
03%
2001 Oct 11-14
68%
26%
06%
2001 May 10-14
65%
27%
08%
March 2015
Sampling: How to Select a Few to
Represent the Many
Chapter 4
Introduction to Political Research
Introduction
•
•
•
•
70% doubt accuracy of polls
How do I know that 70% don’t trust?
Did I ask everyone in the US?
I using a sample to tell you that between 67%
and 73% of people don’t trust samples
• If conducted properly with a high enough
sample size we can make inferences about
billions of people
Introduction to Political Research
Introduction
• But people remain skeptical
• Which is sometimes justified
– Poorly conducted surveys do not provide reliable data
– But if done scientifically, they should be trusted
• Pertinent Questions:
– Who collected it?
– What is sample size?
– How was sample generated?
• “If you don’t believe in random sampling, next
time tell the doctor to take it all”
Introduction to Political Research
Introduction
• There are good polls and bad polls
• CNN:
– Hillary: 62
– Trump: 27
– Scientific Poll: National random sample of 530 registered voters who watched
the debate
– CI = 4.5%
• Online Polls:
–
–
–
–
–
Hillary: 39
Trump: 53
Non-Scientific Poll: Asked people to vote who found the poll
Prone to bias
People that cast votes are not representative, can cast multiple votes, people
can cast votes even if they don’t live in the united states.
– And these poll results were reported!
Introduction to Political Research
Overview
• 8 Main Types of Samples
• 4 are representative
– Which means I can make a prediction of an overall
population based on my sample
– I can construct confidence intervals
• Margin of error
• Because we are not 100% certain (it is an estimate after all)
• 4 are not representative
– There is no know probability of being selected for a
sample out of a population
– Which means I can’t make inferences
Introduction to Political Research
Overview
• Known Probability of Selection (generalization)
– Random
• Known
• Everyone has equal probability of selection
• Size of probability will be based on size of sample vs. overall population
– Systematic
• Known probability
• Probability will vary based on sampling interval
– Stratified
• Known probability
• Probability is based on strata used
– Cluster
• Known probability
• Probability based on cluster units
Introduction to Political Research
Overview
• Non-Probability (difficult to generalize)
– Convenience/haphazard
• No know probability
– Quota
• No known probability
– Purposive
• No known probability
– Snowball
• No known probability
Introduction to Political Research
Generalizability
• What do we do when we generalize our results?
– How applicable to situations outside of our sample
• Random sample is critical
– Probability theory is based on random samples
– Reduces bias
• Generalize Minneapolis study of mandatory arrest for abuse to US?
– 1980
– When police were called for domestic violence they were instructed to 1.
arrest 2. counsel 3. mandatory separation
– Lower recidivism for those arrested
• Interviews of 7 Louisiana Supreme Court Justices
– Generalized to all state supreme court judges
• Generalize innovative reentry program from Cleveland, OH to entire US?
– Inmates begin to meet with coach 6 months prior to release
• Random sample of 1,067 registered voters to US?
Introduction to Political Research
Reasons People are Skeptical
• My phone didn’t ring
• I see bumper stickers and signs all over
• Wrong a few times
– Literary Digest
• FDR v. Landon (1936)
• Incomplete sampling frame (automobile lists)
– 2000?
• Most were correct but within margin of error
• Personal experiences
– Individual experiences in TX vs aggregate trends
Introduction to Political Research
Introduction to Political Research
Why Sample?
• Cost of census is too high
– 13 billion for 2010
– Hired 500,000 people to follow up on 34% of houses
that did not return form
• Estimation is accurate – no need to capture
everyone
– Probability theory
– Confidence Intervals
• Effort by Clinton Administration to change census
process
– Majority do not state that estimates are precise
Introduction to Political Research
Sampling
• Process for selecting sources of the observations used
in a research study
• Sample – a smaller collection of units taken from a
larger collection
• Two Types
– Nonprobability
• Cheap and easy
• Exploratory research
– Probability
• Known probability of being sampled
• Allows for estimation of population parameter and confidence
interval
Introduction to Political Research
Fundamental Question to Always Act
• How representative is my sample?
Introduction to Political Research
Types of Nonprobability Samples
• Haphazard, Accidental or Convenience
–
–
–
–
Easy and cheap but of limited value
Select cases until we achieve our sample size
No specific criteria
Select cases that are available, willing, or look “normal”
• Can lead to selection bias
• May avoid some people because they look threatening
– But those people are still part of overall population we are attempting to
sample from
– Ex. Cable News surveys
– You get information, just not representative information
• You can get very unrepresentative samples
– Good for exploratory research purposes
Introduction to Political Research
Convenience Sample Example
• Your question:
– How do US students intend to vote in the
presidential election?
• Sampling Technique:
– Stand outside of Olin on Friday afternoon
between 9:00am and noon.
– Talk to people until I get 200 responses
• Not Representative
Introduction to Political Research
Types of Nonprobability Sampling
• Quota
– Still fast and cheap to conduct
– Improvement over convenience sampling but still
not great
– First: You select relevant categories
– Second: Determine overall sample size and size
required for each category
– Finally: Select cases haphazardly until you fill
quota
Introduction to Political Research
Example
• Your Question: How do UA students intend to vote in the
presidential election?
– And I think gender matters
– And I know that UA has 60% female and 40% male enrollment
• Sampling Technique
– First: Decide on overall sample (we will use 100)
– Second: Determine the breakdown based on your prior
information (60-40 gender)
– Finally, haphazardly select 60 women and 40 men to interview.
• More accurate data
– So now I am representative on gender
– But I may not be on age, race, religion, etc.
– So still very problematic
Introduction to Political Research
Introduction to Political Research
Types of Nonprobability Samples
• Purposive or Judgmental
– Best used if you have a specific target population in mind and if
that population may be difficult to reach.
– Here you do not want a representative sample as you are not
interested in what the overall population looks like
– Uses the judgment of an expert in selecting cases or selected
with a specific purpose in mind
– Fenno (soak and poke)
• Members of congress who spend most of their time in either districts
or Washington
– Deviant populations (Drug Dealers/Users)
– Tearoom study
• Also raised several privacy concerns
• Don’t want a sample of anyone using a public restroom
Introduction to Political Research
Types of Nonprobability Samples
• Snowball (network)
– Best used for difficult to reach populations
– Each case is connected with another through
direct and nondirect linkages
• Begin with a few cases and then spread or snowball out
– Drug dealers, prostitutes, gang leaders, etc
– Sometimes people use financial incentives
Introduction to Political Research
Snowball Sampling
Introduction to Political Research
Why Use Probability Sampling
• Minimize bias
– They are purely mathematical or mechanical
– Selection due to chance (flip of coin)
• Reduce bias because everyone has chance of selection
– Allow calculation of probability of outcomes with
great precision
• Can obtain a sampling error (e.g., the degree to which a
sample deviates from a population)
• Obtain a representative sample
– Generalizability
Introduction to Political Research
Important Terms
•
•
•
Population – larger pool of cases from which a sample is drawn or taken
– Who or what we generalize to
– Usually unknown
– Does not need to be people: Can be countries, newspaper articles, etc.
– All felons in OH
– All voters in OH
Sampling Ratio – ratio of the size of the sample to the size of the target population
– n/N
– Target Population = 50,000 people
– I sample 150
– Sample Ratio = 150/50,000 which is .3
Sampling Frame – a specific list that closely approximates all elements in the
population
– ODC records, phone book, voting registration, etc.
– I sample from this list
– Incomplete lists lead to biased and inaccurate results
Introduction to Political Research
Important Terms
•
•
Population Parameter – any characteristic of a population that we estimate
– Unknown (data I don’t have)
– Percentage of people that smoke, percentage of people who support
Hillary/Trump, percentage of people who own a gun, percentage of people
who have had an abortion, percentage of people who have been sexually
assaulted
Statistic – information from the sample, most often used to estimate population
parameters
– Estimate of unknown (based on data I do have)
– Point estimate with a bubble of confidence around it
– Percent support for Hillary/Trump is a population parameter
– We do not know the true level of support but we do know within +/- 3%
Introduction to Political Research
Sampling Ratio
• Sampling Ratio – ratio of the size of the
sample to the size of the target population
Introduction to Political Research
Key of Probability Sampling
• RANDOM SAMPLING
– These samples are most likely to produce a sample
that truly represents the unknown population
– Random in sampling world means a technique where
each element in a population has a known and equal
chance of selection
– Probability theory requires random sampling to
generate inferences
• Statistical theory and mathematical proofs are based on a
known probability of selection
• i.e. the match between our sample and the population
• It allows us to know how much our sample estimates deviate
from the population
Introduction to Political Research
Random Sampling
• Need Unbiased Sampling Frame
• A sample drawn in which a random process is used to
select units from the population
• Best to get an accurate representation of the population
– Essential for treatment and control groups
– Difficult to conduct
– Expensive
• Always have a margin of error (e.g., confidence interval)
– N = 1,067 3%
– Confidence Level= on repeated sampling – usually 95%
– Margin of error = 50% +/- 3
Introduction to Political Research
Types of Probability Sampling
• Simple Random Sample
– This is a type of random sample (obviously)
• This is the easiest to do
– Sample elements selected from the frame based on a
mathematically random selection procedure
– Steps:
• 1st: Select Sampling Frame
– Each elements in sampling is numbered from 1 and on until the last
element of sample
• 2nd: Randomly select elements
– A computer does this
– Import sample frame, tell computer the sample size and then it
computes a random list
• 3rd: Locate elements
– Call, go to home, send questionnaire, etc.
Introduction to Political Research
Simple Random Sample
• Accurate sampling frame critical (Literary Digest)
• Even phone book might contain non-response
bias
• Selection based on mathematically random
procedure
– The researchers selects the sampling frame
– Computers randomly select who is selected
• Confidence Intervals =
• CI: sample statistic +/- confidence level (SD)
Introduction to Political Research
Types of Probability Sampling
• Systematic
– This is a type of probability sample
– Elements are selected at predetermined intervals rather
than randomly
– An approximation to random sampling in which you select
one in a certain number of sample elements, the number
is from the sampling interval
– Sampling interval (the size of the sample frame over the
sample size, used in systematic sampling to select units)
• N/n
• If I need a sample of 100 and my sample frame is 1,000, my
sampling interval is 10
• Thus, I select every 10th case
Introduction to Political Research
Types of Probability Samples
• Steps:
–
–
–
–
1.
2.
3.
4.
Locate Sampling Frame
Determine sample size
Calculate Sampling Ratio
Locate every case based on sampling ratio
• Congressional bills, newspaper articles, judicial
decisions, etc.
• Bias – underling order to list (Military Squads,
GPA 1, 51, and 101 different from 50,100, and
150)
Introduction to Political Research
Types of Probability Sampling
• Stratified
• This is a type of random sample
• A type of random sampling in which a random
sample is drawn from multiple sampling
frames, each for a part of the population
• You have prior information about the overall
population (just like in quota)
• You want to be certain to include these cases
Introduction to Political Research
Types of Probability Samples
• Steps
– 1st: Population is divided into subpopulations (strata)
• Based on the criteria you select
– Age, Gender, Race, Religion, etc…
– 2nd: Random sample is drawn from each strata
• This is what separates it from quota
– Example:
• I want a sample of 1,000 students from a University with
10,000 students
• I know that 60% are female and 40% are male
• My 2 strata consist of 6,000 females and 4,000 males
• Next I randomly sample from each group until I have 600
females and 400 males
Introduction to Political Research
Stratified Sampling
Introduction to Political Research
Types of Probability Samples
• Cluster (multi-stage)
• This is a type of random sample
• A multi-stage sampling method, in which clusters
are randomly sampled, then a random sample of
elements is taken from sampled clusters.
– NO SAMPLING FRAME
• No national list of professors working at colleges with less
than 10,000 students or crime victims
– Clusters are temporarily treated as cases themselves
– Each cluster contains cases that will eventually be
sampled
Introduction to Political Research
Types of Probability Samples
•
•
•
•
•
•
Example:
National Crime Victimization Survey
There is no national list of crime victims
DoJ uses cluster sampling
Randomly select Counties (First Cluster)
Randomly select geographical districts (Second
Cluster)
• Randomly select houses (Third Cluster)
• Randomly select anyone over age 12 in the house
(Final Sample)
Introduction to Political Research
Cluster (Multi-Stage) Sampling
Introduction to Political Research
Cluster (Multi-Stage Sampling)
Introduction to Political Research
Introduction to Political Research
Sampling
• Random Digit Dialing = Computer based
random sampling of telephone numbers.
– Not using a telephone book as sampling frame
• Will miss unregistered numbers which in some areas
can be 40%
• Miss people without landlines
• People who have moved
• People who only have a cellphone
– Using all possible numbers in an area
• Area code – exchange – 4 numbers
Introduction to Political Research
Cell Phones
•
•
•
•
It is important to try and sample cellphone only individuals/households
2004: 10% were cellphone only
2016: 47% are cellphone only
If not accounted for, sample is not representative because cell phone only
individuals are different from those with landlines
– Cellphone only users are younger, have less education, less income, etc.
– Missing this group results in sample that is not representative
• They can pay companies for their lists of cellphone only numbers
• Or they can use RDD
– Cellphones have area codes and unique exchanges
– Once you know the area code and the unique 3 number exchange you
randomly generated last 4 numbers
• Both are expensive
– Must pay for list
– Federal law requires that cell phones be dialed by an actual human which
costs money because you have to hire people to make calls and ask questions
Introduction to Political Research
Within-Household Sampling
• Within Household Samples = Random sampling
from within households
– Ask question to determine respondent.
– Not only randomly sample house, you randomly
sample person living there
– Example: “Can I talk to the person over 18 who had
the last birthday?”
• Sampling Hidden Populations
– Hidden Population = A group that is very difficult to
locate and may not want to be found, and therefore,
are difficult to sample.
• Use purposive or snowball
Introduction to Political Research
Size of the Sample
• Depends on the kind of data analysis and desired accuracy of the sample
– Large sample size does not automatically guarantee representative sample
• Bad sampling frame produces inaccurate results no matter how many you sample
• With an accurate sample, your estimates become more accurate the greater your sample
size but there are diminishing returns based on mathematical proofs
• This is the central limit theorem
• Rule of Thumb – to be 95% confident in your estimate with a +/- 3%
margin of error…
–
–
–
–
Diminishing returns
Small populations (under 1000), large sampling ratio (about 50%)
Moderately large populations (10,000), smaller sampling ratio (about 10%)
Large populations (over 150,000), less than 1%
• Magic number, based on mathematical proofs, is 1067
• And the same to say we are 95% confident in our estimate if there are 300,000,000
Introduction to Political Research
Purchase answer to see full
attachment