Commentary
pubs.acs.org/jchemeduc
Pencil−Paper Learning Should Be Combined with Online Homework
Software
David B. Smithrud* and Allan R. Pinhas*
Department of Chemistry, University of Cincinnati, Cincinnati, Ohio 45221-0172, United States
ABSTRACT: For the past eight years, we have used online homework software as a pedagogical tool for undergraduate students
(almost all are second-year students) learning organic chemistry. Being strong proponents of homework, we were surprised to
find that about half of the students who complete 90% or more of the online homework assignments received less than 50% of
the available points on examinations, even though comparable problems were given. Students who combined traditional pencil−
paper problem solving with the online software performed better than students who solely used the online homework package.
Our findings contribute to the ongoing debate between pencil−paper and computer learning with the recommendation that they
should be combined to provide students with the ultimate learning tool.
KEYWORDS: Second-Year Undergraduate, Organic Chemistry, First-Year Undergraduate/General, Communication/Writing,
Computer-Based Learning, Hands-On Learning/Manipulatives, Misconceptions/Discrepant Events
■
INTRODUCTION
Organic chemistry is and always has been a difficult course to
master for most students. A student’s success in a chemistry
course is strongly correlated to on-task studying of the material
through solving homework problems.1 With large undergraduate class sizes, such as the ones at the University of
Cincinnati, hand grading of homework problems and providing
students with valuable feed-back is impractical. Online
homework overcomes most of these obstacles. The computer
software acts as the instructor, providing the correct answers,
guidance, and tutorials.2 Furthermore, the software is a
diagnostic tool for the instructor to monitor individual and
overall performance. Students appreciate the immediate
feedback provided by the software, which reinforces the topic
being taught and fosters a more positive study attitude.3,4
However, it has been found that most students need to receive
points toward their final scores before they will complete online
homework problems.3 Over the past decade, many studies have
shown a positive relationship between online homework and a
student’s total points/course grade.1−6 In general, the total
points/course grade for the students who perform well on the
online homework is higher than for students who do not
complete online homework assignments. In addition, those
students who complete online homework on average do better
on examinations in comparison to those students who do only
written homework.7
For our organic courses, taught by four professors over the
past eight years, we observed the same positive relationship
between online homework and examination performance as
determined by the examination averages. However, looking
more deeply into these data, we unexpectedly discovered that
excellent performance on online homework is not a predictor of
examination performance. Specifically, we find that half of the
students who earn the majority of points on the online
homework are significantly below average on the examinations.
In addition, we find that students who combined traditional
pencil−paper problem solving with the online software
© 2015 American Chemical Society and
Division of Chemical Education, Inc.
performed better than students who solely use the online
homework package.
■
METHODONLINE HOMEWORK SETUP
For those only familiar with online homework packages for a
general chemistry course, please be aware that online
homework for organic chemistry is somewhat different. Similar
to general chemistry problems, the organic package contains
multiple-choice questions, but the possible algorithmic
variations of a single problem are much more limited. In
addition, unlike the general chemistry problems, online
homework for organic chemistry contains a large number of
problems that require using a drawing program to show the
product of a reaction or to show the arrows for a mechanism.
The organic professors at the University of Cincinnati have
used Connect or OWL as homework packages. These programs
were chosen, in part, because they are integrated with course
textbooks and contain a subset of the back of the chapter
problems. Connect and OWL are both interactive homework
packages providing students with feedback, guidance, and
tutorials. The problems are parametrized such that if a problem
is repeated, it is a new problem that investigates the same
concept with different values or molecules. For Connect, we
also provided LearnSmart, which is an interactive tool that
adaptively assesses a student’s knowledge and confidence level
of organic topics. Both Connect7 and OWL8 have been shown
to improve student performances in organic chemistry courses.
Regardless of which package was used by us, online
homework accounted for about 15% of the total points for
the course. For the online homework, we assigned 15 problem
sets with questions based on the material discussed in each
hour and a half class period. To earn all the possible points, a
student needs to solve around 160 problems per semester. The
remaining points in the course were generally divided into
quizzes worth between 0% and about 15%, two or three
Published: October 12, 2015
1965
DOI: 10.1021/ed500594g
J. Chem. Educ. 2015, 92, 1965−1970
Journal of Chemical Education
Commentary
Table 1. Examination Average versus Online Homework Data
Parameter
Exam averagea for all students, %
Exam averagea for students who earned 90% or more of the
online homework points, %
Exam averagea for students who earned 50% or fewer of the
online homework points, %
Exam median for all students, %
Exam median for students who earned 90% or more of the
online homework points, %
Exam median for students who earned 50% or fewer of the
online homework points, %
a
Connect
(N = 1143)
OWL
(N = 2280)
ACS Examination (Connect
Homework) (N = 246)
ACS Examination (OWL
Homework) (N = 219)
48
55
50
56
46
52
52
59
31
33
40
42
49
56
50
55
44
49
50
61
29
33
36
38
Standard deviation for each of the averages is approximately 17%.
Figure 1. Connect percentage versus examination percentage. The left graph is all the students, and the right graph is those students who earned
90% or more of the online homework points. The correlation coefficient for the left graph is 0.450 and for the right graph is 0.205.
examinations worth a total of 40−55%, and a final examination
worth about 30% of the total points. The examinations use
pencil and paper and do not contain multiple-choice questions.
A full range of typical organic chemistry questions are asked
including topics such as nomenclature, physical and chemical
properties of molecules, stereochemistry, products from
reactions, synthesis, and mechanisms.
The authors believe that online homework is a very
promising pedagogical tool. Because we are strong proponents
of homework problems as an instructional tool for organic
chemistry, students are given a large number of attempts and
are given feedback after each attempt. We realize that this
method of administering the online homework enables a
student to acquire all the homework points through guessing.
On the other hand, we felt that limiting the number of attempts
would be more stressful by treating homework as an
examination rather than a learning experience. Our approach
is supported by a recent study that found students were most
appreciative of the immediate feedback, the opportunity of
multiple attempts, and the ease of the Connect program when
solving online homework.7 In two of our large classes
(approximately 350 students), we discovered that most
students completed each problem set using only two to three
attempts (the average is 2.2 ± 0.4 attempts per problem set).
Furthermore, we found that most students spent a significant
amount of time working homework problems, apparently in an
attempt to master organic chemistry.
In this study, we use Pearson correlation coefficients to
measure the linear correlation between the percentage of
examination points versus the percentage of online homework
completed. A statistical analysis was performed on students’ use
of the Connect program (averaged time spent actively working
on the problems and the averaged number of attempts) to
determine if there was a significant difference in the methods
used by students to complete the online homework assignments.
■
RESULTS
For the past two academic years, we used Connect. For our
courses, a total of 1143 students completed organic chemistry.
As can be seen in Table 1, we compare the percentages of
online homework completed to examination scores. We do not
compare percentages of online homework completed to the
total points because the total points include the online
homework, and thus, bias the results. We also exclude quiz
scores since some courses did not give quizzes. As the data
show, those students who earn 90% or more of the points for
the online homework did better on the examinations than the
class as a whole (56% vs 48% of total examination points, Table
1). In contrast, students who complete less than 50% of the
online homework generally do more poorly (31% vs 48% of
total examination points, Table 1). The large standard deviation
of approximately 17% makes the difference in these numbers
less relevant. Because of this rather large standard deviation for
the averages, we also looked at the median value and obtained
similar percentages. Thus, our findings are consistent with
previous literature studies; those students who do well on the
online homework, on average, do better on the examinations
than the class as a whole.
1966
DOI: 10.1021/ed500594g
J. Chem. Educ. 2015, 92, 1965−1970
Journal of Chemical Education
Commentary
Figure 2. OWL percentage versus examination percentage. The left graph is all the students, and the right graph is those students who earned 90%
or more of the online homework points. The correlation coefficient for the left graph is 0.443 and for the right graph is 0.115.
Figure 3. Online homework percentage versus ACS standardized exam percentage. The left graph is for Connect, and the right-hand graph is for
OWL. The correlation coefficient for the left graph is 0.337 and for the right graph is 0.402.
When the examination percentage versus the online
homework percentage is plotted (Figure 1), we find it is true
that students who perform poorly on the online homework also
perform poorly on the examinations. However, we find no
relationship between online homework points and examination
scores for those students who earn most of the online
homework points (Figure 1, right graph). The Pearson
correlation coefficient drops from 0.405 when comparing all
students to 0.205 when comparing only students who solved at
least 90% of the online problems. Put another way, if a student
does poorly on the online homework, he or she will likely
perform poorly on the examinations. However, if a student does
really well on the online homework, one cannot predict how
well that student will perform on the examinations.
Next, we wanted to determine whether this was a general
trend or something specific to the Connect package we are
currently using. In previous years, we used the OWL homework
software. Because we used it for a longer period of time, we
have more data. As can be seen in Table 1, the general trend is
the same as seen with Connect. The higher the percentage of
points earned for the online homework, the higher the
examination average or median; however, the same high
standard deviation is obtained. Plotting these data, as shown in
Figure 2, shows that students who perform poorly on the
online homework perform poorly on the examinations, but
once again, there is no correlation between completing 90% of
the OWL homework and performing well on the examinations.
One could readily argue that the lack of correlation between
online homework performance and examination performance is
due to the fact that the problems given on the examinations are
substantially different than the ones given online. We believe
this is not true. As discussed previously, the problems given on
the examinations are standard organic chemistry questions, and
most of the examination problems are very similar to the
homework problems. As an aid to the students to help them
study, they are told at the beginning of each course that
examination problems are very similar to the online homework
problems. Furthermore, the scores reported were obtained
from multiple classes taught by educators with different
teaching styles.
To further explore the possibility that there is a
disconnection between our examination questions and the
online questions, we compared the online homework scores
with the percentage of questions answered correctly on the
ACS standardized examination (Table 1 and Figure 3). The
same trend as discussed earlier was found with these
standardized examinationsthere is no relationship between
doing well on the online homework, using either Connect or
OWL, and how a student will perform on the ACS
examination. (Because not all classes gave the ACS
examination, we have fewer data points.)
1967
DOI: 10.1021/ed500594g
J. Chem. Educ. 2015, 92, 1965−1970
Journal of Chemical Education
■
Commentary
DISCUSSION
Because the graphs and the correlation coefficients are so
similar, to expedite the discussion of the data, in Figure 4, we
We can test for student apathy because the Connect program
provides the number of attempts and the total time actively
working on each homework assignment. A student would only
have to spend a small amount of time online to randomly click
responses, randomly draw structures, or input correct answers
obtained from a classmate. If student apathy is the key, then we
should observe a correlation between students’ attempts or
time spent on online homework versus examination percentage.
We plotted the total amount of time a student spent actively
engaged on homework problems versus the examination
percentage for students who earned 90% or more of the online
homework points for one large class (134 students out of a
total of 240 students, Figure 5). There is no correlation.
Figure 4. Homework percentage versus examination percentage. The
data shown in Figure 1 and in Figure 2 have been combined, and the
graph has been divided into four quadrants of equal size.
have combined the Connect (Figure 1) and OWL (Figure 2)
data and have arbitrarily divided the graph into four quadrants
of equal size. The students in Quadrant 2 (Q2) are generally
well motivated, well prepared, and most likely would excel in
organic chemistry independently of the style of homework.
Notice that there are virtually no students in Quadrant 4 (Q4),
namely, those students who do poorly on the online
homework, but do well on the examinations.
The students in Quadrant 3 (Q3) are the “nonparticipants”.
They do not complete the homework assignments and
apparently do not study for examinations. This group is the
main reason why the examination average or median for those
who do online homework is higher than the examination
average or median for the class as a whole.
The final group of students lies in Quadrant 1 (Q1). They
score well on the homework assignments but perform poorly
on the examinations. As seen with the entire population of
students, there is no correlation between percentage of
examination scores and percentage of online homework
completed (Pearson correlation coefficient = 0.224). These
students, who comprise approximately 40% of the class, would
benefit the most from a well-crafted homework system. Most
educators likely would agree that teaching students on a oneon-one basis is ideal. Current online homework programs
provide this service conveniently because students can work a
problem multiple times with constant feedback. So why are
there so many students in Q1?
Because of the nature of the material in organic chemistry,
the algorithmic variations for a problem are much more limited
than for general chemistry problems. Thus, an unmotivated
student may just randomly click responses until the correct
answer is obtained. If one considers that the solution for many
problems requires using a template to draw structures or to
provide mechanistic arrows, the likelihood that students
randomly provide correct answers is diminished. Another
possibility, of course, is that a classmate simply provides the
answers. In other words, these students are apathetic; they do
not care about learning the material, only in obtaining the
points.
Figure 5. Time spent on online homework versus examination
average. The data were taken from a single large organic chemistry
class. The correlation coefficient for the graph is 0.000.
Statistically, the same amount of time was spent on homework
for students who received greater than 70% of the examination
points as for students who received less than 40% of the
examination points (1640 ± 500 min and 1640 ± 700 min,
respectively). Similarly, as mentioned earlier, the average
number of attempts per problem set is 2.2 ± 0.4 regardless
of examination scores.
We now turn to the possibility that there could be an
inherent problem with current online homework software that
may hinder learning organic chemistry, and possibly material in
other courses as well. There appears to be a slowly evolving
paradigm shift in learning at all levels from pencil and paper to
computers. Ever since computers have become a standard
resource in the classroom, there have been many interesting
studies exploring the benefits and costs of using computer
software versus traditional pencil and paper in learning.9−14
One possible supposition is that students who solve the
majority of the homework problems using the online software,
but performed poorly on examinations, did not benefit from
drawing organic molecules and reaction mechanisms with a
pencil. Support for this hypothesis comes from a report that
shows learning by handwriting facilitated subjects’ memorization of graphic forms as compared to using a computer
only.9−11 Similarly, these authors9
[C]ompared children’s learning of a series of abstract
graphic forms depending on whether they simply looked at
the forms or looked at them as well as traced the forms with
their index finger. The tracing movements seemed to improve
the children’s memorization of the graphic items.
This type of “motor-memory” is also important in adult
learning, including learning new symbols.12 A recent fMRI
1968
DOI: 10.1021/ed500594g
J. Chem. Educ. 2015, 92, 1965−1970
Journal of Chemical Education
Commentary
(functional magnetic resonance imagining) study on adults13
demonstrated that processing letters produced by writing
versus by typing occurs in different parts of the brain. Writing
letters via pencil−paper produced greater activity in areas that
are involved in imagery, execution, and the observation of
actions. Using pencil−paper has been recently found beneficial
for taking class notes as well. Students who type their class
notes do not retain knowledge as well as students who
handwrite notes because of a shallower processing of the
information.14 Thus, motor-memory is obtained from handwriting or tracing but not from typing. It follows from these
studies that students benefit from drawing new shapes, such as
molecules and reaction mechanisms, with a pencil in
comparison to the palette used in current software.
In contrast, a recent study7 showed that for organic
chemistry, students who used online homework had a higher
percentile score than those who used only pencil−paper for
homework. The author did not discuss if the students who used
only pencil−paper for homework had a solutions manual as a
guide. Furthermore, students who used online homework may
or may not have used pencil−paper to solve the problems prior
to putting the answers online.
To investigate the possibility that handwriting the answer to
a problem is beneficial for examination performance, we asked
students who used Connect if (a) they used only the computer
software to solve problems, or (b) they wrote out the answer
with pencil−paper first and then transferred the answer to the
online program. Out of the 206 respondents, 128 (62%) stated
they first used pencil−paper to solve any online homework
problem (population of students who use pencil and paper with
the online homework = Pp−p/ol), whereas the remaining 78
(38%) used solely the online homework package (population of
students who only used the online package = Pol). Because the
students answered just yes or no to our question, we do not
know how often paper and pencil were used to solve a problem
prior to putting the answer online. The mean examination
percentage for Pp−p/ol is 57 (standard deviation = 18) and for
Pol is 51 (standard deviation = 20). The t-value (148 degrees of
freedom) is 2.17 (tcrit = 1.65), and the p-value = 0.0157, which
indicates a greater than 98% confidence level that the difference
between the means is statistically significant. Similar results are
obtained when comparing the two populations of students
against the performance on the ACS examination. Only a
subset of these student populations took the ACS examination
(Pp−p/ol = 53 students, and Pol = 32 students). The mean
examination percentage for Pp−p/ol is 50 (standard deviation =
15) and for Pol is 43 (standard deviation = 16). The t-value (55
degrees of freedom) is 1.96 (tcrit = 1.67), and the p-value =
0.0273, which indicates a greater than 97% confidence level that
the difference between the means is statistically significant.
Thus, students who initially used pencil−paper to solve at
least some of the organic problems performed better on the
examinations than students who used only the computer to
solve problems. Placing these results into our quadrant system
shows that about one-third of the students who first handwrote
some of their answers (Pp−p/ol) are in Q1, whereas over half of
the students who only used the computer (Pol) are in Q1. Our
goal of getting students into Q2 is better accomplished when
students combine pencil−paper with their online homework
package (68% vs 45% of the students received greater than 50%
of the examination points, Figure 6).
A statistical analysis was performed using the Connect
software diagnostics to see if these two groups of students
Figure 6. Online and handwriting versus online only. Students who
initially handwrote solutions to some of the online problems
performed better on examinations than students who solely used the
online homework package.
could be differentiated on the basis of their use of the program
or if they were just given the answers by friends, instead of
whether or not they used pencil and paper. Both groups of
students solved the same percentage of the homework
problems (Pp−p/ol solved 84 ± 20%, and Pol solved 84 ± 18%
of the homework problems). On average, students who used
pencil−paper with the online homework (Pp−p/ol) spent 1700 ±
600 min to complete the homework problems, whereas
students who used only the online software (Pol) on average
spent 1800 ± 900 min to complete the homework problems.
The average number of attempts to complete each of the 15
weekly assignments was similar as well. For Pp−p/ol, four
assignments averaged a single attempt, seven assignments
averaged two attempts, three assignments averaged three
attempts, and one assignment averaged greater than three
attempts. For Pol, four assignments averaged a single attempt,
seven assignments averaged two attempts, two assignments
averaged three attempts, and two assignments averaged greater
than three attempts. Because the two groups are statistically
very similar, it appears that both populations of students spent a
similar amount of time on homework, took advantage of the
online homework guidance features, and were not just given the
answers by friends. Thus, our results suggest that initially
solving homework problems with pencil−paper appears to be
an important method for learning.
On the basis of these findings (Figure 6), we now
recommend that our students use pencil and paper to do
their online homework, and thus to study for their
examinations. However, because students are naturally reluctant
to complete homework twice, a better solution would be to
develop further online homework software that uses a stylus
and onscreen writing to enable students to hand draw
molecular structures and mechanisms. Such software has been
demonstrated but not applied to online homework software.15,16 The combination of pencil−paper with computer
learning would likely benefit students who study other
disciplines as well.
■
CONCLUSION
Online homework has succeeded in getting students of all
abilities to complete homework problems. We find that
students spend many minutes on each problem and take
advantage of the hints and the repeated attempts. What needs
to be improved are the examination scores for about half of the
students who have earned 90% or more of the online
homework points. In this commentary, we introduce the
possibility that the lack of handwriting may account for part of
the poor examination performance. Handwriting causes motormemory, which appears to be important for organic chemistry
1969
DOI: 10.1021/ed500594g
J. Chem. Educ. 2015, 92, 1965−1970
Journal of Chemical Education
Commentary
(15) Cooper, M. M.; Grove, N. P.; Pargas, R.; Bryfczynski, S. P.;
Gatlin, T. OrganicPad: An Interactive Freehand Drawing Application
for Drawing Lewis Structures and the Development of Skills in
Organic Chemistry. Chem. Educ. Res. Pract. 2009, 10, 296−301.
(16) Silverberg, L. J.; Tierney, J.; Bodek, M. J. Use of Doceri Software
for iPad in Online Delivery of Chemistry Content. J. Chem. Educ.
2014, 91, 1999−2001.
students who are drawing complex molecular compounds. The
authors encourage more studies on the ramifications of learning
online versus the traditional use of pencil and paper. More
specifically, studies on the ramifications of using a mouse and a
keyboard versus using some type of a writing tool will be
helpful. These studies would not only benefit students who take
organic chemistry, but also all students who learn online.
■
AUTHOR INFORMATION
Corresponding Authors
*E-mail: allan.pinhas@uc.edu.
*E-mail: david.smithrud@uc.edu.
Notes
The authors declare no competing financial interest.
■
ACKNOWLEDGMENTS
We thank our colleagues Deborah Lieberman and James Mack
for allowing us to incorporate their student performances into
our data. We also thank the anonymous reviewers of this paper
for many very helpful suggestions.
■
REFERENCES
(1) Leinhardt, G.; Yaron, D.; Cuadros, J. One Firm Spot: The Role of
Homework as Lever in Acquiring Conceptual and Performance
Competence in College Chemistry. J. Chem. Educ. 2007, 84, 1047−
1052.
(2) JCE Staff.. Comparison of Features, Electronic Homework
Management Systems. J. Chem. Educ. 2009, 86, 693.
(3) Parker, L. L.; Loudon, G. M. Case Study Using Online
Homework in Undergraduate Organic Chemistry: Results and Student
Attitudes. J. Chem. Educ. 2013, 90, 37−44.
(4) Drelick, J.; Henry, Z.; Richards-Babb, M.; Robertson-Honecker, J.
Online Homework, Help or Hindrance? What Students Think and
How They Perform. J. Coll. Sci. Teach 2011, 40, 81−93.
(5) Revell, K. D. A Comparison of the Usage of Tablet PC, Lecture
Capture, and Online Homework in an Introductory Chemistry
Course. J. Chem. Educ. 2014, 91, 48−51.
(6) Eichler, J. F.; Peeples, J. Online Homework Put to the Test: A
Report on the Impact of Two Online Learning Systems on Student
Performance in General Chemistry. J. Chem. Educ. 2013, 90, 1137−
1143.
(7) Malik, K.; Martinez, N.; Romero, J.; Schubel, S.; Janowicz, P. A.
Mixed-Methods Study of Online and Written Organic Chemistry
Homework. J. Chem. Educ. 2014, 91, 1804−1809.
(8) Evans, J. OWL (Online WebBased Learning). J. Chem. Educ.
2009, 86, 695−696.
(9) Mangen, A.; Velay, J.-L. Digitizing Literacy: Reflections on the
Haptics of Writing. Advances in Haptics 2010, 385−401.
(10) Naka, M.; Naoi, H. The Effect of Repeated Writing on Memory.
Memory & Cognition 1995, 23, 201−212.
(11) Hulme, C. The Interaction of Visual and Motor Memory for
Graphic Forms Following Tracing. Quarterly Journal of Experimental
Psychology 1979, 31, 249−261.
(12) Longcamp, M.; Boucard, C.; Gilhodes, J.-C.; Velay, J.-L.
Remembering the Orientation of Newly Learned Characters Depends
on the Associated Writing Knowledge: A Comparison Between
Handwriting and Typing. Human Movement Science 2006, 25, 646−
656.
(13) Longcamp, M.; Boucard, C. L.; Gilhodes, J.-C.; Anton, J.-L.;
Roth, M.; Nazarian, B.; Velay, J.-L. Learning Through Hand- or
Typewriting Influences Visual Recognition of New Graphic Shapes:
Behavioral and Functional Imaging Evidence. Journal of Cognitive
Neuroscience 2008, 20, 802−815.
(14) Mueller, P. A.; Oppenheimer, D. M. The Pen Is Mightier than
the Keyboard: Advantages of Longhand Over Laptop Note Taking.
Psychological Science 2014, 25, 1159−1168.
1970
DOI: 10.1021/ed500594g
J. Chem. Educ. 2015, 92, 1965−1970
Workshop 2: Deciphering a Research Paper
What type of information was presented in the introduction? What did you learn from the introduction
that you did not know before?
What is the purpose of the methods section? Why do you think results aren’t reported in the methods
section?
Which figure plots the percentage of homework questions students got correct vs. the percentage they
earned on their exams? What are the results of this plot?
Why was the statistical test the Pearson correlation coefficient used? What does a large value for this
test mean? How can the authors determine whether their results mean anything using this test? (You
will need to research this test, but don’t spend too much time on this.)
How do the statistics for the OWL homework system stack up to the statistics for the Connect
homework system?
Provide a reason why there are almost no students in quadrant Q4 in figure 4. What are the parameters
for this quadrant (essentially what are the axes)?
What are some limitations of online homework systems (meaning, what are some things that writing
with pen and paper may be better for)?
How did students who wrote out their answers while doing online homework compare to those
students who didn’t write anything out first on their exams?
What is the difference between the results section of the paper and the discussion section of the paper?
Why do you think they are separated?
Summarize the author’s conclusions. Do you agree with their conclusions? How do their conclusions
help to explain the graph below, which was prepared from real class data from a CH100 exam given in
the Spring of 2016?
Purchase answer to see full
attachment