Creative Education
2013. Vol.4, No.11, 683-693
Published Online November 2013 in SciRes (http://www.scirp.org/journal/ce) http://dx.doi.org/10.4236/ce.2013.411097
Open Access 683
Assessment of an Interactive Internet Program to Educate
Children Aged 7 - 9 about Science, the Brain and Drugs*
Mary P. Metcalf
Clinical Tools, Inc., Chapel Hill, USA
Email: metcalf@clinicaltools.com
Received October 4th, 2012; revised February 6th, 2013; accepted February 15th, 2013
Copyright © 2013 Mary P. Metcalf. This is an open access article distributed under the Creative Commons At-
tribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the
original work is properly cited.
BrainTrain4Kids.com is an interactive science Internet program for children aged 7 - 9 developed with
funding from the National Institute on Drug Abuse (NIDA). Based on NIDA’s classroom curriculum,
Brain Power!, we adapted and expanded this material to optimize online media. The primary objective of
the curriculum is to provide an early foundation for drug abuse prevention efforts by educating elemen-
tary school-aged children about the brain and how alcohol, tobacco, and drugs can harm it. Secondary
aims include improving knowledge, appreciation for scientific inquiry, and improving/sustaining a posi-
tive attitude towards science. The program employs engaging and entertaining elements including Flash-
based activities and educational games. Assessment of the program occurred using a two-group, random-
ized case/control pre/post trial with a crossover design using a convenience sample. Child participants (N
= 102) from control and case groups had similar knowledge and attitudes towards science at baseline. At
post-intervention, there was a significant increase in knowledge scores for the case group; this increase
was retained at the six-week follow-up. Case group attitudes towards science were more positive immedi-
ately after post-intervention than at baseline, and at follow up than at baseline. BrainTrain4Kids can be an
effective tool for educating children about science and drugs, and has the potential to positively impact at-
titudes. It can be used as a part of a drug abuse prevention program either in schools or at home.
Keywords: Neuroscience Education; Substance Abuse Prevention; Internet Based Education
Introduction
In society, concerns abound that the age of first use of drugs
is decreasing. The National Parents’ Resource Institute for
Drug Education (2002) reports that among 4th graders, 2.1%
had smoked cigarettes in the past year, 5% had drunk beer,
2.7% had used inhalants, and 0.5% had used marijuana. An
increase in all types of drug use was documented by the time
children reached the 6th grade. DuRant, Smith, Kreiter, &
Krowchuk (1999) found that among middle school children,
10.1% of smokers started at age nine or younger. Furthermore,
the CDC (2007) reports that 24% of US students drank alco-
hol for the first time prior to age 13. Experience with alcohol
significantly increases between 4th - 6th grade (Donovan et al.,
2004) and rises with each grade level, with boys more likely to
have alcohol experience than girls. Although most school drug
prevention programs are oriented to a middle school audience
(Drug Strategies, 1999; e.g., NREPP, 2009; Winters, Fawkes,
Fahnhorst, Botzet, & August, 2007), evidence indicates that
drug prevention should actually begin in elementary school.
One study found that the use of alcohol in elementary school
increased the use of alcohol in middle school by nearly three
times (Wilson, Battistich, Syme, & Boyce, 2002). Moreover,
the same study found that early use of tobacco and marijuana in
elementary school increased the use in middle school by five
and four times, respectively. Hence, discouraging children from
using drugs at an elementary school age could be beneficial in
later years.
There is also the problem of children falling behind in the
sciences to consider. A study has documented that fourth grad-
ers’ performance in math and science is decreasing relative to
other countries (NCES, 2003). Moreover, in examining average
science scores for fourth grade US students, 2007 scores were
not measurably different compared to scores in 1995 (Gonzales
et al., 2008). Hence, science education in the US is in need of
improvement.
The problem of drug abuse and erosion of knowledge in sci-
ence can be addressed in tandem with a science-based drug
prevention program. Focusing on early-age prevention, it is
important to examine an effective mode for educating a young
audience. Various health education programs have effectively
used interactive technologies to teach children and youth in
areas such as heart disease (e.g., Lindsay, Christie, Gaw, Pack-
ard, & Shepherd, 1994), asthma (e.g., Lieberman, 2001), HIV
(e.g., Schinke, Orlandi, Schilling, & Parms, 1992), and water
quality (e.g., Yuan, Manderson, Tempongko, Wei, & Aiguo,
2000). Thus, we predict that a technology-based curriculum can
help the efficacy of drug prevention programs. It was found that
the use of animated cartoons in a multimedia intervention sig-
nificantly increased elementary student’s science comprehen-
*Conflicts of Interest: The author is employed by Clinical Tools, Inc. The
development and assessment of the program were funded by a Small Busi-
ness Innovation Research Award contract awarded to Clinical Tools, Inc. by
the National Institutes of Health/National Institute on Drug Abuse.
M. P. METCALF
sion and knowledge (Dalacosta, Kamariotaki-Paparrigopoulou,
Palyvos, & Spyrellis, 2009), suggesting that certain technology
features are engaged to a younger audience. Our rationale is
that, via technology, we can help children learn and appreciate
science. We wanted to explore the practicality of technological
solutions to the delivery of drug abuse prevention programs,
which leads to our main research question: will an interactive,
multimedia, Internet-based version of a science and drug abuse
prevention curriculum be effective in changing knowledge and
attitude of children about science and drugs? To answer this,
we had two aims: 1) create, and 2) evaluate such a program to
investigate whether an Internet curriculum effects changes in
knowledge and attitudes.
We created a science education Internet curriculum, partly
adapted from NIDA’s classroom curriculum entitled Brain-
Power! The NIDA Junior Scientist Program, whose primary
target audience is 2nd - 3rd graders. We enhanced this program
to use the full potential of the Internet and designed it with
features including animated cartoons, Flash movies, quizzes
which provide immediate and customized feedback, 3D mod-
eling with objects that can be manipulated, supplemented in
some instances with audio. There are significant differences
between the two curricula, which necessitated a separate study
to determine the impact of BrainTrain4Kids on children in the
target age range. To our knowledge, no interactive Internet
program is designed solely for educating children aged 7 - 9
about science, neuroscience, and drugs for independent use.
The primary goal is to lay the foundation for future drug abuse
prevention efforts and scientific learning by educating children
at an early age about the brain, the differences between helpful
and harmful drugs, and how alcohol, tobacco and drugs affect
the brain and body. Secondary objectives include improving
knowledge and appreciation for science and scientific inquiry,
and building or maintaining a positive attitude towards science.
Evaluation of the curriculum involved:
Instruments: the development of instruments to assess
knowledge and science/drug attitudes which were created in
consultation with education experts.
Formative Analysis: a small pilot study of those instruments
to determine the instruments’ viability to measure knowledge,
attitude and satisfaction with the program.
Summative Study: a final evaluation, to assess the impact of
the curriculum in a randomized case-control study employing a
test/retest methodology, and in which instrument/inter-item
validity was examined.
Early Age Drug Prevention Programs
Multiple factors such as peers, media, siblings, and parents
influence attitudes towards substances like alcohol. It is argued
that since risk factors are present before initiation, prevention
programs should start in elementary school and be reinforced in
advanced schools (Bell, Kelley-Baker, Rider, &Ringwalt,
2005).
NIDA (2003) outlines principles for school prevention plan-
ning, recommending that prevention commence in preschool to
address risk factors for drug abuse. They indicate that children
with risk factors at age seven, such as poor academic perform-
ance, are prone to engage in substance abuse by age 14 or 15.
Subsequently, they suggest that delaying intervention until
adolescence makes it increasingly harder to overcome risks,
since children’s attitudes are well grounded by then. Addition-
ally, the US Dept. of Education (1998) recommends that par-
ents begin to lay the foundation for later drug abuse prevention
as early as kindergarten.
Numerous studies have shown significant knowledge reten-
tion in children through brief educational interventions, indi-
cating the desirability of school-based health educational pro-
grams (Dressman & Hunter, 2002). Nearly 85% of children
ages 12 - 13 attended drug education classes that included mul-
tiple sessions, with 75.9% reporting having attended in the past
year. Among 9 - 11-year-olds, the proportion of students at-
tending similar drug education classes in the past year drops to
55.3 percent. Furthermore, only 12% of all youth ages 9 - 18
have ever attended such a class outside of school (NIDA,
2000).
Which Drug Abuse Prevention Programs Are of
Proven Effectiveness?
The evaluation of the effectiveness of behavioral interven-
tions such as drug use prevention programs involves quantita-
tive and/or qualitative pre-and post-assessment of one or more
of the core components of knowledge, attitudes and/or beliefs,
and practice. A review by the National Registry of Evidence-
based Programs and Practices (2002) found that there are a
number of community-based or school-drug abuse prevention
programs that were effective. However, Hallfors & Godette
(2002) reported that a majority of schools were implementing
drug abuse prevention programs which were neither evaluated
for effectiveness nor exceeded limited effectiveness. They
found that only 25% of teachers were using evidence-based
programs, and that these programs appear to be more effective,
partly as the teachers were specifically trained to teach them.
The effectiveness of the popular and widely used DARE pro-
gram was questioned; for example, one study conducted by the
Research Triangle Institute for the Department of Justice failed
to show evidence of reduced drug use in the schools and com-
munities where the program has been implemented (Ennett,
Tobler, Ringwalt, & Flewelling, 1994). There is some evidence
to suggest greater impact where teachers, rather than law en-
forcement officers conduct the program.
There is also some evidence to suggest that involvement of a
combined team of families, teachers and other adults in chil-
dren’s drug abuse interventions may have a more positive im-
pact than any one group alone, and that parents can have a posi-
tive impact on drug education and attitudes as well as teachers
(Hanson, Deere, Lee, Lewin, & Seval, 2001). Behavioral health
intervention programs such as “Starting Early, Starting Smart”
can provide effective approaches to delivering integrated be-
havioral health services for preschool as well as older children
(Hanson et al., 2001). In addition, a recent study by the Part-
nership for a Drug-Free America suggests that, among teens, a
willingness to discuss drug use “regularly” in the family was
associated with a significant decrease in drug use. The US Dept.
of Education has produced guidelines that summarize how par-
ents can educate their children about drugs at each age level.
Use of Computer Develivered Health Education for
Youth
In a study by Lieberman (2001), children and adolescents
assume the role of main characters in a disease management
video game, in which they must manage the character’s chronic
Open Access
684
M. P. METCALF
diabetes, like helping their character take appropriate amounts
of insulin. Role-playing may help translate skills to real-life
scenarios. After six months, children who used the video game
experienced a 77% decline in diabetes-related urgent care and
emergency room visits; also, this group improved in self-effi-
cacy, diabetes self-care, and communication with parents about
diabetes. The same study assessed a different video game about
smoking prevention, found to be effective in increasing knowl-
edge about the effects of smoking and attitudes towards not
smoking. Anti-smoking attitudes and intentions at 10 - 12 years
of age atrophy when children reach 13 - 14 years old, a time
when many people pick up smoking.
The use of computer based education has clear advantages in
terms of accessibility for home and school use and low utiliza-
tion cost when compared to teacher-, health educator- or com-
munity-based interventions, relatively few health intervention
programs have, to date, explored the potential of these media as
tools for health behavior interventions. There is some evalua-
tive research to support the view that this form of health educa-
tion and behavioral intervention may be effective, although few
studies have directly compared effectiveness with conventional
approaches. In one study, the short-term impact on knowledge
and interactive multimedia software on preventive alcohol
education was positive in young adults (Reis, Riley, Lokman,
& Baer, 2000). Shegog et al. (2001) found that a new educa-
tional CD-ROM aimed at enhancing pediatric asthma education
in children at ages of 9 - 13 was effective at enabling children
to self-manage their disease. Programs delivered through the
Internet, CD-ROM, and other interactive software can improve
the efficacy of risk communication (Strecher, Greenwood,
Wang & Dumont, 1999). According to Hornung et al. (2000), a
multimedia interactive CD-ROM developed to teach elemen-
tary age children about skin cancer was found to increase
knowledge and awareness of skin cancer and in shaping posi-
tive attitudes towards sun protection, after their use of the soft-
ware.
A computer programs format can encourage users to stay
with the task longer and to absorb more material. Multimedia
can also allow children to choose in a virtual safe environment
(McPherson & Glazebrook, 2001). The multimedia’s recep-
tiveness, partly occurs as technology can be incorporated and
used as a game, far more interesting to children than plain text
(Brown et al., 1997). Piaget’s theory suggests that interactivity
prompts children to take an active vice passive role in learning,
and attention theorists suggest that programs that employ ani-
mation can keep children more alert (as cited in Hornung et al.,
2000).
Use of the Internet in Drug Prevention Programs
The Internet is a knowledge resource widely used for drug
prevention information, with both governmental and non-gov-
ernmental, community, educational and consumer-based or-
ganizations. An expanding range of novel interactive Internet or
CD-ROM based tools is under-exploited for drug use preven-
tion or in the general promotion of health behaviors among
younger age groups. SAMHSA (2000) lists a number of pro-
grams nationwide that it deems “model programs”. Of these,
none offers an Internet component in order to be implemented
and only two, the Too Good for Drugs (TGFD) program and
the Protecting You/Protecting Me (PY/PM) program, integrate
information about the effects of drugs and alcohol into a cur-
riculum for 2nd and 3rd graders. To our knowledge, none have
been converted to Internet-based applications.
BrainTrain4Kids Program Description
Based on NIDA’s classroom curriculum Brain Power! for
2nd - 3rd graders, we adapted material between 2004-2007 to
make it more interactive. In doing so, we made the information
more suitable to an Internet audience through our website. The
value of an Internet-based program could increase access to,
and aid in the dissemination of, such programs. Furthermore,
we created additional material and all elements developed by us
are copyrighted; this includes the games, many of the printables,
the look and feel of the website, the assessment instruments,
and all materials that were not taken directly from Brain Power!,
and thus in the public domain. There were significant differ-
ences between the two programs, encouraging our study to
assess the impact of our Internet curriculum. First, our cur-
riculum differs in its design for independent use by the student
in a non-school setting, as in the home. Unlike Brain Power!,
our program omits information on illicit drugs of abuse, and
instead focuses on tobacco and alcohol. Since the potential of
iatrogenic effect for any prevention program exists, we decided
that it should solely focus on legal substances. We felt that
whereas a classroom-based teacher can adapt a presentation to
students and determine an apt presentation-level related to ille-
gal drugs, an Internet-based program lacks this option. Both
programs use six main lessons, but BrainTrain4Kids focuses
and emphasizes the scientific method, the brain, and prevention
of alcohol and tobacco use. Features unique to our program
include Playground which contains arcade-style games, and
subsequent links to outside sites with brain teasers, puzzles, and
games, as well as the Grown-up Guide, with information on
how to use the website, website content, educational objectives,
all printable and hands-on activities, and additional resources.
Also, children can complete the Junior Scientist Quiz at the
program’s end.
The website www.BrainTrain4Kids.com is fully developed
and users can self-register free of charge via the Internet. Albeit
the program is designed primarily for children aged 7 - 9, we
tailored the site to parents, caregivers, and substance abuse
workers. The program incorporates three minimum themes:
science education, neuroscience, and drug abuse prevention.
Albeit our program is designed for independent use, we do
not propose that children attempt to use the site without paren-
tal supervision. Children can navigate the program from any
location at any time to allow them to work individually and to
advance independently, rather than working under a class-
room’s pace. There is no time limit on how quickly a user must
complete an activity. Since all the material on the website is
designed using Adobe Flash technology, the user can stop at
any time and return to the activity when convenient. Users
navigate at their own pace, and to explore subjects of interest in
as often as desired.
Three different methods of imparting information are used to
engage the learner: the main program which presents and ex-
plains the basic curriculum, interactive educational materials
(online riddles, quizzes, and games that provide immediate
feedback), and offline activities that further review and link
information to real life via science experiments and hands-on
activities. The program employs educational games, taken from
ideas of game theory to reach educational objectives. As noted
Open Access 685
M. P. METCALF
in Cameron & Chudler (2003), in science teaching, there has
been a shift away from didactic teaching, and adaptation to-
wards using an inquiry-based method; this inquiry and hands-
on approach stimulates engagement in reasoning and problem-
solving. Our program begins by introducing the steps of scien-
tific inquiry. BrainTrain4Kids follows the National Science
Education Standards (1996) for levels K-4 in the following
categories:
1) Science as Inquiry: Understanding the abilities necessary
to do scientific inquiry
2) History and Nature of Science: Science as human en-
deavor
3) Unifying Concepts and Processes: Systems, order, and
organization
4) Physical Science Standards: Properties of objects and ma-
terials
5) Life Science Standards: Characteristics of organisms
6) Science in Personal and Social Perspectives: Personal
health
Drug prevention programs in the U.S. have seen a rise in
science-based prevention programs (Winters, Fawkes, Fahnhorst,
Botzet, & August, 2007). For example, a program for high
school students uses a multi-module curriculum, incorporating
science-based information from areas such as forensic science,
pharmacology, nutrition, and toxicology (Lennox & Cecchini,
2008). In a study by Sigelman et al. (2003), the scientific theory
of drug action was used, by explaining the role of the brain in
producing adverse effects due to cocaine and alcohol for 3rd -
6th graders. Additionally, an alcohol prevention program for 1st
- 5th graders incorporates lessons and activities about the brain,
its development, and the effects of alcohol on it, which in-
creased knowledge and changed attitudes afterward (Bell, Kel-
ley-Baker, Rider, & Ringwalt, 2005). Similar to these afore-
mentioned programs, our curriculum uses science information
to impart our drug abuse prevention message.
In our research, we wanted application of facts, not just
learning by rote. Underlying this approach is the constructivist
theory, in which knowledge is best retained when there is active
participation and a base of knowledge to build upon (Miller,
Moreno, Willcockson, Smith, & Mayes, 2005). Our program is
designed linearly so that each part of the curriculum builds on
information presented earlier in the program. Additionally, we
employ a narrative approach so the user is entertained; narra-
tion occurs by a cerebral cortex, named Corty, who is the con-
ductor of the Brain Train, and at some points, by Junior Scien-
tists who travel with Corty as they learn about neuroscience.
Learning can be enhanced through storytelling or a narrative (as
cited in Miller et al., 2005).
The program consists of six train stations that build a
knowledge base encompassing science, the human brain, and
how drugs affect the brain and body. Each station, in sequential
order, focuses on a particular theme: Science Place (scientific
inquiry), Brainville (brain function), Neurontown (how mes-
sages travel through the nervous system), Drugopolis (helpful
and harmful effects of drugs), The Smoke Stacks (risks associ-
ated with smoking and what makes cigarettes addictive), and
New Health City (what drugs do to your brain and body and
how to stay healthy). Children can then learn how drugs and
medicines affect the brain and body, using the knowledge
gained in the previous stations about the brain and nervous
system. This is unique, as few curricula teach the science be-
hind drug addiction to elementary-school children.
Each station has four buildings that children can visit, with
each one focusing on a different topic of science education.
Building 1: Welcome Center, introduces what children will
learn in the station, and Buildings 2 and 3 contain the bulk of
the station’s content while providing interactive lessons and
activities. Lastly, Building 4: Brain Games, contains three
categories of materials (Online Games, Printables, and Hands-
on) that reinforce and supplement previous information pre-
sented in the station. Each building has a unique design aligned
with the station theme. For example, in Station 5 (The Smoke
Stacks), educational objectives focus on preventing tobacco use;
Building 1 resembles lungs and in Building 2 (Tobacco Fac-
tory), the smoke stacks represent cigarettes. After completion of
the six-station program, the user can click on the link to the
final quiz at the end of Building 3 in Station 6 (New Health
City) to earn a Junior Scientist Award and to print a Certifica-
tion of Completion.
Each station must initially be accessed in numerical order.
The buildings within each station are password-protected to
encourage children to complete the buildings in sequence; the
password for each building is given after completion of the
previous building. While the Internet lends itself to a child-
directed pace of discovery, we believe that a logical order is
needed to internalize the educational objectives, especially
when the child is using the program independently. Such a
design allows children to build upon this knowledge and to
maximize their understanding of concepts. This design should
compel children to visit them in order, allowing them to use
knowledge gained in prior lessons to complete the following
building’s activities and games.
Methodology
Formati ve A nalysis
We created the BrainTrain4Kids website as well as new
software for course development. This software enabled re-
search staff to add content through simple, user-friendly editors.
Among the range of research methods used by us is the testing
of the system and product usability. During development,
BrainTrain4Kids was assessed using a series of iterative, for-
mative usability evaluations. The main goal was to gather
qualitative information about the process of interaction between
the user and the tested software application. By replicating the
natural pattern of Internet use with a focus on qualitative be-
havioral data, a relevant and objective assessment of a user’s
experience is obtained. This is crucial for building successful
web applications. Usability testing was conducted at two dif-
ferent stages of the web application development. The general
areas of research included the communication of the site’s pur-
pose, the navigation, design, download time, performance of
the application, and organization of the home page, the folders,
and the separate pages (Nielson, 2000; Nielsen & Tahir, 2001).
A total of seven usability tests were held with parent/child pairs,
in which formalized evaluations solicited feedback from chil-
dren (aged 7 - 9) and their parents. Concurrently, feedback was
sought from professional teachers of this age group, and our
education and content consultants. Child participants evaluated
the website in a usability lab setting; testing was conducted on
all stations and the overall site. After each round of formative
analysis, we employed an iterative process to adapt the website
in response to feedback; information from each round of testing
led to changes that were incorporated into the program before
Open Access
686
M. P. METCALF
the next round of testing. Changes included navigation related
issues, graphics/artwork, adding voice-over to all main program
text, and similar interface changes.
In addition, a small pilot study was conducted prior to the
summative study to assess the instruments and the research
protocol. The instruments were pretested with 30 parent/child
pairs, to determine the instruments’ viability to measure
knowledge, attitude and satisfaction with the program. Each
pair completed a standardized informed consent process, and
was assigned a user name and password. Each knowledge and
attitude item included a follow-up question that asked the child
if they thought they understood the question, or if it confused
them. We also used a short instrument to gather feedback from
the parents including topics such as level of language complex-
ity in the questions themselves, the parent’s perception of the
child’s understanding of the Likert-style answering scale, any
technical problems with delivery, and the length of time that it
took to complete the instruments.
The responses to the knowledge test showed that there was
neither a ceiling nor a floor effect for the knowledge instrument.
Some questions were clearly more “difficult” as determined by
the high number of incorrect answers. Children reported under-
standing the format and the questions, even if they didn’t know
the correct answer. The attitude scale did show a slight ceiling
effect, as anticipated. Average scores for the positive questions
ranged from 4.5 to 4.8 on a 5-point Likert scale, and “negative”
questions such as “Smoking cigarettes is good for your health”
had perfect disagreement. The only question which children
and parents reported difficulty with was “People who exercise
are cool” since it was pointed out that “exercise makes you
hot”.
No technical difficulties were found, and the online instru-
ments passed the usability tests. As there were neither technical
difficulties nor floor or ceiling effects, the knowledge instru-
ment was not revised after the pilot test. We considered revis-
ing the attitude instrument in order to reduce the ceiling effect
that the data suggested. There were compelling arguments
against this route. First, it was unlikely that we would be able to
demonstrate a strong improvement in attitude towards science
with this age group using a simple survey instrument, since
children in this age range are well known to answer questions
“the right way.” While it would be quite interesting to develop
a more versatile instrument (e.g., using open ended questions,
scenarios, or science stories), that was beyond the project’s
scope. Second, the primary goal of the instrument was to check
for any unintended negative changes. The potential for iatro-
genic like effects with any drug use prevention is well estab-
lished in the literature; the attitude survey’s main purpose in the
final study was to assess the presence or absence of such effects.
Findings from the pilot study indicated that the instrument
items and format were usable and sufficiently clear to the target
audience; hence, they were used at the beginning of a sentence.
Summative Study
Recruitment of subjects occurred via a variety of local papers
and appropriate publications. Print materials included Carolina
Parent Magazine and newspapers such as the Santa Fe New
Mexican and the Albuquerque Journal. However, many re-
cruitment advertisements were also posted online, in places
such as Craig’s List, Postaroo, LookSeek classifieds,
OK4Free.com, and online newspaper and news station classi-
fieds in Texas, California, and North Carolina. These adver-
tisements noted that parents must have children between the
ages of seven and nine (or, in the second or third grade), be at
least twenty-one years old, and have regular Internet access.
Parents were asked to contact the researchers via email for
more information about the study. Once we received a request
for more information, we sent the parents a screening document.
Information requested included the gender, ethnicity, race, age,
and Internet use of both parents and child, as well as the child’s
birth month and year, and school grade. The inclusion criteria
were: interest in the project and willingness to participate. Ex-
clusion criteria included an inability to use a computer. The
screening document was also used to obtain adequate represen-
tation from gender and minority groups. Distribution by race is
shown in Table 1; distribution of participant ethnicity is shown
in Table 2.
Participants selected for the study were chosen to mirror the
target enrollment as closely as possible. When the child par-
ticipants (N = 102) were randomly assigned to either the con-
trol or case group, some care was taken to divide the demo-
graphic groups evenly.
Evaluation Instru me nts
We measured the effect of the intervention by developing
two instruments. The outcome measures focused on changes in
1) knowledge and 2) attitude. Scales were developed to meas-
ure these outcomes. Furthermore, satisfaction surveys were
developed to assess components of the website.
Knowledge areas are based upon the objectives of the Brain
Power! curriculum and include: knowledge of basic compo-
nents of scientific inquiry, names and functions of four major
parts of the brain, basic information on what a nerve cell is and
how nerves cells work, the differences between drugs and
medicines, and the effects of nicotine on the brain and lungs.
The knowledge instrument consisted of 21 total items; there
were 16 multiple choice and 5 True/False (T/F) items. The T/F
Table 1.
Enrollment demographics by race.
Racial Category Number of Enrollees
Females Males Total%
American Indian/Alaska Native2 3 5 4.2%
Asian 3 3 6 5.10%
Black or African American 13 5 18 15.3%
Native Hawaiian/Other Pacific
Islander 0 0 0 0%
White 41 46 87 73.70%
No Answer/Other 1 1 2 1.7%
Total 60 58 118 100%
Table 2.
Enrollment demographics by ethnicity.
Ethnic Category Number of Enrollees
Females Males Total%
Hispanic or Latino 6 6 12 10.2%
NOT Hispanic or Latino 54 51 105 89%
No Answer Given 0 1 1 0.8%
Total 60 58 118 100%
Open Access 687
M. P. METCALF
Open Access
688
items included “I don’t know” as a possible response. The
questions focused on recognizing the five main parts of the
brain and functionality, understanding the steps of scientific
inquiry, and negative health effects of tobacco use and alcohol
misuse.
Attitudinal/belief change goals include instilling in the target
audience a general appreciation of science, an appreciation the
brain and of the importance of keeping the brain healthy, and
the belief that drugs can harm the brain. Attitudinal change is
influenced by via interactive tools and features.
The attitude scale was a 16-item instrument that assessed the
child’s attitude towards science (6 items), health (5 items) and
related health behavior issues (5 items). For example, items
asked about science attitudes in general (“I think doing science
is fun”), health/drug prevention (“Drinking alcohol can be
harmful to your brain”), and health behavior (“I think exercise
is fun”). Thirteen items were answered with a five-point Likert-
style scale ranging from NO! to YES!, with the available option
of “I’m not sure what this question means.” In addition, three
items were related to the health behaviors of others, for exam-
ple, “If someone I knew was smoking cigarettes, I would
feel…” These questions had four possible answers: not at all
worried about them, a little worried, very worried, or I don’t
understand this sentence.
Satisfaction scales were designed and measured components
of the website including its utility, clarity, completeness, and
ease-of-use. The measure also included global assessments
including the user’s overall satisfaction, willingness to reuse the
website and to recommend the website to other professionals/
individuals, interest in using Internet-based resource compared
to paper-based, and perceived value. The scale is given after the
curriculum intervention. Our standard for success is based on
how many participants rated the item favorably. We deem any
item that receives less than a 75% favorable rating (or a lower
rating than other items) to indicate weakness. We then reevalu-
ated the website to identify why that aspect was unsatisfactory.
User self-report helped us answer: Will they recommend it to
others, do they want to use it again, is it better than other means,
and will it affect how they seek out additional resources?
Study Desi g n
Parent/child participant pairs were randomly assigned to ei-
ther the control or case group, and assigned individual user-
names and passwords. The study used a two-group, randomized
case/control pre/post trial with a crossover design. Child par-
ticipants were asked to complete assessment measures (knowl-
edge and attitude) at three time points. As shown in Table 3,
the control group completed the knowledge and attitude meas-
ures at baseline, after a six-week wait period with no program
intervention, and finally, a third time after “crossing over” to
use the intervention. This crossover design allowed control
group participants program access after the wait period. Case
group participants were given knowledge and attitude measures
at baseline, after using the intervention, and at a six-week fol-
low-up. Administration of the pretest to the control group at
baseline and after the six-week wait period provided informa-
tion on the test-retest reliability of the measures. The follow-up
test with the case group provided data on whether or not the
effects on knowledge and attitudes were maintained. Partici-
pants spent a maximum of thirteen weeks in the study, but
some participants were unable to finish the study, and others
were discontinued due to concerns over their participation (see
Limitations).
If a parent/child pair was selected for the summative study,
the parent received consent and assent forms via mail. The
parents were instructed to read and sign the consent forms, as
well as to read the assent forms with their child before having
the child sign them. Informed consent could also be completed
by a brief phone conversation with two research members. Af-
ter consent, parents received a link via email to the pretest
(knowledge and attitude measures) instructions, which would
ultimately provide us baseline measures. Child participants
were asked to complete the pretest within one week of this
initial email. After completion, parents were told whether their
child was in the control or case group.
Children in the control group were asked to wait six weeks
without viewing the BrainTrain4Kids website. In contrast, par-
ents of children in the case group were sent a link to the website
and were instructed that their child was allowed to use the web-
site at their own pace over the next six weeks. Six weeks after
baseline in which both control and case groups took the pretest,
all parents received an email asking their child to complete a
second assessment, which contained the same questions as the
pretest. These participants were given one week to complete
this. If the participants did not complete the second assessment,
they were sent a reminder email.
Once the participants completed the second assessment, they
received another email with instructions for the final step. Con-
trol group participants were given a link to the website and
were told that they had six weeks to view the entire site and
finish the course. If the children did not complete the post-test
at the end of six weeks, parents received a reminder email for
their children. The case group was asked to wait six weeks, at
the end of which they were sent an email with a link to the fol-
low-up, and instructions to answer this within a week. Again,
the third assessment for each group was the same as the base-
line and second assessment taken by all participants.
Results
Factor analysis was deemed insufficient for this sample size
and type of analysis. Both knowledge and attitude measures
were tested for internal reliability, using Cronbach’s alpha. A
result was deemed statistically significant when p < .001.
Participants who did not complete the experience: Seventy-two
Table 3.
Summative evaluation timetable.
Group Baseline
(Week 1)
Six-Week Interval
(Weeks 1 - 6)
2nd Assessment
(Week 7)
Six-Week Interval
(Weeks 7 - 12)
3rd Assessment
(Week 13)
Case Pretest BrainTrain4Kids
Intervention Posttest Post-Intervention
Interval Posttest 2
Control Pretest No Intervention Pretest 2 BrainTrain4Kids
Intervention Posttest
M. P. METCALF
of the 102 child participants completed all parts in the study
time frame. An independent t-test analysis was performed to
determine if the participants who dropped out, before com-
pleting all three assessments, should be included. This analysis
proved that there was no difference between the participants
who completed all three assessments and those who dropped
out after the first or second assessment (t(110) = 1.471, p
= .144). This allowed us to include all available data for each
knowledge and attitude measure.
Knowled ge Assessment Sc oring and Item Analysis
After the child participants completed the knowledge instru-
ment, each answer was scored with either a 1 or a 0, for correct
or incorrect answers, respectively. Skipped questions were
treated as incorrect. Each child’s set of knowledge questions
was scored for a possible total of 21. Cronbach’s alpha revealed
that the T/F item 18 (“Some drugs are always helpful”) had a
negative item-test correlation. Due to this finding, responses to
item 18 were excluded.
As the knowledge scores illustrate in Table 4, the average of
correct answers at baseline from the control and case groups,
were 53.75 and 52.67, respectively. Using a repeated measures
ANOVA test, it was determined that there were no significant
differences in the knowledge scores of the control and case
participants at baseline (p = .484). After the second assessment,
the knowledge scores of the case group increased, whereas
those of the control group did not (see Table 4). At this point, a
repeated measures ANOVA test showed that the difference in
knowledge scores of the control and case participants was sta-
tistically significant (p < .001). Additionally, between the first
and second administration of the assessment in the control
group, the knowledge scores did not change significantly (p
= .057). After the third assessment, the scores of the control
group increased, while the case group’s scores remained the
same compared to their scores from the second assessment.
A repeated measures ANOVA test was also used to compare
the knowledge scores of boys versus girls as well as knowledge
scores of children ages 7, 8 and 9. No significant differences
existed among these groups.
Attitude Assessment Scoring and Item Analysis
Attitude scores were determined for the 13 questions an-
swered with the Likert-style format by summing ratings for
each statement. The statements were rated from 1 to 5; a score
of 1 signified strong disagreement; a 5 meant strong agreement.
Generally, agreement was considered a favorable response.
However, some of the statements given were expected to elicit
disagreement rather than agreement (“Smoking cigarettes can
be helpful to your body”). The scores for these questions were
reversed so that children who responded with a 1 received a
score of 5, those who responded with a 2 received a 4, and so
on. This allowed us to standardize the scores so that a high
overall score was considered positive.
Additionally, there were three attitude items regarding health
behaviors of others that consisted of four possible answers: not
at all worried about them, a little worried, very worried, or I
don’t understand this sentence. The responses were given the
following scores: 5 was given to the most favorable response, 3
to “A little worried about them”, and 1 was given to the least
favorable response. It was found that all items had positive
item-test correlations; the internal consistency reliability of the
16-item attitude outcome (across groups and administrations)
was 0.711, indicating an acceptable level of balance between
internal consistency and uniqueness for the items in the scale.
As shown in Table 5, the average of correct answers for the
attitude test at baseline from the control and case groups, were
85.31 and 86.58, respectively. Using a repeated measures
ANOVA test, there was no statistically significant difference
between the control and case groups at baseline (p = .519).
However, after taking the second assessment, these two groups
had significantly different scores (p = .012). The control
group’s score also changed insignificantly from the pretest (p
= .666), but the case group’s score increased significantly (p
< .001).
In the control group, scores on the second and third attitude
assessments were compared using a paired samples t-test. The
scores were found to be significantly different (p = .001), which
suggests significant gains in the participants’ attitudes towards
science and against drugs. In the case group, a paired samples
t-test comparing the second and third assessments did not find a
significant decrease in this group’s scores. However, the scores
from the third assessment of the case group were insignificantly
higher when compared to the pretest scores.
A repeated measures ANOVA test was also used to compare
the attitude scores of boys versus girls and the attitude scores of
children ages 7, 8 and 9. Insignificant differences were found
among the scores of these groups.
Qualitative Feedback from Parents and Children
Parents provided a variety of comments during the research
study both voluntarily and via satisfaction surveys. Parents
indicated that their children found use of the website fun, that
Table 4.
Knowledge scores for all child participants at three time points.
Group N Minimum Maximum MeanStd Dev
Control
Baseline 52 20 85 53.7515.71
Admin #2 49 15 85 49.1813.67
After the Program37 30 100 70.9516.02
Case
Baseline 60 25 80 52.6713.13
After the Program53 25 100 69.0617.92
Follow-Up 36 45 95 68.3314.83
Table 5.
Attitude scores for all child participants at three time points.
Group N Minimum Maximum MeanStd Dev
Control
Baseline 52 56.25 100 85.3111.17
Admin #2 49 65 100 85.879.53
After the Program36 75 100 91 6.65
Case
Baseline 60 56.25 100 86.589.77
After the Program53 70.31 100 90.328.05
Follow-Up 36 70.31 100 88.608.13
Open Access 689
M. P. METCALF
the information was useful, and that, while it contained a sig-
nificant amount of information, it was valuable to expose their
children to the vocabulary and served as a way to initiate family
based conversations about both science and drug abuse and
prevention. Several parents mentioned that while their children
did not like “science”, the children did like playing the Brain-
Train4Kids “game”.
Discussion
Since there was an insignificant difference in baseline scores
between the control and case groups, the children in both
groups were comparable in knowledge and attitudes about sci-
ence and drugs. This allowed comparison of the two groups.
We are confident that there was no measurable testing effect
that could alter our results, because the control group did not
show any significant differences in knowledge and attitude
scores after completion of the second assessment.
In the case group, we found that there was a significant in-
crease in knowledge scores on the second assessment, as com-
pared to their pretest scores and the control group’s second
assessment scores. The control group’s knowledge scores in-
creased significantly after the program intervention, which
shows that this group learned from BrainTrain4Kids after use.
The case group’s knowledge scores at follow-up did not sig-
nificantly differ from scores in the second assessment. Thus,
our findings demonstrate that children in the case group retained
knowledge changes over a follow-up period of six weeks.
After evaluating the impact of the program’s education ex-
perience on children’s knowledge, our results support that chil-
dren can and do learn material related to neuroscience. This
underscores the inclusion of more specific neuroscience in cur-
ricula related to health, science, and substance abuse prevention,
even at early ages. As noted in Cameron & Chudler (2003), a
knowledge base of the brain and how drugs affect the brain
may help children understand other conditions, such as learning
disabilities of students who have ADHD or dyslexia.
Unfortunately, the attitude results were less dramatic. For the
case group, there was a significant increase in average attitude
scores on the second assessment, as compared to their pretest
scores and the control group’s second assessment scores. Fur-
thermore, compared to the second assessment, the average atti-
tude score decreased insignificantly in the third assessment, to a
point where it could not be considered different. Moreover,
case group attitudes towards science were higher at follow-up
than at baseline, but the difference is statistically insignificant
within this sample. One possible explanation is that this is due
to a ceiling effect since the children initially had high scores at
baseline, subsequently making an increase in attitude difficult
to observe. Noteworthy is that use of the program significantly
improved attitudes for both control and case groups, and nega-
tive effects were absent; iatrogenic effects were unseen. Atti-
tudes towards science and drugs were influenced in the hoped-
for direction.
The attitude baseline scores were consistent with other re-
search focused on elementary-aged children towards science,
such as Sorge (2007) review of science attitudes from ages 9 -
14. We are unaware of other current research that discusses the
attitudes of children in the U.S. as young as 7 and 8 towards
science. Our finding, even with such a small sample size, that 7
and 8 year-old girls and boys share the positive attitude with
their older counterparts (aged 9 - 11 in the Sorge study) is a
contribution. One would expect that, in a broader sample, the
overall positive attitude would be lower. Nonetheless, it is en-
couraging that our attitude results align with other studies indi-
cating that negative attitudes towards science are absent in the
early elementary years. Based on a search of current elementary
attitudes toward science, there appears to be research on this
area in other countries, including places such as the UK (e.g.,
Jarvis & Pell, 2005; Osborne, 2003) but a dearth of current data
on attitudes of US elementary-age students towards science still
exists.
Overall, the program is effective in increasing children’s
knowledge of the brain and nervous system and positively im-
pacts children’s attitudes on science and drugs regardless of
race, gender, or age, assuming that all children are between 7
and 9 years old. Furthermore, parents and children reported
high satisfaction with the program. Parents were concerned that
their child might not retain the information if the child did not
continue to use the site after the course, yet the brief follow-up
for the case group indicates this is not necessarily the case. The
real world use of the program will permit children to return to
the website, as often as desired. Many of the interactive games
and the offline activities are designed to attract repeat visits.
Furthermore, it is possible that parents and children did not
realize that they were welcome to return to the website after
program completion. Many parents said that their child found
the website engaging and that the information was helpful and
well-presented. However, many parents also mentioned that the
information was either too complicated or, in some cases, bor-
ing. Another consistent theme included that the program was a
useful starting point for family discussions of the issues. The
overall anti-drug message was well-received, but the details
about brain parts and nervous system functionality were diffi-
cult to grasp for some children. It is interesting that the knowl-
edge results indicated that the children as a group could grasp
more material than the parents envisioned.
Limitations
An unanticipated finding is a cautionary note about Internet-
based research. From our participation inquiries, we believe
that as many as five “parent participants” lacked a child par-
ticipant. For example, one “parent” gave different birth-dates
for a child periodically, without correlation to earlier responses
when queried. One of the dates would have excluded participa-
tion by the “child.” Other “child users” completed online sur-
vey forms after midnight in their stated time zones. Bearing in
mind that the study period was during the typical academic year,
and that the child participants were aged 7 - 9 years, we con-
cluded that these were not actual participants, and hence ex-
cluded the data. When queried by email regarding the time
stamp of the survey forms, two parents did not reply, strength-
ening our belief that there was not an actual child participant or
they were apathetic.
To circumvent these challenges, we developed a series of
questions about the child participant during the informed con-
sent interview, with Institutional Review Board (IRB) approval.
This information was not verified, but the ability of the poten-
tial participant to answer questions became an inclusion criteria.
Questions were child-focused, related to current events, and
tangent to general science learning. Since this was a screening
interview to determine eligibility to participate, the questions
were appropriate and not invasive; for example, Can you tell
Open Access
690
M. P. METCALF
me some other child friendly websites your child uses at home/
school? We noted that some individuals who presented them-
selves as “parents” were unable to answer these questions or to
provide appropriate answers. With a convenience sample, we
opted to omit these potential participants. All together, fewer
than 10 “potential” parent/child pairs were uninvited to partici-
pate or continue with the study. Additionally, we dropped five
participants who turned in surveys with times stamps between
midnight - 6 am, times that would be unreasonable for a child
to be awake to take a survey.
Since this was a convenience sample, the parents were inter-
ested in their child participating, and the child may be more
interested in the topics of science or drug prevention interven-
tion and more motivated to do well. Conversely, the child par-
ticipants might resent their parents forcing this study and re-
spond in an unintentional poor manner. Post hoc analysis de-
termined that the instruments had strong validity, but the pri-
mary measure used was face-validity. Children aged 7 - 9 often
answer attitude questions the way “they think they should”
rather than honestly, or may have insufficient background
knowledge to be able to accurately self-assess.
In a study by Krishna et al. (2003), an internet-enabled mul-
timedia program was developed to educate children about
asthma, by including animated lessons and real-life scenarios;
in addition to increasing knowledge, the program decreased
emergency visits and lowered morbidity. In comparison to this
study, we did not directly assess behavior. It is unclear whether
the knowledge and attitudes gained will ultimately influence or
alter behavior. There was no longitudinal follow-up, and the
incipient follow-up was brief. The program was also short in
duration; frequent use of the program might yield more signifi-
cant findings. Rideout (2001) reports that among young “health
surfers” aged 15 - 24, 39% altered their behavior in response to
observed online health information. Thus, knowledge of online
information can cause people to act. The same rationale that
people can alter behavior based on knowledge was used in our
curriculum, in the effort that a young elementary-age audience
also has the capacity to change attitude by providing them a
knowledge base about science and drugs.
Our program used a targeted approach to cater to children
aged 7 - 9, but this age range may be considered broad. A more
targeted approach to this age group might have been more ef-
fective, or a focus on a high-risk child population. Also, due to
a small sample size, we cannot generalize our results to chil-
dren at-large; regardless, our results are promising.
Conclusion
Alcohol and tobacco use among youth prompts the need for
drug abuse education. Delay in initiation of alcohol or tobacco
use is a significant public health goal. We addressed the prob-
lem of drug abuse by developing and evaluating an interactive
program. We took a traditional educational intervention and
transferred it to a new technology, in this case the Internet. Per-
haps other programs with proven efficacy can be incorporated
into a new technology. Our technology-based curriculum is fun
and maximizes entertainment via games and animations to mo-
tivate and engage the user. It is science-based, and introduced
scientific inquiry to lay groundwork for later problem-solving
while teaching the science behind drug abuse and addiction.
The goal of BrainTrain4Kids was to lay a foundation for later
drug abuse prevention programs and interventions by educating
an elementary school-age audience. This gave children a know-
ledge base for drug abuse prevention efforts by offering educa-
tional games, interactive science lessons, and hands-on activi-
ties that teach about science, the brain, and drugs. We conclude
that our program was effective in increasing general science
and scientific inquiry knowledge, and in instilling positive atti-
tudes toward science. Currently, a decrease in positive attitudes
towards science occurs toward the end of elementary school
(Sorge, 2007). Further research of early elementary-age stu-
dents towards science and health could raise understanding of
these attitudes, especially as educators try to determine how to
affect this decrease. Moreover, this area warrants research as an
atrophy of science understanding may result in an inability for
children and youth to make informed rational health choices.
More research is also needed to assess if teaching the science
behind drug abuse and addiction is effective, with a challenge
to do so at an appropriate age level.
While the conclusions are limited to this specific intervention
and a small sample size, the positive results suggest that inter-
active Internet-based materials such as game-style elements can
be an effective way to deliver drug abuse prevention and to
educate an elementary-age audience. Another implication is the
use of the Internet to be used independently by children, and
that it can be used as a medium to increase science and health
knowledge, while improving attitude. Ergo, these results sug-
gest that the Internet can be an effective tool for developing and
implementing children’s educational tools for other curricula.
However, more research is needed to assess if games can be
effective with other teaching curricula and other age groups, as
in continuing education courses for medical students and
healthcare professionals. One study found that third-year medi-
cal students who used a Jeopardy-style game, vice the lecture
teaching method, to learn about ectopic pregnancy, had an in-
significant difference in knowledge learned between the two
groups. Albeit, students who used the game rated it signifi-
cantly higher in helping to retain information, overall enjoy-
ment, and stimulating faculty/student interaction while learning
about ectopic pregnancies (O’Leary, Diepenhorst, Churley-
Strom, & Magrane, 2005), suggesting that the game format
offers additional advantages. A future challenge will be inves-
tigating how games can be applied to different curricula and
how they can be specifically applied to an Internet curriculum,
designed for independent use. More research on how to maxi-
mize the effectiveness of Internet programs in increasing
knowledge is needed.
As future educational resources become even more restric-
tive, programs such as BrainTrain4Kids can expand the class-
room teacher’s reach without straining limited time. As the
Internet takes a role in health education, appropriate websites
and educational programs can be of avail outside the classroom
as educational tools. This study’s results, complemented by
positive reviews of both parents and children, imply that
BrainTrain4Kids is useful to enable parents and teachers to
educate children about science and drugs. This program can be
a growing resource of drug prevention education and preven-
tion. Our study implies that our program can complement a
drug abuse prevention program in schools or at home; likewise,
it can be a great economical investment to achieve results.
Acknowledgements
This research was funded by a Small Business Innovation
Open Access 691
M. P. METCALF
Research contract, #HHSN271200455234C from NIDA. Brain-
Power! is distributed by NIDA. Clinical Tools, Inc. distributes
Brain-Train4Kids free of charge to users. Between 2003-2005,
Elizabeth B. Sullivan was the Primary Investigator for this pro-
ject, and we appreciate her contribution, as well as that of Meg-
han Hennis, Bart Wodjiouski, Sean Sealy, Lyndsay Kelley,
Kimberly Workman, and Ashley Crimmins. We also thank
Sharyn Levine Rosenberg, who performed the majority of the
statistical analyses.
REFERENCES
Baylor College of Medicine (2004). BrainLink.
http://www.lifescied.org/content/5/2/118.full
Bell, M. L., Kelley-Baker, T., Rider, R., & Ringwalt, C. (2005).
Protecting you/protecting me: Effects of an alcohol prevention and
vehicle safety program on elementary students. Journal of School
Health, 75, 171-175.
Brown, S. J., Lieberman, D. A., Germeny, B. A., Fan, Y. C., Wilson, D.
M., & Pasta, D. J. (1997). Educational video game for juvenile
diabetes: Results of a controlled trial. Medical Informatics (London),
22, 77-89. http://dx.doi.org/10.3109/14639239709089835
Cameron, W., & Chudler, E. (2003). A role for neuroscientists in
engaging young minds. Nature Reviews Neuroscience, 4, 763-768.
http://dx.doi.org/10.1038/nrn1200
Center for Disease Control and Prevention (2007). United States 2007:
Percentage of students who drank alcohol (other than a few sips) for
the first time before 13 years.
http://www.cdc.gov/mmwr/preview/mmwrhtml/ss5704a1.htm
Dalacosta, K., Kamariotaki-Paparrigopoulou, M., Palyvos, J. A., &
Spyrellis, N. (2009). Multimedia application with animated cartoons
for teaching science in elementary education. Computers & Edu-
cation, 52, 741-748.
http://dx.doi.org/10.1016/j.compedu.2008.11.018
Donovan, J. E., Leech, S. L., Zucker, R. A., Loveland-Cherry, C. J.,
Jester, J. M., Fitzgerald, H. E., Puttler, L. I., Wong, M. M., &
Looman, W. S. (2004). Really underage drinkers: Alcohol use among
elementary students. Alcoholism: Clinical and Experimental Re-
search, 28, 341-349.
http://dx.doi.org/10.1097/01.ALC.0000113922.77569.4E
Dressman, L. A., & Hunter, J. (2002). Stroke awareness and knowledge
retention in children: The brain child project. St r oke, 33, 623-625.
http://dx.doi.org/10.1161/hs0202.102728
Drug Strategies (1999). Making the grade: A guide to school drug
prevention programs. Washington, DC: Author.
DuRant, R. H., Smith, J. A., Kreiter, S. R., & Krowchuk, D. P. (1999).
The relationship between early age of onset of initial substance use
and engaging in multiple health risk behaviors among young adole-
scents. Archives of Pediatrics & Adolescent Medicine, 153, 286-291.
http://dx.doi.org/10.1001/archpedi.153.3.286
Ennett, S. T., Tobler, N. S., Ringwalt, C. L., & Flewelling, R. L. (1994).
How effective is drug abuse resistance education? A meta-analysis of
Project DARE outcome evaluations. American Journal of Public
Health, 84, 1394-1401. http://dx.doi.org/10.2105/AJPH.84.9.1394
Gonzales, P., Williams, T., Jocelyn, L., Roey, S., Kastberg, D., &
Brenwald, S. (2008). Highlights from TIMSS 2007: Mathematics and
science achievement of U.S. fourth- and eighth-grade students in an
international context (NCES 2009-001 Revised). Washington, DC:
US Department of Education, National Center for Education Sta-
tistics, Institute of Education Sciences.
Hallfors, D., & Godette, D. (2002). Will the “principles of effectiveness”
improve prevention practice? Early findings from a diffusion study.
Health Education Resea r c h, 17, 461-470.
http://dx.doi.org/10.1093/her/17.4.461
Hanson, L., Deere, D., Lee, C., Lewin, A., & Seval, C. (2001). Key
principles in providing integrated behavioral health services for
young children and their families: The starting early starting smart
experience. Washington, DC: US Department of Health and Human
Services
Hornung, R. L., Lennon, P. A., Garrett, J. M., DeVellis, R. F.,
Weinberg, P. D., & Stretcher, V. J. (2000). Interactive computer
technology for skin cancer prevention targeting children. American
Journal of Preventive Medicine, 18, 69-76.
http://dx.doi.org/10.1016/S0749-3797(99)00115-4
Jarvis, T., & Pell, A. (2005). Factors influencing elementary school
children's attitudes toward science, before, during, and after a visit to
the UK National Space Centre. Journal of Research in Science
Teaching, 42, 53-83. http://dx.doi.org/10.1002/tea.20045
Krishna, S., Francisco, B. D., Balas, E. A., König, P., Graff, G. R., &
Madsen, R. W. (2003). Internet-enabled interactive multimedia asthma
education program: A randomized trial. Pediatrics, 111, 503-510.
http://dx.doi.org/10.1542/peds.111.3.503
Lennox, R. D., & Cecchini, M. A. (2008). The NARCONON drug
education curriculum for high school students: A non-randomized,
controlled prevention trial. Substance Abuse Treatment, Prevention,
and Policy [electronic], 3, 8.
http://www.substanceabusepolicy.com/content/3/1/8
http://dx.doi.org/10.1186/1747-597X-3-8
Lieberman, D. A. (2001). Management of chronic pediatric diseases
with interactive health games: Theory and research findings. Journal
of Ambulatory Care Management, 24, 26-38.
http://dx.doi.org/10.1097/00004479-200101000-00004
Lindsay, G. M., Christie, J., Gaw, A., Packard, C. J., & Shepherd J.
(1994). Educating children about heart disease. Nursing Standard, 8,
32-25.
McPherson, A., & Glazebrook, C. (2001). Double click for health: The
role of multimedia in asthma education. Archives of Disease in
Childhood, 85, 447-449. http://dx.doi.org/10.1136/adc.85.6.447
Miller, L., Moreno, J., Willcockson, I., Smith, D., & Mayes, J. (2005).
An online, interactive approach to teaching neuroscience to adole-
scents.CBE-Life Sciences Education, 5, 137-143.
http://dx.doi.org/10.1187/cbe.05-08-0115
National Center for Education Statistics (2003). Trends in international
mathematics and science study (TIMSS), mathematics and science
achievement of fourth-graders between 1995 and 2003.
http://nces.ed.gov/Timss/results03_fourth95.asp
National Institute on Drug Abuse (2000). Evaluation of the National
Youth Anti-Drug Media Campaign (No. N01DA-8-5063). Rockville,
MD: US Department of Health and Human Services, National
Institute of Health.
National Institute on Drug Abuse (2002). Brain power! The NIDA
Junior Scientist Program.
http://backtoschool.drugabuse.gov/curricula.html
National Institute on Drug Abuse (2003). Preventing drug use among
children and adolescents: A research-based guide for parents,
educators, and community leaders (2nd ed.). Bethesda, MD: US
Department of Education, National Center for Educational Statistics.
National Parents’ Resource Institute for Drug Education (2002). Grades
4-6 pride surveys national summary (2001-02).
http://www.pridesurveys.com/customercenter/01elemns.pdf
National Registry of Evidence-Based Programs and Practices (2002,
revised 2009). SAMHSA’s national registry of evidence-based pro-
grams and practices.
http://www.nrepp.samhsa.gov
National Science Education Standards (1996). National science edu-
cation standards. Washington, DC: National Academy Press
Nielsen, J. (2000). Designing web usability: The practice of simplicity.
Indianapolis, IN: New Riders Publishing.
Nielsen, J., & Tahir, M. (2001). Homepage usability: 50 websites
deconstructed. Indianapolis, IN: New Riders Publishing.
O’Leary, S., Diepenhorst, L., Churley-Strom, R., & Magrane, D. (2005).
Educational games in an obstetrics and gynecology core curriculum.
American Journal of Obstetrics and Gynecology, 193, 1848-1851.
http://dx.doi.org/10.1016/j.ajog.2005.07.059
Osborne, J. (2003). Attitude towards science: A review of the literature
and its implications. International Journal of Science Education, 25,
1049-1079. http://dx.doi.org/10.1080/0950069032000032199
Reis, J., Riley, W., Lokman, L., & Baer, J. (2000). Interactive multi-
Open Access
692
M. P. METCALF
Open Access 693
media preventive alcohol education: A technology application in
higher education. Journal of Drug Education, 3 0, 399-421.
http://dx.doi.org/10.2190/LWMQ-9CQA-B78H-9MA7
Rideout, V. (2001). Generation Rx.com: How young people use the
internet for health information. Menlo Park, CA: The Henry J.
Kaiser Family Foundation.
Schinke, S. P., Orlandi, M. A., Schilling, R. F., & Parms, C. (1992).
Feasibility of interactive videodisc technology to teach minority
youth about preventing HIV infection. Public Health Reports, 107,
323-330.
Shegog R., Bartholomew, R. K., Parcel, G. S., Sockrider, M. M, Masse,
L., & Abramson, S. L. (2001). Impact of a computer-assisted
education program on factors related to asthma self-management
behavior. Journa l of the American Medical Informatics, 8, 49-61.
http://dx.doi.org/10.1136/jamia.2001.0080049
Sigelman, C. K., Bridges, L. J., Leach, D. B., Mack, K. L., Rinehart, C.
S., Sorongon, A. G., Brewster, A. B., & Wirtz, P. W. (2003). The
efficacy of an education program to teach children a scientific theory
of how drugs affect behavior. Applied Developmental Psychology, 24,
573-593. http://dx.doi.org/10.1016/j.appdev.2003.08.001
Sorge, C. (2007). What happens? Relationship of age and gender with
science attitudes from elementary to middle school. Science Educator,
16, 33-37.
Strecher, V. J., Greenwood, T., Wang, C., & Dumont, D. (1999).
Interactive multimedia and risk communication. Journal of the
National Cancer Institute Monograp hs, 1999, 134-139.
http://dx.doi.org/10.1093/oxfordjournals.jncimonographs.a024188
United States Department of Education. (1998). Growing up drug free:
A parents’ guide to prevention. Washington, DC: US Department of
Education, Office of Elementary and Secondary Education, Safe and
Drug-Free Schools Program.
Wilson, N., Battistich, V., Syme, S. L., & Boyce, W. T. (2002). Does
elementary school alcohol, tobacco, and marijuana use increase
middle school risk? Jo urnal of Adolescent Health, 30, 442-447.
http://dx.doi.org/10.1016/S1054-139X(01)00416-5
Winters, K. C., Fawkes, T. Fahnhorst, A., Botzet, & August, G. (2007).
A synthesis review of exemplary drug abuse prevention programs in
the United States. Journal of Substance Abuse Treatment, 32, 371-
380. http://dx.doi.org/10.1016/j.jsat.2006.10.002
Young, N. K. (2000). TIP 38: Integrating substance abuse treatment
and vocational services . Rockville, MD: Substance Abuse and Mental
Health Services Administration.
Yuan, L., Manderson, L., Tempongko, M. S., Wei, W., & Aiguo, P.
(2000). The impact of educational videotapes on water contact be-
havior of primary school students in the Dongting Lakes region,
China. Tropical Medicine & International Health, 5, 538-544.
http://dx.doi.org/10.1046/j.1365-3156.2000.00602.x
List of Abbreviations
NIDA—National Institute on Drug Abuse
CDC—Center for Disease Control and Prevention