Creative Education
2012. Vol.3, No.2, 241-250
Published Online April 2012 in SciRes (http://www.SciRP.org/journal/ce) http://dx.doi.org/10.4236/ce.2012.32038
A Preliminary Investigation into Critical Thinking
Skills of Urban High School Students:
Role of an IT/STEM Program
Mesut Duran1, Serkan Şendağ2
1School of Education, University of Michigan-Dearborn, Dearborn, USA
2Faculty of Education, Mehmet Akif Ersoy University, Burdur, Turkey
Received February 20th, 2012; revised March 18th, 2012; accepted March 31st, 2012
This paper reports the development of critical thinking of urban high school students in an IT/STEM pro-
gram-using information technology (IT) within the context of science, technology, engineering, and
mathematics (STEM). The study used a quasi-experimental time-series design, involving 47 initial par-
ticipants in an eighteen-month intervention period. Data were collected from the Test of Everyday Rea-
soning (TER), which provides an overall score on critical thinking skill (CTS) and five sub-scale scores
(analysis, inference, evaluation, inductive reasoning, and deductive reasoning). Findings indicate that
based on a mean score of 15.77, the average participant scored between the 16th and 19th percentiles at
the beginning of the program compared to an aggregated national sample. Participants who completed the
program and responded to all three time-series TER tests (14) significantly improved their critical think-
ing skills throughout the program. Program completers’ overall post-program CTS test score was more
homogeneous than the pre-program scores with a 20.07 mean score. In addition, data showed significant
improvement in inductive reasoning skills of the program participants during the first nine months with
continuing improvement in the second nine months. In contrast, data presented improved inference skills
during the first nine months with significant gains during the second half of the program. The study esti-
mates the relative effects of IT/STEM experiences with technology-enhanced, inquiry and design-based
collaborative learning strategies on CTS of urban high school students.
Keywords: Critical Thinking Skills; High School Students; Stem Education; Urban Education
Introduction
As the United States continues to compete in a global econ-
omy that demands innovation, US educational institutions give
even more emphasis on the 21st century skills, which include
critical thinking and problem solving, communication, collabo-
ration, and creativity and innovation (The Partnership for 21st
Century Skills, n.d.). In this undertaking, there is a major chal-
lenge: how are millions of students still struggling to acquire
19th-century skills in reading, writing, and mathematics sup-
posed to learn 21st century skills? This is the question the au-
thors of this study investigated in their recent IT/STEM project,
which focused on using information technology (IT) within the
context of science, technology, engineering, and mathematics
(STEM) through technology-enhanced, inquiry and design-based
collaborative learning experiences.
In the last two decades, contemporary work environments
required fundamental changes in the profiles of the work force,
which basically stemmed from the rapid change and transfor-
mation in the nature of information. For societies to survive in
this competitive world, it is necessary to equip individuals with
skills to conduct research, use and transform information, think
critically and reflectively, and make higher order decisions. In
addition, technological changes along with the changes in the
workplace have made critical thinking abilities more important
than ever. In this regard, learning experiences designed to assist
students’ critical thinking focus on skills that are applicable
across different domains of knowledge and the disposition to
use these skills (Halpern, 1999). A Nation at Risk report of
1983 sounded an alarm regarding insufficient and ineffective
attempts to foster higher order thinking skills in schools in-
cluding critical thinking and problem solving (Richardson,
2003). Later in 1991, Richardson reports, the Secretary’s Com-
mission on Achieving Necessary Skills regarded higher order
thinking competencies as complementary for productive work-
places including critical thinking, decision making, problem
solving and reasoning. In short, Richardson argues, the ability
to understand and use information is emphasized rather than
merely possessing it.
Critical Thinking
Thinking is based on relating and drawing conclusions on
notions and events, and involves a variety of different cognitive
processes such as implicating, problem solving, examining,
reflecting and criticizing. Thinking begins with a physical or
psychological inconvenience stemming from lacking the solu-
tion for a problem whose solution becomes the objective for an
individual. Higher order thinking skills like critical thinking,
and problem solving are considered necessary skills for 21st
century individuals. Thus, it is necessary to examine these no-
tions objectively, study on the contents of these skills, and

M. DURAN ET AL.
elaborate on the ways to equip individuals with such skills.
Problem solving involves cognitive, sensory, and psychomotor
domains which help instructors to resort to a large variety of
contexts and materials. Kalaycı (2001) argues that the most
reliable way to equip individuals with problem solving skills is
to integrate it with creative thinking and decision making.
Critical thinking entails awareness of one’s own thinking and
reflection on the thinking of the self and others (Kuhn & Dean,
2004). In this regard, metacognition is defined as thinking
about thinking, a skill originating early in life when children
first become aware of the mind. Metacognition does not always
develop to the extent that we would like (Kuhn & Dean, 2004).
The construct of metacognition can be examined within a de-
velopmental framework during which the metacognitive skills
become more powerful and effective gradually as they operate
increasingly under individuals’ conscious control. Thus, im-
proving metacognitive awareness of what to believe and how to
know, and employing metacognitive control on information
processing strategies are important developmental and educa-
tional goals.
Critical thinking involves critical implication and discussion,
which has a crucial role in activating problem solving and deci-
sion making processes (Chaffee, 1994). Critical thinking is a
constructivist analysis process to examine what is going on in
the environment. This analysis system can be used to define
problems, take actions towards an aim, make decisions and
conduct retrospective evaluations. In order to define, describe,
measure and evaluate the critical thinking process, it is neces-
sary to understand indicators of critical thinking skills (CTS).
The literature has several classifications in this regard. For in-
stance, the Watson-Glaser classification of CTS involves de-
fining a problem, determining possible solutions and strong
assumptions, drawing valid conclusions regarding the solution
and evaluating these conclusions (Demirtaşlı-Çıkrıkçı, 1996;
Kaya, 1997). Such skills include inference, recognition of as-
sumptions, deduction, interpretation, and evaluation. In other
words, critical thinking is a comprehensive thinking way in-
volving analysis, synthesis, and interpretation.
Critical thinking is a type of thinking that closely associates
with reasoning, decision making, and problem solving (Wil-
lingham, 2008). Educators recognize the importance of gaining
the skill of thinking critically. However, the crucial question is
how to teach critical thinking; directly for instance as a core
course or indirectly such as structuring the teaching and learn-
ing process in a way that those instructional activities enable
the learners to employ their higher order cognitive processes.
At high school level, Burkhart’s (2006) study provides evi-
dences from an explicit critical thinking skill development pro-
grams. However, a strong argument is in the domain of teach-
ing critical thinking in an integrated way in different subject
areas (Sendag & Odabaşı, 2009; Willingham, 2008). In terms of
the impact of critical thinking-integrated instructional activities,
the literature provides examples from solving authentic real-life
problems within the context of a subject matter to focusing on
authentic and ill-structured problem solving process and to
constructivist strategies like problem-based learning and pro-
ject-based learning (Jonassen, 1997; Jonassen, 2000; Savery &
Dufy, 1996; Sendag & Odabaşı, 2009).
Garrett (2009) compared teaching critical thinking directly
versus within an integrated instruction, examining the relation-
ship between critical thinking skills and nonperformance activi-
ties in high school choral rehearsals. Nonperformance activities
referred to cognitive domain skills (“remember, understand,
and apply” as lower order thinking skills, “analyze, evaluate,
and create” as critical thinking skills). Findings of the study
indicated a significant, strong positive correlation between the
amount of time spent in nonperformance activities and time
spent engaged in critical thinking skills. There was no signifi-
cant connection between the time spent in developing critical
thinking and critical thinking skills. That is, the time spent in
the context of subject matter where learners needed to employ
higher order cognitive skills showed significant interaction with
critical thinking, while activities that aim to directly develop
critical thinking process had no significant effect on enhancing
critical thinking skills.
Appamaraka, Suksringarm, and Singseewo (2009) conducted
a study with 9th grade students. The study results indicated that
the treatment group who received instructional activities based
on 5Es Learning Cycle model with metacognitve moves out-
performed in critical thinking skills compared to those students
receiving instructional activities based on teachers’ handbook
approach within a six weeks period.
In addition to structuring instructional activities intentionally
for the improvement of critical thinking, sometimes the nature
of the subject matter can be suitable for students to use their
higher order thinking skills. A recent study by Hove (2011)
highlighted that the high school students receiving inference
strategies in novel reading in an English literature class had
improved their critical thinking skills compared to those stu-
dents who did not receive any reading strategies. Similarly,
Ernst and Monroe (2006) found a significant positive effect of
environment-based education on 9th and 12th grade high school
students’ critical thinking skills and disposition toward critical
thinking.
Although it is suggested that a technologically-enriched so-
cial interactive learning environment is essential to enable
learner-learner, and learner-instructor interaction for developing
critical thinking (Sendag & Odabaşı, 2009), critical thinking is
something that can be built individually through the learners’
own higher order cognitive processes. In this context, Devi
(2008) pointed out that high school students’ interaction with
directed learning activities, which is also defined as “minds-on”
activities by the author, in a chemistry class enhanced the stu-
dents’ critical thinking skills.
Other studies in improving high school students’ critical
thinking seem to focus on hybrid, blended learning activities
where the content of the course and technology integration go
hand in hand rather than giving emphasis on the nature of the
course, instructional strategies, and/or explicit critical thinking
development. Bob’s (2009) study presents such a blended ap-
proach. The study was conducted with 186 participants ranged
from 8th to 9th grade and concluded that an online technol-
ogy-based math course employing real-life problems as an in-
structional activity can stimulate critical thinking and reasoning
skills. Jonassen, Carr, and Yueh (1998) suggest that rather than
using the computers to disseminate information, they should be
used in all subject domains as tools for engaging learners in
reflective critical thinking about the ideas they are studying. In
this context, using information technologies within the context
of classroom teaching in which the content requires the use of
specific computer software like SPSS or Minitab in problem
solving or using STELLA modeling software to create complex
numerical models would stimulate student critical thinking.
Other IT/STEM technologies such as robotics, Geographic
242

M. DURAN ET AL.
Information System (GIS) mapping, and multimedia gaming
are gradually emerging at the middle and high school levels,
but literature provides only descriptive studies of how these
technologies have been implemented in the classroom (Parker,
Carlson, & Na’im, 2007). The research literature is even more
limited in describing how IT/STEM experiences impact critical
thinking skills of K-12 students.
Several studies highlight that high school students in the US
are not well-prepared for college and the work force (Bernas-
coni, 2008; Burkhart, 2006; Pittman, 2010) since high schools
failed to adapt their curricula to help students in improving a set
of higher order thinking skills (Paul & Elder, 2008). Therefore,
recent initiatives in support of innovation and capacity-building
for STEM learning are currently focused on informal science
learning outside formal school settings. The National Science
Foundation (NSF) is the leading institution on this front pro-
viding research funding through its Informal Science Education
(ISE) and Innovative Technology Experiences for Students and
Teachers (ITEST) programs. The underlying assumption of
these programs is the possibility of supporting students to gain
the 21st Century skills including critical thinking through ex-
tracurricular activities. Research in this area is at too early a
stage to document the impact of these programs on students’
critical thinking development.
In sum, review of the literature on the development of high
school student’s critical thinking skills highlights the following
arguments:
 Courses designed to explicitly teach critical thinking skills
may enhance that ability.
 Instructional activities such as problem-based learning, in-
quiry-based learning within the context of subject matter
teaching may improve critical thinking.
 Social interaction has a key role for building critical think-
ing; the ability of critical thinking is an individual outcome.
 The nature of the course content could be an important fac-
tor in improving critical thinking.
 A technology-enriched learning environment and technol-
ogy-integrated instruction may help students in improving
their ability to think critically.
 Extracurricular activities may play a role in improving cri-
tical thinking skills.
In accordance with the existing literature and in response to
the growing need for understanding high school students’ criti-
cal thinking skill development, this study aimed at studying the
development of critical thinking skills of urban high school
students in an IT/STEM program. The following specific re-
search questions were examined in the study:
1) What were the initial CTS profiles of the study partici-
pants?
2) Were there any differences in the profiles of the partici-
pants who completed the program and who did not?
3) Was there any significant increase in the CTS test scores
of participants throughout the program?
4) Was there any significant increase in the subscale (analy-
sis, inference, evaluation, inductive reasoning, deductive rea-
soning) test scores of the CTS test during program participa-
tion?
In the following sections, we first describe the IT/STEM
program examined in this study and discuss the conceptual
framework that provided the basis for the project design. We
then describe the study that used a times-series design to deter-
mine the changes in participating students’ CTS during the life
of the program. In the subsequent section, we present the study
findings accompanied with a discussion of those findings. In
the conclusion section, we discuss ways in which IT/STEM
experiences with technology-enhanced, inquiry and design-
based collaborative learning strategies might have effects on
CTS of high school students.
Conceptual Framework
The present study reports from the Fostering Interest in In-
formation Technology (FI3T) project funded by the National
Science Foundation (NSF) through Innovative Technology
Experiences for Students and Teachers (ITEST) program. The
FI3T project is designed to increase the opportunities for un-
derrepresented and underserved high school students, particu-
larly those from disadvantaged urban communities in South-
eastern Michigan, to learn, experience, and more importantly
use IT within the context of STEM. The FI3T project calls for
the investment and robust participation of post-secondary col-
leges and schools, area school districts, and the business, indus-
try, and government sectors. The “Community of Designers”
approach introduced by Mishra, Koehler, and Zhao (2006) pro-
vides a framework for a collaborative partnership among a
range of participants involved in the program.
Community of Designers
As Mishra et al. (2006) describe, the Community of Design-
ers is an environment in which groups of individuals work col-
laboratively to design and develop solutions to authentic prob-
lems. Mishra et al. highlight the essence of this approach with
four key words: community, design, products/solution, and au-
thentic problems.
Mishra et al. (2006) describe “community” as the social ar-
rangement of the approach. Within the context of social con-
structivism the design community lends itself to sustained in-
quiry and revision of ideas. “Design” specifies the activity di-
mension of the approach. Building upon ideas grounded in
situated cognition theory, learning is contextualized in the
process of doing-solving an authentic problem of practice. De-
sign-based activities provide the rich context for learning, sus-
tained inquiry, and revision and are well-suited to develop the
deep understanding needed to apply knowledge in the complex
real-world domains. Whereas “products/solution” stresses the
goal-oriented psychological dimension, “authentic problems”
addresses the motivational challenge, which become the driving
force behind the work of the community. Authentic problems
that project participants face and need to work on provide the
connection between what they learn and what they actually do.
Mishra et al. (2006) highlight that learners have to actively
engage in practices of design, inquiry, and research in collabo-
rative groups to design tangible, meaningful artifacts as end
products of the learning process. The actual process-by-design
is the anchor around which learning happens. This evolving
artifact is also the test of the viability of individual and collec-
tive understandings as participants test their and others concep-
tions and ideas of the project. Mishra et al. also indicate that
implementing a community of designers breaks down into four
stages that each design team experiences over its lifecycle:
identifying participants and problems, forming communities,
providing leadership and support, and working on authentic
problems.

M. DURAN ET AL.
One would argue that in many ways community of designers
framework parallels with the principals of “community of prac-
tice” idea where Lave and Wegner (1991) describe learning
through social engagement in which members share under-
standings regarding what they are conducting and what that
means in their daily life and for their community (cited in
Parker, Malyn-Smith, Reynolds-Alpert, & Bredin, 2010). Fur-
ther referencing Wenger, the authors highlight that “these com-
munities foster mutual engagement among the members, while
they work on a joint enterprise using shared repertories of ter-
minology and skills” (p. 190).
The FI3T Project
Consistent with the aforementioned discussions, the FI3T
project calls for the collaborative engagement of high school
students, K-12 STEM teachers, Undergraduate/Graduate Stu-
dent Assistants (U/GSAs), and STEM content experts from
university, business, industry, and government sectors to create
high-quality learning projects, strategies and curriculum models
for use in after school, weekend, and summer settings through
hands-on, inquiry-based activities with a strong emphasis on
non-traditional approaches to learning and understanding. The
project also distributes online learning activities using the pro-
ject’s Web site and social media sites, and thus establishes a
culture of collaboration and discourse that extends participation
outside the confines of the formal scheduled events.
The FI3T project concentrates on all four areas of STEM cre-
ating four project-based design teams to address IT in science,
engineering, technology, and mathematics. Each design team
normally includes 10 high school students, one STEM area
high school teacher from the participating school district, one
U/GSA and one postsecondary STEM content expert from par-
ticipating higher education colleges and schools. Each design
team, therefore, consists of 13 collaborating members. One
specialized member of the project leadership team leads each
team.
The science team concentrates on three different but related
applications of IT in the sciences; measurement, modeling, and
mapping. Participating students’ learning experiences for IT/
Science include making location measurements using GPS and
integrating the measurements in a GIS system, using tempera-
ture and light sensors in the sciences, and creating mathematical
models using STELLA that incorporate measured quantities
and make predictions.
The technology team focuses on technological tools and lan-
guages for designing and developing Web applications such as
Web-based games and chat-rooms. Participants gain experi-
ences with the basics of visual programming, familiarize them-
selves with integrated development environments such as Vis-
ual Studio and/or Alice, and practice designing and developing
games.
The engineering team emphasizes the basics of robotics and
its applications as related to IT, including modeling robots,
programming robots, and integrating robots into an application
environment such as a manufacturing system or a medical ap-
plication. Engineering related learning experiences involve
using robotics simulation software packages such as IGRIP and
ROBCAD, learning the basics of robotics such as modeling and
programming robots, and learning integration of robots in an
application environment such as manufacturing system or sur-
gery operating room.
The mathematics team focuses on statistical science with
consideration of the two-sample comparison problem, the sim-
ple regression/correlation problem, and the simple analysis of
covariance problem taking examples and assignments from
public health science, environmental science, and manufactur-
ing reliability. Participants use Minitab to create comparative
displays and regression displays and perform appropriate ana-
lyses to test for and estimate effect sizes.
The work of FI3T is accomplished over an eighteen-month
cycle through summer and school-year activities. The first nine
months of the program is primarily a time for capacity-building
among students to increase IT knowledge and skills in STEM-
related fields. The second part focuses on facilitating student
activities in which they engage in designing inquiry-based au-
thentic projects of science-fair quality using what was learned
in the capacity-building phase of the program. The following
sections describe the Capacity Building and Design activities in
detail.
Capacity Building. During this initial phase of the project,
teachers, higher education faculty, graduate students, and busi-
ness partners in their assigned STEM area collaborate to facili-
tate school-year IT/STEM workshops (September to May) for
participating high school students. Offered in two different
levels, each STEM area offers a total of six different three-hour
content-specific IT/STEM workshops on the campus of the
participating postsecondary institution during weekend meet-
ings.
During the fall semester, each STEM area offers two Level 1
workshops to all participating students. These workshops con-
sist of brief presentations followed by hands-on activities to
provide students the opportunity to learn about IT toolsets
within the context of STEM. A second purpose of the Level 1
workshops is to allow participating students to identify specific
areas of interest within STEM fields. During this period, STEM
area teachers, faculty members, and the project leadership team
observe and survey students for their interest in specific STEM
subject areas and assist them to narrow down their interest into
two specific STEM related fields.
During the winter semester, a set of small group in-depth
Level 2 IT/STEM workshops are offered to participating stu-
dents in their identified two STEM areas of interest. Each
STEM area offers four three-hour content-specific IT work-
shops, allowing each student to participate in a total of eight
workshops related to their identified two STEM areas. Through-
out these workshops, participating students have the opportu-
nity to learn advanced use of IT toolsets. Level 2 workshops
also allow students to narrow down their interest into one spe-
cific STEM area and help them to join in a specific IT/STEM
design team.
Design Year. During the second phase of the project, par-
ticipating STEM teachers continue to collaborate with higher
education faculty, graduate students, and business partners to
facilitate IT-supported STEM project activities for high school
students assigned to their IT/STEM design team. The design
year consists of a two-week summer camp and a series of
site-based sessions for each individual design team. The over-
arching task of each design team in this year is to develop in-
quiry-based authentic projects that are of at least science fair
quality using one or more content-specific IT tools explored
during the previous capacity building year and stimulating
ideas/experiences gained during the summer camp.
244

M. DURAN ET AL.
Two-week summer camps for each design team (Science,
Technology, Engineering, Mathematics) take place at the be-
ginning of the design year in which students and teachers meet
and observe the work of scientists and professionals in IT/
STEM fields. Collaborating business, industry, government,
and university sectors host these sessions. Summer camp also
allows students to develop inquiry-based authentic project pro-
posals and share their ideas with other members of their design
team. Summer camp readies the students for the project devel-
opment stage that occurs during the subsequent collaborating
school year.
shrinking population of students, financial instability, school
closings, and a low high school graduation rate.
The study was announced to all students in four different
high schools of the participating school district at the beginning
of the 2009-2010 school year and volunteers were asked to
participate. These four schools were targeted because the FI3T
program had participating teachers from these schools creating
relatively easy access to the student population. Based on
STEM area teacher recommendations, a total of 47 students
were invited to participate in the program including 1 freshman,
26 sophomores, 19 juniors, and 1 senior.
The Test of Everyday Reasoning (TER) test was adminis-
tered to all of these students at the beginning of the program. Of
those, 18 students completed the program, attending all project
activities with almost perfect attendance. The FI3T project pro-
vided these students a certification of program completion. For
the purpose of this study, these students were identified as Cer-
tificated Participants (CP) hereafter and those others as Non-
Certificated Participants (NCP). Out of the 18 CP, 14 com-
pleted all -pre, -mid, and -post program TER tests.
Critical framework for the design year activities is built
around the “cyclic inquiry” model’s 5 major steps (Bruce,
2003). Ask, Investigate, Create, Discuss, and Reflect. During
the summer camp program, each design team facilitates col-
laborative learning experiences where students learn how to
design and conduct inquiry-based authentic IT/STEM projects,
more specifically how to Ask and how to Investigate. Starting
with the new school year, students engage in the remaining
three stages of their project design (Create, Discuss, Reflect)
iteratively. During these stages, each design team facilitates
multiple site-based face-to-face sessions where students work
on their projects and conduct ongoing discussions about design
activities. The design year and student/teacher participation in
the FI3T program ends with a showcase meeting where students
present their projects and discuss their experiences with the
community.
Instrumentation
In order to gather data related to participants’ CTS, the TER
was used. The TER is a member of the California Critical Think-
ing Skills Test (CCTST) Family of Critical Thinking Skills
Tests offered by Insight Assessment (insightassessment. com).
The TER reports an overall score on critical thinking skill (total
score) and 5 sub-scale scores (analysis, inference, evaluation,
inductive reasoning, and deductive reasoning) along with
demographic info and descriptive statistics. The TER is de-
signed for test takers in secondary school or the first two years
of post secondary education, and for adults of all ages in the
general population. The TER is a 35 item multiple choice test
that is administered in 50 minutes. The Flesch-Kincaid Read-
ability Level is 6th grade. No specialized content knowledge is
required. Test questions engage the test-taker’s reasoning skills
using familiar topics and contexts. Different questions progres-
sively invite test-takers to analyze or to interpret information
presented in text, charts, or images; to draw accurate and war-
ranted inferences; to evaluate inferences and explain why they
represent strong reasoning or weak reasoning; or to explain
why a given evaluation of an inference is strong or weak.
Method
Research Design
The study applied a quasi-experimental times-series research
design. Fraenkel and Wallen (2006) describe that quasi-ex-
perimental times-series research design is similar to one group
pre-test and post-test research design but requires repeated
measurements or observations over a period of time. Figure 1
illustrates the conceptual scheme of the study with an emphasis
on the treatment procedures and measurements.
Participants and Settin gs
The school district involved in this study was a major urban
school district located in Southeastern Michigan. It is the larg-
est school district in the State serving nearly 66,000 students
(mostly African American) throughout a major city area. The
district has a long history of challenging issues including
Data Collection and Analysis
The data was collected through TER test, which also included
*
CTS-T: Critical Thinking Skills Test
Workshops
(IT/STEM and Web 2.0)
Communities of Designers
(Community, Design, Products/Solution, and Authentic Problems)
Cyclic Inquiry
(IT/STEM Projects)
Capacity Building Yea
r
(9 months)
Desi
g
n Yea
r
(9 months)
CTS-T
MID
CTS-T
POST
*
CTS-T
PRE
Figure 1.
Structure of the research design.

M. DURAN ET AL.
questions about demographic info; gender, ethnicity, grade
level. The TER was employed three times—beginning, middle,
and end of the program as illustrated in Figure 1. The data was
analyzed using SPSS 15.0 software. Descriptive statistics such
as percentages, standard deviations (SD), and mean were used
to analyze the initial profiles of the CP/NCP participants. The
one way repeated measure ANOVA test was used to analyze
the CTS progress of the CP participants throughout the program.
The critical value (α) was set at .05.
Limitations
The present study has some limitations that need to be taken
into account when considering the study and its contributions.
The participants in this study were self selected based on their
existing interest in IT/STEM fields. These participants may not
be representative of urban high school students. A total of 47
participants in the study is characteristic of what might be con-
sidered a pilot in preparation for a more extended study.
Findings
Initial CTS Profile of the Participants
Data collected from the pre TER test were analyzed to un-
derstand the initial CTS profile of the participants. As shown in
Table 1, a total of 47 participants completed the pretest. Most
of the participants were female (64%), 36% were male. The
majority of the participants (83%) were identified themselves as
African-American. The remaining participants checked either
“Other” (8.5%) or didn’t provide information for the related
section on the test (8.5%). Most of the participants started the
program at their sophomore year (26) or junior year (19), a
freshmen and a senior also included in the initial group.
Based on overall mean score of 15.77, the average project
participant at the beginning of the program scored between the
16th and 19th percentiles compared to an aggregated sample of
aggregated sample for Total (see Figure 2, Insight Assessment,
2010a). In a later report, Insight Assessment (2010b) reported
that based on their aggregated sample the high school national
TER test mean score was 18.5. This indicates that the initial
CTS profile of the project participants was lower than the ag-
gregated national mean score comparing overall scores.
Data also presented overall mean score differences between
CP and NCP groups in which CP’s overall mean (18.17) was
larger than NCP’s overall mean score (14.28). In order to un-
derstand if this difference was statistically significant the au-
thors conducted t-t test between initial total CTS test scores of
CP and NCP. The data showed normal distribution while the
homogeneity test of Levene showed that the variance of both
groups were heterogeneous. Thus, the authors proceed with
non-parametric counterpart of independent sample t-test, Mann-
Whitney U. As shown in Table 2, the test results showed no
significant difference between CP and NCP in relation to their
total CTS test scores. In other words, the CP and NCP groups
were similar at beginning of the program in terms of their over-
all critical thinking skills.
Table 1.
Demographics and Initial CTS scores of CP and NCP.
Gender Ethnicity
Female Male African-AmericanOthers Not Provided
CTS
N %
N % N % N % N % N % MeanSD
CP 18 38 9 30 9 53 16 41 0 0 2 50 18.17 6.12
NCP 29 62 21 70 8 47 23 59 4 100 2 50 14.28 4.19
Overall 47 100 30 64 17 36 39 83 4 8.5 4 8.5 15.77 5.31
Figure 2.
Comparison to high school students aggregated sample from insight assessment aggregated
sample for total.
246

M. DURAN ET AL.
In sum, the project participants started the program with
similar level of critical thinking skills. Compared to an aggre-
gated national data collected by Insight Assessment participat-
ing high school students started the program with relatively
lower overall CTS score.
Changes in Critical Thinking Skills
Data collected through -pre and -post TER tests were ana-
lyzed to understand if there was a significant difference in CP’s
total CTS scores throughout their participation in program. In
order to employ repeated measure ANOVA test, the Shapiro-
Wilk normality test was conducted to test the normality of CTS
scores. As shown in Table 3, CP’s CTS scores showed no sig-
nificant difference from standard normal distribution, allowing
the use of one way repeated measure ANOVA test between pre
and post CTS scores.
As shown in Table 4, the results of the one-way repeated
measure ANOVA test indicated a significant difference be-
tween CP’s pre and post program total CTS scores (F(1 - 14) =
7.37; p < .05, partial η2 = .362). In other words, there was sig-
nificant increase in the CP CTS scores throughout the program
duration. In addition, standard deviation for the CP’s post-pro-
gram CTS scores was considerably smaller than the standard
deviation of the pre-program CTS scores. The results indicate
that the post-program CTS test scores of the CP were more
homogenous than the pre-program CTS scores.
The TER test used in this study has five sub-scales; Analysis,
Inference, Evaluation, Inductive Reasoning and Deductive Rea-
soning. The one-way repeated measure ANOVA test showed
significant increase in the level of CP’s inference and induction
sub-scale scores. Test results showed no significant difference
for deduction, analysis, and evaluation scores whereas total
post-test scores for each of these sub-scales were greater than
the pre-test scores, indicating a degree of increase.
Table 5 shows the test results for Inference sub-scale. As
shown in the table, CP significantly improved their inference
scores throughout the program (F(2 - 12) = 4.135, p < .05, partial
η2 = .408).
As Table 6 presents, CP’s improvement of inference sub-
scale was statistically significant between pre and post (Mean
Dif = 1.929, p < .05), and mid and post (Mean Dif = 1.357; p
< .05). Improvement between pre and mid test scores of infer-
ence was not significantly different. Figure 3 illustrates the
progress from pre to post program measures.
Similarly, repeated measures ANOVA test showed a signifi-
cant increase for Inductive Reasoning sub-scale scores of CP.
As shown in Table 7, CP’s inductive reasoning test scores sig-
nificantly increased from pre to post program (F(2 - 12) = 4.448, p
< .05, partial η2 = .426).
Table 8 shows the comparisons between pre, mid, and post-
test mean scores related to CP’s inductive reasoning scores. As
Table 8 illustrates, increase in inductive reasoning scores was
significant between pre and post measures (Mean Dif = 1.786,
p < .05), as well as pre and mid measures (Mean Dif = 1.214, p
< .05). Improvement between mid and post test scores of induc-
tive reasoning was not significantly different. Figure 4 depicts
the progress from pre to post program measures.
In sum, CP’s CTS test scores significantly improved during
their participation in the program with post-program CTS test
scores becoming more homogenous compared to the pre-pro-
gram CTS scores. In addition, data presented significant im-
provement in inductive reasoning skills of the PC during the
first nine months of the program with continuing improvement
in the second nine months. In contrast, data presented improved
inference skills during the first nine months with significant
gains during the second half of the program.
Discussion and Conclusion
The present study investigated the development of critical
thinking of urban high school students. The new understanding
gained from the study highlights possible impact of IT/STEM
Table 2.
Results of mann-whitney U test related to initial CTS scores.
Groups N Mean RankSum of Ranks U Sig
CP 18 28.83 519.00 174.00.06
NCP 29 21.00 609.00
Table 3.
Test of normality for CTS Scores throughout FI3T program.
CTS Tests of CP Groups Statistic Df Sig
Pre-program CTS Scores 915 14 .189
Mid-Program CTS Scores 954 14 .625
Post-Program CTS Scores 956 14 .665
Table 4.
Results of one-way repeated measure ANOVA test based on pre and post program CTS scores.
CTS Scores Mean Std. Deviation N F df1 df2 Sig Partial Eta Squared
Pre-program 17.07 5.59 14 7.37 1 14 .018 .362
Post-Program 20.07 3.73 14
Table 5.
Results of one-way repeated measure ANOVA test based on inference scores.
Inference Scores Mean Std. Deviation N F df1 df2 Sig Partial Eta Squared
Pre-program 7.21 2.32 14 4.135 2 12 .043 .408
Mid-program 7.78 2.86 14
Post-program 9.14 2.53 14

M. DURAN ET AL.
Table 6.
Pairwise comparisons for inference scores.
Inference Mean Scores Pre Mid Post
Pre (Mean = 7.21) - .571 1.929*
Mid (Mean = 7.78) - - 1.357*
Post (Mean = 9.14) - - -
Figure 3.
Progress of CP’s inference test scores.
Figure 4.
Progress of CP’s induction test scores.
experiences on critical thinking development through technol-
ogy-enhanced, inquiry and design-based collaborative teaching
and learning strategies.
The first objective of this study was to investigate the initial
CTS profiles of the study participants who were coming from a
major urban school district located in Southeastern Michigan.
Majority of the participants (83%) were African-American and
most of them were female (64%). Most of the students were at
sophomores or junior level at the beginning of the program.
Findings indicate that compared to an aggregated national data
collected by Insight Assessment (Insight Assessment, 2010a;
2010b) participating high school students joined the program
with relatively lower overall CTS test results (mean = 15.77),
scoring between the 16th and 19th percentiles on the pre TER
test. Extensive literature review failed to identify if the initial
profile of the project participants was similar or different than
typical high school population from different urban schools or
high school population in general. Further studies in this area
seem necessary.
The authors’ second objective was to investigate if there
were any differences in the CTS profiles of the participants who
completed the program (CP) and who did not (NPC). Findings
presented overall mean score differences between CP and NCP
groups where CP’s overall mean (18.17) was larger than NCP’s
overall mean score (14.28). However, further data analysis
revealed that the difference between these two groups was not
statistically significant. In other words, the CTS profiles of CP
and NCP were similar at the beginning of the program.
The third objective was to investigate if there was any sig-
nificant increase in the CTS of project participants throughout
the program. Findings indicate that CP significantly improved
their CTS throughout their participation in the program. In ad-
dition, participants’ post-program CTS scores were more ho-
mogeneous than pre-program CTS scores. It appears that the
IT/STEM experiences gained through the program significantly
impacted program completers’ critical thinking skills. Without
an experimental study with a control group, the program impact
of this project on critical thinking skill development cannot be
strongly argued. However, the literature provides implications
for the study design and its components regarding development
of critical thinking of high school students.
The study design provides a unique collaborative environ-
ment for high school students in which they work with a group
of experts from K-12, university, and industry to design and
develop solutions to authentic problems. Another main feature
of the program is its emphasis on IT knowledge and skills
within the context of STEM-related fields. In addition, over an
extended period of time (18 months), the project provides par-
ticipants access to year-round enrichment experiences through
technology-enhanced, hands-on, inquiry and design-based ac-
tivities around authentic projects with a strong emphasis on
non-traditional approaches to learning and understanding.
The program structure is closely aligned with the position of
teaching critical thinking in an integrated way in different sub-
ject areas, as argued by Sendag & Odabaşı (2009) and Willing-
ham (2008). The study findings support the findings of previ-
ous research in which focusing on authentic real-life problems
within the context of a content area or focusing on problem solv-
ing process through problem-based and project-based learning
improves students’ critical thinking skills (see Appamaraka et
al., 2009; Ernst & Monroe, 2006; Hove, 2011; Jonassen, 1997;
Table 7.
Results of one-way repeated measure ANOVA test based on inductive reasoning scores.
Inference Scores Mean Std. Deviation N F df1 df2 Sig Partial Eta Squared
Pre-program 7.857 2.597 14 4.448 2 12 .036 .426
Mid-program 9.071 2.813
Post-program 9.642 1.823
248

M. DURAN ET AL.
Table 8.
Pairwise comparisons for inductive reasoning scores.
Inference Mean Scores Pre Mid Post
Pre (Mean = 7.857) - 1.214* 1.786*
Mid (Mean = 9.071) - - .571
Post (Mean = 9.642) - - -
Jonassen, 2000; Savery & Dufy, 1996; Sendag & Odabaşı,
2009).
The final objective of the authors was to investigate if there
was any significant increase in the five sub-scale scores of the
CTS test used in the study, TER. The findings showed no sig-
nificant difference for deduction, analysis, and evaluation
scores of program completers from beginning to the end of the
program. In contrast, there was significant improvement in
inference and inductive reasoning skills.
Inference includes the skills of “querying evidence”, “con-
jecturing alternatives”, and “drawing conclusions” Facione
(2011). In the context of the TER test, “inference skills are used
to draw conclusions based on reasons and evidence” (Insight
Assessment, n.d.). Findings indicate that the project participants
significantly improved their inference skills throughout the
program with significant gain recorded in the second phase
(Design Year) of the project.
During the design phase, the project facilitated IT-supported
STEM project activities in which participating students col-
laborated with K-12 STEM teachers, higher education faculty,
graduate students, and business partners in their IT/STEM de-
sign team. The overarching task of each design team in this
year was to develop inquiry-based authentic projects that are of
at least science fair quality using one or more content-specific
IT tools explored during the previous capacity building year
and stimulating ideas/experiences gained during the summer
camp. Critical framework for the design year activities was
built around the “cyclic inquiry” model’s 5 major steps (Bruce,
2003). Ask, Investigate, Create, Discuss and Reflect. It appears
that inquiry and design-based collaborative learning activities
had significant impact on project participants’ inference skills.
The finding of this study is similar to previous research
found in the literature. Citing several studies (i.e., Chiappetta &
Russel, 1982; Saunders & Shepardson, 1987; Haury, 1993)
Abdelraheem & Asan (2006) reports that inquiry-based learn-
ing enhances students’ academic achievement, especially prob-
lem solving skills, ability to explain data sets, thinking critically,
and understanding of concepts in science teaching. Abdelra-
heem and Asan further argue that by using processes like col-
lecting and classifying data, stating hypotheses, making predic-
tions, and interpreting results of experiments, students infer
knowledge from the information provided. The authors add that
this is essential skill because such a knowledge base is not di-
rectly available in discovery situations and knowledge is to be
inferred (Shute, Glaser, & Raghavan, 1989).
Within the context of TER test, inductive reasoning is de-
scribed as “drawing warranted probabilistic inferences regard-
ing what is most likely true or most likely not true, given the
information and the context at hand” (Insight Assessment, n.d).
The study findings highlight that in contrast to the gain in in-
ference skills during the second half of the program, significant
improvement for inductive reasoning skills was recorded in the
first nine months during Capacity Building year. During this
initial phase of the project, high school students participated in
school-year IT/STEM workshops offered jointly by K-12 STEM
teachers and university faculty in their respective STEM area.
Taught in two different levels, each STEM area offered a total
of six different three-hour content-specific IT/STEM work-
shops on the campus of the participating postsecondary institu-
tion during weekend meetings.
It appears that learning experiences related to IT/STEM
technologies impacted participants’ inductive reasoning skills
significantly. In the context of this study, specific IT/STEM
technologies included GPS and GIS systems, STELLA model-
ing software, Alice 3D programming for game design, robotics
simulation software packages such as IGRIP and ROBCAD,
and Minitab statistical software. Willingham (2008) argues that
having adequate knowledge is essential for individuals to think
critically. Heit (2000) describes inductive reasoning as consid-
ered prediction of the unknown situations using known. Heit
argues that problem solving, social interaction, and motor con-
trol are important activities facilitating such a cognitive process.
IT/STEM workshops provided during the Capacity Building
year aimed at utilizing knowledge acquisition with “hands-on”
learning experiences. Thus the instructional strategies used
during the workshops along with the nature of the subject mat-
ter taught (IT/STEM toolsets) may have triggered participants’
inductive reasoning skills. A great degree of social interaction
among project participants was also evident during the first
phase of the program through whole-group seminar meetings
where students, teachers, content experts, project leaders, and
parents collectively focused on identifying students’ specific
areas of interest in STEM fields and planning for the design
year.
In conclusion, IT/STEM experiences supported through
technology-enhanced, inquiry and design-based collaborative
learning strategies seem to have significant impact on devel-
opment of critical thinking of urban high school students. The
improvement is specifically evident in inference and inductive
reasoning areas of critical thinking. Previous research provides
implications for the study design and its components in devel-
oping critical thinking of high school students. Without an ex-
perimental study with a control group, the program impact of
this study should be considered as preliminary. As a next step,
the authors of this study recommend a more rigorous study to
fully investigate the impact of the design model and instruc-
tional strategies implemented in the program.
Acknowledgements
The study reported in this article is based on a project being
funded by the National Science Foundation (NSF) through
Innovative Technology Experiences for Students and Teachers
(ITEST) program. Co-author, Dr. Serkan Şendağ’s involvement
in the study is supported by the Scientific and Technological
Research Council of Turkey (TUBITAK).
REFERENCES
Abdelraheem, A., & Asan, A. (2006). The effectiveness of inquiry-
based technology enhanced collaborative learning environment. In-
ternational Journal of Technology in Teaching and Learning, 2,
65-87.
Appamaraka, S., Suksringarm, P., & Singseewo, A. (2009). Effects of

M. DURAN ET AL.
learning environmental education using the 5Es-learning cycle with
the metacognitive moves and teacher’s handbook approach on learn-
ing achievement, integrated science process skills and critical think-
ing of high school (grade 9) students. Pakistan Journal of Social
Science, 6, 287-291.
Bernasconi, L. (2008). The jewels of ERWC instruction. California
English, 14, 16-19.
Bob, B. (2009). A problem based on-line mathematics course and its
affect on critical thinking, reasoning skills and academic achieve-
ment. Proceedings of the 31st Annual Meeting of the North American
Chapter of the International Group for the Psychology of Mathemat-
ics Education. Atlanta, GA: Georgia State University.
Bruce, B. C. (2003). Literacy in the information age: Inquiries into
meaning making with new technologies. Newark, DE: International
Reading Association.
Burkhart, L. (2006). Thinking critically about critical thinking: Devel-
oping thinking skills among high school students. Unpublished PhD
dissertation, Claremont, CA: The Claremont Graduate University.
Chaffee, J. (1994). Thinking critically. Boston: Houghton Mifflin.
Chiappetta, E. L., & Russell, J. M. (1982). The relationship among
logical thinking, problem solving instruction, and knowledge and ap-
plication of earth science subject matter. Science Education, 66,
85-93. doi:10.1002/sce.3730660111
Demirtaşlı-Çıkrıkçı, N. (1996). Critical thinking: A scale and a study. 3.
Ulusal psikolojik danişma ve rehberlik kongresi bildiri kitapçığı (pp.
208-216), Adana.
Devi, P. K. (2008). D.A.R.TS using work sheets for developing process
skills and critical thinking with pencil and paper tasks an experiment
study in chemistry senior high school at “colligative properties con-
cept”. VTE Research and Networking, 1. URL (last checked 07
March 2012).
http://ojs.voctech.org/index.php/seavern/article/view/128/121.
Ernst, J. A., & Monroe, M. (2006). The effects of environment-based
education on students’ critical thinking skills and disposition toward
critical thinking. Environmental Education Research, 12, 429-443.
doi:10.1080/13504620600942998
Fraenkel, J. R., & Wallen, N. E. (2006). How to design and evaluate
research in education (6th ed.). New York, NY: McGraw-Hill.
Facione (2011). Critical thinking: What it is and why it counts. URL
(last checked 07 March 2012).
http://www.insightassessment.com/content/download/1176/7580/file/
what%26why2010.pdf
Garrett, M. L. (2009). An examination of critical thinking skills in the
high school choral rehearsal. Unpublished PhD dissertation, Talla-
hassee, FL: The Florida State University College of Music.
Halpern, D. F. (1999). Teaching for critical thinking: Helping college
students develop the skills and dispositions of a critical thinker. New
Directions for Teaching and Learning, 80, 69-74.
doi:10.1002/tl.8005
Haury, D.L. (1993). Teaching science through inquiry. Eric Document
Reproduction Service No. ED 359048.
Heit, E. (2000). Properties of inductive reasoning. Psychonomic Bulle-
tin & Review 2000, 7, 569-592. doi:10.3758/BF03212996
Hove, G. (2011). Developing critical thinking skills in the high school
English classroom. Unpublished Master dissertation, Menominee,
WI: The Graduate School University of Wisconsin-Stout.
Insight Assessment (2010a). Report results, TER, The Test of Everyday
Reasoning, Univ MI Dearborn Education. Dearborn, MI: Author.
Insight Assessment (2010b). Cap score results. Dearborn, MI: Univer-
sity of Michigan-Dearborn.
Insight Assessment. (n.d.). Scales of the TER. URL (last checked 07
March 2012)
http://www.insightassessment.com/Products/Critical-Thinking-Skills-T
ests/Test-of-Everyday-Reasoning-TER/Scales-of-the-TER.
Jonassen, D. H. (1997). Instructional design models for well-structured
and ill-structured problem solving learning outcomes. Educational
Technology Research and Development, 45, 65-94.
doi:10.1007/BF02299613
Jonassen, D. H. (2000). Toward a design theory of problem solving.
Educational Technology Research and Development, 48, 63-85.
doi:10.1007/BF02300500
Jonassen, D. H., Carr, C., & Yeuh, H. (1998). Computers as mindtools
for engaging learners in critical thinking. TechTrends, 43, 24-32.
doi:10.1007/BF02818172
Kalaycı, N. (2001). Problem solving and applications in social studies.
Ankara: Gazi Kitapevi.
Kaya, H. (1997). Critical thinking skills of the students of university.
Unpublished PhD dissertation. Istanbul: Istanbul University Graduate
School of Health Sciences.
Kuhn, D., & Dean, D. (2004). Metacognition: A bridge between cogni-
tive psychology and educational practice. Theory into Practice, 43,
268-273. doi:10.1207/s15430421tip4304_4
Lavy, J., & Wegner, E. (1991). Situated learning: Legitimate peripheral
participation. In C. Parker, J. Malyn-Smith, S. Reynolds-Alpert, & S.
Bredin (2010). The innovative technology experiences for students
and teachers (ITEST) program: Teachers developing the next genera-
tion of STEM talent. Journal of Technology and Teacher Education,
18, 187-201.
Mishra, P., Koehler, M. J., & Zhao, Y. (2006). Communities of design-
ers: A brief history and introduction. In P. Mishra, M. J. Koehler, &
Y. Zhao (Eds.), Faculty development by design: Integrating tech-
nology in higher education (pp. 1-26). Charlotte, NC: Information
Age Publishing.
Parker, C. E., Carlson, B., & Na’im, A. (2007). Building a framework
for researching teacher change in ITEST projects. URL (last
checked 07 March 2012)
http://itestlrc.edc.org/sites/itestlrc.edc.org/files/Researching_Teacher_C
hange_%20in_ITEST%20Projects.pdf.
Parker, C., Malyn-Smith, J., Reynolds-Alpert, S. & Bredin, S. (2010).
The innovative technology experiences for students and teachers
(ITEST) program: Teachers developing the next generation of STEM
talent. Journal of Technology and Teacher Education, 18, 187-201.
Paul, R., & Elder, L. (2008). Critical thinking: The nuts and bolts of
education. Optometric Education, 33, 88-91.
Pittman, K. (2010). College and career readiness. School Administrator,
67, 10-14.
Saunders, W. L., & Shepardson, D. (1987). A comparison of concrete
and formal science instruction upon science achievement and rea-
soning ability of sixth grade students. Journal of Research in Science
Teaching, 24, 39-51. doi:10.1002/tea.3660240105
Savery, J., & Duffy, T. M. (1996). Problem based learning: An instruc-
tional model and its constructivist framework. In B. G. Wilson (Eds.),
Designing constructivist learning environments. Englewood Cliffs,
NJ: Educational Technology Publications.
Sendag, & Odabaşı, H. F. (2009). Effects of an online problem based
learning course on content knowledge acquisition and critical think-
ing skills. Computers & Education, 53, 132-142.
Shute, V. J., Glaser, R., & Raghavan, K. (1989). Inference and discov-
ery in an exploratory laboratory. In P. L. Ackerman, R. J. Sternberg,
& R. Glaser (Eds.), Learning and individual differences: Advances in
theory and research. New York: W. H. Freeman & Company.
Richardson, V. (2003). Constructivist pedagogy. Teachers College
Record, 105, 1623-1640.
doi:10.1046/j.1467-9620.2003.00303.x
The Partnership for 21st Century Skills (n.d.). A framework for 21st
century learning. URL (last checked 07 March 2012).
http://www.p21.org
Willingham, D. T. (2008). Critical thinking: Why is it so hard to teach?
Arts Education Policy Review, 109, 21-32.
doi:10.3200/AEPR.109.4.21-32.
250