Open Access
Open Journal of Social Sciences, 2014, 2, 47-53
Published Online February 2014 in SciRes. http://www.scirp.org/journal/jss
http://dx.doi.org/10.4236/jss.2014.22007
How to cite this paper Purnakanishtha, S., Suwannatthachote, P. and Nilsook, P. (2014) Development and Validation of a
Problem Solving Skill Test in Robot Programming Using Scaffolding Tools. Open Journal of Social Sciences, 2, 47-53.
http://dx.doi.org/10.4236/jss.2014.22007
Development and Validation of a Problem
Solving Skill Test in Robot Programming Using
Scaffolding Tools
Supree Purnakanishtha1, Praweenya Suwannatthachote1, Prachyanun Nilsook2
1Faculty of Education, Chulalongkorn University, Bangkok, Thailand
2Faculty of Technical Education, King’s Mongkut’s University of Technology North Bangkok, Bangkok, Thailand
Received September 2013
Abstract
Problem solving is a crucial skill for students who experience learning and living in the
21st century. To enhance this skill, students need to face a situation setting problem, then
students solve the problem. After students overcome the obstacles, feelings of pride and
success grow in students’ hearts. Successful minds of students will lead the students to
become problem solvers and will be embedded into their thought process. It is quite hard
to find the right way to establish the problem solving skill. Robot programming is a
selective course for the secondary level of education for Thai schools. The activities in
this program provide students a chance to identify the problem, identify and analyze the
cause of the problem, propose a problem solving method and examine the problem
solving result. The problem solving skill test (PSST) included 4 levels of problem solving
skills as well. PSST consisted of 57 multiple-choice items. The test can bridge the content
of robot programming and the problem solving skills. It is useful for evaluating the skill
progress in secondary schools in Thailand.
Keywords
Problem Solving Skill; Scaffoldi ng; As sessm ent
1. Introduction
Problem-solving skills are an important factor in life. Helping students learn how to be problem solvers is a crit-
ical skill. Students have to be able to solve problems. Problems pop up every day. Sometimes they are small and
sometimes large. Fostering problem solving skills through the robot programming course is an alternative way
for many teachers to invent the appropriate methods for the teenager students. This course provided many

S. Purnakanishtha et al.
48
chances for students to build up their problem solving abilities with scaffolding tools which teacher prepared
well in class. These skills are increased by technology-enhanced scaffolding in the field of computer-based
learning environments [1], teacher-enhanced scaffolding [2], and teachers’ roles, which become very crucial to
help students in reaching the goal and interaction between scaffolding and students’ characteristics. This paper
aimed to develop and validate the Problem Solving Skill Test (PSST) in the Robot Programming Course. The
test would be an assessment tool for teachers in Thailand who teach robot programming and would be a great
way to measure the problem solving skill of students who engage in the scaffolding procedure.
2. Problem Solving
Problem solving has long been a goal in education; researchers and theorists have advanced markedly different
conceptions and method of study [3] Numerous problem-solving phases and associated learning activities have
been proposed, reflecting diverse theoretical orientations such as information processing [4], cognitive science
[5] and constructivism [6]. Reference [7] proposed four problem-solving steps: understanding the problem, de-
vising a plan, carrying out the plan, and looking back at work. These activities are often combined with heuris-
tics. Extending [7]’s approach, Reference [8] developed a 5-stage problem-solving model that includes identify-
ing problems and opportunities, defining goals, exploring possible strategies, anticipating outcomes and acting,
and looking back and learning from them. However, Reference [9] introduced the 4 problem solving levels:
identifying the problem level, analyzing and identifying the cause of the problem level, proposing the problem
solving method level and examining the problem solving result level. The scaffolding tools and scaffolding en-
vironment in class will lead students to obtain the ability of problem solving. The PSST provided various situa-
tions to promote students’ problem abilities with 4 questions in each situation, such as: like what is the problem
in this situation? What is the cause of this problem? What should you do to solve this problem? And what is the
result after the problem was solved?
3. Scaffolding in Instruction
Scaffolding refers to a “process that enables a child or novice to solve a problem, carry out a task, or achieve a
goal which would be beyond his unassisted efforts” [10]. They are “forms of support provided by the teacher or
another student to help students bridge the gap between their current abilities and the intended goals” [11]. The
notion of scaffolding instruction was introduced in Vygotsky’s Social Development Theory [12] which held that
learning and development are interrelated in students’ everyday lives. Learning should be matched in some
manner with the students’ development level. The relationship between learning and development was explained
in terms of the zone of proximal development (ZPD), which refers to “the distance between the actual develop-
mental level as determined by independent problem-solving and the level of potential development as deter-
mined through problem-solving under adult guidance or in collaboration with more capable peers” [12]. He also
pointed out the scaffolding should be provided only within the ZPD. Learning activities that are oriented toward
development levels that have already been reached are ineffective and learning activities that are oriented toward
development levels that are too far advanced for the learners’ potential ability are also ineffective. When the
learner interacts with an adult or a more skilled peer within the ZPD, he or she is guided and supported to a
greater competence and becomes capable of performing at a higher cognitive level independently once the
guidance and support are internalized [13]. The scaffolding internalization process enables learners to achieve
the tasks without guidance and support from social interaction. This is a critical process for students’ develop-
ment.
Many researchers have investigated varieties to design and use technologies to scaffold learning.
Reference [14] classified technology-enhanced scaffolds for open-ended, student-centered learning into four
types: conceptual, metacognitive, procedural, and strategic. Reference [15] proposed implicit and explicit scaf-
folds, while [16] distinguished between hard (fixed, stable, pre-set) and soft (dynamic, flexible, adaptive) scaf-
folds. Reference [17] indicated that such adaptive scaffolding enhances student problem solving by fostering.
The scaffolding strategies in robot programming were the covered area of the introduction to robot, robot as-
sembling, movement of robots, robot touch sensor, data transferring of robots and infrared sensors of robot.
Moreover, computer-enhanced scaffolding can assist students in structuring complex tasks by problematizing
content knowledge [18]. Scaffolding has been applied to help articulate and act upon problem-solving process

S. Purnakanishtha et al.
49
and learning activities.
The instruction of robot programming increases the problem solving ability of students. The multiple pre-set
questions, tasks and situations provide a variety of ways for students to identify the problem level, identify and
analyze the cause of the problem, propose the problem solving method and examine the problem solving result.
Students need to process whole knowledge and skills from the scaffolding tools such as teachers, peers, flow
charts, computer enhancement. The process of learning gears them to gain problem solving skills.
This paper shows the validation of the PSST. This test evaluated the problem solving ability enhancement of
students after integrated the scaffolding tools in the class.
4. Methodology
The PSST was designed based on Thai Basic Education Curriculum and focused on the occupation and tech-
nology curriculum for secondary education. The items of the PSST were validated by occupation and technolo-
gy experts in content validation and language use appropriateness. The PSST included 14 situations from 7 topic.
The topic, objective and number of situation are shown in Table 1.
Students gave the correct answer after reading each situation. The PSST construction was based on [9].
It included identifying the problem, identifying and analyzing the cause of the problem, proposing the prob-
lem solving method and examining the problem solving result. The whole test consisted of 57 multiple-choice
items. The PSST was validated by 3 experts in the learning areas of occupations and technology from the basic
education schools. The test was to be suitable for Thai students who registered for the robot programming
course. The test lasted 60 minutes. The numbers of item for each component of PSST are listed in Table 2. The
data were calculated by the Test Analysis Program (TAP).
The preliminary version of the PSST was pilot tested on 20 students of grade 7 in secondary schools in
Thailand. These students had taken the robot programming course in 2012. The students were given 60 minutes
to complete the test. Data collected was also used to investigate reliability and validity of the test. The reliability
of the test was provided for by the use of index of the Kuder-Richardson Formula 20 (KR20). Item indices were
examined for the purpose of item revision. Item analysis was performed in order to determine item difficulty and
discrimination.
Table 1. The topic, objective and number of situation in the PSST.
Topic Objectives Number of Situation
Robot construction To construct the robot 9
Movement of robot To control the movement of robot 1 - 5
Bumping control by switch To avoid bumping of robot by using switch 10 - 11
Data transferring of robot To transfer to data between robot and computer 8
Infrared set up and remote control To set up an infrared and use remote control 12 - 13
Moving away from obstruction
using infrared function
To move away from obstruction using
infrared function 6 - 7
Following the line movement To follow the line of the route 14
Table 2. Number of items for each component of PSST.
Problem Solving Skill Item Number Number of Items
Identifying the problem 1,5,9,13,17,21,26,30,34,38,42,46,50,54 14
Identifying the cause of problem 2,6,10,14,18,22,27,31,35,3,47,51,55 14
Identifying the problem solving method 3,7,11,15,19,23,25,28,32,36,40,44,48,52,56 15
Identifying the result of problem solving 4,8,12,16,20,24,29,33,37,41,45,49,53,57 14

S. Purnakanishtha et al.
50
5. Results
The mean and standard deviation of the PSST on 57 items were 36.6 and 5.63 respectively. Score ranged from
27 to 45. Reliability of the test was measured at 0.68. Indices of item difficulty and item discrimination were
analyzed as shown in Table 3.
The item difficulty indices range from 0.30 (the most difficult item, item 21 and item 23) to 1.00 (the easiest
item, item 5) Even though 0.50 is the ideal difficulty indices, the difficulty indices from 0.20 - 0.80 can be used
to retain the items in a test. The items discrimination indices range from −0.27 (item 19) to 0.83 (item 25) while
item 4, 13, 19, 38, 39, 40, 41, 43, 54 are negative. There are 14 items (Item 5, 6, 10, 15, 22, 28, 30, 31, 32, 33,
36, 44, 56, 57) have item discrimination less than 0.20, thus these items are not appropriate to differentiate be-
tween high and low achiever.
An analysis of the options chosen by students is shown in Table 4.
Table 3. Item indices of PSST items.
Item Number Item Difficulty Item Discrimination Item Number Item Difficulty Item Discrimination
01 0.55 0.43 30 0.85 0.03
02 0.55 0.43 31 0.85 0.03
03 0.55 0.63 32 0.80 0.03
04 0.55 −0.10 33 0.85 0.03
05 1.00 0.00 34 0.75 0.80
06 0.80 0.20 35 0.85 0.40
07 0.85 0.20 36 0.85 0.03
08 0.85 0.20 37 0.75 0.80
09 0.60 0.07 38 0.80 −0.13
10 0.55 0.07 39 0.90 −0.17
11 0.45 0.27 40 0.90 −0.17
12 0.90 0.40 41 0.85 −0.13
13 0.40 −0.07 42 0.35 0.30
14 0.60 0.43 43 0.10 −0.20
15 0.55 0.07 44 0.30 −0.13
16 0.55 0.63 45 0.40 0.30
17 0.60 0.27 46 0.85 0.40
18 0.65 0.23 47 0.70 0.60
19 0.40 −0.27 48 0.55 0.63
20 0.70 0.23 49 0.55 0.47
21 0.30 0.30 50 0.45 0.30
22 0.40 0.10 51 0.40 0.67
23 0.30 0.30 52 0.50 0.47
24 0.65 0.80 53 0.80 0.20
25 0.55 0.83 54 0.55 −0.10
26 0.90 0.40 55 0.60 0.07
27 0.90 0.40 56 0.50 0.07
28 0.90 0.03 57 0.50 0.43
29 0.95 0.20 Mean 0.64 0.25

S. Purnakanishtha et al.
51
Table 4. Analysis of response of students.
Option Option
Item A B C D Omit Item A B C D Omit
01 8 11* 0 1 0 30 17* 1 2 0 0
02 1 11* 3 5 0 31 1 17* 2 0 0
03 3 3 3 11* 0 32 2 1 1 16* 0
04 11* 3 2 4 0 33 17* 2 1 0 0
05 0 0 0 20* 0 34 15* 2 3 0 0
06 3 16* 1 0 0 35 17* 1 1 1 0
07 3 0 17* 0 0 36 2 0 17* 1 0
08 2 0 17* 1 0 37 15* 4 0 1 0
09 6 0 12* 2 0 38 16* 0 2 2 0
10 1 2 6 11* 0 39 18* 0 2 0 0
11 4 3 4 9* 0 40 0 18* 2 0 0
12 18* 0 1 1 0 41 17* 1 1 1 0
13 3 8* 1 8 0 42 7* 9 2 2 0
14 6 12* 1 1 0 43 3 6 9 2* 0
15 1 8 11* 0 0 44 6 6* 5 3 0
16 11* 0 5 4 0 45 5 5 2 8* 0
17 12* 4 1 3 0 46 17* 1 2 0 0
18 13* 2 1 4 0 47 2 2 14* 2 0
19 8* 7 4 1 0 48 2 11* 3 4 0
20 1 14* 4 1 0 49 3 2 4 11* 0
21 6 6* 7 1 0 50 9* 6 4 1 0
22 4 8* 8 0 0 51 4 3 5 8* 0
23 4 6* 2 8 0 52 5 4 10* 1 0
24 13* 5 2 0 0 53 16* 0 1 3 0
25 4 3 2 11* 0 54 11* 3 5 1 0
26 1 1 18* 0 0 55 1 5 12* 2 0
27 1 0 18* 1 0 56 10* 0 7 3 0
28 0 18* 1 1 0 57 7 10* 1 2 0
29 19* 1 0 0 0
The distracters “D” in item 13 has attracted the response from the students at the same number of correct an-
swer. This item needs to be review and rewritten for its intended purpose.
6. Conclusion
The intent of this paper is to develop an instrument to measure problem solving skills with reference to the Thai
secondary school occupation and technology curriculum. Findings from analysis showed that improvement is

S. Purnakanishtha et al.
52
still needed for 23 of the PSST item to ensure instrument is reliable and useful. The qualified 44 items will be
selected to measure the progression of problem solving skill. The development and validation of the PSST will
be beneficial for the further study in the area of robot programming and problem solving. This instrument may
be limited in its usage as it is specific to the Thai secondary school level. The findings also reflect the students’
problem solving skill acquisition in the content area of occupation and technology curriculum. The examples of
the PSST are shown as follows:
References
[1] Bannert, M. (2009) Promoting self-regulated learning through prompts. Zeitschrift Fur Padagogische Psychologie, 23,
139-145. http://dx.doi.org/10.1024/1010-0652.23.2.139
[2] Crawford, B.A. (2000) Embracing the essence of inquiry: New roles for science teachers. Journal of Research in
Science Teaching, 37, 916-937. http://dx.doi.org/10.1002/1098-2736(200011)37:9<916::AID-TEA4>3.0.CO;2-2
[3] Gagne’, R.M. and Briggs, L.J. (1974) Principle of instructional design. 2nd Edition, Holt, Rineheart and Winston, New
York.
[4] Chase, W.G. and Simon, H.A. (1973) Perception in chess. Cognitive Psychology, 4, 55-81.
http://dx.doi.org/10.1016/0010-0285(73)90004-2
[5] Paas, F.G.W.C. and Van Merrienboer, J.J.G. (1994) Variability of worked examples and transfer of cometrical problem
solving skill: A cognitive-load approach. Journal of Education Psychology, 86, 122-133.
http://dx.doi.org/10.1037/0022-0663.86.1.122
[6] Mayer, R.E. and Wittrock, M.C. (2006) Problem solving. In: Alexander, P.A. and Winne, P.H., Eds., Handbook of
Educational Psychology, Macmillia n, New York.
[7] Polya, G. (1957) How to solve it: A new aspect of mathematical method. Doubleday, Garden City.
[8] Bransford, J.D. and Stein, B.S. (1984) The ideal problem solver: A guide for improving thinking, learning, and creativ-
ity. Freeman, New York.
[9] Weir, J.J. (1974) Problem solving is everybody’s problem. The Science Teacher, 4, 16-18.
[10] Wood, D., Bruner, J.S. and Ross, G. (1976) The role of tutoring in problem-solving. Journal of Child Psychology and
Psychiatry, 17, 89-100. http://dx.doi.org/10.1111/j.1469-7610.1976.tb00381.x
[11] Rosenshine, B. and Meister, C. (1992) The use of scaffolds for teaching higher-level cognitive strategies. Educational
Leadership, 49, 26-33.
[12] Vygotsky, L.S. (1978) Mind in society: The development of higher psychological processes. Harvard University Press,
Cambridge.
[13] Hogan, D.M. a nd Tudge, J.R.H. (1999) Implications of Vygotsky’s theory for peer learning. In: O’Donnel, A. M. and
King, A., Eds., Cognitive Perspective on Peer Learning, Lawrence Erlbaum, Mahwah.
[14] Hannafin, M. and Land, S. (2000) Technology and student-centered learning in higher education: Issues and practices.
Journal of Computing in Higher Education, 12, 3-30. http://dx.doi.org/10.1007/BF03032712
[15] Hadwin, A. and Winne, P. (2001) CoNoteS2: A software tool for promoting self-regulation educational research and
evaluation. Educational Research and Evaluation, 7, 313-334. http://dx.doi.org/10.1076/edre.7.2.313.3868
[16] Saye, J. and Brush, T. (2002) Scaffolding critical reasoning about history and social issues in multimedia-supported
learning environments. Educational Technology Research and Development, 50, 77-96.
http://dx.doi.org/10.1007/BF02505026
[17] Azevedo, R. (2005) Using hypermedia as a metacognitive tool for enhancing student learning? The role of self-regu-
lated learning. Education Psychologist, 40, 199-209. http://dx.doi.org/10.1207/s15326985ep4004_2
[18] Reiser, B.J. (2004) Scaffolding complex learning: The mechanisms of structuring and problematizing student work.
Journal of the Learning Science, 13, 273-304. http://dx.doi.org/10.1207/s15327809jls1303_2

S. Purnakanishtha et al.
53
Appendix
Sample of the PSST
Situation: There are many robots in a robot race. Each robot takes turns winning and losing. All robots must run
6 rounds. On the fifth round, a robot of team 1 runs up the hill and stops when half way up.
What is the problem? (identifying the problem)
1) The motor of robot in team 1 is unworkable.
2) The robot fails to run up the hill.
3) The robot doesn’t use the gear shift function.
4) The robot uses low speed.
What is the cause of this problem? (identifying and analyzing the cause of the problem)
1) The path is very rough.
2) Batteries have low energy to run the robot.
3) The hill is very steep.
4) The competitor should practice more.
To solve this problem, how should you do? (proposing the problem solving method)
1) Stop the race.
2) Check the readiness of robot.
3) Change the new batteries.
4) Set the flat path.
From the proposed solving method you choose, what is the result of it? (examining the problem solving result)
1) All robots can run through 6 rounds of the racing.
2) This racing field provides all competition.
3) All Robots have checked the readiness before the beginning of the race.
4) The unworkable motor is not found.