Intelligent Control and Automation, 2015, 6, 147-157
Published Online May 2015 in SciRes. http://www.scirp.org/journal/ica
How to cite this paper: Al-Shabi, M. (2015) Simulation and Implementation of Real-Time Vision-Based Control System for
2-DoF Robotic Arm Using PID with Hardware-in-the-Loop. Intelligent Control and Automation, 6, 147-157.
Simulation and Implementation of
Real-Time Vision-Based Control System for
2-DoF Robotic Arm Using PID with
Department of Mechatronics Engineering, Philadelphia University, Philadelphia, USA
Email: mshabi@philad elphia.edu.jo
Received 1 March 2015; accepted 23 May 2015; published 26 May 2015
Copyright © 2015 by author and Scientific Research Publishing Inc.
This work is licensed under the Creative Commons Attribution International License (CC BY).
Microsoft Kinect sensor has shown the research community that it's more than just an interactive
gaming device, due to its multi-functional abilities and high reliability. In this work, online HIL
(Hardware-in-the-Loop) experimental data are used to apply human motion imitation to a 2-degree
of freedom Lego Mind storm NXT robotic arm. A model simulation of the dc motor used in this ex-
periment is also present in this paper. The acquired input data from the Kinect sensor are
processed in a closed loop PID controller with feedback from motors encoders. The applied algo-
rithms solve the overlapping input problem, conducting a simultaneous control of both shoulder
and elbow joints, and solving the overlapping input problem as well. The work in this paper is
presented as a prototype to assure the applicability of the algorithms, for further development.
HIL, NXT, Kinect, PID, Real-Time, Image Processing, Imitation, 2-DoF
Three-dimensional data recognition and processing are recently a vital research area for many modern applica-
tions in Mechatronic and the field of engineering in general. Kinect, which is a motion detection and recognition
smart sensor developed by the engineers at Microsoft, allows human/computer interaction needless to any phys-
ical controllers. The sensor contains two cameras for image acquisition, an RGB camera that captures colors and
a laser camera that captures depth . The processor inside the Kinect recognizes human movement patterns to
generate a corresponding skeleton model that can be provided in a computer environment such as Matlab
The data provided by the Kinect have opened the door to develop new methods and algorithms. A comparison
between different methodologies to determine the patterns provided by the Kinect was discussed in . Simula-
tions and real-life implementations were presented to validate which method resulted less errors. HGR (Hand-
Gesture Recognition) system was presented in  , where the system recognized the user’s hands and fin-
gers to translate their movement patterns into sign language. The system used skin color matching algorithms
along with HMM (Hidden Markov Models) for training and continuous-motion tracking.
Studying and analyzing gait abnormalities are essential in neurological examinations of walking disorders.
The work in  provided analysis on the effects of Parkinson’s disease on the deviations in human walking.
Two sets of data were used in this experiment. The first one was comprised of walking patterns of healthy
people, where the second contained patterns of patients with walking disorders. The abnormalities were found
and classified by comparing the differences in motion detected between the two sets of data.
In , Kinect was used to control the mobility of a wheeled Lego Mindstorm robot through Matlab. In order
to monitor the robots environment, a mobile phone was mounted on the robot to utilize its camera as feedback to
the user. The system recognized the user’s hand-gestures that were captured by the Kinect sensor. The algorithm
translated those gestures into predefined commands, to be processed and sent over to the robot -.
In this work, an algorithm is developed to extract the angles from the detected skeleton model by the Kinect
sensor. Only two of the angles are then sent to the NXT robot to control two linked servo motors representing
the shoulder and the elbow joints.
2. Mathematical Model
This section describes the mathematical model of the plant, and is divided into two parts:
- The first part describes the transf er fun ctions of the servo motors while;
- The second part dwells into the details of the robots s tatic/dynamic stability and CG (Center of Gravity).
The servo dc motors torque
is proportionally related to its curr ent
through its constant
in Equation (1) . The relation between that current and the voltage applied to the motor is explained through
KVL (Kerchoff Voltage Law) in Equation (2) for the dc motor structure in Figure 1.
Combining Equations (1) and (2) then taking the Laplace transformation to obtain:
Figure 1. dc motor structure .
Considering friction and motor load in Equation (4) then substituting its Laplace transform in (3), to obtain
the relation between the angular velocity and torque in (5).
Substituting Equation s (4) in (3) gives the relation between the armature voltage and the angular speed of the
motor in Equation (6).
Divide both sides by
to obtain the transfer function in terms of voltage to position in Equation (7).
is the angular position of the motor in rad.
is the angular velocity of the motor in rad/sec.
Va is the armature voltage in volts.
Ke is the torque constant in N⋅m/A.
is the motor viscous friction constant in N⋅m⋅s.
La is the field inductance in H.
Ra is the field resistance in Ohms.
J is moment of inertia in kg⋅m2.
Finally, the motor’s model is ready for simulation after substituting values from Table 1 for the constants in
the transfer function in Equation (8).
Vs SS S
Table 1. NXT servo motor parameters .
Parameter Value and unit
Ke 0.47 N⋅m/A
B 1.13 × 10−3 N⋅m⋅s
La 0.15 H
Ra 6.9 Ohms
J 3.228 × 10−6 kg⋅m2
3. Simulation and Preliminary Results
The proposed algorithm is summarized in Figure 2. The Kinect sensor is used as an image acquisition input for
the algorithm’s Matlab Simulink as shown in Figure 3. The acquired skeleton is then viewed as shown in Fig-
ure 4, and the angle of the elbow
is extracted. The data is processed through the created Simulink and sent
to the transfer function of the motor in Figure 5. The sampling time for the acquired data in the simulation is
0.033 seconds to match the number of frames captured by the Kinect per second.
Changing the reference angle of the elbow joint manually by the user, caused the responses of the motor to be
jerky and unstable, thus PID control method as in Figure 6 and Figure 7, is used in this simulation to enhance
and smooth the motor’s response.
The simulation results for the modeled motor shows the response of the motor's angle compared to the user's
input as shown in Figure 8. The PID algorithm applied is resulting smooth responses to the minor changes in
the users elbow angle, eleminating the jerky movement and sucsessfully overcoming the over-lapping input
changes. A delay of approximately 1 second is noticed in the simulation results, which is left to deal with later
on in the hardware implementation in Section 3.
Figure 2. Flow chart of the system.
Figure 3. Kinect Simulink library based on  .
Figure 4. Depth view of the skeleton data showing the elbow angle.
Figure 5. Simulink created for the system.
Figure 6. Th e system’s loop.
Figure 7. PID algorithm flowchart.
Figure 8. Elbow angle input and output motor response.
4. 2-Dof Hardware in the Loop
The system simply consists of the Kinect sensor, an NXT Lego Mindstorm brick two NXT dc servo motors a
computer. The Kinect sensor is connected to the computer, and Matlab works as a host environment for the Ki-
nect to process the skeleton data. The data is processed and sent online to the NXT brick, which only bypasses
the data and works as a drive for the NXT motors. The encoders mounted on the motors send the feedback data
to the NXT brick which sends it back to Matlab, Figure 9.
The 1st motor is mounted on the NXT brick representing the shoulder joint, while the 2nd one is mounted at
the end of the 1st one, representing the elbow joint. The distance between the shoulder motor and the elbow mo-
tor is the arm link. On the shaft of the 2nd motor, a plastic rod is mounted to represent the elbow link, as shown
in Figure 10. The sampling time for the NXT brick was chosen to be 0.02 seconds, which is the minimum time
that the brick can handle for processing the data. This results of an outstanding reduction in the response delay
which can be noticed in both motors for position and speed responses. Even though both motors are identical,
different PID values are implemented for each motor, due to differences in the resultant forces and torques ap-
plied on each one.
5. Results and Analysis
Results for both motors representing shoulder and elbow joints are described in this section. For all simulations,
a sampling time of 0.02 is used. A comparison between input angles and motors angles are described and ana-
lyzed for the 2-DoF arm. Angular velocities are studied to understand the limitations of the motors. The system
is able to respond quickly, and shows good stability to varying inputs with accurate positioning and low over-
shoot. In terms of angles changing over time, the responses of the shoulder and elbow motors to their inputs are
shown in Figure 11 and Figure 12, respectively.
Samples of the data were taken and represented in Figure 13 and Figure 14. The selected data shows that for
the shoulder motor, a maximum over shoot of eight degrees is noticed with 0.33 seconds settling time.
Analyzing the angular velocities of the shoulder motor in Figure 15 and the elbow motor in Figure 16, shows
that both are able to match the rate of change of the human arm angles to certain limit-due to the motors physical
limitation. Yet, the system manages good stability during sudden changes in speed and direction.
The results show that the system is robust and accurate for controlling position. On the other hand, the system
faced limitations for high angular velocity inputs due to the physical limitations of the motors. This problem can
be easily overcome by selecting motors with higher maximum speed and few adjustments to the algorithm used.
Figure 9. Block diagram of the system.
Figure 10. Practical sequence of operation.
Figure 11. Shoulder input angle vs corresponding motor angle.
Figure 12. Elbow input angle vs corresponding motor angle.
Figure 13. Data section for shoulder motor response.
Figure 14. Data section for elbow motor response.
Figure 15. Shoulders angular velocity (input ) and the motors
angular velocity (output).
Figure 16. Elbow angular velocity (input) and the motors angular velocity (output).
The elbow motor showed better response in terms of overshoot and settling time because of its load lower
than the shoulder motor that carried that elbow motor.
The author of this paper would like to thank the Mechatronics department in Philadelphia University for their
support and providing the required apparatus and equipment for this project. Special thanks for Iyad Sawalmeh
and Samer Al-Shaer for their help through this project.
 Microsoft (2009) Project Natal.
 Li, Y. (2012) Multi-Scenario Gesture Recognition Using Kinect. The 17th International Conference on Computer
Games, Louisville, 30 July-August 1 2012, 126-130. http://dx.doi.org/10.1109/cgames.2012.6314563
 Machida, E., Cao, M., Murao, T. and Hashimoto, H. (2012) Human Motion Tracking of Mobile Robot with Kinect 3D
Sensor. 2012 Proceedings of SICE Annual Conference, Akita, 20-23 August 2012, 2207-2211.
 Wang, Y., Yang, C., Wu, X., Xu, S. and Li, H. (2012) Kinect Based Dynamic Hand Gesture Recognition Algorithm
Research. The 4th International Conference on Intelligent Human-Machine Systems and Cybernetics, Nanchang, 26-27
August 2012, 2 74-279. http://dx.doi.org/10.1109/ihmsc.2012.76
 Zhang, C., Xu, J., Xi, N., Jia, Y. and Li, W. (2012) Development of an Omni-Directional 3D Camera for Robot Navi-
gation. The 2012 IEEE/ASME International Conference on Advanced Intelligent Mechatronics, Taiwan, 11-14 July
2012, 262-267. http://dx.doi.org/10.1109/AIM.2012.6266033
 Procházka, A., Vyšata, O., Valis, M. and Yadollahi, R. (2013) The MS Kinect Use for 3d Modelling and Gait Analysis
in the Matlab Environment. Technical Computing 2013, Prague.
 Li, B. (2013) Using Kinect for Face Recognition under Varying Poses, Expressions, Illumination and Disguise. Appli-
cations of Computer Vision (WACV), Tampa, 15-17 January 2013, 186-192.
 Prochazka, A., Kubicek, M. and Pavelka, A. (2006) Multicamera Systems in the Moving Body Recognition. 48th In-
ternational Symposium ELMAR-2006 Focused on Multimedia Signal Processing and Communications, Zadar, June
2006, 45-48. http://dx.doi.org/10.1109/elmar.2006.329511
 Staranowicz, A. and Mariottini, G.-L. (2013) A Comparative Study of Calibration Methods for Kinect-Style Cameras.
University of Texas at Arlington, Arlington.
 Li, Y. (2012) Hand Gesture Recongnition Using Kinect. University of Louisville, Louisville.
 Harrison, T.R. (2011) Chapter 367: Approach to the Patient with Neurologic Disease. In: Longo, D.L., Fauci, A.S.,
Kasper, D.L., Hauser, S.L., Jameson, J.L. and Loscalzo, J., Eds., Harrison’s Principles of Internal Medicine, McGraw
Hill Professional, New York.
 Ivanescu, L. (2014) KEV3. http://www.mathworks.com/matlabcentral/fileexchange/47968-kev3
 Lasenby, J. and Stevenson, A. (2001) Using Geometric Algebra for Optical Motion Capture. Birkhäuser, Boston.
 Ringer, M. and Lasenby, J. (2002) Multiple Hypothesis Tracking for Automatic Optical Motion Capture. Lecture Notes
in Computer Science, 2350, 524-536.
 Dorf, R.C. and Bishop, R.H. (2001) Modern Control Systems. 9th Edition, Prentice-Hall, Englewood Cliff.
 Watanabe, R. (2015) NXT Motor Parameters.
 Kroon, D.-J. (2011) Mathworks. http://www.mathworks.com/matlabcentral/fileexchange/30242-kinec t-matlab