Received 24 September 2015; accepted 22 December 2015; published 25 December 2015

ABSTRACT

This article presents an analysis of the selection process of students from both private and public high schools in the state of Rio Grande do Norte (Northeast of Brazil), in order to form the first group of the Information Technology technical course of the Metrópole Digital project at the Federal University of Rio Grande do Norte (UFRN). The selection instrument consists of a written exam built on the basis of a set of matrices that depict relevant competences and abilities expected from the students. Data concerning the profile of distribution of performance per exam question are presented, together with empirical evidences of the effectiveness of the proposed competence matrices in the selection process. The correlation very fied among the matrices compounding the instrument and scores in courses, and the instrument as a wholeand scores in courses is moderate to strong (r_S = 0.6 and r_S = 0.7; p < 0,05). The association between the criterion variables (Average of Performance in the Basic Moduleof courses) and the explicative variables (Total score in the selection instrument (QTot) and each of the five compounding matrices) is moderate (r_S=0.51, p = 0.001), and the matrices and QTot are jointly responsible for 25% of the variance of the performance in the basic module. The regression coefficient for matrix 1 is 1.38 (CI of 95% = 1.07-1.68); for matrix 2 it was 0.65 (CI of 95% = 036-0.95); for matrix 3, 1.2 (CI of 95% = 0.92 - 1,55); matrix 4 0,76 (CI of 95% = 0.45-1.06); matrix 5 1.08 (CI of 95% = 0.78-1.39). The effects of student-descriptive independent variables on performance in the exam are also presented and discussed, and predictive relationships between performance in the exam and performance in the disciplines of the course are verified. Data collected allow concluding that the instrument of selection proposed fits well the aim of forming a group of students to undertake a technical course in the area of information technology.

Keywords:

Competences in Information Technology, Prospection of Students, Digital Inclusion

1. Introduction

In educational contexts, the popularization of interactive, multimedia technology has profoundly changed the standard ways of performing some traditional activities like writing, communication and planning. It is a new, qualitatively different extension of memory, which introduces other ways of thinking than linear reasoning, based on simulation, experimentation, and a new language involving writing, orality, images and instantaneous communication (Borba & Villareal, 2005) .

However, despite such potential, in the everyday reality of schools computers have mainly been used as tools for information searching, text editing and presentations, with little stimulation of creativity and flexible thinking (Keay-Bright, 2008, Passey, Rogers, Machell, & McHugh, 2004) . Papert (1980) , a pioneer in the use of computers as tools for constructivist learning, proposes the exploratory programming environment Logo as a way of developing, among others, abilities of logical reasoning and problem solving, which form what may be called critical thinking.

Critical thinking refers to a set of executive abilities that allow (Sheffield, 2007): identifying relevant information of the problem; understanding and use the language; evaluating data and information; anticipating conclusions; reconstructing own beliefs; identifying the problem solution; interpreting data; formulating conclusions; producing and defending arguments.

Generally speaking, disciplines specific of the IT field simultaneously stimulate and demand critical thinking. Considering this context, we can highlight two main reasons for offering IT education for youths: 1―IT is a part of youths’ life in contemporary times; 2―Youths with high abilities possess characteristics of cognitive functioning that are important and valued in the IT domain.

Aiming to contribute for the knowledge production in this area, the present article discusses the development of an instrument for selecting abilities and competences in IT, and presents an analysis of the results of the application of this instrument among secondary school students from the state of Rio Grande do Norte, Brazil. Such initiative, coordinated by the post-graduation program in Psychology at the Federal University of Rio Grande do Norte (UFRN), is integrated with the Metrópole Digital project (MD) (http://portal.imd.ufrn.br/instituto-metropole-digital/), financed by FINEP―Ministry of Science and Technology, and executed by UFRN.

Metrópole Digital project is structured from three basic axes: the promotion of social and digital inclusion, through the realization of the aforementioned educational IT program and through the development of a wireless, high-speed metropolitan network; the development of research and technological innovation, through the formation of multidisciplinary research groups at UFRN in the areas of industrial computing e robotics, embedded and dedicated systems, intelligent /biomedical instrumentation and software project and development; the incubation of IT companies, to create a dynamic interaction between companies and university, as well as to transfer technology and innovation to the regional companies and to improve the quality of products developed locally.

One of the main goals of the MD project is to establish an educational program in IT for 1200 youths (between 14 and 18 years of age) in the UFRN campus. Such initiative springs from the necessity of providing the state of Rio Grande do Norte with a structuring project that will allow, simultaneously: 1―Increase the availability of specialized professionals to feed the local software and hardware industries; 2―Increment the local IT park; 3―Contribute, through initiatives in the area of services and production, to the global development of the region, within the context of the set of actions planned in the governmental program PAC (acceleration of growth program) for the state.

The development of specific procedures of selection and evaluation in order to make possible the construction of an instrument for this purpose was a first need of the MD project. Such an instrument should be clearly connected to ability for Informatics and IT in general. This instrument was developed by researchers issued from the Department of Psychology of Universidade Federal do Rio Grande do Norte, Universidade Federal de Pernambuco, and Department of Education from Universidade Federal Rural de Pernambuco.

2. Methodology

A correlational study was performed, analyzing scores from an instrument of selection (MDI Selection Instrument) and scores from disciplines in the course for those students selected; this study tried to find out possible connections between performances in these two sets of scores. ANOVA analyses were also performed in order to verify effects from a set of independent variables over performances in disciplines offered by the complete MDI first year program. The study was initially performed with all students engaged in MDI selection (5758 candidates), both sexes, ages from 14 to 18 years; afterwards, the totality of students selected in order to participate in MDI formation program (1200 students) was engaged.

All scores and personal information used in this study came from MDI institutional data banks, including performance scores in the selection examination and technical formation courses, together with socio-econo- mical descriptive data.

The path towards the development of the above mentioned instrument to search for competences and abilities in information technology demanded a coordinated three-stage procedure.

Shaping of an instrument for selection of students for technical formation in IMD in terms of competences and abilities considered necessary and/or desirable for the selection of students, presented in detail and discussed by Hazin (2011) . Competences here are seen as structuring modalities of intelligence used in the procedural field, or knowing how to do; they can be more or less complex, and are formed of other more specific abilities. In the light of such approach for abilities and competences, these cannot be restricted to information and concepts specifically related to content: rather, they consider strategies of interpretation and management of information, in the sought for solutions for problems.

It is relevant to point that such conception is already present in official instruments of evaluation of the Brazilian educational system. For instance, for the instruments of MEC-INEP (Ministry of Education), namely ENEM, SAEB and Prova Brasil, evaluation is based on the previous establishment of competences and abilities. Such instruments adopt the classic perspective of Perrenoud (2000) on such constructs, for whom competences are the structuring modalities of intelligence. Abilities are derived from the acquired competences, and refer to the knowing how to do. Through actions and operations, abilities are improved and articulated, leading to a reorganization of competences (Perrenoud, 2000) .

In the domains of cognitive psychology and psychometry, the constructs of competences and abilities have been approached differently from the manner described above, in spite of some points of convergence. From the perspective of Primi et al. (2001) , the conception of competence of Perrenoud (which inspired the Brazilian exam ENEM), is similar to what Psychology calls crystallized intelligence, i.e. a corpus of knowledge that, in its turn, when mobilized within interdisciplinary questions, becomes fluid intelligence. From the perspective of the authors, abilities indicate facility to deal with determined types of information, but to be transformed in competences an investment is needed in learning experiences, and especially directing in an articulate manner such abilities as activities of culture (Primi et al., 2001) .

In this sense, the instrument of selection proposed was not characterized by questions on specific school contents, as it traditionally happens with instruments of evaluation of school performance. The phases of elaboration of the instrument are presented next, as well as data related to the results of the application of the instrument, which bring important contributions for the empirical verification of the technical adequacy of the test in terms of aspects related to the theory of psychological measurement, and in terms of empirical support for the proposition of the matrices that composed the instrument in question. The whole process of construction of the instrument can be summarized by Figure 1.

As illustrated in this figure, a set of competences and abilities to evaluate (competences and abilities related to IT environments/situations) was established in phase 1; such competences were decomposed and described in terms of matrices that detailed the competences into abilities in phase 2; in phase 3, abilities were related to descriptors that led to items of the questionnaire. Once such items were proposed and refined, the questionnaire was applied to candidates in phase 4, so as to provide means of selecting the group of students who would take part in the project (phase 5).

The first version of the selection instrument for the Metrópole Digital project proposed thirty multiple-choice questions with a unique correct answer among five options. Such instrument was meant to cover the set of competences and abilities considered necessary for IT students and summarized by Table 1. This table summarizes

the set of talents, vocations and competences that respond to such pragmatic aspect of the evaluation, inspired in indicators from the International Society for Technology in Education (ISTE, 2007) .

Application of IMD Selection Instrument for access at IMD Technical Formation: this procedure was started with a divulgation campaign in all available media and by divulgation visits to local schools. All this strategy aimed to spread out the new intended formation among potentially concerned students and teachers. Enrolment in the selection procedure was free of charge, and available in internet-based sites. Students with ages varying from 14 to 18 years, inscribed in the last year-course of Brazilian fundamental level II and any level from high-school were admissible. IMD selection procedure occurred in the campus of Universidade Federal do Rio

Grande do Norte, and candidates were allowed a three-hours time to answer the 30 questions compounding the selection instrument. The procedure of places distribution was done in a descending classificatory basis, starting from the best performances. 70% of these places were reserved to students issued from public school system. The final score used for classification of candidates was result of the average of standardized z-scores of the number of right answers in each of five competences. The Final Classificatory Score was then the average z-score of the five competence-matrices. In cases of ties, the oldest student was selected.

Data analysis: Data issued from candidates identification profiles, like age, gender, school origin and so on, together with scores from the instrument itself and scores from students performances in courses composing IMD curricula were submitted to summary statistics, including averages and cluster analysis, together with correlational analysis and inferential hypothesis testing analysis based on ANOVA models.

The next section presents the results of the application of this selection instrument for a population of 5758 respondent candidates.

3. Results

1200 students (69.4% male and 30.6% female) were selected from the total of 5758 candidates inscribed in IMD Technical Formation selection procedure and therefore submitted to the selection instrument. An adequate interpretation of the profile of selected candidates concerning gender must considering that 51.7% of the enrolled candidates were female, and 48.3% were male candidates. Such data point to the need for further studies that contribute to the comprehension about the differences in gender as far as IT competences and abilities are concerned.

Results regarding the school origin and school life of the candidates, in terms of public and private schools, are summarized in Table 1. It is important to highlight that the higher percentage of selected candidates coming from public schools (whole or part of formal education) is influenced by the inclusion policy adopted by UFRN, which gives a bonus for students from the public system, multiplying their scores by 1,1. The criterion to receive such bonus is to have successfully undertaken the last three years of school education in the public system. Still, Table 2 shows that the percentage of selected candidates from private schools (27%) is higher than the percentage of enrolled students from the private system (25.6%), while the public system was responsible for 54.6% of enrolled students and 48% of selected students.

Statistical techniques were chosen in order to verify correlational links between performance scores in MDI selection instrument and scores in disciplines for those selected in the previous stage. On the other hand, ANOVA models were used in order to establish possible links between students profile variables (like gender and familiar income level) over performance in the IMD instrument selection and in the courses compounding the technical formation offered. This last target was completed by the use of cluster analysis tools.

The profile regarding the average level of difficulty of the instrument questions is summarized by Figure 2. It is important to highlight here that Question 24 was cancelled, being thus considered correct for all candidates (100% correct answers in the graph).

[1] Percentage of the total contingent of candidates. [2] Percentage of the total contingent of selected candidates.

As it can be read from the graph in Figure 2, the easiest question was number 9, belonging to matrix 2 (Communication and Collaboration), with an average rate of correct answers above 90%. Such question is reproduced in Figure 3 below.

The most difficult question, on the other hand, with an average rate of correct answers of approximately 5%, was question 18, from matrix 3 (Search and management of information). Such question is reproduced in Figure 4 below.

Figure 5 shows a distribution close to normal for the frequency distribution of the total of correct answers in the instrument; such profile of distribution represents a technical positive indication in terms of the quality of this psychometric instrument (Primi, 2003) . The same normal distribution profile is observed among the frequency distribution profiles of the total number of correct answers regarding the competences and abilities matrices that composed the instrument. This is summarized by graphs in Figure 6, Figure 7.

Finally, it is important to note that the competences matrices proposed in the instrument presented specificity, as shown by the data in Table 3. Such data show low correlation coefficients between matrices (which attests the specificity of each matrix in relation to the others), and homogeneously high correlations between the matrices and the instrument as a whole, which attests the participation of each matrix in the global construct intended by the instrument (competences and abilities in IT).

When analyzing data from the performance of the students in terms of the selection instrument and grades obtained in written exams throughout the technical course, through a tool for multidimensional descriptive analysis, two large groups were detected that suggest important aspects regarding the evaluation of the relationship between the selection instrument and the course disciplines. For such analysis, the scores of students’ performance in all instruments analyzed (selection instrument and disciplines) were categorized, so as to perform a Cluster Analysis¹ for categorized data, in the categories superior (third of distribution above the region of the mean), medium (third of distribution in the region of the mean), and inferior (third of distribution below the region of the mean). Such analysis showed the existence of two groups or clusters, one of them aggregating 35.3%

of the subjects (Cluster 1) and the other aggregating the remaining 64.7% (Cluster 2). There was therefore no lost of any participant in terms of being inserted in one of the two clusters detected. The variables with the most relevant contributions for the partition above are summarized in Figure 8. Cluster’s partition was done based upon the hierarchical contribution of categorized variables, as shown by the summary of the most important variables contributing for clusterisation below (Figure 8). As will be discussed below, Cluster 1 was basically composed by low-performance students (in both IMD instrument and disciplines), while Cluster 2 was composed by-medium level performance students.

As suggested in such figure, such variables were constituted by ten disciplines plus the global performance (total of right answers in the selection test), codified as QTot categorized (the distribution of frequency of answers was divided in three categories of frequency, as mentioned above) and the global performance average (Written Exam categorized); the disciplines were, in decreasing order of contribution: Data Banks, Technical English II, Desktop Development, Security in Networks, Technical English I, Applied Mathematics, Logic in Programming, Communication and Expression, Computer Networks, Introduction to Electronics and Projects in Electronics. In general, the two obtained clusters refer to inferior performance in Cluster 1, and medium performance in Cluster 2, both for performance in the instrument of selection and for performance in the disciplines aforementioned. Figure 9 illustrates the partition principle in relation to the categories of performance in the selection instrument (QTot), with a predominance of inferior performance in Cluster 1 (83% of such performance category in this cluster), and medium performance in Cluster 2 (63.6% of such performance category in this cluster).

The obtained clustering allowed verifying the contribution of categories of performance in the disciplines for each of the identified clusters. The disciplines of Data Banks and Technical English II contribute to the division of groups with inferior (Cluster 1) and medium (Cluster 2) performance; Desktop Development and Technical English I contribute equally for the division with inferior performance (Cluster 1) and medium performance (Cluster 2); Applied Mathematics, in its turn, did not contribute to the division of groups, considering that both clusters presented inferior performance. In addition, the performance in the selection instrument contributed equally for the division of groups commented here, with categories of inferior performance concentrated on Cluster 1 and categories of superior performance concentrated on Cluster 2 (see Table 4).

The composition of performance in the selection instrument and in the disciplines aforementioned was complemented by the realization of the Analysis of Variance (ANOVA), having as independent variable the global score in the selection instrument (QTot), categorized according to the aforementioned system (levels inferior, medium and superior), and as dependent variables each numerical score (non-categorized) and each one of the disciplines that proved relevant for the clustering obtained. Such analysis made evident the statistically significant effect of the performance in QTot on all disciplines analyzed, excepting Reconfigurable Electronics and Electronic Projects, as shown in Table 5.

The analysis above confirms the tendency suggested by the clustering, in terms of the existence of connection between the selection instrument (QTot) as independent variable and the disciplines offered in the course as dependent variables. Some discussions on tendencies and organization of data arise from the cluster analysis performed. The medium and inferior performances were more frequent, but the superior performance was also present in the groups, although in less quantity. Such data is suggested by the analysis of correlations between the performance in the instrument and the disciplines of the course, with moderate coefficients, which allow sup-

posing that many students that had good performance in the selection instrument did not necessarily remain in the best-performing group in the course (and vice-versa). This is probably due to other contextual factors (notably teaching-learning processes and tutor-student relationships), to be empirically verified in further research. To sum up, the selection instrument could identify the medium and inferior performances in the course, but the instrument did not statistically predict the students who had superior performance.

Finally, an analysis was performed to identify, from the competences and abilities matrices that composed the selection instrument, which ones were the most effective in terms of the global strength of discrimination of the instrument, i.e. for establishing a predictive link between performance in the instrument and performance in the disciplines of the course. This was performed through regression analysis, where the criterion variable was the final average of performance in the basic module and in the course as a whole with the emphases on Web and Electronics, and the explicative variables were the global performance in the selection instrument (QTot) and the performances in each of the five matrices that composed the instrument (see Table 6).

The association between the criterion variables (average of performance in the basic module) and the explicative variables (QTot and the 5 matrices) is moderate (R = 0.51, p = 0.001), and the matrices and QTot were jointly responsible for 25% of the variance of the performance in the basic module. The regression coefficient for matrix 1 was 1.38 (CI of 95% = 1.07 - 1.68); for matrix 2 it was 0.65 (CI of 95% = 036 - 0.95); for matrix 3, 1.2 (CI of 95% = 0.92 - 1.55); matrix 4 0.76 (CI of 95% = 0.45 - 1.06); matrix 5 1.08 (CI of 95% = 0.78 - 1.39). As the confidence intervals did not include a negative value, it can be concluded that the regression coefficients for the matrices are positive. The standardized regression coefficients indicate that matrix 1 and matrix 3 are stronger. Nevertheless, all of them are positive and significantly related to the performance in the basic module.

The same analysis, having as criterion variables the advanced module in Web and the advanced module in Electronics made evident that the only matrix that was not strong in terms of predictive influence on the disciplines and averages of the course was matrix 2 (Communication and Collaboration). When analyzing the regression with the averages of the basic module and emphases, it was observed that matrix 1 (systemic thinking) and matrix 3 (search and management of information appropriate for specific tasks) were more statistically efficient for the prediction of success in the disciplines of the course. In general, the contribution of the matrices and of the global score of the instrument QTot) presents higher prediction strength on the performance in the advanced module in Electronics, followed by the performance in the basic module and finally in the advanced module in Web.

4. Conclusions and Final Considerations

The present research seeks to contribute to the development of the comprehension of the set of competences and abilities involved in selection and education in IT, through the critical analysis of the selection instrument

proposed and used in the first selective process of the MD project, and its relationship with the technical course in IT. The research has social and cultural context in the development of information technologies and the consolidation of a digital culture that reconfigure education and work.

Results from the experience of the application of the first version of the evaluation instrument allow making conclusions about the psychometric robustness of the instrument, in terms of the normal profile of the distribution curve of the frequency of correct answers to the proposed questions, as well as about the empirical basis for the general division of the measured construct, the competences and abilities in IT, distributed in five matrices of proposed competences and abilities. In this sense, the matrices show a reasonable independence between themselves, in terms of correlation indices. Such results are of special interest, due to the inexistence, in Brazil, of a selection instrument for the referred construct and for the population of Brazilian students. The first version of the present instrument, given the mentioned results, served as platform for the proposition of versions 2 and 3, which have already been executed.

Finally, a link is demonstrated between the performance in the instrument and the performance throughout the course, being it in global terms, or in terms of verifications per group of disciplines. Such data, jointly with the data related to the follow-up of students’ progress after the course, will allow completing the evaluation of the pertinence of the proposed instrument.

It is expected that the results from the development of the tool in question contribute to the selection and education of youths, modifying the perspective of use of IT for this population. On the other hand, the efforts in operationalizing the instrument represent the concretization of a fruitful dialogue between distinct domains of knowledge production, namely psychology and technological disciplines of Computer Science and Computer Engineer. The final result of this work moves towards an interdisciplinary perspective, where human phenomena are approached taking into consideration the inherent complexity, which is characteristic of human development, organization and functioning.

Cite this paper

Juliana Teixeira da CâmaraReis,IzabelHazin,Jorge Tarcísio da RochaFalcão,LucianoMeira,PedroBendassolli,AmandaGuerra,Taciana PontualFalcão,Apuena VieiraGomes,Marcel Vinícius MedeirosOliveira,José Ivonildo doRêgo, (2015) Development and Results of an Instrument to Search for Competences and Abilities in Information Technology. Creative Education,06,2384-2396. doi: 10.4236/ce.2015.622244

References

NOTES

¹Cluster Analysis is a technique of multidimensional description of data that allows detecting groupings (or clusters) of modalities of variables. It is a multivariate analysis that therefore allows grouping modalities of variables in homogeneous groups, if statistically possible (cf. Mooi & Sarstedt, 2011: pp. 237-284 ).