Creative Education
Vol.06 No.10(2015), Article ID:57259,12 pages
10.4236/ce.2015.610093

Perspectives for Defining Student Teacher Performance-Based Teaching Skills Indicators to Provide Formative Feedback through Learning Analytics

Edgar Krull, Äli Leijen

University of Tartu, Tartu, Estonia

Email: edgar.krull@ut.ee, ali.leijen@gmail.com

Copyright © 2015 by authors and Scientific Research Publishing Inc.

This work is licensed under the Creative Commons Attribution International License (CC BY).

http://creativecommons.org/licenses/by/4.0/

Received 13 April 2015; accepted 12 June 2015; published 18 June 2015

ABSTRACT

The aim of this study was to analyze the possibilities for performance-based evaluation of student teachers’ teaching skills in their field practice using learning analytics (LA)―a computer-based instructional system for documenting assessment results, and providing specified feedback on the achievement of expected objectives and suggestions for corrective and further learning activities. The research literature on LA applications in pre-service teacher education reveals that its application at this level of sophistication calls for reliable specific performance-based indicators of teaching to model teaching competences. Such defined indicators should allow for the statistical modeling of teachers’ progression in teaching skills. Our analysis of the literature regarding defining teaching and teacher competence standards and identifying performance-based indicators for assessing progress in teaching skills, led us to conclude there was no formal procedure for deducting performance-based indicators of teaching competence, except expert ratings coordinated by the general ideas of teaching quality. The available performance-based assessment matrices (representing performance levels by teaching tasks) of teaching competence that are used by teacher educators tend to be context-sensitive and call for adaptation when used in the conditions of another country. Also, the reporting of teaching activities (typically using a portfolio format) and quantifying the entries using performance indicators for LA requires a highly formalized coding system to preserve the semantics of the initial data in the models of progress in teaching skills. For avoiding the risk of introducing too many atomic indicator behaviours of student teachers, focusing only on essential teaching competencies via the application of the emerging concept of core practices is suggested.

Keywords:

Teacher Education, Teaching Practice, Teaching Competences, Professional Development, Competency-Based Instruction, Performance Indicators, Core Practices

1. Introduction

The aim of this study was to prepare the ground for optimizing student teachers’ professional learning in school practice by applying learning analytics to provide adaptive formative evaluation of their progress. This approach could provide many advantages for monitoring and supporting student teachers’ progress, but may also be misleading if the entered data is unreliable or data processing is inadequate. Therefore, we focused on the practice of defining and using teaching skills performance indicators for the application of learning analytics (LA). First however, we summarize current knowledge on the practice of using LA for educational purposes.

2. Learning Analytics

LA is a computer-based instructional system for documenting assessments of learning results, providing specified feedback on achievement of expected objectives and dispensing suggestions for corrective and further learning activities (e.g. Clow, 2013; Ferguson, 2012 ). Applying LA calls for quantified and valid entry data on the progress of learning in order to provide analytical feedback via statistical modeling. In this study we focused on the possibilities for providing representative (valid in terms of monitoring the progress of teaching competence) entry data on teaching quality to enable the use of LA procedures. According to Clow, LA is not currently considered as “… a solid discipline with established methodological approaches but an occasional … field of enquiry, picking up ‘shiny’ techniques, tools and methodologies, including web analytics … social network analysis … predictive modeling, natural language processing and more” (pp. 685-686). It is therefore necessary to specify which of these versions is suitable for promoting teaching skills. The more specific definition of LA given by Liaqat, Hatala, and Gasevic (2012) is instructive, as they regard it as “… the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs” (p. 470). To achieve meaningful results, entry data must be representative of the progress of teachers’ learning towards expected outcomes. High quality input data is considered as a prerequisite for the successful functioning of LA in many analytical articles (e.g. Dringus, 2012; Greenberg & Buxton, 2008; Greller & Drachsler, 2012 ). As Clow (2013) explains:

The choice of what is measured―in learning analytics terms, the selection of metrics―is critical. If an educational system is designed to optimise metrics that do not encompass learning, it is likely that learning will be optimised away. For those who care about learning, the choice is to attempt total resistance to the regime of metrics or to take a more pragmatic course and insist on the inclusion of appropriate metrics that do reflect learning…. If assessment does not reflect and reward those aspects of learning that are valued, a learning analytics system that improves assessment scores will not improve those aspects of learning (p. 692).

In traditional approaches the assessment of student teachers’ progress in acquiring teaching skills is typically based on their supervising teachers’ or university tutors’ subjective ratings of specific performances and their summative, mostly intuitive evaluation of teaching competence in general. Often the integral and intuitive evaluation of teaching competence dominates assessments of the meeting of performance criteria for specific teaching skills. The functioning of LA, however, is based on reliably measuring the quality of specific teaching performances. These performances need to be formally representative or content valid regarding the teaching competence expected to be acquired. The degree to which LA makes it possible to capture progress in teaching competence in an integrative way is limited by the algorithm or method “… applied in it … [which] is reductive by nature in that it simplifies reality to manageable set of variables” (Greller & Drachsler, 2012: p. 50) .

LA can be a powerful means for processing and summarizing documented evaluative judgments, producing results comparable with statistically expected profiles of teaching competences, but it should not be viewed as a replacement for an all-inclusive and integral evaluation of teaching competence. As Greller and Drachsler (2012) pointed out, LA provides many new opportunities for education but “… it is still too early to base education fully on LA approaches alone, and we expect it never will be possible to do so” (p. 55). It cannot be expected that LA captures progress in acquiring teaching competence in its integrity in the same way as a human tutor, but it does enable the collection and processing of data (including drawing profiles of progress) on the performance of particular teaching tasks to allow predicting the level of achievement.

For exploiting most effectively the advantages of LA for pre-service teacher education, certain preconditions regarding the definition of teaching competences must be fulfilled. First of all, there is a need for a clear definition of the expected teaching competences of student teachers, preferably covering all the major facets of teaching. These competences should be reliably measurable and quantifiable in order to assess and analyze student teachers’ progress. Data collection also needs to be feasible within reasonable time limits. If these conditions are met, LA allows automatic mapping or modeling of individual student teachers’ profiles of professional development as related to expected (computer generated) professional profiles. The results provide students with “just in time” feedback on their learning progress. However, even if a teaching competence can be reliably and integrally described in operational terms, the models created on the basis of teaching performance indicators by LA procedures can only be approximations of this competence (as pointed out by Greller & Drachsler, 2012 ), or in the case of student teachers’ school practice, approximate or simplistic models of teaching competence with which their progress in acquiring teaching skills is compared. Yet, the possibilities for valid modeling of teaching quality on a performance-based level that are needed for applying LA are not clear, because there are no definitions of teaching skills at this specification level. Therefore, an assessment of the possibilities for valid performance-based competence indicators is needed. The rest of this article reports our attempt to solve this issue by analyzing the latest research findings on defining teaching competence requirements and quantitative performance-based indicators for these competences.

The specific aim of this study was to analyze empirical and theoretical studies that define teaching quality and performance-based teacher competences, and collected quantitative data on student teachers’ progress in acquiring teaching competences. Assessing the latest research findings on these topics allowed us to maximize the validity of quantified entry data for LA procedures regarding performance-based indicators of teaching competence, in order to increase the effectiveness of pre-service teacher education.

3. The Current State-of-the-Art Definitions of Performance-Based Competence Requirements for Student Teachers

3.1. Studies on Characterizing Effective Teaching

Our analysis of the research literature on teacher education showed that attempts to define the knowledge and skills necessary for the teaching profession started almost one hundred years ago, when researchers began to compile lists of teaching skills on the basis of teacher inquiries into what makes a good teacher. This resulted in the production of long and varying lists of the attributes of “good” teaching that had little use for teacher education (Good, 1996) . Since the 1960s more specific models for describing approaches to teacher education or defining the attributes of good teaching have appeared. For example, recently Cochran-Smith (2014) listed four consecutively appearing research questions regarding teacher education aimed at discovering the secrets of good teaching: (1950 - 1960) what are the attributes of good teachers?; (1960 - 1980) what teaching processes lead to effective teaching?; (1980 - 1990) what should teachers know, be able to do?; (since 2001) is nature (teachers are born) versus nurture (teachers are educated) more important? In the first case the main research question is how born teachers can be recruited?; in the second case it is how teacher candidates can learn to teach?

Another list of research traditions for discovering effective teaching1 (with approximate periods) is provided by Good (1996) : (1960s) the focus on teacher personality; (mid-1960s - 1970) the search for a teacher-proof curriculum; (1968 - 1990s) investigation of teaching in naturalistic settings; (1960 - 1990s) relating teacher behaviour to student learning; (1960s - 1990s) examination of how teachers utilized classroom time; (1970s - 1990s) centring on teacher cognition; (1980s - 1990s) focusing on student mediation of teaching; (1990s) studies on teaching for understanding.

These research questions for teacher education and research traditions of teacher effectiveness illuminate the complexity of teaching as a professional activity and point to the fact that construction of all-comprehensive models of effective teaching is a controversial and challenging task. It can be argued that a common trend for the research of teacher education and teacher effectiveness is moving away from attempts to discover specific and simplistic indicators of good teaching (e.g. specific teacher behaviour or classroom time utilization) to indicators embracing teacher professionalism in all its aspects (e.g. teacher learning as personal-professional growth or teaching for understanding). As Good (1996) points out, each research tradition has yielded relevant ideas for the evaluation of teaching and by relying on a paradigm of inclusion researchers can take advantage of all the positive aspects of former research.

The issue of creating models of good teaching leads to questions of a more general character: to what extent is good teaching identifiable?; does it have a permanent character regarding working with different students and in different contexts, i.e. are there certain general professional skills of teaching or are skills context dependent? The first position (that there exists a universal competence of teaching) was, for example, supported by Medley (1985) and Stodolsky (1985) . Shulman (1992) , however, pointed out that teaching quality depended on the subject being taught and the situation in which teaching took place. Even Medley (1982) found that correlation between quality teaching activities of a teacher in different contexts was less than 0.3. Consequently, it is rather difficult to construct a model of teaching activities with indicators that would enable to compare teachers or even assess teaching effectiveness of teachers in different school conditions.

This task becomes even more complicated if we take into consideration that different stakeholders in educational systems and even teacher educators might have differing expectations for the professional competences of teachers. For example, several scholars (Doyle, 1990; Joyce, 1975; Zeichner, 1983) have proposed five major profiles of ideal teachers: good employee; junior professor; fully functioning professor; innovator; reflective practitioner. Furthermore, teaching at different school levels and of different subjects might require variations in the needed teaching competences and lead to specific competence requirements for different teacher certifications. This is the main reason why major teacher certification systems like NBPTS in the USA introduce specified standards for teachers of different school levels and subjects.

Despite these theoretical concerns warning against using simplistic or biased models of teaching competences and consequently competence requirements, there have been standards with indicators in use for assessing the competences of student and even experienced teachers in many countries. As stated by Arends (2006: p.17) , “… effective teaching is dependent upon a knowledge base and clear definitions of what constitutes effective teaching”. The standards and competencies describing “effective teaching” have usually been defined in general terms that leave a lot of flexibility in order to take into account specific contextual features. A need for “… formulation of clear professional standards and also criteria to assess them” is also emphasized by Zgaga (2011: p. 31) based on a survey of teacher education in 12 European countries.

However, the most crucial problem with finding trustful lists of standards or competences for teaching is that typically they are not provided with any research-based evidence of their validity. Our search for research in this field published during the last 20 years yielded only a few approaches to defining teaching competences that provided (at least some) estimations of their validity. These approaches included: the analysis of former research (e.g. Looney, 2011 ); reaching a consensus of practitioners (e.g. Dutch teacher competence requirements); empirical research (NBPTS standards); deducting more specific standards from validated general standards (INTASC standards).

3.2. Theory-Driven Characterization of Effective Teachers

One of the rare attempts to theoretically define teaching quality on the basis of the analysis of former research is made by Looney (2011) . This article points out the relevance of teacher evaluation for improving the quality of teaching: “… any system for teacher evaluation needs to be tied to a clear set of standards and competences … [as there is] … no single, widely accepted definition of teacher quality”; and provides a six-point list of the most effective teacher attributes as “… a reflection of the complexity of teaching and learning” research (p. 441). According to Looney (2011) , the most effective teachers:

Ÿ are intellectually able. Verbal skills are particularly important;

Ÿ have good knowledge of the subject-area(s) and competences they are teaching, as well as a broad repertoire of teaching methods and strategies to meet diverse student needs;

Ÿ develop positive relationships with their students and recognize the crucial role of motivation and emotions in learning;

Ÿ have strong classroom management skills, including clarity in the presentation of ideas, well-structured lessons and appropriate pacing;

Ÿ are skilled assessors. They use assessment ‘formatively’ to monitor students and provide timely and specific feedback on what students need do to improve performance and meet learning goals;

Ÿ work collaboratively with their peers to develop a positive school climate, to improve overall school performance and engage in mutual support and professional learning (pp. 441-442).

However, the most frequent practice of defining good teaching in many countries is teacher competence requirements or standards (e.g. Bourgonje & Tromp, 2011; Roth, 1996 ). Three of them that are validated through research or the documentation of development procedures based on research or professional literature are introduced in the following section with a focus on their validation.

3.3. Teacher Competence Requirements and Standards

3.3.1. Dutch Teacher Competence Requirements

In Europe, many elaborate systems of teacher competence requirements have been developed. For example, in the Netherlands, public discussions over such a system’s suitability lasted for almost six years until being approved by the Dutch Parliament in 2006 (Bourgonje & Tromp, 2011) . The Dutch teacher competence requirements are defined two-dimensionally as combinations of teacher roles and situations in which they implement these roles. The competence requirements include four roles (interpersonal, pedagogical, organizational and expert in subject matter and teaching methods) and four types of contexts (with students, with colleagues, the working environment and by themselves). The sets of teacher roles and types of situations form a 4 by 4 matrix. According to SBL (the Association for the Professional Quality of Teachers), seven partial competences are sufficient to cover all essential aspects of teacher competence (Bourgonje & Tromp, 2011) . These seven competences are defined as:

Ÿ Interpersonal competence in creating a pleasant, safe and effective classroom environment;

Ÿ Pedagogical competence to support children’s personal development by helping them to become independent and responsible;

Ÿ Subject knowledge and methodological competence that demonstrates substantial knowledge of their subject and appropriate teaching methods (including pedagogical content knowledge);

Ÿ Organizational competence in organizing curricula that support student learning;

Ÿ Competence to collaborate with colleagues and thus contribute to a well-functioning school organization;

Ÿ Competence to collaborate with those in the school environment who also play a role in students’ well-being and development (e.g. students’ parents or guardians, colleagues at educational and youth welfare institutions);

Ÿ Competence to reflect and develop as professionals over the long term (Snoek, 2011) .

The seven competences are specified by introducing competence requirements and indicators (Bourgonje & Tromp, 2011) . The Dutch model has been tried out with teachers in primary, secondary and vocational education (SBL, 2004) . However, as the described requirements have not been tested with scientific rigor, and it is unsurprising that the issue of subjectivity of evaluation has arisen, i.e. different evaluators assess differently. Therefore, such teacher competence requirements should serve rather as coordinating guidelines or ideas for teacher education programs, but not directly as a tool for assessing teaching.

3.3.2. NBPTS Standards

Probably the most thoroughly investigated conception of teacher standards and related procedures for teacher certification belong to the National Board for Professional Teacher Standards (NBPTS) in the United States of America. All specific standards and procedures of NBPTS are based on five core propositions about good teaching. According to NBPTS, effective teachers:

Ÿ are committed to students and their learning;

Ÿ know the subjects they teach and how to teach those subjects to students;

Ÿ are responsible for managing and monitoring student learning;

Ÿ think systematically about their practice and learn from experience;

Ÿ are members of learning communities (NBPTS, 1987) .

NBPTS certification is performance-based. Candidates applying for certification present portfolios containing four videos of teaching performance in different contexts, along with detailed explanations of observable teaching activities. However, for understanding the instruments that provide validity to NBPTS certification procedures, it has to be taken into account that the competence requirements of these standards are specified by school levels and the subjects taught (25 certification areas; see NBPTS, 2014 ) and all specific standards are provided with detailed assessment guidelines for experts.

A comprehensive study by Bond et al. (2000) that aimed to investigate the validity of the NBPTS teacher certification procedures by comparing the instructional practices and outcomes of teachers who have been certified by the agency with those of teachers who have applied for certification but were not certified, yielded a detailed list of teacher attributes in which certified teachers typically outperformed non-certified teachers. The list of expert teachers’ attributes was further developed on the basis of a meta-analysis of a large number of studies and its validity was confirmed by relatively high inter-rater agreement on the relevance of the listed attributes and by demonstrating the superiority of certified teachers in comparison with uncertified teachers in the case of 11 of 13 attributes. These 11 attributes were: use of knowledge; having deep representations; problem solving; improvisation; challenge of objectives; creating classroom climate; multidimensional perception; monitoring learning and providing feedback; respect for students; passion for teaching and learning; motivation and self-efficacy.

Another study aimed at validating the NBPTS certification standards was carried out by Vandevoort et al. (2004) . The study revealed that students with higher learning achievements were taught by certified teachers. In this study academic performance of elementary students in the classrooms of 35 certified teachers and their non-certified peers in 14 Arizona school districts were compared. Four years of results from the Stanford Achievement Tests in reading, mathematics and language arts, for grades three through six, were analyzed. In the 48 comparisons (four grades, four years of data, three measures of academic performance), using gain scores adjusted from students’ entrance ability, the students in the classes of National Board Certified Teachers surpassed those of non-Board certified teachers in almost three quarters of comparisons.

3.3.3. INTASC Standards

Alongside of investigating the validity of competence requirements for practicing teachers, there have been a few attempts to put under scrutiny the competence requirements of student teachers. It is obviously wrong to consider competence requirements for experienced and student teachers to be the same thing. In the case of student teachers’ school practice, the main focus should be on formative evaluation of their progress in learning to teach, rather than on the final professional competences to be achieved (Andrews & Barnes, 1990) . Therefore, the performance-based objectives to be achieved during teaching practice cannot be derived directly from the standards for expert teachers. Nevertheless, awareness of the main competences of expert teachers might be helpful for coordinating and guiding teacher learning. Of interest in this respect is the work done by the New Teacher Assessment and Support Consortium (INTASC) in the USA.

The model of core standards (10 INTASC principles) for student teachers grew out of the five propositions about effective teaching adopted by the NBPTS (Arends, 2006) . A recent revision of these standards was launched to develop an updated version. The beta version of these standards is publicly available and aims at student and as well as practicing teachers (Council of Chief State School Officers, 2013) . All ten standards define expected teaching competences in terms of performances, essential knowledge, critical dispositions, and descriptions for progression. However, the most interesting feature of these draft standards is the descriptions of progression for the listed standards that outline three levels of professionalism in teaching. The main categories of these standards are:

Ÿ learner development;

Ÿ learning differences;

Ÿ learning environments;

Ÿ content knowledge;

Ÿ application of content;

Ÿ assessment;

Ÿ planning for instruction;

Ÿ instructional strategies;

Ÿ professional learning and ethical practice;

Ÿ leadership and collaboration (Council of Chief State School Officers, 2013: pp. 16-47) .

Other relevant examples of attempts to validate performance based requirements that teacher candidates must demonstrate are criteria for the hiring of first year teacher candidates for getting job at the University of Nebraska at Kearny (Ziebarth-Bovill, Kritzer, & Bovill, 2012) and the procedures of the Performance Assessment for California (PACT) (e.g. Pecheone & Chung, 2006, 2007 ).

3.4. Difficulties in Defining (Inter-) National Standards for Effective Teaching in Details

Whereas in European countries little empirical evidence has been reported on the effectiveness of the use of teaching standards, in the USA rigorous evaluations of certification systems based on such standards have emerged. However, many experts have expressed that the results of those studies are not sufficient to draw firm conclusions about their effectiveness (Cantrell, Fullerton, Kane, & Staiger, 2008; Hakel, Koenig, & Elliott, 2008; Ingvarson, 2009) .

What is considered as good teaching in one country is not necessarily good teaching in another. Furthermore, expectations of teachers’ roles might differ within a country. This might explain why there are not many international validation studies of teacher competence requirements or teaching standards regarding measurable teaching performance.

The diversity regarding what aspects constitute good teaching and the scarcity of reliable research on teacher competence requirements or certification procedures points to the complexity of teaching as a professional activity and the difficulties of modelling a good teacher in a reliable way. For example, if the teaching quality for learning a foreign language and progress in it at a basic level can be quite reliably and satisfactory described as a gradual acquisition of vocabulary and phrases in a LA-based approach to learning foreign languages (e.g. Ennet, 2014 ), then the teaching quality cannot be reduced to the sum of single performances without assessing the quality that these performances produce integrally. The alternation in approaches to defining attributes of good teaching, as exemplified by periodization of the related research (see Cochran-Smith, 2014; Good, 1996 ), only confirm that an all-comprehensive capturing of what constitutes quality teaching is a very challenging task.

Due to the lack of valid models of good teaching with clearly identified variables that allow rigorous discrimination between poor and good teaching, prospects for optimizing student teachers’ study paths is limited, as descriptions of expected integral teaching competence and identification of partial competences underlying this competence tend to be subject to different interpretations and are context dependent. Furthermore, considering that educators’ models and representations of good teaching and its component skills are never ideal or unchangeable, an adequate assessment of teaching should always involve an interpretative component (Tigelaar & Van Tartwijk, 2010; Van der Schaaf, Stokking, & Verloop, 2008b) , which certainly complicates applying LA procedures.

4. Tools for Performance Based Assessment of Teaching Competences

4.1. Specifying Criteria for Performance-Based Assessments of Teaching

Irrespective of the theoretical problems with ensuring rigorous validity when modeling good teaching and identifying teaching skills components, teacher educators have always tried to imagine and describe what they mean by good teaching and which teaching skills have to be developed for learning how to teach. Typically this has been done by following national competence requirements or teacher standards as a coordinating framework for specifying tasks or activities for student teachers to perform in their field-practice in order to achieve the anticipated teaching competences. However, this does not necessarily mean that performance indicators are derived directly from standards.

Depending on individual traditions of teacher education, the teaching activities and component skills to be mastered and the corresponding assessment criteria are more or less specific. These specifications are often validated on the basis of negotiations and agreements among the teacher educators involved (Van der Schaaf & Stokking, 2011) . An example of specifying teaching competences defined at a general level into specific indicator activities is CCSSO’s (Council of Chief School Offices, 2013) Model Core Teaching Standards and Learning Progressions for Teachers. Its ten standards of teaching competences (see Section 3.3. for detail) are defined in terms of the behaviours that characterize the achievement of the sub-competences listed under each standard. For example, the Standard # 6: Assessment is specified by 9 statements for performances during assessment, 7 for essential knowledge and 6 for critical dispositions. The first statements for these three domains are: (1) the teacher balances the use of formative and summative assessment as appropriate to support, verify, and document learning; (2) the teacher understands the differences between formative and summative applications of assessment and knows how and when to use each; (3) the teacher is committed to engaging learners actively in assessment processes and to developing each learner’s capacity to review and communicate about their own progress and learning (Council of ..., 2013, p. 30).

However, the most valuable feature for assessing student teachers’ progress in acquiring teaching competences are the definitions for progression indicators of the standards. For example, Standard # 6 provides three aspects or dimensions of progress in the assessment competences: (1) the teacher uses, designs or adapts multiple methods of assessment to document, monitor and support learner progress appropriate to the learning goals and objectives; (2) the teacher uses assessments to engage learners in their own growth; (3) the teacher implements assessments in an ethical manner and minimizes bias to enable learners to display the full extent of their learning (Council of ..., 2013, pp. 31-33).

Every aspect of progress for the standards is described with three gradually refined levels of proficiency. For dimension two (the teacher uses assessment to engage learners in their own growth), the nature of these levels can be comprehended from the first sentences of the corresponding descriptions: (1) the teacher engages each learner in examining samples of quality work …; (2) and the teacher engages learners in generating criteria for quality work …; (3) and the teacher engages learners in giving peers feedback on their performance using criteria generated collaboratively. In total, the whole model offers descriptions of 21 progress dimensions with 79 indicators of performance suitable for documenting in a quantified format student teachers’ progression in acquiring the required teaching competences.

A similar approach for preparing an assessment matrix for applying LA methodology has been proposed by a Dutch-Estonian team (WATCHME, 2014) . The matrix was developed on the basis of Dutch and Estonian teacher competence requirements in terms of how their performance indicators for teacher education were specified. The original list of seven Dutch competences was shortened to four professional roles and then extended to five based on Estonian competences. Each of the five professional roles was specified through descriptions of tasks or activities (also, in terms of Entrusted Professional Activities2―EPAs) that a student teacher was expected to be able to perform. The model was validated in both countries using the Delphi method. The obtained final versions differed to some extent: the Dutch model contained 5 roles, 11 professional activities and 34 performance indicators with 4 performance levels; the Estonian model had 5 roles, 12 professional activities, 40 performance indicators and 5 performance levels.

Owing to being defined at very specific levels, the progression indicators of CCSSO’s Model Core Teaching Standards and Learning Progressions for Teachers and WATCHME’s matrices of performance indicators observe the specific traditions and needs of teacher education programs. Therefore, for these indicators that were developed for the conditions of one country or region to be applied in another may require adaptations to the native teacher education traditions and conditions.

4.2. A Core Practices Approach as a Way of Adjusting Teacher Learning Tasks to Promote Essential Teaching Competences

Alongside the problem that specific performance-based assessment matrices of teaching competences are context dependent and not always transferable to other conditions, these matrices tend to become non-compassable. The potential evaluation matrix for the CCSSO model standards would involve documenting progression in 79 indicator performances (the document itself does not refer to such a matrix). Simultaneously observing large numbers of teacher attributes and teaching skills without losing an integral sight of a student teacher’s progression in acquiring teaching competence is complicated.

A solution to this atomizing of teaching competences could be the application of the recently adopted concept of core practice ( Grossman, Hammerness, & McDonald, 2009) . This approach is different from prior attempts to orient teacher education around practice. Forzani (2014) points out in a comparison of this innovative concept with former practice-centred teacher education that “… it [core practice] is playing out in both the decisions teacher educators are making about what novices should learn-and in particular in how they decompose practice into learnable parts-and in the pedagogical approaches used in professional training” (p. 360). This author goes on to explain that unlike former practice-based teacher education where researchers tried to decompose teaching into specific performances “… core or high-leverage practices … have used carefully developed criteria to identify a smaller number of items, recognizing that the short duration of teacher education must be used strategically” (ibid., p. 363). This means that teaching practice tasks should be carefully selected in a way that implementation of these tasks supports linking theory with practice and promotes competence in teaching.

It should be noticed that the core practice approach differs radically from competency-based teacher education (CBTE) based on the Stanford taxonomy of teaching tasks that contained hundreds of teaching tasks organized into 19 categories (Baral, Snow & Allen, 1968; Bush, 1968) and which can be seen as forerunner of the assessment matrices introduced above. The CBTE approach was based on a belief that learning to teach consisted of mastering specific teaching activities through training, which does not leave much place for reflection or experimentation in Dewey’s sense (1904) .

The concept of core practice as applied to teacher education has a high potential for increasing the coherence between general competency requirements of the teaching profession and selection of practice-based teaching tasks that through implementation are expected to lead to meeting the requirements. Therefore, paying more attention to applying the ideas of core practice in addition to taking heed of national competence requirements for defining and selecting teaching tasks for student teachers’ school practice is justified. This would increase the effectiveness of students’ learning to teach and also validate the performance-based assessment of teaching competences by introducing more authentic performance indicators to represent teaching in its integrity as a logical whole.

4.3. Data Collection Tools for Performance-Based Assessments of Student Teachers’ Teaching Skills

Besides defining performance-based criteria for the assessment of student teachers’ progress in learning to teach, the identification of tools to allow the collection of the necessary data for assessment procedures is not an easy task. If the aim is to provide student teachers with feedback for the formative evaluation of their competences by applying LA procedures, the evidences of their teaching skills should be collected in reasonable time limits.

Typically different documents providing information on teaching competences are drawn together into student teacher’s portfolios. For example, the use of portfolios for assessing teaching skills has been studied by Van der Schaaf, Stokking, and Verloop (2008 a, 2008b) . In recent years more and more different versions of electronic e-portfolios are used. One such e-portfolio called EPASS (2014) is offered by Maastricht University, The Netherlands. Collected evidence can include lesson plans, student teacher self-evaluation placement forms, lesson observation forms from supervisors, lesson observation forms from other student teachers and reflection reports.

Though portfolios have been introduced as tools to collect evidence about the development of various competencies, the implementation of portfolios has met with mixed success, especially when they are not tailored to show what really happens in the workplace (Driessen et al., 2007; Van der Schaaf et al., 2008a) . Arends (2006) observes that “To be valid, the work included in the portfolio must cover the range of standards and the scoring must be reliable” (p. 105) and provides two examples of how the Universities of Nôtre Dame and Connecticut have developed procedures to make portfolios a valid and reliable summative assessment of teacher candidates and student teachers’ performance. A main problem with using portfolios as tools for assessing teaching competences, however, is that composing them is time consuming and the quality of self-reflection on teaching activities is often unsatisfactory (Tigelaar & Tartwikjk, 2010) .

An interesting example of using e-portfolios is The Performance Assessment for California Teachers (PACT) system that uses a multiple measures approach based on two types of portfolio-based assessment strategies (Pecheone & Chung, 2006) . The assessment procedures are defined by the California Teacher Performance Expectations (Commission on Teacher Credentialing, 2013) . The formative assessment of prospective teachers is implemented through embedded signature assessments that occur throughout the teacher education program. A summative assessment of teaching competences takes place during teaching practice (Pecheone & Chung, 2006).

The purpose of the embedded signature assessment (ESA) is to provide formative feedback to the student teachers and teacher educators. Many generic guiding questions were formulated for creating developmental performance assessment portfolios that meet psychometric rigor (Larsen & Calfee, 2005) :

The ESAs are campus-specific assignments chosen from standard criteria that track a teacher candidate’s growth over time. The ESA label signifies that the assessments (1) already were part of preparation courses (embedded) and (2) provided significant snapshots of teacher candidate competency over time (signature). ESAs include case studies, lesson plans, observations, classroom management plans, and other assignments or activities that fulfill certain selection criteria (p. 151).

Larsen & Calfee (2005) also note that “… for universities using an e-portfolio system, ESAs push the boundaries one step further. An e-portfolio is a computer-based collection of artefacts showcasing a teacher candidate’s growth over time in key teaching competencies” (pp. 155-156).

The PACT summative assessment of teaching events (TE) uses multiple sources of data (teacher plans, teacher artefacts, student work samples, video clips of teaching, and personal reflections and commentaries) organized in four categories of teaching: planning; instruction; assessment; reflection (PIAR). To complete TE, candidates plan and teach a learning segment (i.e., an instructional unit or part of a unit), videotape and analyze their instruction, collect student work and analyze student learning, and reflect on their practice. TEs are designed to measure and promote candidates’ abilities to integrate their knowledge of content, students and instructional context in making decisions and to stimulate teachers’ reflection on their practice (Pecheone & Chung, 2006) .

TE scoring is based on a task-based model that follows the design of the portfolio in a sequential manner. The candidates’ performance is rated for each task on the basis of guiding questions (GQ) on a 4-point scale. As a result, a detailed score profile is generated that provides information at the GQ and PIAR task levels and can be used for developing individual induction plans (Pecheone & Chung, 2006) .

As in the introduced complex systems of performance-based assessment of teaching competences, the definition of performance indicators (in form of assessment matrices) and data collection procedures, when based on teaching portfolios, tend to be tightly interconnected and take on an interpretive character though the application of LA tools calls for quantitative data input. Therefore, the formulation of indicators that are relevant for the evaluation of learning (in this case of learning to teach) play a critical role in translating learning into numbers for applying LA (Greller & Drachsler, 2012) .

5. Discussion and Conclusive Remarks

The idea of applying LA for optimizing student teachers’ learning in school practice by providing adaptive formative evaluation of students’ progress is promising, but also challenging for many reasons. A central issue in using statistical modeling to analyze progress in teaching (as a very complex professional activity) is related to creating models of quality teaching and identifying main performance indicators, and the technical capabilities of LA procedures to collect and process data and produce quantitative models. Furthermore, the information needed for making analytical conclusions on the progression of teaching skills and for proposing optimal study paths has to be available with minimum delay to enable effective learning guidance.

Our analysis of research on defining effective teaching and teacher competence shows that there are no formal rules or guidelines for characterizing quality teaching (e.g. Looney, 2011 ) or national teacher standards and competence requirements. Nor do these characteristics or standards provide the rules for specifying indicator performance that are needed for assessing teachers’ or student teachers’ teaching competence. As performance- based assessment of teaching competence or progress is in itself not feasible without the introduction of performance indicators, these indicators are typically introduced on the basis of expert agreements coordinated by standards or general competence requirements, as well as by experts’ understanding of the issues of teachers’ professional development under specific conditions. Performance-based teacher evaluation matrices, such as those offered by the CCSSO Model Core Teaching Standards and Learning Progressions for Teachers or the WATCHME project, cannot therefore be automatically assumed to be valid in any context. The necessary validation study would start from an investigation if teaching of student teachers that is meeting all performance- based requirements (in terms of indicator performances) proves to be competent on the basis of expert ratings as well. The specification of performance-based indicators of teaching competence should follow the principle that underlies the defining of specific performance-based learning objectives on the basis of general aims or goals of education in the context of objective-driven approaches to instruction―decisions regarding the coherence of specific objectives with coordinated general aims that are typically based on expert opinions (e.g. Gronlund, 1988; Airasian, 1994 ).

Applying LA procedures can be helpful for validating the used model of teaching competence attained via performance indicators by providing processed data for external interpretation by experts. For example, the graphical presentation of progress profiles and typology of progress might be beneficial for more general interpretations and validation of the underlying model. Yet as Greller and Drachsler (2012) warned, there is always an on-going challenge to formulate indicators from the available datasets that are relevant to the evaluation of the learning process, but do not represent real teaching competence in its integrity, i.e. embracing all relevant professional skills.

Another issue besides modeling teaching competence is collecting data on teaching performance indicators for applying LA. The most frequently used data collection procedure is using portfolios that gather together artefacts and documents from different sources. Many versions of electronic portfolios such as EPASS (2014) are used. There have been many attempts to develop procedures for increasing the validity of portfolio assessments. The main problems are that composing portfolios for assessing teaching competences and procedures are time consuming, and student teachers’ self-reflection of teaching activities is often unsatisfactory. Many promising procedures for portfolio-based formative and summative assessment of teaching are offered by PACT―a system involving embedded signature assessment (ESA) and assessment of teaching events (TE). For ESA an e-portfo- lio (a computer-based collection of artefacts) showcasing a teacher candidate’s growth over time in key teaching competencies is used.

Probably the most challenging problem when using a portfolio-based assessment as input for LA is not having reliable assessment scales of defined performance indicators to enable conversion of interpretative ratings into quantitative data. Here the strong and weak sides of all three options of scaling―introduction of relatively independent indicators of performance levels, cumulatively defined indicators of performance levels (e.g. CCSSO Model Core Teaching Standards) and using Liker-type scales for assessing the quality of indicator performan- ces―should be taken into consideration to find optimal solutions.

Many of the issues raised in connection with validating models of good teaching and their component skills to identify progress in mastering teaching skills by student teachers in the context of LA can be solved in an integral way if the practice of decomposing of teaching into atomistic performances are replaced with a more careful selection of teaching tasks by decomposing practice into learnable parts for student teachers, as foreseen with the concept of core practice which is increasingly acquiring the attention of teacher educators.

Application of LA procedures might be a powerful tool for providing student teachers with relatively quick feedback on their progress in acquiring teaching competences, however, the realities of meaningfully describing teaching competences in quantitative terms and of analytically modeling these competences should be taken into account.

Acknowledgements

This study was conducted within the framework of the “Workplace-Based e-Assessment Technology for Competency-Based Higher Multi-Professional Education” (WATCHME) project supported by the European Commission 7th Framework Programme (grant agreement No. 619349).

References

  1. Andrews, T. E., & Barnes, S. (1990). Assessment of Teaching. In W. R. Houston, M. Haberman, & J. Sikula (Eds.), Handbook of Research on Teacher Education (pp. 569-598). New York: Macmillan Publishing Company.
  2. Airasian, P. W. (1994). Classroom Assessment (2nd ed.). New York: McGraw-Hill.
  3. Arends, R. I. (2006). Performance Assessment in Perspective: History, Opportunities, and Challenges. In S. Castle, & B. D. Shaklee (Eds.), Assessing Teacher Performance: Performance-Based Assessment in Teacher Education (pp. 3-22). Lanham, MD: Rowman & Littlefield Education.
  4. Baral, D. P., Snow, R. E., & Allen, E. W. (1968). A Taxonomy of Teaching Behaviors: Progress Report (Research and Development Memorandum No. 36). Stanford, CA: Stanford University School of Education. ERIC Document Reproduction Service No. ED049157.
  5. Bond, L., Smith, T., Baker, W. K., & Hattie, J. A. (2000). The Certification System of the National Board for Professional Teaching Standards: A Construct and Consequential Validity Study. Greensboro, NC: Center for Educational Research and Evaluation, The University of North Carolina at Greensboro.
  6. Bourgonje, P. & Tromp, R. (2011). Quality Educators: An International Study of Teacher Competences and Standards. Education International/Oxfam Novib. http://download.ei-ie.org/Docs/WebDepot/Quality%20Educators.pdf
  7. Bush, R. N. (1968). Second Annual Report (Annual Report No. 2). Stanford, CA: Stanford Center for Research and Development in Teaching. ERIC Document Reproduction Service No. ED024642.
  8. Cantrell, S., Fullerton, J., Kane, T. J., & Staiger, D. O. (2008). National Board Certification and Teacher Effectiveness: Evidence from a Random Assignment Experiment (NBER Working Paper 14608). Cambridge: National Bureau of Economic Research.
  9. Clow, D. (2013). An Overview of Learning Analytics. Teaching in Higher Education, 18, 683-695. http://dx.doi.org/10.1080/13562517.2013.827653
  10. Cochran-Smith, M. (2014). A U.S. Perspective on Teacher Preparation Research: Past, Present and Future. A Paper Presented at the EERA Conference “The Past, the Present and the Future of Educational Research in Europe”, Porto, 1-5 September 2014.
  11. Commission on Teacher Credentialing (2013). California Teaching Performance Expectations. http://www.ctc.ca.gov/educator-prep/standards/adopted-TPEs-2013.pdf
  12. Council of Chief State School Officers (2013). Interstate Teacher Assessment and Support Consortium (InTASC) Model Core Teaching Standards and Learning Progressions for Teachers 1.0: A Resource for Ongoing Teacher Development. Washington DC: CCSSO.
  13. Dewey, J. (1904). The Relation of Theory to Practice in Education. In C. A. McMurry (Ed.), The Third Yearbook of the National Society for the Scientific Study of Education. Part I. (pp. 9-30). Chicago, IL: The University of Chicago Press. https://archive.org/details/r00elationoftheorynatirich
  14. Doyle, W. (1990). Themes in Teacher Education Research. In W. R. Houston, M. Haberman, & J. Sikula (Eds.), Handbook of Research on Teacher Education (pp. 3-24). New York: Macmillan Pub. Co.
  15. Driessen, E., Van Tartwijk, J., Van der Vleuten, C., & Wass, V. (2007). Portfolios in Medical Education: Why Do They Meet with Mixed Success? A Systematic Review. Medical Education, 41, 1224-1233. http://dx.doi.org/10.1111/j.1365-2923.2007.02944.x
  16. Dringus, L. P. (2012). Learning Analytics Considered Harmful. Journal of Asynchronous Learning Networks, 16, 87-100.
  17. Ennet, P. (2014). The Study Aid Being Developed by Estonians Accelerates Almost 10 Times Learning of Foreign Languages (Eestlaste loodav õppevahend kiirendab keeleõpet kuni kümme korda). http://teadus.err.ee/v/haridus/6a25856b-99b8-49c2-b867-b195925d27f9
  18. EPASS (2014). About EPASS. http://www.epass.eu/en/over-epass/
  19. Ferguson, R. (2012). The State of Learning Analytics in 2012: A Review and Future Challenges. Technical Report, Knowledge Media Institute, The Open University, UK. http://kmi.open.ac.uk/publications/pdf/kmi-12-01.pdf
  20. Forzani, F. M. (2014). Understanding “Core Practices” and “Practice-Based” Teacher Education: Learning from the Past. Journal of Teacher Education, 65, 357-368. http://dx.doi.org/10.1177/0022487114533800
  21. Good, T. L. (1996). Teaching Effects and Teacher Education. In J. Sikula (Ed.), Handbook of Research on Teacher Education (2nd ed., pp. 617-665). New York: Simon & Schuster Macmillan.
  22. Greenberg, S., & Buxton, B. (2008). Usability Evaluation Considered Harmful (Some of the Time). In CHI 2008 (pp. 111-120). Florence: ACM Press.
  23. Greller, W., & Drachsler, H. (2012). Translating Learning into Numbers: A Generic Framework for Learning Analytics. Educational Technology & Society, 15, 42-57.
  24. Gronlund, N. E. (1988). How to Construct Achievement Tests (4th ed.). Englewood Cliffs, NJ: Prentice-Hall, Inc.
  25. Grossman, P., Hammerness, K., & McDonald, M. (2009). Redefining Teaching, Re-Imagining Teacher Education. Teachers and Teaching: Theory and Practice, 15, 273-289. http://dx.doi.org/10.1080/13540600902875340
  26. Hakel, M., Koenig, J., & Elliott, S. (2008). Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington DC: National Academies Press.
  27. Ingvarson, L. C. (2009). Developing and Rewarding Excellent Teachers: The Scottish Chartered Teacher Scheme. Professional Development in Education, 35, 451-468. http://dx.doi.org/10.1080/19415250903016707
  28. Joyce, B. (1975). Conceptions of Man and Their Implications for Teacher Education. In K. Ryan (Ed.), Teacher Education, Part II: The 74th yearbook of the National Society for the Study of Education (pp. 111-145). Chicago, IL: University of Chicago Press.
  29. Larsen, L. R., & Calfee, R. C. (2005). Assessing Teacher Candidate Growth over Time: Embedded Signature Assessments. The Clearing House, 78, 151-157.
  30. Liaqat, A., Hatala, M., & Gasevic, D. (2012). A Qualitative Evaluation of Evolution of a Learning Analytics Tool. Computers & Education, 58, 470-489. http://dx.doi.org/10.1016/j.compedu.2011.08.030
  31. Looney, J. (2011). Developing High-Quality Teachers: Teacher Evaluation for Improvement. European Journal of Education, 46, 440-455. http://dx.doi.org/10.1111/j.1465-3435.2011.01492.x
  32. Medley, M. (1982). Teacher Competency Testing and the Teacher Educator. Charlottesville, VA: University of Virgia, Bureau of Educational Research.
  33. Medley, D. (1985). Issues and Problems in the Validation of Teaching and Teacher Professional Behaviours. Proceedings of the Annual Meeting of the American Educational Research Association, Chicago, 31 March-4 April 1985.
  34. Mulder, H., Ten Cate, O., Daalder, R., & Berkvens, J. (2010). Building a Competency-Based Workplace Curriculum around Entrustable Professional Activities: The Case of Physician Assistant Training. Medical Teacher, 32, e453-e459. http://dx.doi.org/10.3109/0142159X.2010.513719
  35. NBPTS (1987). The Five Core Propositions. http://www.nbpts.org/five-core-propositions#sthash.rdAKZJ2w.dpuf
  36. NBPTS (2014). Certificate Areas. http://www.nbpts.org/certificate-areas
  37. Pecheone, R. L., & Chung, R. R. (2006). Evidence in Teacher Education: The Performance Assessment for California Teachers (PACT). Journal of Teacher Education, 57, 22-36. http://dx.doi.org/10.1177/0022487105284045
  38. Pecheone, R. L., & Chung, R. R. (2007). Technical Report of the Performance Assessment for California Teachers (PACT): Summary of Validity and Reliability Studies for the 2003-04 Pilot Year. California: PACT Consortium. http://www.pacttpa.org/_files/Publications_and_Presentations/PACT_Technical_Report_March07.pdf
  39. Roth, R. A. (1996). Standards for Certification, Licensure, and Accreditation. In J. Sikula (Ed.), Handbook of Research on Teacher Education (2nd ed., pp. 242-278). New York: Simon & Schuster Macmillan.
  40. SBL (2004). Competence Requirements Teachers. http://www.bekwaamheidsdossier.nl/cms/bijlagen/SBL_Introduction.pdf
  41. Shulman, L. (1992). Research on Teaching: A Historical and Personal Perspective. In F. Oser, A. Dick, & J. Patry (Eds.), Effective and Responsible Teaching (pp. 14-29). San Francisco, CA: Jossey-Bass.
  42. Snoek, M. (2011). Teacher Education in the Netherlands. Balancing between Autonomous Institutions and a Steering Government. In M. V. Zuljan, & J. Vogrinc (Eds.), European Dimensions of Teacher Education―Similarities and Differences (pp. 53-83). Ljubljana: Faculty of Education, University of Ljubljana, Slovenia and the National School of Leadership in Education. https://www.pef.uni-lj.si/fileadmin/Datoteke/Zalozba/pdf/Zuljan_Vogrinc_Teacher_education.pdf
  43. Stodolsky, S. (1985). Teacher Evaluation: The Limits of Looking. Educational Research, 13, 11-18.
  44. Tigelaar, D. E. H., & van Tartwijk, J. (2010). The Evaluation of Prospective Teachers in Teacher Education. In P. Peterson, E. Baker, & B. McGaw (Eds.), International Encyclopedia of Education (3rd ed., pp. 511-517). Amsterdam: Elsvier.
  45. Van der Schaaf, M., Stokking, K., & Verloop, N. (2008a). Teacher Beliefs and Teacher Behaviour in Portfolio Assessment. Teaching and Teacher Education, 24, 1691-1704. http://dx.doi.org/10.1016/j.tate.2008.02.021
  46. Van der Schaaf, M., Stokking, K., & Verloop, N. (2008b). Developing and Validating a Design for Teacher Portfolio Assessment. Assessment & Evaluation in Higher Education, 33, 245-262. http://dx.doi.org/10.1080/02602930701292522
  47. Van der Schaaf, M. F., & Stokking, K. M. (2011). Construct Validation of Content Standards for Teaching. Scandinavian Journal of Educational Research, 55, 273-289. http://dx.doi.org/10.1080/00313831.2011.576878
  48. Vandevoort, L. G., Amrein-Beardsley, A., & Berliner, D. C. (2004). National Board Certified Teachers and Their Students’ Achievement. Education Policy Analysis Archives, 12, 46. http://dx.doi.org/10.14507/epaa.v12n46.2004
  49. WATCHME (2014) WATCHME Workplace-Based E-Assessment Technology for Competency-Based Higher Multi-Professional Education. http://cordis.europa.eu/project/rcn/189095_en.html
  50. Zeichner, K. M. (1983). Alternative Paradigms of Teacher Education. Journal of Teacher Education, 34, 3-9. http://dx.doi.org/10.1177/002248718303400302
  51. Zgaga, P. (2011). Why We Need to Learn One from Another and Work Together? In M. V. Zuljan, & J. Vogrinc (Eds.), European Dimensions of Teacher Education―Similarities and Differences (pp. 11-31). Ljubljana: Faculty of Education, University of Ljubljana, Slovenia and the National School of Leadership in Education. https://www.pef.uni-lj.si/fileadmin/Datoteke/Zalozba/pdf/Zuljan_Vogrinc_Teacher_education.pdf
  52. Ziebarth-Bovill, J., Kritzer, J., & Bovill, R. (2012). The Essential Criteria for Hiring First Year Teacher Candidates. Education, 133, 125-138.

NOTES

1Also, terms like teaching competence or competency, professional competence, basic teaching competence, teaching skills, teaching knowledge and effective teaching are used as approximate synonyms.

2An EPA is a critical part of professional work that can be identified as a unit to be entrusted to a trainee once sufficient competence has been reached (Mulder, Ten Cate, Daalder, & Berkvens, 2010) . The identification and definition of these teaching criteria is the subject of another study.